Hello, world model

This is the first entry in my field notes — a place to think out loud about the things I'm working on: world models, exploration, and how agents decide what is worth learning.

#What I want to figure out

A curious agent is rewarded for reducing its uncertainty. The classic way to measure that is information gain about the parameters $\theta$ of its world model after seeing a new transition $s'$:

$$ \mathrm{IG}(s, a) = H\big[p(\theta \mid \mathcal{D})\big] - \mathbb{E}_{s' \sim p(\cdot \mid s, a)}\, H\big[p(\theta \mid \mathcal{D} \cup \{(s, a, s')\})\big] $$

Methods like VIME and Plan2Explore are built on versions of this idea. But not every bit of uncertainty is equally useful. Some of it is about parts of the world that will never change what the agent does. The question I keep coming back to:

Is general model uncertainty the right curiosity signal — or only the uncertainty that is relevant to the agent's decisions?

#What you'll find here

• Notes on papers I'm reading (VIME, Plan2Explore, OPAX, LOVE, …)
• Small experiments, with code
• Write-ups of projects like physics-informed world models and reward hacking in RL

A tiny example of the kind of code that will show up:

import gymnasium as gym

env = gym.make("MountainCar-v0")
obs, _ = env.reset(seed=0)
for _ in range(200):
    obs, reward, terminated, truncated, _ = env.step(env.action_space.sample())
    if terminated or truncated:
        break

More soon.