can a browser be deterministic?
- browsers
- testing
- dst
- agents
last weekend i had built a tiny DST. to summarize, put all the luck behind one seed and every bug replays perfectly. so this weekend i wanted the same thing but for web agents. the problem to solve could be a scenario where an agent fails on a page, passes on the next run, and youre stuck with "cant reproduce sorry" again (just like the last weekend)
so the question was pretty simple: if i load the same page 100 times, do i get the same page 100 times?
short answer: yes, on one machine. but it took four fixes, and i fell into three different traps, and the pixels didnt match on a different linux distro
so, let's get started, shall we? :)
code: github.com/bit2swaz/safeconduct/spikes/determinism
the luck in a page
a web page is full of luck (technical: nondeterminism):
- the clock:
Date.now(),performance.now(), timers,requestAnimationFrame - randomness:
Math.random(),crypto.getRandomValues(),crypto.randomUUID() - the network: slow responses, responses in a different order, data that changes between visits
- frames: when Chrome paints decides where an animation is in your screenshot
and some of these are pretty sneaky too. seeding Math.random really does nothing for crypto.randomUUID() because it has its own generator. and a web worker is a another separate JS world, so a patch in the page never really reaches it. a worker gets its own fresh copy of every global, so the Math.random in there is the original, unpatched one
the spike
i made a fake shop page that uses every kind of luck: it shows the time, a random session id, a random "pick for you", a number from a worker, two API calls that append to a list in arrival order, a spinning loader (a CSS animation), a canvas, and a promo that shows up after 300ms
then, we render it a 100 times, and hash three things each time. a hash turns a whole page into one short string, so "are these 100 renders the same?" becomes "is there only one unique string?"
- the DOM (the html)
- the accessibility tree (simple: what an agent "sees")
- the screenshot (the pixels)
why three? the DOM is what the page's code sees, the a11y tree is what most agents read, and the screenshot is what a vision agent looks at. an agent can trip on any of them
if we only get 1 unique hash, that means the page and the browser as a whole is deterministic. then we turn the fixes off one at a time and count again
the fixes
1. recorded network. load the live site once, save every response, and serve the saved bytes on every run through the Fetch domain of the Chrome DevTools Protocol (CDP, the API that tools like playwright use to drive Chrome). anything not in the recording gets blocked (thats an "off-recording" request). why not just fetch it live? because then the live site's luck comes right back in
2. seeded rng. replace Math.random, crypto.getRandomValues and crypto.randomUUID with one seeded generator, in the page and in workers. same seed means the same numbers in the same order on every run. to reach workers, i wrap the page's Worker constructor so every new worker runs the same seeding code before its own script
3. virtual time. Chrome has this built in: Emulation.setVirtualTimePolicy. the page gets a fake clock that only moves when you give it a "budget". when nothing is happening, Chrome skips ahead for free, so a 5 second timer fires in ~7ms of real time. this is safe because if nothing is running, nothing can notice the time jumping. so Chrome jumps straight to the next timer
4. manual frames. launch Chrome with --deterministic-mode and it stops painting on its own. you call HeadlessExperimental.beginFrame to draw each frame and you hand it a fake frame time too. that matters because animations use the frame time to decide where they are, so a real timestamp would let the real clock sneak back in
the results
| config | dom | a11y | png |
|---|---|---|---|
| all fixes | 1 | 1 | 1 |
| no recorded network | 100 | 100 | 100 |
| no seeded rng | 100 | 100 | 100 |
| no virtual time | 100 | 100 | 100 |
| no manual frames | 1 | 1 | 2–8 |
| no fixes | 100 | 100 | 100 |
basically unique hashes over 100 renders. the "no manual frames" png count changes from run to run, more on that below
all four fixes on 100 renders return one DOM, one a11y (accessibility) tree, and one screenshot. and i actually checked the screenshot myself and its the actual finished page with the spinner mid-spin and everything. that check matters a lot as well because a page that never rendered also gives only 1 hash (more on that in trap 3)
drop any one of the first three and every single render is different. thats because each one feeds text straight into the page: the time, the random ids, the order of the API results. one changed character and the whole hash changes
drop manual frames and the html is still fine but the pixels differ. the html is fine because the frames only decide when things get painted, not what the page says. so what changes in the pixels? for this, i saved every screenshot and diffed them, here's what happened:
the only pixels that differ are the spinner, caught at a different angle each time. the CSS animation runs on Chrome's own real clock, so where the spinner is depends on the exact moment the screenshot lands. thats also why the count moves around: 2, 3, 3 and 4 on a quiet machine, and 8 while docker was busy in the background
trap 1: the order matters
my first try at virtual time just hung. i basically gave Chrome a 1000ms budget and then navigated. but on an empty tab nothing is pending so Chrome just fast-forwarded through the whole budget instantly and the page loaded into frozen time. with the budget gone the clock stays paused forever so none of the page's timers ever fire and the page never finishes loading
the lesson: pause first, navigate, then hand out time in small steps
trap 2: other threads race the clock
when i first tried virtual time with one big 1000ms budget, the worker's answer landed at a different time on every run. heres why:
chrome skips ahead whenever the main thread is idle. but a worker boots on its own thread in real time. so the clock jumps ahead while the worker is still starting up and so the message lands wherever it lands
the funny part is that the deterministic browser paper solved this exact problem. before the main thread starts a worker, it puts a placeholder in its event queue at a fixed future time, so the clock cant move past the placeholder until the worker's real message replaces it:
figure 3 from Cao et al., "Deterministic Browser", CCS 2017
Chrome's virtual time has no placeholder for workers. so what i do instead is hand out time in small 16ms steps (about one frame at 60fps) so the clock can only run a little ahead. with small steps, the worker answered at the same virtual time in every run
fyi that just shrinks the race, it doesnt eliminate it completely because say if a worker ever takes longer than one step to boot (say, on a slow or busy machine), then the clock can still run past it
but workers werent the only thread racing. without manual frames, the number of steps until the page was done changed between runs (it was 19 diff counts in 100 runs). my first guess was that maybe beginFrame just gave the other threads time to catch up. so i put my theory to the test (frames.mjs) with 30 runs each:
| frames | page waits on | unique step counts |
|---|---|---|
beginFrame | requestAnimationFrame | 1 |
beginFrame | setTimeout | 1 |
| chrome | requestAnimationFrame | 18 |
| chrome | setTimeout | 1 |
turns out my guess was wrong. the culprit is requestAnimationFrame. normally Chrome's compositor (which is basically the part that paints) fires animation frames on its own real clock and virtual time doesnt touch it. with --deterministic-mode, a frame only happens when i call beginFrame so i decide which step every frame lands on. swap rAF (requestAnimationFrame) for setTimeout and the steps are stable even without manual frames
and the same test found one more leak: performance.now() still wobbles by ~0.1ms between runs even with everything on (31.9, 32, 32.1). my guess for this is that Chrome blurs that clock on purpose so a page cant use a precise clock to time Spectre style attacks. the paper clearly states that this defense doesnt work because an attacker can just measure many times and average out the blur. and here it costs determinism too. my test page never prints performance.now() so the hashes didnt catch it
hopefully ill write more on the compositor and that clock in another write up, theres a lot more going on in here than i had initially expected
trap 3: things that only look deterministic
two times i got a perfect "1 unique hash" that was a lie:
- the frozen spinner. with a fake clock and no frames, the CSS animation sat at frame 0 forever. same hash every time, because nothing was rendering. a page that never moves is deterministic and useless
- the recording that wasnt saved. my first cross machine run disagreed with itself. every process re-recorded the "live" site which returns a random deal price, so every process replayed a different page. i fixed this by recording once, saving it to a file, and then replaying the same file everywhere
key thing to take away? always look at what you' re hashing
does it survive another machine?
i dont have a second machine, so i ran the all fixes config in docker on different distros and settings with a 100 renders each. each container brings its own OS files, fonts and libraries, which is most of what differs between two machines (but the CPU and kernel stay the same so this is not a full test). all 9 setups took about 5 minutes in total. every row was deterministic inside itself. the question is whether the rows agree with each other:
| setup | dom | a11y | png |
|---|---|---|---|
| my machine (ubuntu 26.04) | d5532b | d32fba | d3193c |
| ubuntu 24.04 | d5532b | d32fba | d3193c |
| ubuntu 24.04, 1 cpu | d5532b | d32fba | d3193c |
| ubuntu 24.04, 0.5 cpu + 1gb ram | d5532b | d32fba | d3193c |
ubuntu 24.04, --ipc=host | d5532b | d32fba | d3193c |
| ubuntu 24.04, other timezone + locale | d5532b | d32fba | d3193c |
| ubuntu 22.04 | d5532b | d32fba | 22c656 |
| debian 12 | d5532b | d32fba | 22c656 |
| debian 12 + noto fonts | d5532b | d32fba | 22c656 |
same chromium build everywhere (playwright 1.64.1)
the good thing is that the DOM and a11y tree matched everywhere. even at half a cpu where everything runs slower, absolutely nothing changed. thats virtual time and manual frames doing their job, making sure real speed never leaks in. the timezone and locale didnt matter either because the page prints the time as UTC and the clock itself is fake
but... the pixels (screenshot) split into two groups. the only pixels that changed were text. chrome said every system drew it with "Liberation Sans" but the font file was a different version: 2.1.5 on the newer ubuntus, 1.07.4 on ubuntu 22.04 and debian 12. a new font version can change the shape and spacing of letters just a teeny tiny bit, and that alone changes the pixels. so i put 2.1.5 on debian to prove it... and i got a third hash. the font settings (fontconfig) were different too. same font name, but different pixels. fontconfig decides things like anti-aliasing and hinting which change how each letter is drawn
and adding noto fonts changed nothing. i asked Chrome which font it really drew with, and it was still Liberation Sans, even tho noto became the system's default sans-serif. this is because Chrome asks for Arial and Linux swaps Arial for Liberation Sans, which is its look-alike. (to check, i told fontconfig to swap Arial for a monospace font instead, and Chrome's sans-serif text turned monospace) obviously, a font that is installed but never used cant change a pixel
takeaway: pixels depend on the whole OS font setup and not just one package. (i also tried ARM64 under emulation but Chrome's GPU process crashed under the emulator before it rendered anything, so ARM is untested)
so, can a browser be deterministic?
- on one machine, yes. DOM, accessibility tree and pixels, 100 out of 100
- across diff machines, the DOM and a11y tree hold. the pixels dont (only because the OS font setup sneaks in)
- so the "render contract" im going with is to hash the DOM + a11y tree everywhere, and only compare pixels inside the same pinned container image. a pinned image ships the exact same fonts and font settings every time which is the singular thing that broke the pixels
and the limits: one synthetic page, no web fonts, no video, no iframes, no service workers, and the fact that my worker fix only covers classic workers which in turn leads to module workers not being able to load my seeding code the same way, performance.now() still wobbles, and ARM is untested. overall, real sites will find new ways to be lucky
btw this is stage 0 of something im building: a replay browser for AI agents, so an agent's bad run can be replayed exactly, the same way a DST seed replays a bug. more on that soon ^^
read before starting: Yinzhi Cao et al., "Deterministic Browser", CCS 2017