Desk Paintings

Two quality complaints drove round 7: the rooms read fuzzy near 1:1 on retina glass, and the baked-in screentone needed a custom notch filter to remove. Both fixes turned out to be embarrassingly direct. The fuzz: the 2K masters were being upscaled by big displays, so round 7 re-renders at 4K (the API accepts imageSize: "4K", uppercase K — and has been caught silently ignoring it, so the script’s callers verify the returned dimensions). The dots: one clause in the remaster prompt —

— and the screentone simply isn’t painted. A 2× zoom on any flat region comes back smooth, the descreen filter retired undefeated, and the diff-based sprite pipeline got dramatically easier (no more drifting carrier phase lighting up 30% of every diff). The remaster is a re-render of the descreened v3 room used as a reference image, so the composition survived within a quarter percent — the screen quad moved from (30.09, 23.24) to (30.11, 23.24).

Bonus: the clean 4K paint compresses absurdly well. The shipped 3520px-wide webp weighs 259 KB — smaller than the 2200px jpeg it replaces.

Round 7’s big idea: stop asking the model to re-paint a character mid-pose into the room, and instead take the characters off the room. Each one gets a zoomed crop-edit in the one phrasing the model always obeys — an erasure (“REMOVE the small yellow dog… paint the casing and the night window that were behind him”). The erase renders reconstruct what was behind beautifully: Jake’s lamp stem and hanging pull-chain came back intact, Pochita’s spine text continued correctly on the second try.

Pasting those renders back as whole rectangles failed, though: every re-render drifts its color grade a little, and on a flat night sky that reads as a hard blue box — one paste clipped the moon’s halo. The fix is to clip each paste to the character’s own diff mask (threshold, opening by reconstruction, dilate, feather — the sprite recipe): the plate stays pixel-identical to the painting everywhere except under the character outline, and whatever tone drift remains hides exactly beneath the character’s rest cutout, which is cut from the same diff.

The cast then returns as webp layers: a rest cutout per character (diff of painting vs empty plate — the whole character, alpha feathered) plus pose patches (diff of a pose re-render vs the painting — just what moved). Patches ride CSS clocks on top of the rest layers: two wag bursts and a blink-single-then-double for Pochita, lazy kicks for Jake, a quick stilted rattle for the kodama. The fairy needed no frames at all — her cutout slides down inside an overflow: hidden rect whose bottom edge is exactly where the books start occluding her, so she just… ducks.

The verdict (v4b): reviewed on the real machine, every idle loop except the blink read as puppetry — two poses swapping is a flipbook with too few pages, however clean the masks. So the layers were benched, the characters were painted back in, and the round’s real win got kept instead: Pochita and the fairy were moved — erased from their old spots via the plates and re-added peeking out of the dark recess under the monitor, where both orientations can actually see them. One blink patch ships. The erase→plate→cutout machinery earned its keep as an editing tool, not an animation rig.

v3 swapped worlds with a screen-space glitch. v4 treats the whole room as the screen: both paintings become WebGL textures and a fragment shader plays one of three warps from the exact point where you struck the glass. RIP ripples the room outward (a damped sine on the distance field, RGB sampled at three phase offsets), then tears it open along a seam perturbed by fbm noise — the far world shows through the widening tear behind a hot rim, then settles flat. SHOCK is a single chromatic shockwave; MELT drops the room in 68 lagged strips, Doom-style.

The choreography contract is what makes it seamless: the DOM keeps showing the old world underneath, the canvas mounts transparent (no black flash while textures load), and the shader’s final frame is a plain cover-fit render of the destination — so the DOM swap at the end changes nothing visible. The cozy-side texture is composited on the fly from the empty plate plus every rest cutout, so nobody vanishes mid-rip. Cover-fit math in the shader mirrors the CSS; reduced-motion users skip straight to the swap; the v3 glitch survives as a bar option.

The brief asked: could the characters be generated some other way so they can really animate? The survey, for the record. Video generation — Veo 3.1 does image-to-video through the same Gemini API key (Lite at ~$0.03–0.05/s, Fast at ~$0.15/s, 8-second clips); Runway, Kling and Pika are the same idea on other stacks. Great for clips, wrong for this: you get 720p+ video of the whole scene re-interpreted, camera and paint included, and extracting a loopable, alpha-matted, style-locked sprite out of that is harder than the problem it solves.

Cutout tools — rembg (with its isnet-anime model) and BiRefNet are the standard background removers, and for a character on a plain ground they’re excellent. But they guess a matte from appearance. The diff-alpha approach here gets the matte from evidence: the same room rendered with and without the character differs in exactly the character’s pixels, registration-perfect, in-style, free.

So the shipped mechanism is: pose variants via the image model’s keep-clause edits (one thing changes per render), masks via morphological diff, playback via CSS clocks — plus the clip-rect trick for the fairy’s duck, which is pure geometry. Frame-swap puppetry at 2–3 poses per character reads as an idle loop; the earlier attempt to swap whole characters between renders shimmered (bodies drift between generations), which is why the patches are minimal diffs against the very painting they sit on.

The portrait remaster shipped with the live desktop visibly narrower than the drawn glass, and the cause took a magnifying glass to find: the quad-measuring walk stops when the luminance turns “bright,” and the painting has a bright window reflection painted ON the glass, right of centre. The walk hit the reflection’s highlight, called it the bezel, and reported a screen 48.6% wide when the true dark glass runs 61.2% — so the desktop rendered 13% too narrow.

The fix is to read full luminance profiles through the screen instead of trusting a single threshold walk: glass sits at 35–65, the reflection ramps to ~185 and falls back to 53, and only then comes the true bezel at 160–205 behind an ink line. Same story on the deck mirror, whose right edge had been called 12% early. Every quad now gets a red-rectangle render over the painting before it ships.

Portrait lean-in still cuts the painting into three slices with only the tube stretching — and now that Pochita and the fairy live in the recess directly under the screen, the bottom slice shows them immediately: characters up top, readable desktop, peekers below. No skipping required.

Six rounds, each one narrowing. Round 1 explored style. Round 2 locked the rule that made the whole project possible — the CRT must face the viewer dead-on, so its screen is an undistorted rectangle a live desktop can be composited into:

Rounds 3–6 stopped generating and started editing. A single sentence — call it the keep-clause — front-loads every edit prompt, and the changes are numbered after it:

The cyberdeck mirror is the same room re-lit, generated from the finished cozy painting so the screen lands in the same rectangle:

There’s a sharp line through what a whole-image edit will obey. Erasing works every time: remove the toad, remove the duplicate fairy, remove all print from the mug. Retypesetting never works. Asked to move “UP!” up a line, the model just hands back the original glyphs — three separate generations, including one on a zoomed crop of nothing but the mug. The sticky note did the same thing: told to drop its pwd: label, it reproduced it faithfully.

The trick is to hand it a zoomed square crop with exactly one instruction, phrased as an erasure. crops.mjs cut takes the square out; crops.mjs paste puts the edited one back through a feathered alpha (a white rect inset and blurred), so the middle is the model’s new pixels and the rim melts into the original. Gemini returns a 512² crop as 2048², so the paste always resizes.

It also cuts the other way. The sticky-note crop happened to contain the calculator, which came back with a scribble across its LCD, so the original calculator gets pasted straight back over it.

After the model wouldn’t fit LITT UP! on one line, the mug was blanked (an erasure — it obliged instantly) and the print typeset by hand. Not painted on flat, though: wrapped.

Model the mug as a vertical cylinder seen slightly from above. A horizontal circle on it projects to an ellipse with semi-axes (a, b), measured off the blank mug’s own rim and base — a = 127px, b = 26px in the landscape painting. A point at angle θ from dead-centre lands at

So the print sags toward the middle (that b·cos θ peaks at θ=0) and compresses toward the rim (dX/dθ = a·cos θ). Both fall out of inverting that map per output pixel and sampling a flat text strip whose x axis is θ, 3×3 samples off a 4× strip so the squeezed edge glyphs stay clean. Multiply-blend it on and the painted ceramic shading shows through the ink. The line ends up ~10px higher than its middle — exactly the sag of the mug’s own base.

This lofi style comes with a fine screentone baked in: a checkerboard of dots on a diagonal lattice. On these gallery cards it’s invisible. Behind The Machine III the room is drawn near 1:1 on a retina display, and it reads as a grid laid over everything but the CRT glass.

The obvious fix — average each pixel over its cell in a 5×5 lattice and subtract — fails completely. A 2D DFT on a flat patch says why: the carrier period is 5.12px, not 5.00, so the dots slide out of phase with any integer pixel grid. Measured, the phase-average cancels to ~0.25 grey levels while the local dots swing ±18.

What works is an adaptive notch — complex demodulation, the standard way to pull a narrowband carrier with drifting amplitude out of a signal. Find the carrier, lowpass img · e^(-i2πk·x) to recover its slowly-varying complex amplitude, then subtract that band back out. The lowpass is a tent, so the notch is narrow: the dots die, the linework (broadband) lives. A dot lattice rings at every reciprocal-lattice point m·k₁ + n·k₂, so all of those get notched too.

The carrier is found coarsely by DFT on the flattest 256² window and then refined on the whole image from the phase ramp of the demodulated signal — a wrong k shows up as a linear phase drift, so correct and repeat. Tuning the notch width is a straight trade: at radius 6 the dot band drops 87% while broadband detail (book-spine linework) loses 2%.

Pochita’s blink is a second whole painting, generated from the first, in which only his eyes changed. Overlaying a rectangle cut from it doesn’t work — the model repaints everything a little, so the books behind him visibly shift each time he blinks.

Instead the frame is masked down to only the pixels that actually changed: channel-max diff against the base, threshold, then an opening by reconstruction (erode to find seed blobs, regrow each seed inside the raw threshold) — a plain morphological open eats thin real strokes along with the noise, reconstruction doesn’t, because noise has no surviving seed. Dilate, feather, bbox, bake the mask into the webp’s alpha. The blink ships as 3.4 KB.

This only works on descreened images. Against the raw paintings the drifting halftone lights up ~30% of any region with diffs up to 247 — the change is invisible underneath it.

Two other bits of morphology went in the bin along the way: sharp’s chained .blur().threshold() calls don’t execute in sequence (they’re pipeline parameters — the later call silently overrides the earlier), and .blur() on a large sparse single-channel raw buffer returned all zeros. Everything here is hand-rolled separable min/max filters.

A 16:9 painting on a phone is a stripe. So portrait gets its own painting, recomposed vertically: the monitor rides a riser of stacked FOUNDATION paperbacks, the banker’s lamp peeks over its top edge, moon window above. Each orientation also has its own deck mirror and its own blink sprite, since every rect is in painting-percent coordinates.

The screen rectangle can’t be eyeballed — a live desktop sits in it. measure-quad.mjs finds it from the pixels by walking outward from a seed until the luminance stops being dark glass. Two thresholds, because the two edges differ: left and right run into the bright bezel face, so a high threshold there walks straight past the window reflection painted on the glass; top and bottom meet the bezel’s inner lip, which is only mid-grey itself, so those need a low one or the walk leaps the lip and lands on the dark outer trim 20px too far. Five parallel probes, median.

Leaning in on a phone would still leave a letterbox, so the painting is cut into three horizontal slices: the strip above the tube, the tube itself, and the desk below. Only the middle one stretches. The outer two are hard-capped (22% and 14% of the viewport) so the screen wins the height, and the top slice is anchored to its bottom edge — what survives the cap is the strip right above the tube, so the kodama and Jake stay perched on it. The live desktop inside never distorts; it keeps a uniform pixel scale and simply gains logical height.

The kodama’s clockwork head. A circular crop of the painting itself, positioned over its own head so rotating it turned the face like clock hands with no seam — 12s of ticked wind-up, spring release, damped wobble, all in CSS keyframes, no asset. It was technically the nicest thing here. It looked like a puppet.

Jake’s swinging legs. Three generations in a row swung the same leg no matter how the prompt described the other one, so the cycle became rest → low swing → high swing → rest. Also cut.

Hypnotoad. The real broadcast toad, a transparent 15-frame gif from the show, looping under a still frame that lifted for ~2.6s every 11s so he only spasmed at long intervals. He never appeared on phones and he never belonged in this room.

The running cracks. The chipped corner still generates its radial hairlines; they’re simply not drawn. A clean bite reads as a chip. Hairlines shooting across the desktop read as a broken monitor.