Sign in with Google, connect a repository, and describe what you want built and
what it has to beat. The loop runs in your CI on a branch of that repo — your code never
leaves it, the model runs on your key or your Claude subscription, and every generation that
changes something arrives as a pull request you review like any other.
Sign in. Google only, and access is by invitation while MacroEvo is in beta.
Connect a repository. MacroEvo installs as a GitHub App, so it
gets exactly the repos you pick and revoking it is one click. Nothing is granted by default.
Start a loop. Four steps: what you’re building (pick a use case and it fills in
the rest), the goal and the benchmark it has to beat, the three
trap questions, and where it runs. The only required answers are the goal
and the benchmark.
Watch the dashboard. Every round posts a status (running →
won/lost) and, for visual rounds, both images side by side — that
comparison is the only way to tell whether the loop is converging or just spinning.
Stop anytime. Stop is cooperative first (the loop checks before starting a new round
and reports its work), then a hard kill as backstop.
Don’t skip the harness question. A loop without a working harness
either burns its early rounds building infrastructure or — worse — captures the wrong surface,
and every round it collects is inadmissible evidence. When an example shows
build, expect a slow, infrastructure-heavy round 1 before real
judging starts.
Four things decide whether a loop works
Everything else is scaffolding. Get these wrong and the loop will run happily for hours and produce nothing you can use.
The benchmark
The reference your work is screened against — and the single thing most likely to
make a loop worthless. It has to be named, fetchable and comparable: something the critic
can actually obtain and put side by side with yours.
Given a vague benchmark — "award-winning sites", "production quality" — a critic invents a
comparison and approves everything. The loop still runs. It just stops meaning anything.
The judge
What the critic physically does to decide. Judge type, not topic, is what groups
this work: a game and a landing page are the same problem — capture a frame, compare it blind.
A benchmark tells you what to beat; the judge is how the comparison is actually
made, and getting it wrong is how a loop collects rounds of evidence that turn out to be
inadmissible. There are four, set out in
Judge types.
The harness
The command that captures your work so it can be judged — a headless screenshot, a
benchmark run, a held-out eval. Without one, a loop spends its opening generations building
infrastructure, or worse, captures the wrong surface and collects rounds of inadmissible
evidence. That failure is unrecoverable: you find out when you try to ship.
The traps
Three questions asked before a loop starts, compiled into hard fail conditions
rather than advice — and restated whenever a loop resumes, because a loop that forgets where its
work ships will quietly start judging the wrong thing again.
Where does it ship?
The critic judges it there. Anything captured from another tool is not evidence, however
good it looks.
What varies?
Every round also judges cases the benchmark doesn't cover, so winning can't just be overfitting
to the one scored example.
What budget?
A fail condition, not a preference. Quality benchmarks are one-directional, and a loop will spend
anything to win one.
How a loop reports itself
Everything on the dashboard is something the loop said about itself while it ran.
It reports over an internal CLI that lives inside the runner image and is called by the headless
session, not by you — there is nothing to install and nothing to type. It is worth knowing the
shape of, because it is what the round history means and it is what you are reading when a run
goes strange.
When
What the loop records
At the start
Registers itself with your goal, your benchmark and the repo it is
working in — this is the card that appears on the dashboard.
Mid-round
A running status and the critic’s latest single biggest gap. This
is the line on the card that keeps changing.
Closing a round
A terminal won or lost with a one-line
summary, plus one entry per judged piece — and, on a visual round, both images, so the
comparison is on the page rather than in the loop’s description of it.
Before each new round
Checks whether you have asked it to stop, and winds down
cleanly if you have.
At the end
A final status: won, lost, or stopped.
A round only counts once it closes with a terminal status. One
abandoned mid-build never counts, so a crashed turn cannot silently inflate the round tally —
and a round stuck at building on the dashboard means the turn ended before its
critic reported, not that the work was lost.
Judge types
What the critic physically compares. This is the organizing axis — topic ("games",
"landing pages") says nothing about how a round gets decided; judge type says everything.
Judge
How it works
Blind A/B
Capture ours, put it beside the reference with labels stripped, ask which is real. Binary — never a score out of ten, because scores drift upward every round.
Measured
Run the benchmark, compare figures on identical inputs and pinned hardware. The critic can't hallucinate a number — the most reliable judge there is when the measurement setup is honest.
Naive-agent completion
Spawn a clean agent per task with ONLY the interface under test; record whether it succeeds and where it stalls. The critic is a simulated user, not a reviewer.
Blind read
Strip bylines from ours and the reference; judge which one the intended reader understands faster — not which is more impressive.
Harness registry
ready means a working implementation exists to point
at. build means round 1 has to write it first — worth knowing before
you start, because it predicts a slow opening round.
Harness
Captures
Status
Headless browser screenshot
What a visitor actually sees, from a chosen viewpoint or viewport.
ready
Lighthouse + viewport screenshots
Design at 1440px and 390px, plus load performance.
ready
Recorded clip + extracted frames
Motion, timing and feel — things a still frame can't lose on.
build
Held-out eval runner
Pass rate on cases the builder never saw.
build
Naive-agent runner
Whether a fresh agent completes real tasks with only your interface.
build
Benchmark runner
p50/p95 latency and cost per run, plus output equivalence.
build
Blind pairing
Two artifacts, labels stripped, re-encoded identically, randomly assigned. Also crops any watermark band off both so a visible stamp can't be an authenticity tell.
ready
Examples
The curated set MacroEvo ships with — the same list the four steps
offers. Pick one on the left to see how its loop moves information: what you set, who builds,
what turns the artifact into evidence, who judges it, and where the verdict goes.
🏠
Blind A/B
3D in a browser
Goal
photoreal 3D rooms for <property>, built from its listing photos, that look real inside the three.js walkthrough in the browser
The benchmark
the property's own listing photos, judged blind against headless-Chromium screenshots of the real viewer from the same standing point — never an offline render — while holding 60fps
Judge — Blind A/B against a real artifact
Capture ours, put it beside the reference with labels stripped, ask which is the real one. Binary — 'which is better', never a score out of ten, because scores drift upward every round.
Harness — Headless browser screenshot ready
What a visitor actually sees, from a chosen viewpoint or viewport. Drives the REAL page/viewer in headless Chromium. Prefer a GPU backend: a CPU rasteriser renders a 3D scene at ~3fps regardless of how cheap the asset is, so a frame-rate budget measured there is noise.
Trap: Winning in Blender. Offline renders carry procedural materials and path-traced light that glTF cannot export, so the models ship as untextured white boxes.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
🎨
Blind A/B
A landing page
Goal
a landing page for <product> that a visitor would believe came from a top-tier design team
The benchmark
<a named competitor's live page>, screenshotted at 1440px and 390px and compared directly, plus a Lighthouse score no worse than theirs
Judge — Blind A/B against a real artifact
Capture ours, put it beside the reference with labels stripped, ask which is the real one. Binary — 'which is better', never a score out of ten, because scores drift upward every round.
Harness — Lighthouse + viewport screenshots ready
Design at 1440px and 390px, plus load performance.
Trap: Winning at desktop width only, or winning on looks while the page paints in four seconds. Score mobile and load time in the same round or they never improve.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
🕹
Blind A/B
Game feel
Goal
a browser game whose core loop feels as responsive and juicy as a shipped title
The benchmark
<a named game>'s real gameplay footage — same clip length, same actions performed
Judge — Blind A/B against a real artifact
Capture ours, put it beside the reference with labels stripped, ask which is the real one. Binary — 'which is better', never a score out of ten, because scores drift upward every round.
Harness — Recorded clip + extracted frames build in round 1
Motion, timing and feel — things a still frame cannot lose on.
record N seconds at a fixed input script, then extract frames at intervals
Trap: Judging a screenshot. Feel lives in motion, input latency and juice; a still frame cannot lose the comparison.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
🎬
Blind A/B
A generated episode
Goal
a 2-minute episode of <series> that a viewer watches to the end without bailing
The benchmark
a named show's real 2-minute segment, compared on keyframes and on a full watch-through with audio
Judge — Blind A/B against a real artifact
Capture ours, put it beside the reference with labels stripped, ask which is the real one. Binary — 'which is better', never a score out of ten, because scores drift upward every round.
Harness — Recorded clip + extracted frames build in round 1
Motion, timing and feel — things a still frame cannot lose on.
record N seconds at a fixed input script, then extract frames at intervals
Trap: Per-shot quality that dies in assembly. Face, wardrobe and location continuity ACROSS cuts is what separates an episode from a reel.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
🧠
Measured
A system prompt
Goal
a system prompt for <agent> that gets the task right on inputs it has never seen
The benchmark
a HELD-OUT eval set of real cases with known-good answers, scored pass/fail by a separate grader that never sees which prompt produced which answer
Judge — A measured number
Run the benchmark and compare figures on identical inputs and pinned hardware. The critic cannot hallucinate a number, which makes this the most reliable judge there is — when the measurement setup is honest.
Harness — Held-out eval runner build in round 1
Pass rate on cases the builder never saw.
run every candidate against the same held-out split; report pass rate + failures
Trap: Tuning on the eval set. Split it before round 1 and never let the builder see the held-out half, or you are measuring memorisation.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
🔬
Blind A/B
A figure generator
Goal
a system prompt/rules/tool harness for an SVG scientific-figure generator that produces publication-quality figures from a text description ALONE, judged against real published figures it is never shown
The benchmark
a held-out split of real published-figure images the builder never sees — each one rendered from the candidate SVG via the generator's OWN renderer and blind-paired (labels stripped) against the real reference image; win rate over the held-out split is the score, never the training subset
Judge — Blind A/B against a real artifact
Capture ours, put it beside the reference with labels stripped, ask which is the real one. Binary — 'which is better', never a score out of ten, because scores drift upward every round.
Harness — Held-out reference-figure eval, blind-paired build in round 1
Each held-out case's rendered svg blind-paired against its real published reference figure; win rate over the held-out split is the round's score. The two ingredients already exist and are 'ready' on their own — a Playwright SVG->PNG renderer and the blind_pair primitive — but nothing yet wires a fixed description<->reference-image corpus or the train/held-out split around them. Round 1 is that wiring, not the visual judging.
for each held-out case: generate an SVG from its text description ALONE -> render to PNG -> blind_pair.py <case> <round> ours.png reference.png
Trap: Tuning the prompt on the images it's allowed to see, or writing the text descriptions by having a vision model look at the reference and describe its literal rendering (palette, layout) rather than its scientific content — either one lets the 'never shown the image' separation leak, and the blind comparison stops being blind. Freeze the split and vet the descriptions before round 1.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
🔌
Naive agent
An agent-facing interface
Goal
an MCP tool surface (or SDK) for <product> that a fresh agent uses correctly with no help and no docs
The benchmark
a clean agent, given ONLY the tool definitions, completes N real end-to-end tasks — no hints, no repo access, no retries on your say-so
Judge — Naive-agent task completion
Spawn a clean agent per task with ONLY the interface under test, and record whether it succeeds and where it gets stuck. The critic is a simulated user, not a reviewer.
Harness — Naive-agent runner build in round 1
Whether a fresh agent completes real tasks with only your interface.
one clean agent per task, no repo access, log success and the step it stalled on
Trap: Grading it yourself. If the critic can read your repo it is not naive, and it will forgive names and defaults a real client could never guess.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
⚡
Measured
Faster and cheaper
Goal
cut <endpoint or pipeline> latency and cost per run without changing what it returns
The benchmark
the current build's own numbers — p50/p95 latency and $/run on the same inputs and the same machine — with output that stays provably equivalent
Judge — A measured number
Run the benchmark and compare figures on identical inputs and pinned hardware. The critic cannot hallucinate a number, which makes this the most reliable judge there is — when the measurement setup is honest.
Harness — Benchmark runner build in round 1
P50/p95 latency and cost per run, plus output equivalence.
fixed input set, pinned hardware and cache state, before/after on both
Trap: Measuring on the wrong machine or a warm cache. The same 3D asset reads 3fps on a CPU rasteriser and 120fps on the GPU; pin the config or the number is noise.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
✍️
Blind read
Writing vs. an author
Goal
a 2000-word explainer on <topic> for readers who are smart but not specialists
The benchmark
three actual published pieces by <named author or publication>, same length and format
Judge — Blind read
Strip bylines from ours and the reference, then judge which one the intended reader understands faster — not which is more impressive.
Harness — Blind pairing ready
Two artifacts, labels stripped, re-encoded identically, randomly assigned. Also crops any watermark band off BOTH images so a visible stamp can't be used as an authenticity tell.
blind_pair.py <piece> <round> <ours> <reference>
Trap: A critic who can tell which is yours, or an expert judging writing meant for non-experts. Strip the bylines and state the reader.
How the information moves. Amber is what you set, blue is the build path, violet is judgement, and the dashed red line is the loop closing — the gap the critic names is the next generation’s brief.
Hosted mode (GitHub App)
Your code never leaves your repository. MacroEvo sends the brief and receives the results; the model runs on your key or your Claude subscription, and each generation that changes something arrives as a pull request you review like any other.
Local loops run on your machine against a repo you point at. Hosted loops run in
your CI, in a repo you've granted access to — MacroEvo installs as a GitHub App rather than
holding a personal token, so access is exactly the repos you pick and revoking it is one click.
Create the App once (public — "Any account" — so it's installable outside its own org)
with these permissions: Contents (read & write — the loop works in a branch),
Workflows (write — separate from Contents), Pull requests (read & write —
results arrive as a PR), Actions (read & write — dispatch + read results). Events:
installation, installation_repositories.
Set the webhook URL to https://<service>/api/webhooks/github and a random
webhook secret.
Generate a private key and set MACROEVO_GITHUB_APP_ID,
MACROEVO_GITHUB_APP_SLUG, MACROEVO_GITHUB_PRIVATE_KEY (PEM, newlines
escaped as \n) and MACROEVO_GITHUB_WEBHOOK_SECRET.
Copy runner/workflow.yml into the target repo as
.github/workflows/macroevo.yml, and add MACROEVO_API,
MACROEVO_TOKEN and either ANTHROPIC_API_KEY or
CLAUDE_CODE_OAUTH_TOKEN as repository secrets.
Repos appear in the launch picker as soon as the App is installed — the webhook keeps
the list in step, so removing a repo on GitHub removes it here too.
Common pitfalls
Symptom
Cause
Every round wins
The bar is vague, so the critic invented a comparison instead of fetching a real one. Name something specific and fetchable.
Round 1 takes forever
Expected when the harness status is build — the loop is writing the capture infrastructure before it can judge anything.
Loop keeps "winning" but the result looks wrong when you check it
Judging the wrong surface — confirm the harness captures where the artifact actually ships, not a proxy.
Quality plateaus then quietly regresses
No budget trap set, or the resumed loop forgot it. Traps are restated on every resume for exactly this reason.
Score keeps climbing but only on the one example you gave it
No "what varies" case-set — add held-out variants the bar doesn't cover.