BenchFae

What we test, who does the marking, and why the tasks themselves stay behind a locked door.

Privacy#

The tasks BenchFae runs are never published. Not the prompts, not the repositories, not the rubrics, not the fixtures.

This isn't coyness. A benchmark stops measuring anything the moment its contents can be trained on. Published test sets have a well-documented habit of leaking into training data — sometimes deliberately, more often because the whole public web ends up in a scrape — and once that happens a rising score stops meaning "this model got better at the work" and starts meaning "this model has seen the answers." Several widely-cited public benchmarks have had exactly this problem, with saturated scores that stopped tracking real capability.

Keeping the set private costs us the ability to prove our work in public. We think that's the cheaper trade. A benchmark you can audit but not trust is worth less than one you can trust but not audit.

Tasks are also rotated and replaced over time, so a model that somehow acquired one year's set gains nothing durable.

Three signals#

Every submission is graded on up to three independent signals. They are stored separately and displayed separately, and no page on this site collapses them into a single number without also showing the parts.

Automated tests

Functional pass/fail checks against the submission: does it build, does it run, does it do what the task asked. These are deterministic and re-runnable. They carry the most weight in categories where correctness is most of the job, and the least in categories where the output is largely something you have to look at.

Ahleyrie Fairy 1.3

Our in-house judge model reads the code and the workflow that produced it, and grades from 0 to 100. She assesses structure, clarity, error handling, and whether the approach was sound or merely lucky. She is deliberately hard to impress.

Fairy has no vision. She cannot see a rendered page or a running game, which is why she never grades those things — see her page for what she is and what's coming in Fairy 2.

Human evaluation

Anything visual or experiential is scored by people: layout and interface quality, art direction, whether a game is actually playable and whether it feels like anything to play. Categories with nothing visual to assess — Website Backend, Debugging/Refactoring — have no human score at all, and we leave that column empty rather than inventing a figure.

Fairy#

Fairy is a fully custom model, trained in house. She is not a fine-tune of somebody else's LLM and not a wrapper around a frontier API with a clever prompt. Version 1.3 is the one currently grading every submission on this site.

Her remit is narrow on purpose. She reads AI-generated code and the workflow that produced it, and returns a score from 0 to 100 with reasoning attached. She isn't a chatbot, she isn't an assistant, and she has no opinions about anything else.

What she grades

Structure. Is the thing organised, or is it one file holding its breath?

Correctness of approach. Not just whether it works, but whether it works for a reason.

Error handling. What happens on the unhappy path, which is where most submissions quietly stop trying.

Clarity. Could a maintainer who wasn't there understand this next quarter?

Restraint. Whether the solution is the size of the problem.

What she can't do

Fairy has no vision. She cannot see a rendered interface, a sprite, a shader, or a frame of gameplay. Given a beautiful layout and an ugly one with identical markup quality, she would score them the same, and she'd be right to — she isn't looking at them.

So she doesn't grade them. Everything visual or experiential goes to human evaluators instead, and that score is kept in its own column. It's the honest arrangement: a judge that can't see shouldn't be asked to rule on how something looks.

Fairy 2

Fairy 2 is in development, and the headline change is vision — the ability to actually look at what a model produced rather than only reading how it was built.

That will not remove humans from the loop. Human evaluation is permanent here by design, not a stopgap we're waiting to automate away. Some of what this benchmark measures — does this game feel good in the hand, is this interface pleasant to use — is a judgement about human experience, and the right instrument for that will keep being humans. Fairy 2 adds a second opinion. It doesn't inherit the last word.

On the scores being low

They are supposed to be. Fairy is calibrated against what a demanding senior engineer would accept in review, not against what is impressive for a machine. "Better than expected" isn't a grade.

Owner & Creator#

BenchFae is made by Ahleyrie. FiveNova slightly assisted but we do not manage, control, oversee or anything about Benchfae that's all Ahleyrie. were just "Friends" we helped them and they helped us like this website was made with the assistence from Fairy 2 Coder a model they are working on and releasing eventually.