Five problems are hiding in this company. Can you find them?
Fernwell is a fictional B2B software company we built. We planted five real problems in its data, plus five red herrings that look like problems but aren't. Then we gave the same data to plain Claude and to INVAR. Try it yourself first.
THE BRIEF
What Fernwell is trying to do
Fernwell sells onboarding software to customer success teams: 1,239 customers and $637K in monthly recurring revenue. Its leadership wrote down four goals for the year. A problem matters if it stands in the way of one of them.
From Fernwell's own doc, Mission and FY2026 goals. The data covers April to September 2026.
THE EVIDENCE
Every report each tool gives you on its own
This is what a team can export from Stripe, HubSpot, PostHog and its Slack question log separately, plus five company docs. No records are joined across tools. It is exactly what plain Claude was given.
YOUR CALL
Ten leads. Five are real.
Each lead is true in the data. Some point at a real problem; others are red herrings: seasonal noise, a tiny sample, a dip that recovers on its own. Mark each one, then reveal.
EVIDENCE BAG · SEALED
Two investigators worked this case.
Plain Claude and INVAR each filed their findings. Make your calls on the leads, then break the seal to see who solved it.
THE REVEAL
Same data. Same model. Two answers.
Both sides ran on Claude Opus 5.5. Plain Claude got the reports above and was asked for the most important problems and what to do. INVAR read the same company's data and joined the records across tools first.
PLAIN CLAUDE
2 of 5 planted problems found 9 findings: 3 on the 2 problems it found, 2 red herrings, 4 others. See how each one countsINVAR
5 of 5 planted problems found 7 findings: the 5 planted problems, plus 2 smaller real ones it flagged for review. 0 false, 0 red herrings. See all 7Plain Claude is good at what one tool can show.
It caught the Salesforce sync trouble and the workflow editor rollout, because each shows up in a single export. It missed the three that only appear when records are joined: which plan the busy accounts are on, which campaign brought them, what the billing system actually charges. It said as much itself:
“That cannot measure activation by signup cohort or by acquisition source.”PLAIN CLAUDE, FINDING 9 · RECOMMENDING SOMEONE BUILD THE JOINED VIEW INVAR STARTS FROM
Problem by problem
Both sides are quoted word for word from the run file. Nothing is edited. Open a problem to compare the answers.
Loading the run…
Read both answers in full
Every finding, in the order each side ranked them, labelled by how it counts. Five problems were planted, but both sides were free to report anything else they saw. INVAR's two extras are real patterns in the data that nobody planted, which it marked "needs review" at low confidence. Plain Claude's other findings describe the data rather than name a problem the answer key can check.
Plain Claude's full answer
INVAR's full report
HOW WE RAN IT
The fine print, in plain sight
What each side got
Plain Claude (claude-opus-5-5) got the exports above and this instruction:
It was asked for up to 10 problems, most important first, each with its numbers and a specific recommendation.
INVAR read the same company's records, joined them across tools, and worded its findings with the same model. Its numbers come from the data, never from the model: a sentence with a number the data doesn't support is dropped.
How we scored it
Each finding was checked against Fernwell's answer key, which was written when the data was generated. Our automatic grader credited plain Claude with 1 of 5, because it filed the workflow editor finding under a different label. We read every answer and credit 2. Pricing counts as a miss: Claude saw prices running below list, but never found that the doc and the billing system disagree.
Run on 13 Oct 2026, code . Download the raw run file.
What this doesn't prove
Fernwell is the company we built INVAR on, so INVAR has seen it before. It isn't a blind test. A fair one uses a company nobody on the team has looked at.
We run those too: worlds written by a separate agent, scored once, with the pass bar signed beforehand. Across four of them INVAR found 18 of 20 planted problems (95% interval roughly 70–97%). It has failed some harder tests: experiments with hidden costs are caught only about 60% of the time.
YOUR COMPANY, NEXT
Run the same investigation on your own data.
A free 30-day pilot on one question that matters to you. Connected in under 15 minutes, read-only, deleted within 5 days when it ends.
Join the pilot