What it tests

Y Combinator’s Fall 2026 batch has companies. Every model answers one question about each of them: if this startup sold its core product in India, which one sector regulator would it mainly answer to? It picks one of six answers and gives a short reason.

A model does well when its answers match an answer key that was written without looking at any model’s answers. The ranking uses accuracy only. Speed is shown, but it doesn’t decide the order.

Where the companies come from

The profiles come from Firecrawl Alexandria’s Y Combinator data: one search for the Fall 2026 batch, then each company’s full profile, 25 at a time. Fetching the whole batch cost Firecrawl credits. No website was crawled and nothing was added by hand.

Each model sees four fields from a profile: the name, the one-line pitch, the industry and the first 800 characters of the description. This is the exact message for the first company:

The question and its six answers

Every model gets the same definitions, word for word:

The reply has to be JSON with a reason and one of the six categories. The reason comes first, so the category follows from it rather than the other way round. Every model gets the same system prompt:

How each model is called

Each model is called directly through its own provider’s API by one script on one machine. No gateway, router or cache sits in between.

The same for every model:

• The same system prompt, the same company text, temperature , and a 400-token limit on the reply.
• Reasoning is off, or at the lowest setting the provider allows, so no model gets extra thinking time the others don’t.
• A rate-limit or server error is retried up to five times, with a longer wait each time. The question is never reworded, and only the call that answered is timed.
• A reply that fails, or can’t be read as one of the six categories, counts as “No answer”, which is always wrong. It’s never filed under “DPDP only”, the most common correct answer, because that would reward a model for failing to answer.
• Answers are saved as each API returned them, one file per model. None was edited by hand, and no model was run again to replace a weak result.
• API keys stay on the machine that ran the test. They aren’t in this page or in any published file.

The line-up is the one from alpie-pong, an earlier speed comparison: models of about Alpie Seed’s size or larger, each from a different lab, that can answer without reasoning first. Qwen 3.8 27B was in the first line-up, but Cloudflare’s free daily allowance ran out before it finished a full run. It’s left out rather than shown with partial results.

The answer key

An AI assistant labelled every company from a written rubric. It used only the four fields the models see and didn’t look at any model’s answers. Each label has a confidence () and a one-line rationale, all of which are on the results page. Nobody who knows Indian regulation has reviewed the key yet.

The rubric settles borderline cases with four rules:

1. Selling to banks, insurers or hospitals doesn’t make a company regulated. Only a product that moves the money, sells the policy or makes the diagnosis does.
2. Label the product the company sells today, not what it might expand into.
3. A main regulator outside the six (energy, real estate, food, labour, vehicle safety) maps to “DPDP only”.
4. Dual use goes to the more specific regulator: a defence drone is Space & Defence, a clinical-grade wearable is Health.

The companies are split into development companies and held-out ones, with a fixed random seed and every category spread across both. Development companies are used only to choose the prompt. Every score is on the held-out ones.

How it’s scored

of companies are “DPDP only”. A model that gave that answer every time would get accuracy, so accuracy alone can’t rank models. The headline measure, chosen before any model was scored, is macro-F1: an F1 score for each category, averaged, so a rare category counts as much as a common one. The always-“DPDP only” answer scores .

• 95% range: the held-out companies are resampled 2,000 times and every model is rescored each time. When two models’ ranges overlap, the gap between them isn’t reliable.
• Accuracy: share of held-out companies where the answer matches the key.
• Regulated caught: of the companies the key gives a sector regulator, the share where the model named that same regulator.
• Wrongly flagged: of the companies the key marks “DPDP only”, the share where the model named a sector regulator anyway.
• Per answer: the median time for one call, from request to reply.
• Replay finish: when the last answer lands in the replay, explained below.

Cards and table rows are ordered by macro-F1, and the leading card gets the page’s one accent colour. Whoever leads each other measure gets a badge, so a model that wins on speed or recall still gets the credit.

Choosing the prompt

There were two candidate prompts. The first, v1, asked the question with the definitions but no rules for borderline cases. The models mostly erred by naming a sector regulator for companies that only sell to regulated industries. Prompt v2 adds the rubric’s first three rules, so the models answer the same question the key answers.

Both prompts ran at temperature 0 on the development companies with all models. The rule, fixed before those runs were scored: the prompt with the higher average macro-F1 across all models wins, so no single model decides the wording.

The replay

The race replays real, recorded answer times. Each provider’s key has its own rate limit, so the real runs sent different numbers of requests at once. Replaying them as they happened would show rate limits, not models. The replay sends requests at a time for every model, in the same company order, with each answer taking exactly as long as it really did. Retry waits are left out.

Every call was timed from one Mac in India, so the distance to each provider’s servers is part of the time. Sarvam’s API is in Mumbai, which gives it a shorter round trip than providers abroad.

What keeps it fair

• Same prompt, same company text, same answer format and same temperature for every model.
• The prompt was chosen by the average over all five models, on companies that aren’t scored.
• The ranking measure was fixed before scoring, and the leaders on other measures are credited.
• A failed reply counts as wrong, never as the most common answer.
• Rate limits are taken out of the replay, so a model on a free key doesn’t look slow because of its plan.
• No answer was edited, re-run selectively or dropped.
• Every card looks the same. The accent follows whichever model leads, whoever it is.

What it can’t tell you

• The key is a draft. A language model wrote it, so it may share blind spots with the models it judges. An expert review could change the scores.
• The held-out set is small. One answer moves macro-F1 a lot, which is why the ranges are wide.
• One run per model. Temperature 0 makes a run repeatable, but providers update hosted models without notice, so a later run may differ.
• The categories weren’t chosen blind. The six were settled after a first look at this batch and at an early Alpie Seed run on it: SEBI and IRDAI were merged and Space & Defence was added. Every model got the same six.
• It started as an Alpie demo. That’s why Alpie Seed was the first model run. The rules above are there so that start can’t tilt the result.
• It’s hypothetical. These are US startups judged as if they sold in India, so the key is a judgement, not a regulatory filing.
• Timing comes from one place. Speeds from another country would differ.

The steps, in order

1. List Alexandria’s Y Combinator tools and prices (free): discover.py
2. Fetch the batch: fetch_companies.py --batch "Fall 2026"
3. Write the rubric, then label every company and fix the split: labels/
4. Run both prompts on the development companies with every model: compare.py --prompt v1|v2 --temperature 0 --split dev
5. Pick the prompt by the five-model average: select_prompt.py
6. Run the chosen prompt on all companies: compare.py --prompt v2 --temperature 0
7. Score and build the pages: score.py, results.py, race.py

Running it again needs a Firecrawl key and a key for each provider.