01
Package
Connect a repo — Teev rebuilds it exactly as it stood before each merged fix, with the fix hidden as the answer key. Tasks that can't grade themselves reliably are thrown out.
Free during early access
Turn your merged PRs into a benchmark
Teev builds a private coding benchmark from your team's own pull requests, then tests every model and coding agent against it. See which setup is cheapest for each kind of work, without giving up quality.
Private repos included · no card · your code and keys never reach Teev
Results · run 04-1182
Illustrative — not a customer result
Codex leads — 7 of 10
70% pass rate, 2 tasks ahead of Claude Sonnet 4.5 (5 of 10).
1
gpt-5-codex · codex · high
7 / 10
$0.41
2
claude-sonnet-4-5 · pi · medium
5 / 10
$0.63
3
qwen3-coder · qwen · high
4 / 10
$0.02
4
kimi-k2 · kimi · medium
3 / 10
$0.04
Cost is per task, not per token. Every profile attempted the same ten tasks in the same sealed environment.
Not a public leaderboard
You know benchmarks. This one's yours
Public benchmarks test models on someone else's code, and the models have usually trained on the answers. Yours is built from work your team actually merged and graded by your own tests.
Public benchmark
Your Teev benchmark
Tasks come from
Open-source repos
Your merged pull requests
Models have seen the answers
Often
Never, on private repos
Tests what you actually run
Rarely
Every model, coding agent and effort level
Tells you the cost
No
Cost per solved task
How it works
Three steps, one afternoon
A setup is a model, the coding agent it runs in, and an effort level — for example, a Claude model in Claude Code on high effort.
02
Compare
Every setup attempts the same tasks in the same sealed container, several times each — no open internet, so nothing can look up the fix.
03
Review
Cost per solved task and solve rate for every setup, broken down by type of work and complexity — ties stay ties, never an invented winner.
Security
We never see your code or your keys
Teev runs on your machine or in your own cloud. Your runner does the work and sends us scores and pull request facts, never code or credentials.
Your cloud account
The runner
- GitHub token
- Model API keys
- Source code
- Patches, diffs, logs
- Test results, artifacts
Nothing in this box leaves it
Teev
The control plane
- Your account and team
- Run history
- Pull request metadata
- Pass, fail and cost
No route into your network · no credential of yours
What we receive
repo · PR number · title · author (hashed) · files changed · lines added · pass / fail · cost · duration
What we never receive
source code · diffs · logs · issue text · API keys · GitHub tokens
Your keys stay with you
Model keys stay on your runner or with your cloud identity. GitHub access goes straight to your runner. We refuse credentials outright.
No way in
Your runner calls out to ask for work. There's no inbound connection, and it only talks to a fixed list of addresses you can inspect.
Where your code does go
Agents send code to your model provider, on your key, through your own gateway if you use one. That's the same as using the agent today, and none of it touches Teev.
Results
For every kind of work, the cheapest setup that still does it
Nobody doubts frontier models are the best available. The question is whether you need the best for a dependency bump, a null check, or a migration shaped like the last eleven.
By type of work
Bug fixes, migrations, dependency bumps, features. See which setup wins each one.
By complexity
From one-line fixes to multi-file changes. Find where cheaper setups hold up and where they break.
By cost per solved task
Not per token. A cheap model that loops and retries can cost more per fix than the one you avoided.
You leave withA default setup, the specific work worth routing elsewhere, and a number for what your current setup costs you.
No skin in the game
See a real report before you sign up
We've run Teev on a public open-source repo and published all of it: tasks, setups, solve rates, cost, and the tasks we threw out.
We don't resell models or take referral fees. If the setup you already pay for wins, that's what the report says.
Early access
Free while we're early
We want a small number of teams using Teev properly and telling us what's missing.
The full product, private repos included, and a direct line to the founder.
Honest feedback, and a short call after your first run.
Want us to run it with you?
We'll set it up alongside your team and turn the results into defaults and routing your developers actually follow. Everything we produce lives in your Teev workspace.
Talk to us →Questions
Questions
Getting started
What is Teev?
A tool. It builds a coding benchmark from your own merged pull requests and tests models and coding agents against it. If you want help running it or acting on the results, we offer that too, but the tool works on its own.
What do I need?
A GitHub repo with tests, Docker Engine or any Docker-compatible runtime (OrbStack, Colima, Podman, Docker Desktop), and API keys for the models you want to test.
How long does setup take?
Install and connect a repo in minutes. Packaging tasks and running a first comparison usually takes a few hours, depending on your repo's build.
How many pull requests do we need?
Fewer than you'd expect — Teev tells you how many tasks your history can build before you run anything.
Which languages and repos work?
Any repo whose tests run in a container.
GitLab or Bitbucket?
GitHub today.
Benchmarks and results
How is this different from SWE-bench or a public leaderboard?
Same idea, different data. Public benchmarks use open-source tasks the models have often trained on. Yours uses your private work, so no model has seen the answers, and it measures cost as well as quality.
How do you know a task is fair?
Each task's hidden test must fail before the fix and pass after it. Flaky tasks are dropped. So are tasks where the issue already spells out the fix, because that tests copying, not coding.
Why run each task more than once?
Agents don't give the same answer twice. Several attempts per task let us show a range, not a lucky single score.
What if two setups tie?
We show a tie. A two-task gap on a small benchmark is often noise, and we won't dress it up as a winner.
How are tasks sorted by type and complexity?
By what changed — files touched, lines changed, tests added, and the linked issue.
What happens when a new model comes out?
Re-run the same benchmark. Tasks are frozen, so results are directly comparable, and a re-run costs very little.
Which models and coding agents can we test?
Models from OpenAI, Anthropic and Google directly; open-weight models like Qwen, Kimi, GLM and DeepSeek through OpenRouter; and anything in Amazon Bedrock or Azure AI Foundry. Coding agents: any harness you can run in a container.
Security
Do you see our code?
No. Your runner does the work on your infrastructure and sends us scores and pull request facts only.
Where do our API keys go?
Nowhere near us. They stay on your runner or with your cloud identity. We refuse credentials outright.
Can we route model calls through Bedrock or Azure?
Yes, and it's the strongest setup: your keys never enter the test container at all.
Can our security team check the runner?
Yes. You can read the install script before running it, and the runner only talks to a fixed list of addresses.
Cost
Is it really free?
Yes, during early access, private repos included. No card.
What happens after early access?
We haven't set pricing. Early users hear first, well before anything changes.
Who pays for model usage during a run?
You do, directly to your model provider on your own keys. We never mark it up.
We're committed to one vendor. Is this still useful?
Yes. The coding agent and effort level change cost as much as the model does, and you'll know what each fix actually costs.
Couldn't we build this ourselves?
You could. The hard part is making tasks trustworthy: rebuilding old environments, proving each task grades itself, filtering out bad tasks, and keeping it reproducible as models change.
Do model vendors pay you?
No. No reselling, no referral fees, no partnerships.
Find out what your coding agents are actually worth
Build your benchmark and run your first comparison free. We'd rather you saw the evidence than read our claims about it.