Ask an AI agent to build the same application twice, changing only the framework, and the bill may not stay the same. That is the practical question behind developer Nolan Lawson’s framework experiment: how much can JavaScript tooling influence AI coding agent cost? His test illustrates why an agent may consume different numbers of tokens, tool calls, and minutes even when the requested product remains essentially unchanged.
The important lesson is not that one framework has permanently defeated another. It is that framework conventions shape the work an agent must perform. Familiar APIs, predictable file structures, concise generated code, accessible documentation, and clear error messages can shorten a run. Ambiguous setup steps, version mismatches, verbose output, or repeated debugging can lengthen it. Those effects are specific to the task, model, agent, prompt, dependency versions, and pricing snapshot used in the experiment.
What Nolan Lawson’s experiment was designed to measure
Lawson’s reported experiment gave an AI coding agent the same application specification across separate JavaScript framework runs. Holding the requested application steady made it possible to observe how the framework changed the agent’s path through planning, implementation, tool use, testing, and repair.
The recorded dimensions included three measurements that teams should treat separately:
- Token usage: the input and output tokens processed while the agent interpreted instructions, inspected files, generated code, read command output, and corrected errors.
- Elapsed build time: the wall-clock time from the beginning of a run until the agent declared the implementation complete or reached the experiment’s stopping condition.
- Estimated API cost: a conversion of token usage into currency using the model provider’s prices applicable to that setup and pricing snapshot.
The experiment also exposed framework-specific work that a simple dollar total can hide: scaffolding projects, selecting packages, navigating conventions, running commands, interpreting compiler messages, and revising generated files.
Anyone citing a precise leaderboard should first verify the tested framework list, model identifier, agent and version, prompt, task specification, token-accounting method, cache treatment, retry policy, and pricing assumptions in Lawson’s original publication record. Those details are part of the result. This article does not reproduce a numerical ranking because values detached from that source metadata can imply a level of generality the experiment does not support.
An animated view of the reported app-building race
Reported experiment: same app specification, separate framework runs
Framework run A
Framework run B
Framework run C
Completion condition
Why a framework can change AI coding token usage
Conventions reduce the agent’s decision space
A strongly conventional framework may tell the agent where routes, components, tests, and configuration belong. That can reduce exploratory commands and deliberation. A flexible framework may require more choices about architecture and packages. Flexibility can be valuable for experienced developers, but an agent may spend additional tokens evaluating options or changing direction.
The opposite can also happen. A convention-heavy framework may require boilerplate, specialized syntax, or several coordinated files. If the model knows a lighter framework well, it might complete the same feature with less generated code. The outcome depends on the task and the model’s learned familiarity, not simply on whether a framework is opinionated.
Generated code becomes future input
Output volume matters twice. First, the provider may charge for generated tokens at a different rate from input tokens. Second, code created early in the session can return to the model’s context when the agent reads files, reviews a diff, or diagnoses a failure.
A framework that needs more configuration, adapters, or component ceremony can therefore increase later input. Concise code is not automatically better code, but a larger working set can raise AI coding token usage during an agentic run.
Models tend to be more reliable when APIs are well represented in documentation and examples available to their training or retrieval tools. Clear, current documentation can help an agent choose valid packages and syntax on the first attempt.
New releases complicate the picture. An agent may generate patterns from an older major version, install a newer package, encounter an error, search documentation, and rewrite the implementation. One compatibility loop can outweigh the token savings from a compact framework API.
Tooling determines the length of debugging loops
Agents rarely produce an application in one uninterrupted response. They create files, install dependencies, run a development server or build, inspect errors, and patch code. Each command produces text that may be fed back into the model.
Actionable diagnostics can shorten that cycle. Noisy stack traces, cascading type errors, stalled commands, or unclear configuration failures can make it longer. This is one reason framework performance in an agent benchmark cannot be reduced to browser speed or bundle size.
Token and cost counters: what the experiment’s numbers represent
| Reported measurement | What belongs in the counter | What it does not prove |
|---|---|---|
| Input-token counter | Prompt, context, file contents, tool results, errors, and repeated context | Developer labor or application quality |
| Output-token counter | Plans, explanations, generated code, patches, and tool instructions | Runtime speed or bundle efficiency |
| Estimated cost counter | Billable tokens multiplied by the applicable rates, with caching handled as specified | A permanent price for that framework |
| Elapsed-time counter | Planning, generation, commands, installs, tests, waits, and repair loops | Human review time or long-term maintenance |
Token accounting deserves particular care. Some agents display aggregate tokens, while others expose model calls individually. Prompt caching may receive separate pricing. Tool output can be truncated, summarized, or submitted again in a later request. A failed run that is manually restarted may also disappear from a polished final total unless the benchmark explicitly preserves it.
For a realistic AI app development cost estimate, count the entire workflow: successful calls, retries, planning messages, tool calls, failed installations, test output, and abandoned attempts.
Framework comparison graphic: what actually varied
| Framework characteristic | Possible effect on the reported agent runs | Reason the effect may reverse |
|---|---|---|
| Project conventions | Fewer architectural decisions | More required files or specialized patterns |
| Code volume | Less output and less context to reread | Abstractions may require explanation or supporting code |
| CLI and scaffolding | Fast, repeatable setup | Interactive prompts or version drift can cause retries |
| Documentation | Quicker API selection and correction | Search results may mix incompatible versions |
| Diagnostics | Shorter debugging loops | Verbose or cascading errors add context and calls |
| Model familiarity | More accurate first-pass generation | Popular but outdated examples can reinforce old patterns |
API cost is not developer time—or application performance
An inexpensive agent run can still create an expensive engineering task. A developer may need hours to review insecure code, repair inaccessible interfaces, simplify abstractions, or add missing tests. Conversely, a run with a larger API bill may produce code that is easier to understand and integrate.
Keep five cost categories separate:
- API cost: what the model provider charges for billable usage.
- Agent build time: how long the automated session takes.
- Developer time: review, correction, integration, security analysis, and deployment work.
- Build performance: installation, compilation, testing, and bundling speed.
- Runtime and maintenance: user-facing performance plus the cost of operating and changing the code over months or years.
A JavaScript framework comparison that optimizes only the first category risks shifting costs into the other four. Runtime performance should be evaluated with appropriate browser and application measurements rather than inferred from model usage; the web.dev performance guidance provides a useful starting point.
Why your results may differ from Lawson’s
Even a careful reproduction can produce another order. Models differ in framework knowledge, instruction following, tool use, and verbosity. Agent configurations differ in context management, planning behavior, permission settings, retry limits, and whether they automatically run tests.
Prompts also matter. A short request gives the agent more architectural freedom. A detailed specification may increase initial input while preventing expensive revisions. Repository size changes how much context must be searched, and an established codebase may impose local conventions that outweigh framework defaults.
Project scale can reverse an apparent advantage. Boilerplate is a large share of a tiny demonstration app but a small share of a substantial product. Features such as authentication, internationalization, server rendering, data caching, testing, and accessibility may expose framework differences that a compact experiment never reaches.
How teams should benchmark AI-assisted JavaScript development
- Select representative tasks. Use a small feature, a defect, a refactor, and a test-writing task drawn from your actual codebase.
- Freeze the environment. Record model and agent versions, dependency lockfiles, system prompt, permissions, hardware, and tool configuration.
- Track every call. Capture input, output, cached tokens, retries, failed runs, tool calls, and elapsed time.
- Use multiple runs. Agent behavior is nondeterministic. A single lucky or unlucky run is not a stable benchmark.
- Apply completion criteria. Require passing tests, a successful production build, linting, and the requested behavior instead of trusting an agent’s declaration.
- Review the result. Score correctness, security, accessibility, readability, dependency choices, and maintainability.
- Calculate total cost. Combine API charges with developer review time and expected operational consequences.
Frequently asked questions
Does a cheaper framework run mean the framework is better for AI coding agents?
No. It means that framework required less billable model usage under the tested task and configuration. Another model, prompt, feature, or repository could produce a different result, while code quality may favor a different option.
Are token cost and AI agent build time the same thing?
No. A run can wait on package installation, compilation, or a server without consuming many tokens. Another run can generate and process many tokens quickly. Track currency, tokens, tool calls, and wall-clock time independently.
Can framework performance predict AI coding agent cost?
Not reliably. Fast runtime rendering or compilation may help certain tool loops, but agent cost also depends on code volume, diagnostics, documentation, context handling, and model familiarity. Browser performance and agent efficiency require different benchmarks.
What is the best way to lower AI app development cost?
Provide precise acceptance criteria, pin dependency versions, keep tool output focused, run targeted tests, limit unnecessary context, and monitor retries. Most importantly, benchmark recurring tasks from your own repositories rather than adopting a generalized leaderboard.
The responsible conclusion
Lawson’s experiment makes a useful cost visible: framework choice can alter how much reasoning, code generation, tool output, and debugging an agent needs to complete the same stated application. That is a meaningful engineering and budgeting signal.
It is not a universal verdict on JavaScript development. AI coding agent cost changes with the model, agent, prompt, versions, pricing, task size, and definition of completion. Teams should reproduce the method on representative work, preserve full token and retry records, and evaluate maintainability and correctness alongside speed. A single experiment—however illuminating—cannot settle the right framework choice for every project.
