Use the cheapest model that does the job

Savings: ~96%, when comparing GPT-5.4 nano to GPT-5.5 ($0.20/$1.25 vs $5.00/$30.00 per MTok (million tokens)).

For well-defined tasks, small models can be just as good as larger models. Don’t use Opus when Haiku can do the job! Common tasks for small models include summarization, classification, translation and being sub-agents for larger models. If you’re building an agentic workflow, you could use a different sized model for each step depending on complexity. For a chatbot, you could use a model router that picks an appropriate model for each request. Besides the price, some models are more token-efficient than others. They can give equally intelligent answers using fewer tokens. Artificial analysis combines price and token efficiency into a cost per intelligence metric.

Trade-off: could underestimate the difficulty of the task and pick a model that isn’t smart enough, which then needs multiple attempts or downright fails.

Use the cheapest provider and region

Savings: up to 60% less. For example, Kimi K2.6 costs $1.30 per Mtok (blended cost at 7:2:1 cache-input-output) on Azure compared to $0.60 on Novita.

Check if a given model is available from other API providers or in a cheaper cloud region. This matters most for open weights models. Closed models tend to have identical prices across providers.

Trade-off: could be slower, data privacy concerns, cheaper providers may quantize more aggressively.

Use batch inference

Savings: 50% of the request.

If you don’t need the results immediately, you can send requests in a batch and poll for the results later. OpenAI, Anthropic, Google (Gemini), Azure OpenAI, AWS Bedrock, and Mistral all offer this: typically 50% off input and output, with results within about 24 hours. You upload a JSONL file (or equivalent), wait for the job to finish, then download the responses.

Trade-off: need to wait for completion, which rules out any real-time use cases. Async also adds complexity.

Use a flat rate subscription

Savings: Up to 95% vs API list price.

Some providers offer agents (model + tool loop) at a flat rate subscription, rather than models at API rates. Claude Pro and ChatGPT Plus ($20/mo) bundle far more inference than the fee suggests. In a June 2026 experiment, SemiAnalysis ran long agentic coding sessions on every paid tier until weekly limits bound, then priced the consumed tokens at API rates. A fully used Claude Pro plan equated to roughly $400 of API spend (20× the subscription fee). ChatGPT Plus reached about $700 (35×).

If your use case supports it, use a flat rate subscription instead of an API-based model. Keep in mind that the providers intend them for single-user, human-in-the-loop use cases.

Trade-off: Quotas can change, not meant for automated or multi-user workloads, Anthropic has blocked third-party platforms like OpenClaw from routing subscription auth.

Request short responses

Savings: Most of the output tokens.

Save output tokens by asking the model to keep it short. Depending on provider and model the verbosity and max_tokens parameters can also be used to control the length of the response. If you have a long system prompt, consider taking situational instructions from it and packaging as a skill instead. Then the model will only load the full prompt when it’s relevant.

Trade-off: could lose detail.

Use low or no reasoning

Savings: Significant number of output tokens.

Reasoning tokens are billed as output tokens, you just don’t see them in the response. If your task isn’t heavy on logical thinking, turn off reasoning or set it to low.

Trade-off: less intelligence.

Use a token-efficient language

Savings: ~25–55% of input and output tokens when rewriting Japanese prompts in English on current Western models.

LLM billing is per token, and tokenizers are not language-neutral. OpenAI, Anthropic, and most API models use byte-pair encoding vocabularies trained mainly on English. They merge common words into single tokens. Japanese kanji, hiragana, and katakana mostly stay separate. The same meaning therefore costs more in Japanese than in English. Let’s look at an example with OpenAI’s o200k_base tokenizer (as used by GPT-5.x).

Matching colors link equivalent phrases: e.g. English bullet + points (two tokens) map to six Japanese tokens, including one dashed box. That box is a byte-level split: the tokenizer cut the kanji 箇 across two tokens, and both are billed even though the first holds no complete character. According to a benchmark by Mason AI Lab, across six task types, Japanese averages ~1.7× English; Chinese ~1.3×. Chinese-native models (Qwen, DeepSeek) close most of the gap for Chinese.

So when you have the choice, write your instruction in English. The highest payoff is for repeated system prompts.

Trade-off: not possible if your task requires a specific language, translating a prompt could lose nuance.

Use fewer and leaner tools

Savings: mostly input on tool schemas, plus output when you trim what tools return.

Each tool adds a name, description, and JSON Schema to the prompt. With MCP, that overhead lands before the agent does any work and is rebilled every turn. The official GitHub MCP server alone costs ~17,600 input tokens (mcp-compressor), so a one-word message can already bill 17,601 tokens.

Drop MCP servers you are not using, load full schemas only when needed (Anthropic tool search, mcp-compressor), or replace large tool lists with skills (~800 tokens when loaded versus tens of thousands for a big MCP server). Where you can, prefer CLI over MCP: a gh command is hundreds of tokens, not tens of thousands of schema. On output, trim tool descriptions and unused parameters. The compact JSON alternative TOON can also shrink JSON tool results by roughly 20–60%. For more detail, see my Agent Connector Comparison and AXI, a set of design principles for efficient CLIs for agents.

Trade-off: the model may miss a tool it needs, and CLI or skills need shell access.

Shrink documents that you send

Savings: 50–90% of input tokens.

Instead of sending a large document as a whole, run it through a RAG pipeline (chunk, embed, retrieve relevant chunks) and send only the relevant chunks. Another option is to equip an agent with a bash terminal and let it slice the document with grep, awk, sed, or tail to find the relevant parts.

Trade-off: could lose information, shrinking adds complexity.

Use images sparingly

Savings: 80% or more of input tokens.

Images added to the prompt add significant input token costs. A single 1024×1024 image costs ~765 input tokens on GPT with detail: "high", and it is rebilled on every turn the image stays in context. There are many ways to save on images:

• Use detail: "low" instead of detail: "high" on the OpenAI API. On low, the cost is a fixed 85 tokens regardless of size. On high it’s 85 + 170 x (number of 512x512 tiles). That amounts to a 9x reduction on a 1024x1024 image. Avoid this for OCR tasks because letters could become unreadable.
• Downscale images before uploading them to the provider. Check the provider’s documentation first because they have their own downsizing logic which could conflict with your custom logic.
• Strip images from chat history after the first turn if it’s unlikely they’re important again at the next turn.
• Use a text representation instead. If not available, run the image through a vision language model to extract text and other information and save that representation. Reuse it every time you’d normally send the image.
• If the image was originally rendered from code, send the code directly instead of the image. For instance, send the DOM or accessibility tree of a website, or the code for a Mermaid diagram.

Trade-off: OCR, DOM extraction, and cropping add pipeline complexity. Aggressive downscaling or low detail can miss small text and UI elements.

Maximize input cache hits

Savings: 90% of repeated input tokens.

Most providers offer substantial rebates on repeated input tokens. The input must match exactly and be at least a minimum length (1,024 tokens on OpenAI). The repeat request must also arrive before the cache expires. On OpenAI, extended caching retains prefixes for up to 24 hours (prompt_cache_retention: "24h"; the only mode on GPT-5.5). On Claude, the default TTL is 5 minutes, with an optional 1-hour TTL via cache_control.

If the time is up, your context window is full, or your provider doesn’t offer good rebates for cached inputs, consider compaction instead. Let the model summarize the relevant parts of the chat history and start fresh with that as the new context. Some providers also reprice the whole request past a context threshold (e.g. GPT-5.5 above 272K input jumps from $5/$30 to $10/$45 MTok), so try to stay below it.

Trade-off: on Claude, time pressure to answer within the cache window.

Minimize chat turns

Savings: ~10% with input cache and much higher without.

Every chat turn (user asks, model answers) requires sending the entire existing chat history to the model again. It’s cheaper to ask multiple questions in one turn than one by one.

Let’s go through an example: someone asks three questions (Q1, Q2, Q3) about a document (context) and receives three answers (A1, A2, A3).

One question per turn

block-beta
    columns 7
    block:call1["API call 1"]:7
        columns 7
        c1["Context"] q1a["Q1"] a1a["A1"] space:4
    end
    block:call2["API call 2"]:7
        columns 7
        c2["Context"] q1b["Q1"] a1b["A1"] q2a["Q2"] a2a["A2"] space:2
    end
    block:call3["API call 3"]:7
        columns 7
        c3["Context"] q1c["Q1"] a1c["A1"] q2b["Q2"] a2b["A2"] q3a["Q3"] a3a["A3"]
    end

    classDef context fill:#e9ecef,stroke:#6c757d,stroke-width:2px
    classDef question fill:#cfe2ff,stroke:#0d6efd,stroke-width:2px
    classDef answer fill:#e2d9f3,stroke:#6f42c1,stroke-width:2px
    classDef cachedContext fill:#e9ecef,stroke:#198754,stroke-width:2px,stroke-dasharray:8 4
    classDef cachedQuestion fill:#cfe2ff,stroke:#198754,stroke-width:2px,stroke-dasharray:8 4
    classDef cachedAnswer fill:#e2d9f3,stroke:#198754,stroke-width:2px,stroke-dasharray:8 4

    class c1 context
    class q1a question
    class a1a answer
    class c2 cachedContext
    class q1b cachedQuestion
    class a1b cachedAnswer
    class q2a question
    class a2a answer
    class c3 cachedContext
    class q1c cachedQuestion
    class a1c cachedAnswer
    class q2b cachedQuestion
    class a2b cachedAnswer
    class q3a question
    class a3a answer

All questions in one turn

flowchart TB
    subgraph call1["API call 1"]
        direction LR
        c1["Context"]:::context
        q1["Q1"]:::question
        q2["Q2"]:::question
        q3["Q3"]:::question
        a1["A1"]:::answer
        a2["A2"]:::answer
        a3["A3"]:::answer
        c1 --> q1 --> q2 --> q3 --> a1 --> a2 --> a3
    end

    classDef context fill:#e9ecef,stroke:#6c757d,stroke-width:2px
    classDef question fill:#cfe2ff,stroke:#0d6efd,stroke-width:2px
    classDef answer fill:#e2d9f3,stroke:#6f42c1,stroke-width:2px
    classDef cachedContext fill:#e9ecef,stroke:#198754,stroke-width:2px,stroke-dasharray:8 4
    classDef cachedQuestion fill:#cfe2ff,stroke:#198754,stroke-width:2px,stroke-dasharray:8 4
    classDef cachedAnswer fill:#e2d9f3,stroke:#198754,stroke-width:2px,stroke-dasharray:8 4

Each box is a separate part of the chat sent on that API call. Dashed green border = cached input, which is rebated on most providers. Suppose the chat starts with a 5,000-token context document, each question is 50 tokens, and each answer is 300 tokens. Turns 2 and 3 get cache hits on the shared prefix. At GPT-5.5 standard API pricing ($5.00/M fresh input, $0.50/M cached input, $30.00/M output):

All questions in one turn:

Combining the three questions saves 9% in this example ($0.058 → $0.053). Fresh input is the same either way; the gap is rebilled cached history on turns 2 and 3. Without input caching, the same setup would save about 51%, because repeated history is billed at the full input rate.

Trade-off: need to know your questions ahead of time.

Cache whole requests

Savings: 100% of repeat requests.

If you keep sending the same request over and over, add a layer to cache the request and response. I found myself with this issue when using an LLM for data labeling and re-ran it during experiments. This works best for exact repeated requests. Alternatively, you could fetch a cached answer based on a minimum cosine similarity of the request and the cached request.

Trade-off: maintaining the cache, potential for irrelevant responses when using similarity search.

Make workflow steps deterministic

Savings: 100% of the request.

If you have an agentic workflow, see if you can identify steps that are always the same in each run. These don’t have to be controlled by an LLM. Instead, turn them into a deterministic function, which is cheaper, faster and more reliable. Prototype with one LLM call when the problem shape is unknown. Productionize by moving anything that became repeatable (validation, routing, extraction on standard layouts) out of the model and leaving the LLM for the shrinking tail. If the task requires fuzzy intelligence, but not a whole LLM, see the next section.

Take invoice extraction: vendor PDFs have wildly different layouts, so v1 is often one vision call per document. At volume that gets expensive and non-deterministic. Production peels off structured cases and reserves the model for the tail.

One vision call per PDF: LLM on every run

flowchart LR
    A[Invoice PDF] --> B[LLM vision]
    B --> C[JSON fields]

    classDef llm fill:#e2d9f3,stroke:#6f42c1,stroke-width:2px
    class B llm

Parse e-invoices, Doc AI on the rest, LLM on low-confidence fields: LLM only on exceptions

flowchart LR
    A[Invoice PDF] --> B{E-invoice XML?}
    B -->|Yes| C[Parse fields]
    B -->|No| D[Doc AI]
    D --> E{Low confidence?}
    E -->|Yes| F[LLM re-read]
    E -->|No| G[Validate]
    C --> G
    F --> G
    G --> H[JSON fields]

    classDef deterministic fill:#e9ecef,stroke:#6c757d,stroke-width:2px
    classDef nlp fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    classDef llm fill:#e2d9f3,stroke:#6f42c1,stroke-width:2px

    class B,C,G deterministic
    class D nlp
    class F llm

Trade-off: loss of flexibility and need to reorganize the workflow.

Switch to classic NLP models

Savings: ~90% less compute when self-hosting.

If you’re using an LLM for classic NLP tasks, such as classification, sentiment analysis, named entity recognition or translation, consider a dedicated NLP model instead. Encoder models like ModernBERT (149M parameters) run a single forward pass over the input, without added instructions. A 7B LLM is ~50× larger and requires additional prompt tokens as input. That translates to at least a 90% compute savings if you self-host both models. Savings on hosted variants is highly dependent on the provider. To learn more about mixing LLMs, classic NLP models and rule-based systems, read Atomic NLP by Explosion AI.

Trade-off: loss of flexibility, may need to fine-tune a model for your task, inference is faster but platform support is more limited.

Share data to get free tokens

Savings: up to 11 million tokens per day.

OpenAI grants free daily tokens if you share inputs and outputs with them (1M/day on frontier models, 10M/day on mini models; input + output combined, pooled across each group). OpenRouter offers a 1% discount on billed usage for a similar opt-in.

Trade-off: data privacy concerns.

Apply for token grants

Savings: 100% of inference cost while credits last (amount varies by program).

Several providers offer LLM API or cloud credits (that can be used for LLM inference) through dedicated programs. Awards are competitive and eligibility varies.

• Researchers: OpenAI, Anthropic, Google, GenAI4EU (EU), NAIRR Pilot (US).
• Nonprofits: Anthropic, OpenAI, Microsoft, AWS.
• Startups: Anthropic, Google, Microsoft, AWS, Together AI, NVIDIA, Mistral (EU).

Trade-off: application required, not guaranteed, capped.

Tools

DIY from start to finish

graph LR
    Prompt[Prompt including tool definitions] --> LLM
    LLM -->|tool call| Tool[Local function or provider API]
    Tool -->|result| LLM

Tools are function definitions and configurations that are passed directly to the model along with the prompt. The model can output a structured response with the function’s arguments. Now, there are two ways to execute the tool: either the client passes the arguments to a local function, or the API provider executes it inside the API. An example of the latter is OpenAI’s web search tool in the Responses API.

Tools, especially when executed by the client, are the most customizable approach. You have full control over the definitions and executions. Authoring a tool isn’t hard: you write a function, e.g. in Python or TypeScript, and pass it to a tools parameter in a request to the model. The more difficult part is making sure the agent understands your tool, which is a context engineering problem.

If you’re using more than a few tools, tool search or other logic becomes necessary to avoid bloating the context window. An example of this is Pydantic AI’s tool search. You can also write your own logic to control which tools are available at each chat turn. This can act as a system to hide irrelevant tools to save tokens or it can encode state: for example, a data analysis agent might first see tools to list and filter datasets, pick a dataset and only then see tools to analyze it. This logic can be powerful, but it lives inside of your custom harness rather than a reusable connector. Another downside is that hiding tools runs the risk of the agent missing opportunities to use them.

The biggest con of tools is that they are hard to reuse. Let’s say I write a tool to work with an agent that I have built with Pydantic AI. I can’t just take that tool and let a Cursor agent use it. MCP, the next connector, solves this by turning tools into shared servers.

Main use case: custom tools with maximum control.

MCP: Model Context Protocol

The USB-C port for agents – Anthropic

graph LR
    Config[mcp.json] --> LLM
    LLM <-->|JSON-RPC| Server[MCP server]
    Server <--> Backend[Backend API]

MCP servers are essentially shared versions of tools: instead of defining functions inside one harness, you expose them through a reusable server. This server can run locally or remotely. In addition to tools, MCP supports prompts and resources (data sources), but those are rarely used. MCP servers can be shared privately within organizations and publicly at https://registry.modelcontextprotocol.io and other MCP registries.

Almost every major SaaS product has an MCP server, letting agents connect without writing custom code. Not all of those are well designed; many are simple wrappers around existing APIs. In particular, they are not token-efficient in their descriptions and require multiple calls to get something done, each of which is an inference call to the LLM. As a consumer of the MCP server, you can’t change the tool names or descriptions.

Security is decent for trusted, least-privilege servers, but approval gates are mainly a client responsibility and tool descriptions, schemas and outputs are prompt-injection surfaces.

Adding too many MCP servers or servers with too many tools bloats the context window. As an example, the official GitHub MCP server exposes 94 tools consuming 17,600 tokens (source). So just writing “hi” to an agent using that server will cost 17,601 input tokens, plus the cost of the response. To avoid this, Anthropic came up with tool search, which exposes a tool search tool instead of the full tool set. So when you write “hi”, the agent doesn’t even have to load the GitHub MCP tools, and if you ask it “why is CI failing” it can use search to find list_workflow_runs and get_workflow_run specifically rather than loading all 94 tools.

It’s straightforward to create a local MCP server and doesn’t take much more than creating a tool. Sharing it adds significantly more work for packaging, authentication, rate limiting, deployment and ongoing observation.

Main use case: sharing tools across agents and connecting to remote services.

A2A: Agent to Agent Protocol

Agent collaboration in full fidelity

sequenceDiagram
    participant AgentA as Agent A
    participant AgentB as Agent B
    participant Own as Own tools / skills / MCP

    AgentA->>AgentB: discover agent card
    AgentA->>AgentB: task
    AgentB->>Own: use tools / skills / MCP
    Own-->>AgentB: result
    AgentB-->>AgentA: response

A2A connects agents with each other. Each agent has an agent card that describes its capabilities. Another agent discovers the card and can delegate tasks to the agent. The executing agent uses its own tools, skills, MCP servers or any other capabilities and reports back to the delegating agent. The difference to packaging an agent as a tool is that A2A treats the executing agent as a stateful remote collaborator. This unlocks multi-turn interaction (e.g. asking for more information) and long running tasks with task IDs and lifecycle states such as working, completed or failed. How the executing agent uses its own capabilities is not shared with the delegating agent. Internal logic is not shared. The agents can run in different environments, using different LLMs and harnesses and be controlled by different organizations.

A2A is not a replacement for MCP, it adds a new dimension above it. It’s a powerful solution that promises maximum expressiveness, token efficiency and scalability from the perspective of the delegating agent. But A2A hasn’t taken off like MCP. One explanation is that exposing an agent is more involved than exposing a tool: agents are not deterministic, continuously consume tokens, are stateful, and can’t be versioned as easily as an MCP server that’s a front to an API. In addition the frontend support is still limited. Most common agent harnesses and chat apps people actually use, including ChatGPT, Claude, Claude Code, Codex and Cursor, don’t expose A2A endpoints. Others, like Microsoft Copilot Studio and LangDock support it. In contrast, all of them support MCP.

Main use case: multi-turn interaction between agents

CLI: Command Line Interface

The best agent architecture is already sitting in your terminal – Ashka Stephen

graph LR
    User -->|"use the XYZ CLI"| LLM
    LLM -->|bash command| Shell
    Shell -->|stdout| LLM

The typical training material for LLMs contains a lot of examples of using CLIs and writing bash commands. So LLMs are already proficient with standard bash commands and CLIs out of the box, for example grep, git, curl, uv, docker and psql. This gives agents access to a huge existing software ecosystem without agent-specific integration work and with optimal token efficiency. It’s also expressive: shell commands can be piped into each other. Here’s an example that combines the git log command with bash built-in commands to count commits by author:

git log --format='%an' | sort | uniq -c | sort -nr

The tradeoff is security. To use CLIs directly, the LLM needs shell access, which by default gives it access to the entire host system. Granular approval is hard because there are countless commands, and they can be composed in unexpected ways. The solution is to sandbox the agent’s shell, only giving it access to files that it needs to complete the task. Another option is to emulate the shell, as done by Vercel’s just-bash.

Naturally, the CLI ecosystem centers on software development, cloud and system administration tools. Outside of these domains, there are fewer CLIs to reuse, though Data Science at the Command Line show how far standard unix utilities can go for data analysis. For example, awk can extract columns and compute aggregations from text streams, while jq can filter and reshape JSON from API. These tools can also be used for RAG, as shown in an article by Vercel.

Main use case: using developer, sysadmin and unix tools with an agent that has shell access

Skills

Loose prompt and script bundles with built-in tool search

graph LR
    Skill[SKILL.md] -->|name + description upfront| LLM
    LLM -->|load full SKILL.md on demand| Skill
    LLM -->|run script| Scripts[Scripts / resources]
    Scripts -->|result| LLM

Skills are CLI plus a prompt template. They have built-in tool search: each skill is a directory with a SKILL.md file and optional scripts/resources. A minimal SKILL.md example, featuring YAML frontmatter and a script:

---
name: word-count
description: Count words in a text file. Use when the user asks for a word count.
---

Run `python scripts/count.py <path>` and report the total.

The scripts are any code that can be executed by a shell, for example a Python script. SKILL.md is a Markdown file with a name and description, and optional scripts/resources. In addition to providing the script arguments, skills can also function as a prompt that describes how to use multiple scripts in a workflow.

This low ceremony is a large part of their appeal: a useful skill can be just a Markdown file plus optional scripts. In addition, skills have built-in progressive disclosure for token efficiency: the LLM gets the skill name and description upfront, and loads the full skill content on demand. While skills were originally invented by Anthropic to be used with Claude, the biggest registry is ClawHub for OpenClaw agents, featuring over 3000 skills. By now, skills work with most CLI agents and IDEs, but not with apps like ChatGPT.

The limits show up when the workflow gets bigger: skill results can’t be chained directly, scripts need to be packaged as CLI tools, and the pattern is a poor fit for very large tool sets or remote execution. Security-wise, skills inherit the same shell access risk as CLI. They can make it worse by adding a supply-chain surface: users may install skills from registries without fully auditing them. In February 2026,Snyk found 76 malicious skills employing prompt injection and other techniques on ClawHub and found critical vulnerabilities in 13% of all skills.

Main use case: simple, discoverable connector bundles for local agents.

Code

Build your own tools on the fly

graph LR
    Agent -->|"write code"| Code
    Code -->|"type check"| TypeChecker[Type checker]
    TypeChecker -->|"feedback"| Agent
    Agent -->|"run code"| Code
    Code -->|"result"| Agent

Code is CLI taken further: instead of only running existing commands, the agent writes a small program for the task. This approach is the most expressive, letting the agent write whole programs in a single turn of the conversation. It taps into the vast software engineering ecosystem: LLM pretraining and reinforcement learning on coding problems, code editors, type checkers, linters, test runners and all of the tools that developers use to build software. But its usefulness isn’t limited to software development - for instance, Claude Cowork creates PowerPoint files by writing a Python script that defines slides using python-pptx and runs it.

Writing code is easy for the model, but securely running arbitrary code requires a sandbox, dependency controls and permission boundaries. User approvals are even more difficult than for CLI tools because it can write arbitrary code rather than execute predefined commands. Users who are not developers can’t judge the safety and correctness of the code.

Main use case: coding agents, with apps like Claude Cowork exploring the potential for general purpose agents.

Computer Use

When there is no API

Many apps lack an API. In these cases, the agent can simulate mouse clicks and keyboard inputs to interact with the app. The agent receives a representation of the UI, which can be a screenshot, an accesibility tree (used by screen readers), or a simplified browser DOM.

graph LR
    App[App State] -->|"representation"| Agent
    Agent -->|"mouse and keyboard input"| App

The agent can interact with any app, which creates a magic “the agent uses my computer like I do” experience. Computer use has universal reach across websites, web apps and desktop apps, but not a reusable connector ecosystem. Each UI is a fresh comprehension and control problem.

Frontier models have become increasingly capable. The benchmark OSWorld-Verified measures whether agents can complete real desktop tasks across apps. Humans complete 72.4% of tasks, while GPT-5.5 and Opus 4.7 manage 78.0% and 78.7% respectively. Still, the agent relies on a secondary representation that wasn’t made for agents, and the pattern doesn’t scale well to multiple apps. A typical mistake is missing UI elements that are not visible in the initial screenshot and require scrolling.

Scalability is where computer use scores worst. The full app or browser needs to be running. Token usage is high especially when screenshots are involved. In an experiment by reflex.dev where an agent had to navigate an admin interface either by computer use or using API calls, computer use was 45x more expensive. Another difficulty is handling secrets: an agent with access to a password manager could accidentally paste credentials into a wrong field.

Main use case: interacting with apps that lack an API.

Conclusion

Each connector has a unique use case:

• Tools: custom tools with maximum control.
• MCP: sharing tools across agents.
• A2A: multi-turn interaction between agents.
• CLI: using developer, sysadmin and unix tools with an agent that has shell access.
• Skills: simple, discoverable connector bundles for local agents.
• Code: maximum expressiveness for coding agents and experimental general purpose agents.
• Computer use: interacting with apps that lack an API.

It’s easy to wrap one and the same function as a tool, local MCP server or skill, so it’s not a problem to change the representation of the connector later. Only use A2A for advanced multi-turn interaction between agents and check compatibility first. CLI, skills and code offer the fullest expressiveness, but require sandboxing. For now, I’d only recommend them for interactive use. Skills are deceptively simple but can be abused. Computer use scores worst in the table above, but is the only way to interact with apps that lack an API.

No AI

Manual coding speed is highly dependent on how well a developer has memorized the programming language they’re using. Writing code manually repeats the basics such as loops, conditions and indexing over and over. You solve small logic problems all the time and become fluent in a programming language. After writing a piece of code, you know the ins and outs of it.

Strengths: learning on micro and macro level, drilling the basics, control, ownership

Weaknesses: slow, limited to familiar programming languages, high cognitive load

Autocomplete

Using the autocomplete with a coding extension for an IDE or a VSCode fork provides a very nice speed boost. You can skip over the common small logic problems. Knowing the language is still important. Autocomplete also doesn’t do architecture, it focuses on the micro. As the generated code comes in small chunks, developers naturally check it as they go.

The speed of the coding model is critical for this style; if you have to wait, the flow state is broken. The recent GPT-5.3-Codex-Spark is promising near-instant answers with decent intelligence.

Early on, I used to write comments specifically to trigger the autocomplete to write the implementation. With agents that take direct prompts as an instruction, I wouldn’t recommend this specific technique anymore. Overall I see autocomplete as a nicely balanced style.

Strengths: speed up over manual coding, good control

Weaknesses: not as fast as agents, less learning on micro level

Single agent

Write a prompt, perhaps use planning mode and let a CLI agent or an agent embedded in an IDE go off reading docs, writing code and running tests until it reports back with a result. I’ll share productivity tips in a section below.

Strengths: fast, work in unfamiliar languages

Weaknesses: loss of ownership, control and learning, wait for agent to complete

Multi agent for parallel features

Kick off multiple agents, each working on a different task. This is a style favored by Peter Steinberger, developer of OpenClaw, famously programming from his smartphone. I tried it with OpenAI’s Codex app for a React + NextJS side project and found that I could run the app in a browser, write instructions to Codex and see the update in real time. It completely moved my focus to the app’s functionality and UX, rather than the code. This generated an ungodly amout of changes that I dreaded to review, so I kept vibing instead.

This is fantastic for hackathons, demos and experiments. Personally, I don’t feel confident doing this for a serious project, where I am responsible for each line of code. I’d also run out of token budget before the end of the month.

Strengths: absurdly fast

Weaknesses: extreme loss of ownership, control and learning, token usage

Multi agent for exploration

Kick off multiple agents to do the same task independently, then review their solutions and pick the best one. This makes sense for complex tasks with many possible solutions that have advantages and drawbacks that only become clear during implementation.

Cursor implements this explicitly by letting you compare the results of different models.

This is a rare, but promising style that I haven’t built much experience with yet.

Strengths: can improve quality of architecture, learning on the macro level

Weaknesses: time consuming to review, loss of learning on micro level, token usage

Plan mode

Everything you don’t specify has to be guessed by the LLM. That’s similar to a stakeholder formulating a business requirement and a developer having to fill in all the technical details. For complex changes, I’m a fan of plan mode, where the agent answers with a to do list, rather than jumping into action. It’s a forcing function for clarity and a dedicated place for the developer to review architecture. If the plan doesn’t match what you meant, you catch it in 30 seconds instead of 10 minutes reviewing generated code. That’s also token-efficient.

Simple changes don’t need plan mode, but if I already know how I want something implemented, I’ll mention it in the prompt, e.g. “implement this with an env var” or “upsert the entry to the database”. For UI, pasting screenshots of your own app or inspirations works surprisingly well.

Keep in mind that LLMs are still optimized to please. Current leaders aren’t quite as sycophantic as the notorious GPT-4o, but they’re still not pushing back on bad ideas like a real senior engineer would. If you ask an LLM to find a bug, it will try really hard to constitute a bug, whether there is one or not. If you ask whether something could be simpler, it will try to simplify that bit, perhaps at the expense of complexity elsewhere.

Context engineering by example

Context engineering is the art of managing the LLM’s context window and seeding it with all relevant information at the start of an interaction. The best context engineering doesn’t happen in the prompt, but in the code base:

• Making the project agent legible helps, i.e. using a monorepo, paying attention to consistent variable names and co-locating related information such as content and style using Tailwind CSS.
• Existing code serves as examples of how to write. Agents are great at picking up patterns and repeating them. For example, if existing code has full test coverage, an agent will likely write a test for new code as well.
• Readmes and other notes explain the purpose of the project, conventions and instructions to run it. Anything that helps a new hire also helps an agent. Rule files like copilot-instructions.md or AGENTS.md are more specifically aimed at agents. Effort invested here is leveraged by every agent run. Less is more: a few well maintained rules beat an overstuffed and rotting tome. Turning AGENTS.md into a table of contents linking to other notes worked well for a large AI engineering project by OpenAI. They took it further with custom linters by the principle: “Human taste is captured once, then enforced continuously on every line of code”. The emphasis is on human taste: Using an LLM to generate the instructions defeats the point. An benchmark study about solving Python Github issues found that LLM-generated AGENTS.md actually marginal decreased agent performance while increasing token usage by 20%.

These help with the intial context loading. Later, as the chat fills with prompts and responses, it’s important to be mindful of context window size and purity. Wrong information and failed attempts mislead LLMs. Claude’s phrase “You’re absolutely right” has become a meme because of this. When you see it, your context is rotten, and it’s time to start a new thread. One way to wrap up a thread is manual compaction: asking the LLM to summarize the problem, what it has tried and learned. The answer can be both educational and useful as a starting point for a new attempt. Anthropic has a full guide on context engineering for agents.

Harness engineering

Beyond context, you’re designing the agent’s development environment: the information it gets, the tools to unblock itself, and the guidance that keeps it aligned with your plans. Models are trained to hill-climb via RLHF and don’t give up easily, but they need instruments and feedback. Watch for situations where the LLM is blind to a problem and add the tools it needs to resolve problems without human intervention.

Among tooling, type checking deserves special mention: it lets agents easily catch errors statically. Theo Browne makes a strong case for TypeScript as an optimal language for agentic coding, as it guarantees type safety in frontend, backend and on their interfaces. Matt Honnibal gives practical Python advice focused on type safety and testability through pure functions. Linters, unit tests and tools to let agents look at the app’s UI are also important.

The most ambitious projects take multi-agent setups and long runs that span multiple context windows with compaction steps. Anthropic’s guide on effective harnesses covers failure modes and fixes, OpenAI’s case study is another good read. This is the forefront of agentic engineering.

Resisting fragmentation

While an AI chat window formally places the user in control, the dynamic can flip in practice: agents going off making unwanted changes, notifications interrupting the user’s work and the user becoming the agent’s QA help. It’s also easy to get lazy and let the agent do all the thinking.

Developers reporting on their experience emphasize controlling the dynamic:

Try to avoid workflow loops where you go off and do things, paste the error back into the LLM, and then go do whatever it tells you. I’ve seen this anti-pattern called a “reverse centaur”: you want to be the head of the human/horse hybrid, not the ass. – Matt Honnibal

“I found that it was my job as a human to be in control of when I interrupt the agent, not the other way around.” – Mitchell Hashimoto

Keeping agents running in the background while you’re working on something else is efficient, but can also be exhausting context switching. A study tracking AI use at a tech company found that AI use increased multitasking, such as manually coding while orchestrating agents working on other tasks. Workers reported increased cognitive fatigue as they took on extra tasks outside of their core competence and filled natural breaks with AI usage.

Taking responsibility

Opening a pull request is saying: I believe this code is good enough to be merged and is worthy of the reviewer’s time. But popular open source projects are facing an onslaught of low quality vibe coded pull request. As a consequence, new norms about AI-generated code are forming. Maintainers are rightfully angry when someone expects them to review an AI-generated mess. Reviewing is harder than generating.

The solution is to increase the burden of proof in regards to relevance and correctness. Simon Willison wrote: “Your job is to deliver code you have proven to work” and suggests more comprehensive tests, screenshots or even a video attached to the PR. I’d add that it’s also a developer’s job to understand the code and be able to answer questions about it during review.

Part of packaging code up for review is using Git. I find that staging changes for a commit is a natural point for human control to review generated code. Conceding git control to agents would remove this valuable loop.

Specialty benchmarks with narrower focus

• Coding: SWE-bench verified. Fix real GitHub issues from Python repositories. While highly relevant for evaluating coding capabilities, most enterprises will adopt existing AI coding tools (Claude Code, GitHub Copilot, Cursor) rather than develop custom coding agents. Leading score: 74.4% (Claude Opus 4.5, end of 2025).
• Web automation: WebArena, Mind2Web. Navigate and complete tasks on real websites. Web browsing is partially covered in GAIA.
• GUI automation: OSWorld, OfficeBench, AndroidWorld. Control Windows/Mac/Linux/Android via a graphical user interface. Only relevant if the agent must use a GUI instead of APIs. GUIs add a failure mode.
• Safety: FORTRESS. Tests safeguard robustness vs over-refusal. Important for production deployments but not the focus of this article.

Which types of agents are ready for enterprise use?

Let’s consider a business that wants to automate a task. According to the survey of Pan et al. (2025), increasing productivity is the most common motivation. The baseline for accuracy is a human worker doing the task, who can also make mistakes. Unlike standard software, the expectation shouldn’t be 100% accuracy, but an acceptable trade-off for the benefits of automation. GAIA provides a human baseline of 92%, which is just 2 percentage points ahead of the best models.

I propose three stages of readiness:

1. Internal tools reporting to humans, such as deep research, data analysis, information extraction, documentation and coding agents are ready now. The current highest scores for GAIA, BFCL and SWE-bench at the end of 2025 are 90%, 77.5% and 74.4%, respectively. Agents provide a profitable trade-off between accuracy and productivity. The time that humans spend checking results must be less than the time savings from automation.
2. Customer facing tools, such as customer service agents. The challenge here is consistency, not capability. τ-bench shows models hitting 80% pass^1 but dropping significantly on pass^8, meaning the agent might handle a request perfectly one day and fail the next. Tight monitoring and a swift escalation path to a human is necessary.
3. Long running autonomous work, such as inventory and portfolio management, scheduling, management of other agents over multiple tasks is not ready yet. Vending-Bench shows that even the best models show massive variance across runs, and there’s a risk of hitting meltdowns where they spiral into bizarre behavior.

GAIA: General AI Assistant

GAIA (Mialon et al. 2023) is the most important general benchmark. It consists of 466 high quality questions at three difficulty levels, annotated by humans. The authors value quality over quantity and it took them about 2 hours to make each question. The questions have unambiguous verifiable answers and require multi-step reasoning.

As a test, I asked Raycast AI to answer the level 3 difficulty example question from the paper.

It ended up doing 9 web searches and 1 call to a calculator tool. It got the answer wrong due to a false assumption.

Most tasks take 1 to 3 tools and 3 to 12 steps to solve. Questions can be solved by looking up information or by having it memorized. Answering the questions requires the following tools:

• Web search. It relies on information from trusted websites that are unlikely to disappear, such as arXiv and Wikipedia.
• Reading different file types: PDF, Excel, PowerPoint, CSV, image, audio and even video.
• Handling multimodal data, OCR (optical character recognition) and using Google Street View.
• Coding.

In contrast to other benchmarks, GAIA challengers are whole agentic systems that package one or more LLMs, the tools and an agentic loop. The current leader is SU Zero by Suzhou AI Lab. This enables measuring more innovation on the agentic loop and tools, but also means the scores can’t be used directly when choosing an off-the-shelf LLM.

GAIA is the most general benchmark and worth paying attention to, though at risk of saturation as the highscore already reached 90%. Like other benchmarks, GAIA questions and the majority of source materials are in English. Expect a drop in performance on other languages.

Berkeley Function-Calling Leaderboard (BFCL) V3

The Berkeley Function-Calling Leaderboard (Patil et al. 2025) measures LLMs’ ability to invoke functions/tools accurately across multiple programming languages and agentic scenarios. Function calling is the foundation of agentic AI: every API call to update a CRM, check inventory, process a payment, or query a database relies on accurate function calling.

The benchmark consists of 4441 question-function-answer triplets across Python, Java, JavaScript, SQL and REST API calls. They vary in complexity from simple function calls to multi-turn interactions. The augmented multi-turn questions added in V3 are especially challenging, as they require models to recognize situations in which additional information has to be requested from the user, or there isn’t a function that does the job.

Here’s one of the simplest questions: “Can you fetch me the weather data for the coordinates 37.8651 N, 119.5383 W, including the hourly forecast for temperature, wind speed, and precipitation for the next 10 days?” The correct answer is requests.get(url="https://api.open-meteo.com/v1/forecast", params={"latitude": "37.8651", "longitude": "-119.5383", "forecast_days": 10}).

The current leader is Claude Opus 4.5 at 77.5% overall accuracy (pass^1, end of 2025). It scores particularly well on the web search subscore at 84.5%. The leaderboard also reports cost, which ranges from less than $1 with small open weights models to $355 with Grok-4.

τ²-bench: Tool-Agent-User Interaction

τ²-bench (Yao et al. 2024) is a multi-turn support bot benchmark. It measures agents’ ability to interact with simulated users and follow domain-specific policies while using tools in multi-turn conversations. The original τ-bench (2024) introduced the retail and airline domains. τ²-bench (2025) added a telecom domain where both user and agent have access to different tools and must coordinate, better simulating expert-novice gaps in technical support scenarios.

The chat above is an example conversation that shows how user and agent exchange information and coordinate tool calls. An ablation study in which the support agent had access to all tools showed a significantly higher success rate, indicating that user-agent communication is a common failure mode. The full chat histories of the benchmark runs are available on the website.

The leading overall score is 85% (Gemini 3 Pro) for pass^1. Its domain subscores are 85% for retail, 73% for airline and 98% for telecom. The telecom score surprised me, as it’s so far above the best score reported in the paper introducing the domain, which listed Claude 3.7 Sonnet at 49% pass^1. However, other models also listed with scores in the high 90s for the telecom domain.

Quantified reliability with pass^k

The τ²-bench leaderboard reports pass^k metrics for most entries. The graph below shows the dropoff pattern from pass^1 to pass^4 for the current top models that report pass^k metrics and pooled across all domains.

Vending-Bench 2

Vending-Bench (Backlund and Petersson 2025) measures LLM agents’ ability to maintain coherent decision-making over extended time horizons by managing a simulated vending machine business.

The agent needs to use the internet with Perplexity to find hypothetical suppliers, order from them via email exchanges with simulated suppliers, stock the vending machine, set prices and collect earnings. It has access to a scratchpad, key-value database and vector database to help it remember information.

Evaluation and alternative settings

Since version 2, models are solely scored by their cash at the end of the simulation. The leading score is $5,478 (Gemini 3 Pro). The score doesn’t have a clear ceiling. Recent models made massive improvements over their predecessors, such as GPT-5.1 ($1,473) to GPT-5.2 ($3,591) and Gemini 2.5 Pro ($574) to Gemini 3 Pro ($5,478). According to an analysis of Andon Labs, a good player should be able to make at least $63k by finding the best suppliers and negotiating low prices.

In addition to the standard mode, there are two alternative settings:

• Arena mode, where multiple agents compete against each other in a multi-agent environment. They can exchange emails and act cooperatively or competitively. They display sophisticated behavior, such as negotiating payments for sharing information, joining their supplier orders for bulk discounts and even consignment deals.
• Project Vend, where a Claude agent managed a physical vending machine. It ended up with a huge loss after players persuaded it to give away items for free and stocking unusual items, such as tungsten cubes.

Loss of coherence

In the original vending machine benchmark paper (Backlund and Petersson 2025), all models had runs that derailed completely. Even Claude 3.5 Sonnet, the best performer, only increased net worth in three of five runs. In its worst run, it sold zero items. Failure means the agent stops doing basic tasks: restocking the machine, ordering from suppliers, or collecting earnings. These are individually trivial actions, but over hundreds of days, agents lose track of what they’re supposed to be doing.

When this happens, agents don’t degrade gradually. They melt down. Once an agent misinterprets its situation, it tends to spiral rather than self-correct. In one example, Claude 3.5 Haiku escalated a supplier dispute into increasingly unhinged emails demanding “QUANTUM NUCLEAR LEGAL INTERVENTION.” In another, Claude wanted to report the $2 daily fee it was charged to the FBI as an “ONGOING CYBER FINANCIAL CRIME.”

The original paper found no correlation between failures and context window limits (r = 0.167). Models failed well after their memory became full, suggesting breakdowns stem from something other than forgetting. Despite having access to scratchpads, key-value stores and vector databases, agents still lose coherence over time. Vending-Bench 2 added better planning tools including note-taking and reminder systems, yet coherence failures persist across all tested models.

These coherence failures share a cause: the agent must hold its own state together across a long, growing context. The longer the run, the more can go wrong. A bad input from an earlier turn can derail reasoning, a compaction step can erase important information, or the attention mechanism can fail to retrieve it from a long sequence.

Takeaways for enterprise agents

The same problems likely affect any long-running autonomous agent. Enterprise deployments need circuit breakers that detect anomalous behavior patterns and escalate to human review before meltdowns compound. External state management and periodic human checkpoints remain necessary even when agents have memory tools. Agents interfacing with external parties need skepticism built in, since those that over-trust their environment get exploited. For enterprise reliability, the relevant question isn’t “can it succeed?” but “how often does it fail catastrophically?” Even top models show high variance across runs.

Reinforcing loop, part I: Taming the tornado

Ousterhout coined the term tactical tornado, which describes a programmer that adds features quickly, but at the expense of code quality. Most AI coding assistants are tactical tornados. Given too long a lead, they turn a codebase into spaghetti. This is understandable. The AI is thrust into an unfamiliar project, and it has been trained to please its user, which often means achieving whatever is asked. That leads to tacking on the requested feature wherever it fits, rather than stepping back to suggest a redesign that would accommodate the feature more elegantly.

I find it useful to relate to coding agents as a smart, eager new intern. It works best when the project’s architecture is already set up by a senior engineer, and it’s given a well scoped task and all relevant context. Let’s break these requirements down further. Part I focuses on setting conditions where AI can operate safely and effectively.

Reinforcing loop, part II: Reinforcing design through consistent practice

So far we’ve seen how good design enables better AI output. But the loop works in reverse too - AI makes it cheaper to maintain good design practices.

On the question of how much to plan in advance (the waterfall approach) versus how much to start and then adjust as you go (agile), Ousterhout opined that a middle way is best. With AI available, I suggest moving more towards agile than before, because writing code has become cheaper.

Ousterhout also advocates an investment mindset, meaning to spend a little more time upfront on design and comments which repays soon in added productivity. AI improves the economics of this:

• The investment has become cheaper and also more valuable, since it benefits AI in addition to human developers. - - Tasks that feel like drudgery, such as updating comments or adding unit tests, can be delegated. Everyone now has infinite interns to pawn off tasks to.

Takeaways

• Better design improves AI output; AI then accelerates development and enables bolder changes.
• Design first: Deep modules, simple interfaces, and clear names make AI effective.
• Comments-first workflow: Write comments, let AI draft tests, then implement.
• Senior engineer in the loop: Boost human developers rather than attempting to replace them.

Finally, don’t let AI sap the joy out of programming! Keep your coding skills sharp. Turn off tab completion from time to time, especially when you’re working with something new. “A Philosophy of Software Design” does a great job of making principles explicit, but a lot of tacit knowledge and taste remains that needs to be honed by actively coding.

Book to prompt

The book provides handy summaries on the last pages: a list of principles (how to design well) and a list of red flags (to detect bad design and trigger rethinking). I suggest reviewing them, perhaps adding items related to your specific project, and then adding the list into a .cursorrules if you’re using Cursor or the equivalent for other assistants.

Converting agentic workflows to fixed workflows

In city planning, there’s the concept of a “desire path”. Rather than walking the long, intended path, people take a shortcut. There’s an urban legend that when Dwight D. Eisenhower was in charge of an extension of Columbia University, he let students walk on grass until natural paths had formed, and then had them paved.

This concept could be applied to agentic workflows. Start by giving an agent the freedom to choose tools and order of execution. Observe which paths are taken and which result in success. Then pave those paths by making them fixed workflows, enabling greater reliability.

Start with solid software engineering

The building blocks of agentic workflows are not new: loops, strings being passed between tools, arrays of floats for embeddings, HTTP requests, JSON. So the established concepts of writing clean code, enforcing type safety, having a test environment and running automated tests apply. A good agentic app starts with a good app.

Agent frameworks are optional

Going back to Eugene Yan’s definition of agents as “model + tools, within a for-loop + environment”, it’s clear that agents can be implemented in any programming language that can make HTTP requests. In Building Effective Agents, Anthropic notes:

Consistently, the most successful implementations weren’t using complex frameworks or specialized libraries. Instead, they were building with simple, composable patterns.

So should you use an agent framework or not? Let’s examine the tradeoffs:

As a case study, the AI test automation company Octomind wrote an article comparing LangChain to vanilla Python, explaining why they moved away from frameworks altogether.

Trace every step

Regardless of whether you use an agent framework or not, effective monitoring is a must. The first thing to put into place is a system that logs every step of the workflow: user inputs, transformed inputs, tool choices, tool calls, tool call results, reasoning tokens, output, latency, token usage, etc. The best and easiest time to set this up is right at the start of the project. As the project grows, set up a dashboard and alerts for critical metrics.

There’s no substitute for manual inspection

Manual inspection of data has probably the highest value-to-prestige ratio of any activity in machine learning.

Greg Brockman, President and co-founder of OpenAI

Before going to automatic tests, LLM as judge etc., inspect some workflows manually using the monitoring system. Check the input, the tool calls, the intermittent reasoning tokens, and the output. Many problems can be diagnosed this way. In addition, it peels away some of the “magic” that agentic frameworks by showing the prompts and tool calls. Hamel Husain put it well in an article, asking “Show me the prompt” of every LLM library. Manual inspection keeps yielding insights, even after automatic evals are in place.

Dataset preparation

The dataset on HuggingFace consists of 23,516 sentences. After removing duplicate sentences, 20,896 unique examples are left. The distribution of classes is uneven, with more examples of texts without an adverse events. To balance the classes, I’m subsampling the negative examples down to 4,271 cases. Balanced classes prevent the models from overfitting to the majority class and let us compare the models using a simple accuracy metric.

Then, the dataset is split into 60% training, 20% validation and 20% test sets. The validation set is used to tune hyperparameters and implement early stopping. Splits are stratified by class to ensure a 50:50 split between positive and negative examples in each split. The final example count is:

Model selection

I’m comparing ModernBERT-base and ModernBERT-large as the structured language models with Llama 3.2-3B-instruct as the LLM.

For inference, about 1.5 to 2x the model size is required to store the attention cache, calculate layer activations and other intermediate results. The A10G GPU used for this experiment has 24GB memory, so both models fit. The memory footprint can be reduced by half by using FP16 or INT8 precision, which is common for inference.

Setup 1: Fine-tuning ModernBERT

I’m using the transformers library to fine-tune ModernBERT base and large on the training set. Schmid (2024) from Hugging Face wrote a helpful guide which I adapted for use on Modal. The models are optimized on binary cross-entropy loss for 5 epochs. Training took about 2 minutes for ModernBERT-base and 3.5 minutes for ModernBERT-large.

Setup 2: Few-shot learning with Llama 3.2-3B and DSPy

I’m using DSPy (Khattab et al. 2023) to automatically select an optimal set of examples for few-shot learning. That’s a more objective approach than manual prompting and usually results in equally good or better accuracy. In my first trials, DSPy didn’t manage to write a suitable system prompt as it didn’t understand the adverse drug reaction task from examples alone. So I added the prompt: “Determine if the following sentence is about adverse drug reactions:” to the examples. This increased the accuracy by about 15 percentage points.

DSPy settings:

• 20 few-shot examples plus 5 bootstrapped (AI generated) examples
• Optimized for accuracy using MIPROv2 (minibatch size 50, minibatch full eval steps 10, num trials 3)
• 25 threads for calls to the LLM, which is hosted using FastAPI and vLLM on Modal

The optimized predictor is available as a JSON file in the Weights & Biases project.

Setup 3: Fine-tuning Llama 3.2-3B

I’m using the torchtune library and a fine-tuning configuration to train a LoRA adapter on the training set. It targets the attention and feed-forward layers of the model. The adapter is a smaller set of weights that are added to the model at inference time. LoRA training incurs less training cost than full fine-tuning of all weights, but may result in worse accuracy. The LoRA settings used for training are available in the W&B project and the training config file for torchtune. Training took about 8 minutes on the A10G.

Accuracy and speed

ModernBERT-base and ModernBERT-large performed similarly. Large has a 3.2 percentage point advantage in F1 score but at the cost of 27% slower inference. The few-shot approach doesn’t need nearly as much training data, but also results in a less accurate model. It also ran slowly, and this didn’t change when I increased the number of threads used by DSPy to communicate with the vLLM server. I suspect it’s due to inefficient batch inference code in DSPy. Higher speeds could be achieved with a more efficient batching approach.

The clear winner of the experiment is the LoRA fine-tuned Llama 3.2-3B. It’s the most accurate and the fastest. This is down to vLLM being extremely well optimized and using CUDA graph capturing ahead of inference time. If that preparation time of 30 seconds is added, it’s examples per second go down to 42.

Cost and effort

All setups can be trained for under one dollar and in less than 15 minutes. The differences are negligible. What matters more is the time spent setting up training and inference. The transformers library and the tutorial made it very easy to fine-tune ModernBERT and run inference. A major plus is that due to its low size, it can run on CPU at good speed too. DSPy was more involved because it required setting up a vLLM server too. This step is easier when using a managed service like Fireworks AI. Fine-tuning Llama 3.2-3B was the most involved step, as it required formatting the data in a chat format and going through the detailed configuration of the torchtune library and vLLM. Still, it only took a few hours. This step is also easier with a managed service.

Implications for NLP

Implications for adverse event monitoring

Content and layout

1. Use the appropriate diagram type. Flowcharts are the most common, but there are many others such as sequence diagrams, class diagrams, and user journey diagrams.

2. Model the key components, rather than every possible detail. Keep the diagram at a single level of abstraction. Use multiple diagrams if needed.

My rule of thumb is that you need to be able to print the diagram on a single A4 sheet while keeping things readable. – Geert Bellekens, enterprise architect

3. Avoid crossing lines. Layout engines do a good job of this. If they fail to find a layout that avoids overlaps, it’s a sign that the diagram is too complex. Also prefer vertical and horizontal lines over diagonal ones, as they give the diagram a more professional look.

Styling

1. Label all components and arrows. A relationship may be obvious to you, but not to others. Use one or two word labels in a sans-serif font at a size readable for aging eyes.
2. Use consistent shapes, arrows and colors. This helps readers scan the diagram quicker. Use stylistic elements like thicker lines for primary flows and thinner for secondary flows, dotted or dashed lines for optional or future relationships, and different shapes for different types of components.
3. Use color sparingly and meaningfully, e.g., to highlight critical paths. Don’t rely on color as the only way information is conveyed. Use labels and shapes to ensure the diagram works in black and white too. 1 in 12 men are colorblind.
4. Consider using a hand-drawn style in early stages of a design. It conveys that the diagram is a draft and not a final product.

1. When prompting isn’t sufficient

If the desired behavior can be achieved by a prompt, you don’t need to implement DPO. Prompts are easier to change and different instances of a model can run with different system prompts. However, prompts add tokens on every call, which makes them slower, more expensive and fills up the context window. They’re also more limited in what behaviors they can achieve and may be ignored by the model. In those cases, DPO is a more robust solution and can of course be be combined with prompts.

2. When you can’t generate optimal answers at scale

Humans typically have an easier time determining which of two answers is better than developing the best answer on their own. This is especially true for complex outputs, multi-turn conversations, and all matters of style where there are multiple acceptable answers. As an example, it’s faster to judge which of two email texts sounds better than writing the perfect email.

3. When you want to preserve previous behavior

DPO is a more measured treatment than SFT and can be regulated with the hyperparameter. This makes it a good choice if the model is already trained and you only want to make small changes. In contrast, SFT is a more aggressive treatment that overrides previous behavior, for example, a model trained on a named entity recognition task would start speaking JSON rather than English.

Use cases

Let’s consider common LLM use cases and the criteria listed above to decide between SFT+DPO and SFT only.

Tasks that fully leverage LLM’s free-form input and output tend to benefit most from DPO.

Database

I’m using DuckDB to create an in-memory database which will be made available to the agent.

import duckdb

1con = duckdb.connect()

I’ll insert a set of reports into the database. The data included is fictional and was generated by an LLM. The data consists of 40 reports like this:

import polars as pl
from great_tables import GT

reports = pl.read_csv("data/reports.csv")
GT(reports.head(5))

To make the title searchable, I’ll embed it using an OpenAI embedding endpoint. The result will be stored in a new column with 1536 dimensions.

from openai import OpenAI
from tqdm import tqdm


def embed_text(text: str) -> list[float]:
    client = OpenAI()
    model = "text-embedding-3-small"
    return client.embeddings.create(input=text, model=model).data[0].embedding


title_embeddings = [embed_text(title) for title in tqdm(reports["title"])]

reports = reports.with_columns(
    pl.Series(
        name="title_embedding",
        values=title_embeddings,
        dtype=pl.Array(inner=pl.Float64, shape=1536),
    )
)

Now, I’ll insert the data including the embeddings into the database. The embeddings are stored in a fixed-size ARRAY column. The co-location of the structured data and the embeddings in the same table is convenient for our use case.

con.execute(
    """
    CREATE OR REPLACE TABLE reports AS
    SELECT
        id::integer AS id,
        year::integer AS year,
        institute::varchar AS institute,
        country::varchar AS country,
        topic::varchar AS topic,
        title::varchar AS title,
        title_embedding::float[1536] AS title_embedding
    FROM reports;
1    """
)

con.execute("INSTALL vss;")
con.execute("LOAD vss;")

con.execute(
    "CREATE INDEX titles_hnsw_index ON reports USING HNSW(title_embedding) WITH (metric='cosine');"
)

I also create a hierarchical navigable small world (HNSW) index on the title embeddings. This enables approximate nearest neighbor search in O(log n). It’s enabled by the vss extension. Note that persistence to disk is experimental, so I wouldn’t recommend it for production yet.

Agent

Let’s set up an agent powered by the Groq inference API. It serves a range of open source models. Specifically, I’ll use the llama-3.3-70b-versatile model released by Meta on December 6th. Artificial Analysis has a detailed report showing that it advanced the speed-accuracy trade-off. The model has tool calling capabilities, which are critical for our use case.

from pydantic_ai import Agent

agent = Agent(
1    model="groq:llama-3.3-70b-versatile",
    system_prompt="You are a market research expert and answer questions using a database of reports.",
)

result = agent.run_sync("Who are you?")
print(result.data)

I am a market research expert, providing insights and analysis based on a vast database of reports and studies. My expertise spans various industries, including consumer goods, technology, healthcare, and finance. I can help answer questions, provide data-driven insights, and offer market trends and analysis to support business decisions.

My database includes reports from reputable sources, such as market research firms, academic institutions, and industry associations. I can access a wide range of topics, including market size and growth, consumer behavior, competitor analysis, and emerging trends.

What specific area of market research would you like to explore?

Tools

The agent’s job will be to answer questions based on the reports in the database. It needs a way to access the database. We can give it a tool, meaning a function that it can call, to query the database. First, it needs a database connection.

from dataclasses import dataclass


@dataclass
1class AgentDependencies:
    db: duckdb.DuckDBPyConnection


2deps = AgentDependencies(db=con)

Next, let’s give the agent a tool to search the database of reports. Based on the user’s question, it can choose which field to search. The result is always a markdown-formatted table with one row per report.

import json
from typing import Literal
from pydantic_ai import RunContext
from pydantic import validate_call, Field


1def df_to_str(df: pl.DataFrame) -> str:
    return json.dumps(df.to_dicts())


2@agent.tool
3@validate_call(config={"arbitrary_types_allowed": True})
def search_reports_by_field(
4    ctx: RunContext[AgentDependencies],
5    field: Literal["id", "year", "institute", "country", "topic"],
    value: str = Field(
        description="The value to search for in the field. Case insensitive."
    ),
) -> str:
    base_query = """
        SELECT id, year, institute, country, topic, title 
        FROM reports 
        WHERE {}
    """

    if field in ["id", "year"]:
        value = int(value)
        where_clause = f"{field} = ?"
    else:
        where_clause = f"lower({field}) = lower(?)"

    final_query = base_query.format(where_clause)
6    df = ctx.deps.db.execute(final_query, [value]).pl()

    if df.shape[0] == 0:
7        return "No reports found. Try a different field or value, or use the title similarity tool."
    return df_to_str(df)

This lets the agent execute searches based on the exact match of a field.

deps = AgentDependencies(db=con)
result = agent.run_sync("Which reports do we have from Germany?", deps=deps)
print(result.data)

We have four reports from Germany:

1. "Global Electric Vehicle Market Outlook 2018-2023" by Research DNA GmbH (2018) - Automotive topic
2. "Digital Advertising Spend Analysis" by Tech Innovations Ltd. (2020) - Media topic
3. "Beverage Market Competitive Analysis" by Research DNA GmbH (2022) - FMCG topic
4. "Medical Imaging Equipment Market Size" by Tech Innovations Ltd. (2024) - Healthcare topic

Let me know if you'd like more information about any of these reports.

It works, the agent found the 4 reports from Germany. Let’s check the exact tool call:

agent.last_run_messages

[ModelRequest(parts=[SystemPromptPart(content='You are a market research expert and answer questions using a database of reports.', part_kind='system-prompt'), UserPromptPart(content='Which reports do we have from Germany?', timestamp=datetime.datetime(2024, 12, 18, 17, 38, 58, 663721, tzinfo=datetime.timezone.utc), part_kind='user-prompt')], kind='request'),
 ModelResponse(parts=[ToolCallPart(tool_name='search_reports_by_field', args=ArgsJson(args_json='{"field": "country", "value": "Germany"}'), tool_call_id='call_be61', part_kind='tool-call')], timestamp=datetime.datetime(2024, 12, 18, 17, 38, 58, tzinfo=datetime.timezone.utc), kind='response'),
 ModelRequest(parts=[ToolReturnPart(tool_name='search_reports_by_field', content='[{"id": 1, "year": 2018, "institute": "Research DNA GmbH", "country": "Germany", "topic": "Automotive", "title": "Global Electric Vehicle Market Outlook 2018-2023"}, {"id": 12, "year": 2020, "institute": "Tech Innovations Ltd.", "country": "Germany", "topic": "Media", "title": "Digital Advertising Spend Analysis"}, {"id": 21, "year": 2022, "institute": "Research DNA GmbH", "country": "Germany", "topic": "FMCG", "title": "Beverage Market Competitive Analysis"}, {"id": 32, "year": 2024, "institute": "Tech Innovations Ltd.", "country": "Germany", "topic": "Healthcare", "title": "Medical Imaging Equipment Market Size"}]', tool_call_id='call_be61', timestamp=datetime.datetime(2024, 12, 18, 17, 38, 59, 27429, tzinfo=datetime.timezone.utc), part_kind='tool-return')], kind='request'),
 ModelResponse(parts=[TextPart(content='We have four reports from Germany:\n\n1. "Global Electric Vehicle Market Outlook 2018-2023" by Research DNA GmbH (2018) - Automotive topic\n2. "Digital Advertising Spend Analysis" by Tech Innovations Ltd. (2020) - Media topic\n3. "Beverage Market Competitive Analysis" by Research DNA GmbH (2022) - FMCG topic\n4. "Medical Imaging Equipment Market Size" by Tech Innovations Ltd. (2024) - Healthcare topic\n\nLet me know if you\'d like more information about any of these reports.', part_kind='text')], timestamp=datetime.datetime(2024, 12, 18, 17, 38, 59, tzinfo=datetime.timezone.utc), kind='response')]

Here, the model correctly translated the user’s question into the tool call with the arguments {"field": "country", "value": "Germany"}.

To make it easier to evaluate the agent’s output and also make its results useable by other tools, we can create a response model that includes the ids of the identified reports.

from pydantic import BaseModel


class AgentResponse(BaseModel):
    text: str = Field(
        description="Answer to the user's question in informal language. Don't include the report ids."
    )
    relevant_report_ids: set[int] = Field(
1        description="Set of 'id' integer values of the reports that are relevant to the user's question. Only include ids retrieved by the search tools. Never make up ids. Not all ids returned by the search tools are relevant."
    )


typed_agent = Agent(
    model="groq:llama-3.3-70b-versatile",
    system_prompt="You are a market research expert and answer questions using a database of reports.",
    result_type=AgentResponse,
2    result_retries=3,
)

The AgentResponse model is used to validate the agent’s output. It will always include a set of integer ids. In an app, these could be used to provide links to the reports.

result = typed_agent.run_sync("Which reports do we have from Germany? Tell me their titles and ids", deps=deps)
print(result.data)

text='The reports from Germany are titled Global Electric Vehicle Market Outlook 2018-2023, Digital Advertising Spend Analysis, Beverage Market Competitive Analysis and Medical Imaging Equipment Market Size.' relevant_report_ids={32, 1, 12, 21}

Now we have an agent that returns a type-checked structured response. Note that I’ve omitted the re-registration of the tool to the new agent instance for brevity.

However, requests may not exactly match the fields in the database, so let’s also add the ability to search for similar titles.

@typed_agent.tool
@validate_call(config={"arbitrary_types_allowed": True})
def search_reports_by_title_similarity(
    ctx: RunContext[AgentDependencies],
    title: str = Field(
        description="The title of the report to search for with vector similarity."
    ),
) -> str:
    # Embed the title given by the user
    try:
        title_embedding = embed_text(title)
    except Exception as e:
        return f"Error embedding title: {e}"

    # Search for similar titles
1    title_embedding_str = "[" + ",".join(map(str, title_embedding)) + "]"
    query = """
        SELECT id, year, institute, country, topic, title
        FROM reports
        ORDER BY array_distance(title_embedding, ?::FLOAT[1536])  
        LIMIT 5;
2    """
    df = ctx.deps.db.execute(query, [title_embedding_str]).pl()

    return (
        df_to_str(df)
3        + "\n\n These reports have titles similar to the query, but may not be relevant to the user's question."
    )

Let’s ask the agent about a topic that is not in the database to see how it uses the title similarity tool.

result = agent.run_sync("Do we have reports about quantum computing?", deps=deps)
print(result.data)

<function=search_reports_by_field {"field": "topic", "value": "quantum computing"}</function>

That worked as expected.

Comparison to other libraries

PydanticAI is a late entrant to the agent framework space. It joins several established libraries including:

There are dozens of other libraries with fewer stars. In addition, there are libraries specialized for RAG like LlamaIndex and Haystack. The competition landscape doesn’t show signs of consolidation or slowing down.

Development team

Pydantic Services, the company behind Pydantic, has raised a $12.5m Series A in October 2024. This is great news for the project: funding pays for full time developers. It also raises the question of how Pydantic will make money, and the answer to that is Logfire subscriptions. This is a good model that gives long-term stability to the project and follows the lead of LangChain with its commercial product, LangSmith. I just hope that the integration remains optional. While Logfire looks great, my team already uses Weave by Weights & Biases, and having to switch would be a barrier to adopting PydanticAI.

Review

I’m looking forward to an opportunity to build a full-scale application with PydanticAI. The best place to get started is the PydanticAI documentation.

Preview photo by MagicPattern on Unsplash

Experiment setup

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#ffffff',
    'primaryTextColor': '#2d3748',
    'primaryBorderColor': '#90cdf4',
    'lineColor': '#64748b',
    'secondaryColor': '#ffffff',
    'tertiaryColor': '#ffffff',
    'fontSize': '22px',
    'labelFontSize': '18px',
    'edgeLabelFontSize': '18px'
  }
}}%%
graph TB
    %% Define styles
    classDef default fill:#ffffff,stroke:#90cdf4,stroke-width:2px
    classDef highlight fill:#fdf2f8,stroke:#ed64a6,stroke-width:3px
    classDef api fill:#ffffff,stroke:#4fd1c5,stroke-width:2px
    
    subgraph Data ["1️⃣ Data"]
        D1[SemEval Dataset] --> |"Transform"| D2[DSPy Examples]
    end
    
    subgraph Definition ["2️⃣ Model Definition"]
        M2[Pydantic Models] --> |"Define Structure"| M1[DSPy Signature]
        M1 --> |"Initialize"| M3[Predictor]
        M4[Language Models] --> |"Power"| M3
        A1[OpenAI API] --> |"Provide"| M4
        A2[Fireworks.ai API] --> |"Provide"| M4
    end
    
    subgraph Optimization ["3️⃣ Optimization"]
        O1[Evaluation Function] --> |"Guide"| O2[MIPROv2 Optimizer]
        M3 --> |"Optimize"| O2
        D2 --> |"Train"| O2
        O2 --> |"Output"| O3[Optimized Predictor]
    end
    
    subgraph Evaluation ["4️⃣ Evaluation"]
        O3 --> |"Test"| E1[Test Set Evaluation]
        O1 --> |"Measure"| E1
        E1 --> |"Log"| E2[Weights & Biases]
    end

    %% Apply styles
    class O2 highlight
    class A1,A2,E2 api
    
    %% Links between subgraphs
    linkStyle default stroke:#64748b,stroke-width:2px

The steps will be explained in the following sections.

SemEval 2014 Task 4

I’m using the SemEval 2014 Task 4 dataset by Pontiki et al. (2014). The dataset is available on Hugging Face. This is a cleaned version of the original XML files consisting of train and test splits. The small number of examples with the “conflict” label are excluded, as is common in the literature.

import polars as pl

url = "hf://datasets/psimm/absa-semeval2014-alpaca"

train = pl.read_parquet(url + "/data/train-00000-of-00001.parquet")
test = pl.read_parquet(url + "/data/test-00000-of-00001.parquet")

from great_tables import GT

overview = (
    train.vstack(test)
    .group_by(["split", "domain"])
    .agg(examples=pl.len())
    .sort("split", "domain", descending=True)
)
GT(overview).tab_header("SemEval 2014 Task 4 Dataset").cols_label(
    split="Split",
    domain="Domain",
    examples="Examples",
).cols_align(align="right", columns=["examples"])

The dataset contains a similar number of restaurant and laptop reviews.

The goal is to choose the optimal prompt and few-shot examples to maximize the F1 score of the aspect extraction and classification. To achieve this, DSPy needs to be able to evaluate the metrics and a training set to learn from.

Pydantic models for ABSA

We create classes to represent the input and output of the task using the data validation library Pydantic. This helps with validating the data and provides a structured output format for predictor. The Field class is used to describe the expected data type. Their descriptions match the ones used in (Simmering and Huoviala 2023). This is a form of prompting, but DSPy also supports automatically setting the structure’s descriptions using the optimize_signature optimizer. In this experiment I’ll stick with the original descriptions and only vary the normal prompt and few-shot examples.

from typing import Literal
from pydantic import BaseModel, Field


class Input(BaseModel):
    text: str = Field()


class Aspect(BaseModel):
    term: str = Field(
        description="An aspect term, which is a verbatim text snippet. Single or multiword terms naming particular aspects of the reviewed product or service."
    )
    polarity: Literal["positive", "neutral", "negative"] = Field(
        description="The polarity expressed towards the aspect term. Valid polarities are ‘positive’, ‘neutral’, ‘negative'."
    )

    def __hash__(self):
        """
        Make the aspect hashable to enable set operations in evaluation.
        Hash is case-insensitive.
        """
        return hash((self.term.lower(), self.polarity.lower()))

    def __eq__(self, other):
        """
        Define equality for case-insensitive comparison.
        """
        if not isinstance(other, Aspect):
            return False
        return (
            self.term.lower() == other.term.lower()
            and self.polarity.lower() == other.polarity.lower()
        )


class Aspects(BaseModel):
    aspects: list[Aspect] = Field(
        description="An array of aspects and their polarities. If no aspects are mentioned in the text, use an empty array."
    )

The __hash__ and __eq__ methods will be helpful for evaluation, because they allow for use of set operations to compare gold and predicted aspects.

Transform dataset to DSPy examples

Each row in the dataset needs to be turned into an instance of the dspy.Example class. The with_inputs method is used to tell DSPy which column contains the input. Other columns are used as expected model outputs.

import json
import dspy


def to_example(row):
    return dspy.Example(
        text=row["input"],
        aspects=Aspects(aspects=json.loads(row["output"])["aspects"]),
    ).with_inputs("text")


trainset = [to_example(row) for row in train.to_dicts()]
testset = [to_example(row) for row in test.to_dicts()]

Let’s look at the first example.

trainset[0]

Example({'text': 'I charge it at night and skip taking the cord with me because of the good battery life.', 'aspects': Aspects(aspects=[Aspect(term='cord', polarity='neutral'), Aspect(term='battery life', polarity='positive')])}) (input_keys={'text'})

Creating a DSPy typed predictor

In DSPy, a module is a language model and a way of prompting. They can also consist of multiple requests and also include external tools such as a vector database for retrieval augmented generation. In this example, we have a single request using few-shot examples and chain of thought.

In order to be able to parse the output as a dictionary, the LLM must output valid JSON. Therefore I’ll use a Typed Predictor in DSPy, which is similar to structured outputs via instructor or a similar library.

class AbsaSignature(dspy.Signature):
    text: Input = dspy.InputField()
    aspects: Aspects = dspy.OutputField()


predictor = dspy.ChainOfThought(AbsaSignature)

We also need to choose a language model. DSPy works with OpenAI, Anthropic, Ollama, vllm and other OpenAI-compatible platforms and libraries. This is powered by litellm under the hood.

For this article, I’ll use OpenAI’s gpt-4o-mini as well as the 70B version of Meta’s Llama 3.1 hosted on fireworks.ai. Fireworks.ai generously supplied me with credits as part of the Mastering LLMs For Developers & Data Scientists course.

# FIREWORKS_AI_API_KEY environment variable must be set.

lm = dspy.LM(
    api_base="https://api.fireworks.ai/inference/v1/",
    model="fireworks_ai/accounts/fireworks/models/llama-v3p1-70b-instruct",
    temperature=0.0,  # best for structured outputs
    cache=True,
    max_tokens=250,
)
dspy.configure(lm=lm)

Specify the evaluation function

An evaluation function takes an example and a prediction and returns an F1 score. A true positive is a predicted aspect that is also in the gold answer, a false positive is a predicted aspect that is not in the gold answer, and a false negative is a gold answer aspect that is not predicted. Here are the precision, recall, and F1 score functions.

def precision(tp: int, fp: int) -> float:
    # Handle division by zero
    return 0.0 if tp + fp == 0 else tp / (tp + fp)


def recall(tp: int, fn: int) -> float:
    return 0.0 if tp + fn == 0 else tp / (tp + fn)


def f1_score(tp: int, fp: int, fn: int) -> float:
    prec = precision(tp, fp)
    rec = recall(tp, fn)
    return 0.0 if prec + rec == 0 else 2 * (prec * rec) / (prec + rec)

Next is the evaluation function which compares the gold and predicted aspects. To count as a true positive, both the term and the polarity have to be correct. As it is conventional on this benchmark, the case where both the gold answers and the prediction are empty is treated as a correct prediction of no aspects.

def evaluate_absa(example: dspy.Example, prediction: Aspects, trace=None) -> float:
    gold_aspects = set(example.aspects.aspects)
    pred_aspects = set(prediction.aspects.aspects)

    tp = len(gold_aspects & pred_aspects)
    fp = len(pred_aspects - gold_aspects)
    fn = len(gold_aspects - pred_aspects)

    if len(gold_aspects) == 0 and len(pred_aspects) == 0:
        tp += 1  # correct prediction of no aspects

    return f1_score(tp, fp, fn)

Let’s try the evaluation function with a single example. We expect the F1 score to be 1.0, because the prediction matches the gold answer exactly.

example = dspy.Example(
    text="The pizza was great, but the service was terrible",
    aspects=Aspects(
        aspects=[
            Aspect(term="pizza", polarity="positive"),
            Aspect(term="service", polarity="negative"),
        ]
    ),
).with_inputs("text")
prediction = predictor(text=example.text)
evaluate_absa(example, prediction)

1.0

Optimizers

DSPy has a variety of optimizers, loops that change the prompt and/or few-shot examples and evaluate the performance. They’re analogous to optimizers like SGD and Adam in PyTorch. The choice of optimizer depends on the task, the amount of labeled data and the computational resources available. As we have a large labeled dataset, it’s not necessary to have the model bootstrap artificial examples. Our 2023 paper found that fine-tuning yields the best results, but the goal of this article is to showcase DSPy’s prompt optimization.

The most powerful optimizer available for a prompting approach for this task is MIPROv2 (Multiprompt Instruction PRoposal Optimizer Version 2) by Opsahl-Ong et al. (2024). MIPROv2 uses Bayesian optimization to find an optimal combination of few-shot examples and prompt instructions.

optimizer_settings = dict(
    metric=evaluate_absa,
    num_threads=12,  # make parallel requests to Fireworks.ai
    max_errors=1000,  # keep going even when invalid JSON is returned
)
optimizer = dspy.teleprompt.MIPROv2(**optimizer_settings)

The final step is to call the compile method, which starts the optimization process. After about 5 minutes, the best prompt and few-shot examples are saved to a JSON file.

# Define settings for the comilation step of the optimizer.
compile_settings = dict(
    minibatch_size=50,  # evaluate changes on a subset of the validation set
    minibatch_full_eval_steps=10,  # evaluate on the full validation set after every 10 steps
    max_labeled_demos=4,  # the number of few-shot examples to use
    max_bootstrapped_demos=1,  # not required because we have labeled examples, but setting it to 0 causes an error during sampling
    num_trials=3,  # how many combinations of few-shot examples and prompt instructions to try
    seed=42,  # for reproducibility
    requires_permission_to_run=False,  # skip confirmation dialog
)

We save the optimized predictor to a JSON file. It’s a small config file listing the chosen few-shot examples and the optimized prompt.

optimized_predictor = optimizer.compile(
    student=predictor, trainset=trainset, **compile_settings
)
optimized_predictor.save("configs/absa_model.json")

Let’s check if we can load it again:

optimized_predictor = dspy.ChainOfThought(signature=AbsaSignature)
optimized_predictor.load(path="configs/absa_model.json")

Again: “Show me the prompt”.

print(optimized_predictor.extended_signature.instructions)

You are a product reviewer tasked with analyzing customer feedback for laptops and netbooks. Given the fields `text`, which contains a customer review, produce the fields `aspects`, which should include the specific features or aspects of the laptop or netbook mentioned in the review, along with their corresponding sentiment or polarity.

and show me the chosen few-shot examples:

for demo in optimized_predictor.demos[:3]:  # first 3 examples
    print(demo["text"])
    print(demo["aspects"])

-Called headquarters again, they report that TFT panel is broken, should be fixed by the end of the week (week 3).
{"aspects":[{"term":"TFT panel","polarity":"negative"}]}
But we had paid for bluetooth, and there was none.
{"aspects":[{"term":"bluetooth","polarity":"negative"}]}
The powerpoint opened seamlessly in the apple and the mac hooked up to the projector so easily it was almost scary.
{"aspects":[{"term":"powerpoint","polarity":"positive"}]}

Comparison to the 2023 manual prompts

The DSPy runs are competitive with the manually crafted prompts from the 2023 paper. In contrast to the manual prompt, DSPy instructions are relatively short and emphasize the use of few-shot examples to illustrate the task.

Impact of hyperparameters

To understand which factors significantly influence the F1 score, we’ll run a simple linear regression analysis. The manual runs are excluded. To analyze the impact of the model choice, we’ll create a boolean variable for gpt-4o-mini and treat llama-v3p1-70b-instruct as the baseline.

import statsmodels.api as sm
import pandas as pd

# Prepare data for regression
reg_df = results_df.filter(
    ~pl.col("run_name").str.contains("manual"),
    pl.col("model").str.contains("gpt-4o-mini") | pl.col("model").str.contains("llama"),
).with_columns(  # exclude manual prompts
    pl.col("model").str.contains("gpt-4o-mini").alias("is_gpt4_mini"),
)


# Convert to pandas and ensure numeric types
X = reg_df.select(
    ["max_demos", "chain_of_thought", "num_trials", "is_gpt4_mini"]
).to_pandas()

# Convert boolean columns to int
bool_columns = ["chain_of_thought", "is_gpt4_mini"]
for col in bool_columns:
    X[col] = X[col].astype(int)

y = reg_df.select("f1").to_pandas()

# Add constant for intercept
X = sm.add_constant(X)

# Fit regression
model = sm.OLS(y, X).fit()
n = len(reg_df)
r2 = model.rsquared

# Print results using GT
df = pd.DataFrame(
    model.summary().tables[1],
    columns=[
        "Parameter",
        "Coefficient",
        "Std Error",
        "t",
        "p>|t|",
        "[0.025",
        "0.975]",
    ],
)

df = df.iloc[1:]  # remove row with repeated column names

GT(df).tab_header(
    title="Hyperparameter Analysis", subtitle="Dependent variable: F1 score"
).cols_align(align="right").tab_source_note(f"n={n} runs, R²={r2:.2f}")

Few-shot examples

More examples are generally better, as indicated by the positive coefficient in the regression. However, the top runs didn’t use more than 20 examples, indicating that there are diminishing returns.

Chain of thought (CoT)

Runs where the model was instructed to perform an intermediate reasoning step yielded worse results than those without. This is an unusual result - typically CoT helps LLMs achieve better results, for example the main advantage of OpenAI’s o1-preview over gpt-4o is the advanced CoT that is built into it. However, on this structured task and using DSPy’s Predictor and ChainOfThought classes, CoT seems to be detrimental.

Model choice

• gpt-4o-mini-2024-07-18 seems to have an edge over llama-v3p1-70b-instruct, but the confidence interval is wide.
• gpt-4o-2024-11-20 performs better than the other models that were tested. I expect that performance of similar sized models such as Llama 3.1 405B will be similar. Due to cost considerations, I’ve skipped the optimization of a large model with DSPy.
• gpt-3.5-turbo-0125 performed better than gpt-4o-mini-2024-07-18, but worse than the deprecated gpt-3.5-turbo-0613 performed during the experiments for the 2023 paper (57.45 vs. 65.65 F1 Score).

Number of trials

Using more trials is associated with higher F1 scores. However, the table also shows setups with identical results at 15, 30 and 60 trials. Going beyond 60 trials isn’t likely to be helpful.

1. More detailed guidelines

A longer prompt can provide more context to the model, letting it perform a task more accurately or represent a brand or character more faithfully. Consider including information like this:

• Background information about the website, app or task that the model is embedded in.
• Behavioral constraints, like not using certain words or phrases. For example telling the prompt to avoid starting answers with “Certainly!”, to make it sound less AI-like.
• Style guidelines, like using a certain tone or level of formality, whether to address the user by first or last name, or to use emojis.
• Characterization, giving the model a personality or role to play. For example, a chatbot for a bank could be characterized as a friendly and professional customer service agent.
• A more detailed task description, like a list of steps to follow or a description of the desired output.
• Information about the user, like their name, location, or preferences.
• A translation glossary, if the model is used in a multilingual setting.

If you’re looking for inspiration for a chatbot prompt, check the recently revealed prompts for Anthropic’s Claude.

2. Many-shot in-context learning

Few-shot examples can be included in the prompt for in-context learning (ICL). These examples can teach the model about the rules for the task, the desired output format, intermediate reasoning steps and handling of edge cases. Commonly this is done with 1 to 5 examples, but with prefix caching it’s possible to include 50, 100 or even more examples.

Agarwal et al. (2024) ran this experiment with Gemini 1.5 Pro across several tasks. Many-shot ICL outperformed few-shot learning in all cases. For sentiment analysis they went as far as 2048 examples in the prompt, achieving an increase in 18.2 percentage points over a 32-shot prompt. In many of their experiments the limiting factor wasn’t the context size, but the number of available examples.

This allows a prompting approach become closer to fine-tuning, but without the need for training or a model store. Bertsch et al. (2024) made the comparison between many-shot ICL and LoRA (Hu et al. 2021) on 5 classification tasks and conclude that “finetuning is more data-hungry than ICL”. In their experiments with Llama2-7b, many-shot prompting outperformed fine-tuning up to about 1000 examples (see figure 2 of their paper).

3. More RAG context

A key design parameter in retrieval augmented generation (RAG) is the number of text chunks to retrieve from a source. With a larger context size, more and longer text chunks can be included in the prompt. This increases the likelihood that the information required to answer the query is present in the prompt.

Leng et al. (2024) tested RAG answer correctness on 13 open source and proprietary LLMs.

As the graph above shows, answer correctness increased with longer context all models up to 4k tokens and up to 32k tokens for most models. This is driven by the boost in retrieval (see experiment 1 in the article).

However, the “lost in the middle” problem can occur, a phenomemon first found by (Liu et al. 2023), where information presented in the middle is not used as well as information presented at the beginning or end. It can be measured by the “needle in a haystack” method, meaning that a piece of information is hidden in a long text and the model has to find it. The longer the text, the harder it is to find the information. The RULER benchmark by (Hsieh et al. 2024) extended this to more complex tasks and introduced the concept of an effective context length, which is shorter than the technical context length of a model.

Retrieving more information also increases the risk of including irrelevant information. If chunk ranking works correctly, lower ranking chunks are less likely to be relevant and adding them reduces the density of relevant information. Levy, Jacoby, and Goldberg (2024) found that irrelevant information isn’t neutral, it’s detrimental to model performance on a question-answering task.

4. More functions for agentic models

Models used as agents are given function signatures in a JSON schema. Each of these has to be sent to the model as part of the prompt. The more functions and the more arguments they have, the longer the prompt. With lower prompt costs, it’s becoming more economical to have agents with many different and more detailed functions in their repertoire.

Common functions include:

• Send a task to a sub-agent
• Web search
• Query a database by using text-to-SQL
• Redirect to a human agent
• Call a REST API, e.g. to send an email or schedule a meeting
• Execute code in Python, JavaScript or another language

The Berkeley function calling leaderboard (Yan et al. 2024) offers detailed benchmarks for a variety of function calling tasks.

More alternatives at higher levels

There tend to be more alternatives the higher you go in the stack. Recently, I’ve compared 10 different libraries for structured LLM outputs, all at the highest level of abstraction. In contrast, there is no widely used alternative to Nvidia GPUs and CUDA for the hardware and acceleration levels.

Too much abstraction?

There is such a thing as too many layers of abstractions. Hamel Husain put it well in his article: “Fuck You, Show Me The Prompt”. Make sure you know which tokens are actually being sent to the LLM, and whether there’s more than one round-trip involved in getting a response. For education, too many layers can also hinder understanding. Andrej Karpathy is known for re-implementing the GPT architecture for education, for example nanoGPT, which is GPT2 in ~600 lines of Python.

Fine-tune, don’t train from scratch

Training LLMs from scratch is almost never worth it for organizations whose main business isn’t providing foundation models for others. It requires far too much training data and GPU hours. As an example, even the smallest of Meta’s Llama 3.1 models was trained for 1.46M GPU hours (source). In contrast, fine-tuning a LoRA adapter for that model can be done in less than 1 GPU hour on an H100.

When working with lower-level libraries like PyTorch, it’s therefore necessary to start by copying the architecture of an existing LLM and loading its weights. Tweaks like a new output layer must be done carefully in order to preserve the usefulness of the learned weights. This is in contrast to less compute-intensive machine learning models, where training one’s own model from scratch is common. For these reasons, starting from a high-level framework like Hugging Face Transformers is more common for working with LLMs.

High level of abstraction

Choose a higher level of abstraction if you:

• Are a beginner seeking quick first successes
• Work at a startup focused on product-market fit
• Are a researcher in a different field wishing to use LLMs
• Want to integrate LLMs without deep ML expertise
• Are already committed to a specific cloud ecosystem
• Have no need for deep customization of models (you’d know if you did)

The danger of choosing a too high level of abstraction is that you may hit a wall when you need to do something the tool doesn’t support. For example, OpenAI’s API doesn’t support reinforcement learning from human feedback (RLHF), only supervised fine-tuning. If you need RLHF, you’d have to switch to a lower level of abstraction.

Low level of abstraction

Opt for a lower level of abstraction if you:

• Are a researcher or engineer pushing LLM boundaries
• Require fine-grained control over the model
• Need on-premises or on-device deployment
• Desire a deep understanding of the underlying technology
• Prioritize code and model portability
• Have engineers familiar with distributed systems and GPU programming

The danger of choosing a too low level of abstraction is that you may spend too much time on infrastructure and not enough on the actual problem you’re trying to solve. For example, if you’re building a prototype for a meeting summarization chatbot, your time is better spent talking to project managers than optimizing your distributed training setup.

Cost can go both ways

High level tools can add a tax, but prices have been decreasing quickly. Managed services and API providers can leverage economies of scale and have highly optimized infrastructure. This can be difficult to achieve with a DIY stack. For example, a privately owned GPU deployed for inference may be underutilized outside of business hours, while a GPU at a cloud provider services other customers.

Keep your training data portable

The linear progression from low to high abstraction is a simplification. As the ecosystem matures, interoperability increases. For example, Hugging Face Transformers abstracts away the model architecture, but you can still access the PyTorch model and adjust it. Then that model can be deployed to AWS SageMaker. Not all combinations are possible though - for example a GPT model fine-tuned on OpenAI’s API can only run on that account. When it’s cheap to do so, use solutions that have as little lock-in as possible. Especially your training data should remain portable. In a time where research labs one-up each other weekly with better base models, being able to switch to a new model quickly is an advantage.