Building an agent loop: tool calls, retries, failure modes

Building an agent that can call tools and handle failures is harder than the demos suggest. The loop that orchestrates the LLM's decisions, tool executions, and retries is where most production agents fall apart. Here’s how to build one that survives real-world conditions.

The shape of an agent loop

Every agent loop follows the same skeleton: a user message enters, the LLM decides whether to respond directly or invoke a tool. If it calls a tool, you execute that tool, feed the result back into the conversation, and repeat until the LLM produces a final response. Two exit conditions exist: the model generates a content message (not a tool call), or you hit a maximum iteration limit.

Here’s a minimal TypeScript while-loop that captures the pattern:

type Message = { role: 'user' | 'assistant' | 'tool'; content: string; tool_call_id?: string };
 
async function agentLoop(
  initialMessages: Message[],
  tools: Record<string, (args: any) => Promise<string>>,
  maxIterations = 10
): Promise<Message> {
  const messages = [...initialMessages];
  let iterations = 0;
 
  while (iterations < maxIterations) {
    const response = await llmChat(messages, Object.keys(tools)); // provider-specific call
    const choice = response.choices[0];
    const msg = choice.message;
 
    if (!msg.tool_calls || msg.tool_calls.length === 0) {
      return msg; // final response
    }
 
    for (const toolCall of msg.tool_calls) {
      const toolName = toolCall.function.name;
      const args = JSON.parse(toolCall.function.arguments);
      const result = await tools[toolName](args);
      messages.push({
        role: 'assistant',
        content: null,
        tool_call_id: toolCall.id,
      });
      messages.push({
        role: 'tool',
        content: result,
        tool_call_id: toolCall.id,
      });
    }
    iterations++;
  }
 
  // If we exit because of max iterations, return a fallback or error
  throw new Error('Agent loop exceeded maximum iterations');
}

The loop itself is simple; the complexity lives in the tool schema, retry logic, and failure handling around it.

Tool call schema: mapping JSON schema to provider definitions

Every LLM provider expects tool definitions in a specific format. The common currency is JSON Schema for parameters, but the envelope differs. For OpenAI you use the functions or tools parameter; for Anthropic it’s tools with a input_schema field; for open-source models via Ollama or vLLM the shape varies further.

A concrete get_weather tool with parameters looks like this in JSON Schema:

{
  "name": "get_weather",
  "description": "Get current weather for a location",
  "parameters": {
    "type": "object",
    "properties": {
      "latitude": { "type": "number" },
      "longitude": { "type": "number" },
      "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
    },
    "required": ["latitude", "longitude"]
  }
}

When you send that to OpenAI, you wrap it in a tools array:

const tools: ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get current weather for a location',
      parameters: { /* JSON Schema */ },
    },
  },
];

Anthropic uses a different envelope:

const tools: Tool[] = [
  {
    name: 'get_weather',
    description: 'Get current weather for a location',
    input_schema: {
      type: 'object',
      properties: { /* ... */ },
      required: ['latitude', 'longitude'],
    },
  },
];

Open-source models served through a compatible API (like vLLM with OpenAI-compatible endpoints) often accept the OpenAI shape. The key is to write a small adapter that converts your internal tool registry into the provider’s format. This is essential when you want to switch providers without rewriting every tool.

For more on the JSON Schema side, see Reliable Structured Outputs from LLMs Using JSON Schema.

Retry strategies for tool calls

Tool calls can fail for three common reasons: network errors (timeout, DNS), rate limits (HTTP 429), or malformed tool outputs (the tool itself returns an error). Each warrants a different retry strategy.

Exponential backoff with jitter is the baseline. Implement it as:

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 3,
  baseDelayMs = 1000
): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt === maxRetries) throw err;
      const delay = baseDelayMs * Math.pow(2, attempt) + Math.random() * 1000;
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw new Error('Unreachable');
}

For rate limits specifically, parse the Retry-After header if present and use that instead of the computed delay. OpenAI’s API returns a Retry-After value in seconds on 429 responses; Anthropic uses a similar pattern.

A more nuanced approach uses dynamic retry limits based on remaining context window. If you have only 200 tokens left before hitting the model’s max tokens, retrying a long tool call might be pointless. Track token usage per iteration and clamp retries accordingly.

The failure modes nobody mentions

Documentation covers happy paths. Real agents hit these four failure modes repeatedly:

Infinite tool call loops. The LLM calls a tool, you return a result, and it calls the same tool again with identical arguments. This is surprisingly common when the tool output doesn’t satisfy the LLM’s implicit criteria. Detection requires tracking tool call hashes: if the same tool name and arguments appear more than twice consecutively, break the loop and escalate. Another tactic: limit consecutive tool calls of the same name to a small number (e.g., 3).

Stale context. Tool results can be long. When you append them, the prompt may exceed the model’s context window. Naive truncation (dropping earlier messages) can lose the original user request. Instead, summarise or compress older tool results. Some providers support context caching—see Prompt caching: cache keys, misses, and getting a useful hit rate for how to use it effectively.

Tool output validation errors. The LLM may return invalid JSON, missing required fields, or values outside allowed enums. Catch this before passing to the tool. If the JSON is malformed, re-prompt the LLM with the message: “The JSON you provided was invalid. Please retry with valid JSON.” This often works because the model can correct itself after seeing the error.

State corruption from concurrent tool calls. Some agents fire multiple tool calls in a single LLM response (parallel tool calling). If tools mutate shared state (e.g., a database), concurrent execution can cause race conditions. You have two options: execute tools sequentially (safe but slower) or use a transaction-like mechanism. For most agents, sequential execution is simpler and avoids subtle bugs.

Handling tool output errors gracefully

When a tool itself fails (e.g., API returns 500, database query times out), you have three choices:

1. Retry the same tool with the error message as feedback. This works if the error is transient. Downside: latency increases.
2. Fall back to a default value (e.g., return {"error": "service unavailable"}). The LLM can then work around the missing data. Downside: the LLM may treat the fallback as real data and produce incorrect results.
3. Skip the tool entirely and let the LLM proceed without the information. This often leads to the LLM getting stuck or hallucinating a substitute.

A practical suggestion: log the error and re-prompt the LLM with the error description, then let it decide. For example:

The tool `get_weather` returned an error: "HTTP 503 Service Unavailable". 
You may try again, use a different tool, or respond with what you know.

This gives the model agency while being transparent about the failure. It works well in practice because LLMs are good at adapting to partial information.

Framework vs custom loop: a comparison

When to use a framework: you need complex multi-tool agents, built-in retry policies, or streaming support without reinventing wheels. LangChain provides a ToolExecutor and retry middleware; Vercel AI SDK gives you a clean streamText with tool call handling.

When to roll your own: you have a simple agent (2–5 tools), need full control over error handling, or want to avoid framework bloat. The hand-rolled loop above is ~30 lines; adding retries and logging adds another 50. That’s often less code than understanding LangChain’s AgentExecutor internals.

Observability and debugging the agent loop

Without visibility, an agent loop is a black box. Log every step:

• Each tool call request (tool name, arguments, timestamp)
• Each tool call response (status, duration, output size)
• Token usage before and after the call
• Iteration count and cumulative latency

Use a correlation ID that flows from the initial user message through all iterations. Structured logging (JSON lines) makes it easy to replay a session. For example:

{"correlation_id":"abc123","iteration":2,"event":"tool_call","tool":"get_weather","args":{"lat":51.5,"lon":-0.12},"duration_ms":340}
{"correlation_id":"abc123","iteration":2,"event":"tool_response","tool":"get_weather","status":"success","tokens_used":45}

For distributed tracing, use OpenTelemetry to instrument the LLM call and tool execution. Export traces to Jaeger or a cloud backend. A simple dashboard showing average iterations per conversation, tool failure rates, and token consumption can catch regressions before they reach users.

The OpenAI API documentation on function calling and Anthropic’s tool use are the definitive references for provider-specific details. For retry backoff patterns, the AWS documentation on exponential backoff and jitter remains the canonical guide.

Key takeaways

• An agent loop is a simple while-loop that feeds tool results back to the LLM; the hard parts are detecting infinite loops, managing context windows, and handling tool output errors.
• Convert an internal tool registry to provider-specific schemas using a small adapter; don’t couple your tools to one provider’s format.
• Retry tool calls with exponential backoff and jitter, and consider dynamic limits based on remaining context.
• Log every iteration with correlation IDs and structured output; observability is the difference between a debugged agent and a mystery.