AgentOpt

AgentOpt is a pre-deployment checker for AI agents. You give it a small JSON description of your agent's workflow — its steps, how they connect, and where it loops — and it reports structural problems, simulates cheaper configurations, and audits process documents against a best-practice library. It also reads OpenTelemetry captures from a Claude Code session and reports what that session actually used.

Two surfaces, one system: a web app at yisnotx.com (sign in with Google) and a REST API at https://api.yisnotx.com (send a key in an X-API-Key header). The API is also available as an MCP server with seven tools. Everything the app does is an ordinary HTTP POST, so it works as a step in CI, in n8n/Zapier/Make, as a LangSmith evaluator, or as a tool inside another agent.

What it does

Find structural problems in an agent workflow

POST /analyze/inline. Static checks over the workflow graph: retry loops with no iteration cap, unreachable steps, cycles, uncapped parallel fan-out, missing contracts between steps. No AI model is involved and no data leaves the request — it runs on your JSON in under a second, which makes it usable as a CI gate that fails a build on critical findings. Findings are tagged with the MAST failure taxonomy where they map to it.

Find a cheaper configuration

POST /simulate/inline. Varies tool-call count, loop cap, and context size, and reports the cheapest configuration's cost saving in dollars and percent — arithmetic on versioned per-token and per-call pricing (tiers.yaml), including context snowballing and prompt-cache discounts. Takes 10–30 seconds because the causal model is built and sampled per request.

We never compute a success probability for you — you are the success model. There is no success-rate output to read. If you tell us what a successful run is worth to you versus a failed one (success_value_usd, optional), the response adds a break-even figure: how many points of success the cheaper configuration could afford to lose and still be worth it. You decide whether that's a risk you'll take; we only ever report the arithmetic. Every response also says whether its cost figures are declared, trace-calibrated, or (if neither) withheld entirely — see cost_basis.quantity_basis — since a fabricated resource quantity would make a dollar figure meaningless. Upload a trace bundle (POST /upload/traces) to unlock real numbers if your workflow file doesn't declare enough resource detail on its own. Accepted trace formats: a raw OTel/OTLP export, a LangSmith run export, a Langfuse trace/observation export, or a .zip of any mix of these. Workflow files can be hand-written, or exported directly from LangGraph, n8n, or Flowise/Langflow — see the guide for the field-level detail per format.

Compare two runs

POST /diff/traces. Upload two trace bundles — a baseline and a candidate, in any of the formats above — and get the arithmetic difference in tool calls, tokens, iterations, and cost per run, with no workflow file or simulation involved. Useful for checking whether a change you made actually helped, using your own eval runs as the evidence. Claude Code sessions have their own equivalent, POST /cc_report/diff_sessions, rather than going through this endpoint. The response includes a reusable fingerprint for the baseline so you can diff several candidates against it without re-uploading the same bundle each time.

Download an experiment pack

POST /experiment_pack. Takes one suggested simulation from a /simulate response and turns it into a downloadable config patch plus a short experiment card explaining what changed and why, with instructions for verifying it against your own eval set via /diff/traces. It formats an existing recommendation — it does not run a new simulation.

Audit a process document

POST /audit. Reads a PDF describing a workflow (a runbook, SOP, or design doc) and checks it against a fixed general best-practice library (18 practices, always run) and, optionally, your own standards document (standards_paths), returning a scored report where every finding against your document cites the literal clause it came from. apg_path (your agent's workflow file) is required — there is no audit without an actual agent to audit against. This endpoint calls an AI model — once per document supplied — and it uses your provider credentials — OpenAI, Groq, Together, Mistral, or a self-hosted Ollama — passed per request. There is no offline mode.

Certification is evidence-gated: without a workflow file, without a standards document, or with any critical practice failing, the report is explicitly a recommendation tier with a disclaimer, never a certificate.

Report on a Claude Code session

POST /cc_report/analyze, the Claude Code session tab in the app, or a published GitHub Action. See the section below.

Analyzing a Claude Code session

Claude Code can stream what it is doing — turns, tool calls, tokens, cost — over OpenTelemetry. AgentOpt ships a one-command local recorder that catches that stream, and then reads the resulting file into a per-session report.

1. Record. In one terminal:

   curl -O https://api.yisnotx.com/download/recorder.py
   python3 recorder.py
   

   One file, Python standard library only — no install, no virtualenv, no account, and it needs nothing newer than Python 3.9. It listens on 127.0.0.1:4318 and writes to ~/.agentopt/capture/. Nothing is uploaded, and prompt text, code, and file paths are replaced with [scrubbed] before anything is written to disk. The file is deliberately readable: you are being asked to run a listener over your own telemetry, so the scrubbing rules should be checkable rather than taken on trust.

   By default every session appends to one rolling file, which is what makes comparing sessions over time possible. Add --per-session to start a fresh file per session.id instead (announced in the terminal as it switches), and --gzip to write .jsonl.gz instead of plain .jsonl once a capture gets large.

2. Work. In a second terminal, set CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_LOGS_EXPORTER=otlp, OTEL_METRICS_EXPORTER=otlp, OTEL_EXPORTER_OTLP_PROTOCOL=http/json, and OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318, then run claude as usual. The variables affect only that terminal.

3. Read it. Upload the capture file in the app's Claude Code session tab, or POST its contents to /cc_report/analyze. A .jsonl.gz capture is accepted directly — no need to decompress it first.

Recording is entirely local and needs nothing from AgentOpt; only step 3 involves the service at all.

What you get: per session — turns, session duration, tool calls (with a per-tool breakdown, which is also where a subagent count comes from), tokens, estimated cost, the share of tool calls that succeeded, the context-growth rate across turns, and the prompt-cache hit rate. Subagents spawned by the Task tool are attributed correctly: each subagent run collapses to a single tool call whose tokens and duration come from the session's own subagent_completed totals, so its internal tool calls do not inflate the main loop's turn count.

A capture file often holds more than one session. The app previews a rolling file's sessions locally with no network call, lets you choose which to include, and only then calls POST /cc_report/sessions to list every selected session (turns, tool calls, cost, tokens, approximate start time) or POST /cc_report/diff_sessions to compare the cost of two sessions directly — same response shape as /diff/traces above.

What you do not get: recommendations. Claude Code does not report whether your task actually succeeded, and without that signal there is nothing for the cost simulation to optimize against. This path is a measurement, not advice.

The cost figure is not your bill. Cost figures come from Claude Code's own telemetry, priced at standard metered API list rates for the model used on each turn. They do NOT reflect negotiated/enterprise discounts, Claude Team/Enterprise seat billing, or Claude Pro/Max subscription usage (which draws down a plan quota, not dollars) — treat this as a metered-API-equivalent estimate, not confirmation of what was actually billed.

How this differs from what Claude Code already shows you

Claude Code's own /cost gives you a running total for the session you are sitting in, in the terminal, and it goes away when the session does. The numbers themselves are the same numbers — AgentOpt does not recompute pricing, it reads what Claude Code reported. What the AgentOpt path adds is everything around them:

So: use /cost to see what the session in front of you is costing. Use AgentOpt when you want that as a durable, shareable, machine-readable artifact — to compare sessions, to watch tool success rate over time, or to post a usage report on every pull request that ran a Claude-Code-powered step.

What AgentOpt deliberately does not do

• It does not judge output quality. Every metric is a count, a rate, a cost, or conformance arithmetic. Anything requiring a judgment call about whether an answer was good is emitted as a suggestion for the caller to compute, never computed here.
• It does not watch your agent in production. It is a pre-deployment and post-hoc analysis tool, not a runtime monitor.
• It does not need your agent's source code. It works from a workflow description, an OpenTelemetry trace bundle, or a PDF.
• It does not store your prompts. The Claude Code recorder scrubs before writing; the API keeps uploads only for the life of the request.

Getting started

• Guide — the full walkthrough: describing your agent, the first API call, the Claude Code recipe, CI integration, and the current limits.
• API reference — every endpoint, with schemas.
• Privacy — what is stored and what is not.
• Open the app — sign in with Google, then generate a named API key from the Keys panel for scripts and CI.