AgentOpt guide

AgentOpt checks an AI agent before you ship it. Point it at your agent's workflow and it answers three questions:

• Checks — does the structure contain loops that may never stop, steps nothing can reach, or runaway nesting?
• Suggested simulations — is there a cheaper configuration, priced against versioned rates (never a predicted success rate)?
• Audit a document — does a written process document match best practice?

The first two live together in the app's SimRun tab — one workflow upload, two sections (a structural fact-check, then disclaimed cost hypotheticals), so you never upload the same file twice to ask both questions.

It also reports on a Claude Code session you've already run, so you can see what a coding agent actually cost in turns, tool calls, and tokens.

Everything runs on your agent's structure and declared resource counts. AgentOpt never calls your tools, your model, or your users.

Start here (10 minutes, nothing to install)

1. Go to yisnotx.com and sign in with Google. That's the whole setup — no key to paste. New accounts are approved automatically up to a tester cap; if you see a waitlist message, an admin approves you shortly.
2. Click through the demo buttons:
   • SimRun → load the example agent (it has three bugs planted on purpose) → Run checks for the structural findings, then Run simulation for cost (allow 10-30 seconds)
   • Audit a document → load the example document (precomputed, needs no AI key)
3. Repeat any tab with your own agent. Download the blank workflow template from the sidebar and fill it in.
4. Tell us what happened in the Feedback tab — both the did-it-work form and the free-text "this broke / this confused me" box.

That last step is the point of the test phase. The structural checks and cost recommendations get more accurate as real results come back.

Describing your agent

AgentOpt reads a small JSON file describing your agent's steps, the edges between them, and any loops.

On LangGraph — export it directly:

import json
from agentopt.adapters.langgraph import from_dict

graph_export = app.get_graph().to_json()   # your compiled LangGraph app
apg = from_dict(graph_export, source="my_agent")
apg_json = json.loads(apg.model_dump_json())

Start and end nodes, tool nodes, and agent nodes are mapped automatically. Refine afterwards by annotating loops with max_iterations or nodes with resources.

On n8n — export the workflow (Settings → Download, or the clipboard export) and upload the JSON file directly; /analyze and /simulate detect the format automatically (n8n-nodes-base.*/@n8n/n8n-nodes-langchain.* node types, a connections object keyed by node name). Sticky notes are dropped; trigger nodes become the start node; AI Agent tool/model/memory bindings are mapped to a delegation edge rather than a sequential step.

On Flowise or Langflow — export the chatflow and upload the JSON directly; same auto-detection (a flat nodes/edges list, each node's data.category — "Multi Agents", "Tools", "Chat Models", "Memory" — drives the node-type mapping).

On LangSmith Fleet — download the agent's config export (a small zip: config.json, tools.json, and AGENTS.md) and upload the zip directly to /upload/apg; it's auto-detected from config.json's metadata (not a single JSON blob like the exports above). A Fleet agent is one system prompt plus a tool belt plus triggers, not a node graph, so it maps to a single-loop pattern (a turn-loop agent node, one tool node per declared tool, wired back into the loop) rather than a general workflow structure. Cron schedules and trigger config carry no resource-quantity information and are attached as descriptive metadata only — they never become a simulated loop cap.

On anything else (CrewAI, AutoGen, a hand-rolled loop) — write it by hand:

{
  "id": "my_crew",
  "name": "My CrewAI pipeline",
  "nodes": [
    {"id": "__start__", "type": "start", "label": "Start"},
    {"id": "researcher", "type": "agent", "label": "Research agent"},
    {"id": "search", "type": "tool", "label": "Web search tool",
     "resources": {"tool_calls": 1}},
    {"id": "writer", "type": "agent", "label": "Writer agent"},
    {"id": "__end__", "type": "end", "label": "End"}
  ],
  "edges": [
    {"source": "__start__", "target": "researcher", "type": "sequential"},
    {"source": "researcher", "target": "search", "type": "sequential"},
    {"source": "search", "target": "writer", "type": "sequential"},
    {"source": "writer", "target": "__end__", "type": "sequential"}
  ],
  "loops": []
}

A hosted tool like Tavily is just a tool node with a declared tool_calls count. AgentOpt reasons about the count; it never calls the tool.

Report on a Claude Code session

If you use Claude Code (or the Claude Agent SDK), it can stream what it's doing — turns, tool calls, tokens, cost — over OpenTelemetry while it runs. AgentOpt ships a one-command recorder that catches that stream locally. No collector to install, no config files, no account, nothing uploaded.

What you get: a per-session breakdown of turns, session duration, tool calls (with a per-tool breakdown — also where a subagent count comes from), tokens, cost, how often tools succeeded, the context-growth rate across turns, and the prompt-cache hit rate. What you don't get: recommendations. Claude Code doesn't report whether the task actually succeeded, and without that signal the recommendation and cost-simulation logic has nothing to optimize against. This is a measurement, not advice.

Before you run it — what gets written to disk:

• The recorder listens on localhost only and writes to a local file. Nothing leaves your machine.
• Scrubbing is on by default. Prompt text, code snippets, and file paths are replaced with [scrubbed]. Kept: event names, session ids, model names, tool names, token counts, costs, timings — the numbers the report needs.
• --no-scrub keeps everything including prompt content. Local debugging only. Never share a --no-scrub file.

Terminal 1 — download and start the recorder:

curl -O https://api.yisnotx.com/download/recorder.py
python3 recorder.py
# prints the output path, e.g. ~/.agentopt/capture/claude-telemetry-<stamp>.jsonl

One file, standard library only — no pip install, no virtualenv, no account, and it needs Python 3.9 or newer, which macOS and every Linux distribution already have. Open it before you run it if you like; the scrubbing rules are near the top and the whole thing is about 250 lines.

By default every session you run gets appended to the same rolling .jsonl file — that's deliberate, since comparing sessions is more useful than isolating them (see the table below). Two flags change that:

• --per-session starts a new file the moment it sees a new session.id, named claude-<session>-<stamp>.jsonl, and prints ! new session detected -> writing ... each time it switches. Use this if you want a one-session-per-file layout instead (e.g. to hand a single session's file to someone without the rest of your history).
• --gzip writes .jsonl.gz instead of plain .jsonl — combine with --per-session for claude-<session>-<stamp>.jsonl.gz. Useful once a capture file gets large.

python3 recorder.py --per-session --gzip

Terminal 2 — turn on telemetry and use Claude Code normally:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_LOGS_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# optional: flush faster so short sessions are captured in full
export OTEL_LOGS_EXPORT_INTERVAL=5000
export OTEL_METRIC_EXPORT_INTERVAL=10000

claude

These variables only affect the terminal you set them in. Other Claude Code sessions are untouched, and a new terminal turns telemetry back off.

Work as usual, exit Claude Code, wait about 10 seconds for the final flush, then Ctrl-C the recorder. It prints one line per batch received, so you can watch it working.

Then read the session. Either drop the file into the Claude Code session tab in the app, or post its contents to the API:

jq -Rs '{capture_jsonl: .}' ~/.agentopt/capture/claude-telemetry-<stamp>.jsonl \
  | curl -X POST https://api.yisnotx.com/cc_report/analyze \
      -H "X-API-Key: $AGENTOPT_API_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-

You get, per session: turns, duration, tool calls (with a per-tool breakdown), tokens, estimated cost, tool success rate, context-growth rate across turns, and prompt-cache hit rate.

Note the split: recording is local and needs nothing from us; reading the result is what the API does. If you only want to confirm the recorder is capturing something, the capture file is plain JSONL — wc -l it.

If you recorded with --gzip, send capture_jsonl_gzip_b64 (base64 of the .jsonl.gz bytes) instead of capture_jsonl — the app's file uploader accepts .jsonl.gz directly and does this for you:

base64 < ~/.agentopt/capture/claude-<session>-<stamp>.jsonl.gz \
  | jq -Rs '{capture_jsonl_gzip_b64: .}' \
  | curl -X POST https://api.yisnotx.com/cc_report/analyze \
      -H "X-API-Key: $AGENTOPT_API_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-

About the cost number

The dollar figure is Claude Code's own self-reported number, priced at standard metered API list rates for whichever model handled each turn. It does not reflect negotiated/enterprise discounts, Claude Team/Enterprise seat billing, or Claude Pro/Max subscription usage (which draws down a plan quota, not dollars) — treat this as a metered-API-equivalent estimate, not confirmation of what was actually billed.

(This is the same text every surface uses — the API returns it verbatim in each response's cost_disclaimer field, sourced from agentopt.adapters.claude_code.COST_DISCLAIMER, so it can't drift out of sync here.)

How this differs from Claude Code's built-in /cost

Claude Code already has a /cost command. It gives you a running total for the session you are sitting in, printed in the terminal, and it is gone when the session ends. The numbers are the same numbers — AgentOpt does not recompute pricing, it reads what Claude Code itself reported over telemetry. What this path adds is everything around them:

Use /cost to see what the session in front of you is costing right now. Use this when you want that as a durable, shareable, machine-readable artifact — to compare sessions over time, to watch tool success rate as your CLAUDE.md and hooks change, or to post a usage report on every PR that ran a Claude-Code-powered step.

Where the subagent difference actually bites. If your session used the Task tool, a naive read of the telemetry double-counts: the subagent's own turns look like main loop turns and its tool calls look like yours. AgentOpt brackets each subagent run (tool_decision(Agent) → subagent_completed → tool_result(Agent)) and collapses it to one tool call, taking tokens and duration from the session's own reported totals. Turn counts stay comparable between sessions that used subagents heavily and ones that didn't.

In CI

A published Action posts a capture file to AgentOpt and writes the report to the run's Job Summary. It's a thin HTTP client — no AgentOpt source in the image.

You need an API key: sign in at https://yisnotx.com, generate a named key from the sidebar's API keys panel, and add it as a repo secret named AGENTOPT_API_KEY.

name: Claude Code usage report
on:
  workflow_dispatch:
    inputs:
      capture_file:
        description: "path to the captured .jsonl (relative to repo root)"
        required: true

jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker://ghcr.io/mvingian/agentopt-cc-report:latest
        env:
          CAPTURE_FILE: ${{ inputs.capture_file }}
          AGENTOPT_API_KEY: ${{ secrets.AGENTOPT_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Set COMMENT_ON_PR: "true" (plus permissions: pull-requests: write) to post the report as a PR comment on pull_request runs. The comment is posted with your own GITHUB_TOKEN; no AgentOpt credential is involved in that part.

You can also point this at a Claude-Code-powered CI step — start the recorder as an earlier step in the same job, let the Claude Code step run, then report — so an automation's resource usage isn't a black box.

A missing capture file, missing key, or network error produces a warning and skips the report. It never fails your build over a reporting problem.

Calling the API from your own code

Get a key

Sign in at https://yisnotx.com and generate a key from the sidebar's API keys panel. Keys are named — one per script or CI job — so you can revoke a single one later without disrupting anything else. It's shown once; copy it then. Send it as an X-API-Key header.

One call, one answer

/review/inline runs the structural check and the cost check together and replies with a single plain-language verdict. Start here to confirm your key works. Allow 10-30 seconds for the simulation step:

curl -s --max-time 60 -X POST https://api.yisnotx.com/review/inline \
  -H "X-API-Key: YOUR_AGENTOPT_KEY" \
  -H "Content-Type: application/json" \
  -d "$(python3 -c "
import json
apg = json.load(open('examples/buggy_agent/apg.json'))
print(json.dumps({'apg_json': apg}))
")"

Against the example agent you should get "verdict": "fix_before_deploy" and a summary naming the planted bugs. Possible verdicts:

401 means the key is missing or wrong. 429 means you hit the rate limit; wait a minute.

Structural check only (fast enough for a CI gate)

Returns in under a second and never calls a model.

import requests

resp = requests.post(
    "https://api.yisnotx.com/analyze/inline",
    headers={"X-API-Key": "YOUR_AGENTOPT_KEY"},
    json={"apg_json": apg_json, "format": "json"},
    timeout=15,
)
critical = [f for f in resp.json()["findings"] if f["severity"] == "critical"]
if critical:
    raise SystemExit(f"AgentOpt found {len(critical)} critical issue(s): {critical}")

Non-zero exit on critical findings turns this into a pre-merge or pre-deploy gate in any CI system.

Cost check

Takes 10-30 seconds — it builds and samples a causal model per call, and results are not cached. Set your client timeout accordingly.

AgentOpt never predicts a success rate — you are the success model. /simulate reports cost arithmetic only: what a cheaper configuration would save, in dollars and percent. Optionally supply success_value_usd (what a successful run is worth to you minus a failed one) to also get breakeven_success_drop_pct — how many points of success the saving could afford to lose and still be worth it. That's arithmetic on the number you supplied, not a prediction about your agent.

resp = requests.post(
    "https://api.yisnotx.com/simulate/inline",
    headers={"X-API-Key": "YOUR_AGENTOPT_KEY"},
    json={"apg_json": apg_json, "model_tier": "mid", "format": "json"},
    timeout=45,
)
sim = resp.json()
min_cost = next(r for r in sim["optimization"] if r["query"] == "min_cost")
print(min_cost["optimal_config"], min_cost["cost_usd_p50"], min_cost["cost_saving_pct"])

optimal_config, cost_usd_p50, cost_saving_usd_per_run, cost_saving_pct, and breakeven_success_drop_pct (None unless you supplied success_value_usd) are flat sibling fields on each entry. optimal_config itself holds only the knob values.

Check cost_basis.quantity_basis ("declared" / "trace_calibrated" / "default") before trusting a dollar figure: if it's "default", every cost field in the response is null — the resource quantities behind it were fabricated fallbacks, not your agent's real numbers. Annotate resources in your workflow file, or upload a trace bundle — a raw OTel/OTLP export, a LangSmith run export, a Langfuse trace/observation export, a Jaeger trace export, or a .zip of any mix of these (POST /upload/traces, then pass the returned traces_path) — to unlock real figures.

Report what actually happened

The cost figures above are exact arithmetic — there's nothing to calibrate there. What /calibrate_feedback is still for: outcome labels to fit the scenario weight vectors behind depends_on_unverified_assumption (per-agent, via /refit, opt-in, ~30 labels).

requests.post(
    "https://api.yisnotx.com/calibrate_feedback",
    headers={"X-API-Key": "YOUR_AGENTOPT_KEY"},
    json={
        "run_id": "run_001",
        "apg_id": "my_crew",
        "predicted_config": min_cost["optimal_config"],
        "actual_success": 1.0,   # your measurement — AgentOpt never judges this itself
    },
)

predicted_delta_success is optional and omitted here on purpose — post-0.4.0 there's usually no success prediction left to compare against; pass it only if you're still verifying a prediction from before this reframe. run_id is a free-text label. There's no delete endpoint and the log is append-only, so avoid submitting placeholder values repeatedly.

Everything you can call

/refit also exists. It fits the internal success model to one agent, from outcome labels you supply — roughly 30 of your own labelled runs is enough for a usable fit, and nobody else's data is involved. That model is never shown to you directly as a success rate; it's what powers the depends_on_unverified_assumption scenario check on /simulate. /refit is a different thing from /calibrate_feedback, which reports outcomes into a shared log the maintainer uses to improve the starting assumptions for everyone; that one retrains nothing of yours.

Full reference with request and response schemas: https://api.yisnotx.com/docs

Without writing code (n8n, Zapier, Make)

Any tool that can send an HTTP request works: POST the URL with an X-API-Key header and the JSON body shown above. In n8n, two editor quirks will cost you an hour if you don't know them:

• Build HTTP nodes with Import cURL, not by hand. Pasting large JSON into a Set node's value box silently mangles it — dropped braces and colons, truncation mid-word — with no error until a later node fails.
• Use "Send Body → Using JSON", not "Using Fields Below". The per-row editor stringifies every value on the wire regardless of its preview, so numbers arrive quoted and objects arrive as [object Object], which the API rejects. Wrap object-valued expressions in JSON.stringify(...).

A third gotcha, for IF nodes: force the condition's type selector to Boolean. Left on the default, a non-empty result can coerce to true regardless of the actual value and misroute a clean graph into the failure branch. A number comparison — {{ $json.findings.filter(f => f.severity === "critical").length }} not equal to 0 — sidesteps this entirely.

Set the cost-check node's timeout to 45000 ms; the default HTTP timeout will cut it off. And note the api. subdomain: yisnotx.com alone is the app and returns 405 on POST.

The AgentOpt nodes don't consume anything from a surrounding workflow — the agent JSON is baked into the request — so you can attach them anywhere, in any order.

LangSmith: wrap the structural check in a function registered as a custom evaluator and return the findings as the evaluator's score and comment.

Auditing a document

The audit tab reads your process document (a PDF) and your agent's workflow file, and checks the process against a fixed general best-practice library (18 practices, always run). Optionally also supply your own standards document (ISO 9001, AS9100, an internal SOP, a regulation) to additionally check against the literal text of that document. It's the only part of AgentOpt that uses an AI model, and it uses your model, not ours — once per document supplied.

Pick a provider in the sidebar (OpenAI, Groq, Together, Mistral, local Ollama, or a custom OpenAI-compatible endpoint) and paste your own key. The key travels with that one request, goes straight from our server to the provider you chose, and is never stored.

Your agent's workflow file is required — there is no audit without an actual agent to audit against, and no partial or offline mode. A standards document is optional: without one, the fixed library alone still runs. Every finding checked against a standards document cites the literal clause it came from; AgentOpt never invents a judgment the text doesn't make, and never claims to certify against a third-party standard by name — only that your process conforms to what's quotable in the document you supplied.

Two ways to try it without a key: the cached demo document pair, or Groq's free tier (console.groq.com — the Groq preset is pre-wired).

Findings come back in one of two tiers. Verified requires that you also supplied your own standards document and no critical practices failed; everything else is a recommendation and says so. AgentOpt never issues a verified result without the evidence to back it.

Data handling

• Uploaded files land in a temporary directory on the API server, are readable only by that service, and disappear on restart or redeploy. Re-upload rather than reusing an old server path.
• Audit PDFs are additionally sent to whichever AI provider you configured — your key, your provider, your account's terms.
• Please don't upload confidential documents during the test phase.
• Your LLM provider key is never stored.
• Claude Code captures never leave your machine unless you explicitly send one to the CI endpoint.
• Usage is logged (timestamp, endpoint, status, duration, account) to understand how the tool is used during the test phase.

Full detail: https://api.yisnotx.com/privacy

Known limits during the test phase

• The cost check takes 10-30 seconds and isn't load-tested at scale. Concurrent requests beyond a small number get a clean 429 rather than a pile of timeouts.
• Uploaded files are ephemeral, as above.
• Feedback submissions are append-only with no de-duplication.
• A waitlisted account finds out it's been approved by signing back in — there's no email notification yet.

Getting help

Everything in this guide works against the hosted service — there's nothing to install except, optionally, the local Claude Code recorder. If something here doesn't behave as described, or you want a capability that isn't covered, say so in the app's Feedback tab; it goes straight to the maintainer along with enough context to reproduce it.