← All writing

Agentic AI · FIX protocol · Incident response

The on-call engineer who never sleeps: building an AI agent to triage FIX trading incidents

How I replaced the first forty minutes of every FIX incident with an agent that investigates, cites its evidence and hands a human a diagnosis.

It's 10:14 on a Tuesday and an alert fires: reject rate high on session ACME_FIX44. Anyone who has run a FIX-based trading platform knows what happens next, and knows that the hard part is not the fix. It's finding it.

Someone on call drops what they're doing. They open the log search and filter by session. They scroll through rejects, copy a tag number, look it up in the FIX spec, and check whether the client reconnected recently. They query the orders database, check the message broker for consumer lag, ask in Slack whether anyone deployed something, and search the wiki for a runbook that may or may not be current. Forty minutes later they have the answer: the client started sending PutOrCall as C instead of 1 after restarting their engine. The fix takes five minutes. Finding it took the rest.

As VP of Technology at Intick, a block trading venue for futures and options, I owned the FIX connectivity into our counterparties. Incidents like this weren't rare, and they weren't hard. They were slow, because the knowledge needed to solve them was spread across six systems and a few senior people's heads. So I designed and shipped an AI agent to do the first pass. Not a chatbot that summarises logs, but an agent that investigates the way a good engineer does: it forms a hypothesis, pulls the evidence it needs and stops when the evidence is conclusive.

Two swimlanes. Today: alert, then log search, FIX spec, orders DB, broker admin, deploy history, wiki runbook, then diagnosis at about 10:55. With an agent: alert, triage agent making 5 tool calls across the same systems, report in the Slack thread in under a minute, engineer verifies and acts.
The engineer stops being the orchestrator. The agent does the running around; the human still makes the call.

Why FIX incidents are slow to triage

No single step is difficult. The problem is that every incident could live in one of four layers, and you don't know which until you look:

Each layer lives in a different system. FIX logs in one place, order state in PostgreSQL, broker health behind an admin API, runbooks in Confluence, deploy history in CI. The engineer's real job during triage is orchestration: knowing which system to check next based on what the last one said.

That orchestration is exactly what agents are good at.

What an agent actually is

Strip away the marketing and an agent is three things:

  1. A language model that reasons about what it's seeing.
  2. Tools: ordinary functions you write, such as "get rejects for this session in this window" or "look up this FIX tag".
  3. A loop: the model asks for a tool, your code runs it and returns the result, and the model decides what to do next. Repeat until it has an answer.
The model on the left, your code in the middle, your systems on the right. Your code sends the prompt, tool list and incident data to the model; the model replies with a tool_use request; your code executes it against read-only systems and returns a tool_result. This repeats until the model stops asking.
The model never touches your database. It can only ask.

The model can't execute anything. It can only request a tool call, and your code decides whether and how to answer. That one fact shapes the whole design, and it's what makes this safe enough for a trading environment. The list of tools you send with each request is, quite literally, the agent's permission boundary: if a tool isn't offered, it can't be called.

The same incident, investigated by an agent

Here is how the agent handles that incident. The client name and details are changed, but the shape is typical.

The alert arrives. Before the model is involved at all, plain code parses the alert payload and pre-fetches the obvious context: recent session events, a reject summary and broker health. The model's starting picture:

Then it investigates.

Six rows, each with a question, the tool call the model requests and the result. Baseline shows a logon with sequence reset and 37 rejects on tag 201. lookup_fix_tag(201) returns PutOrCall 0 or 1. get_session_rejects shows 201=C. get_client_dictionary shows no override. recent_deploys shows nothing. search_runbooks returns the client enum override procedure. A final report states the root cause with evidence and next steps.
Five tool calls, well under a minute, and every claim in the report points at a specific message.

The report it posts back to the Slack thread:

Root cause (high confidence): ACME began sending PutOrCall (201) as C/P instead of FIX 4.4 0/1 after reconnecting at 10:01 with a sequence reset, most likely a client-side engine upgrade. No deploys or config changes on our side.

Evidence: seq 14 at 10:03:41 rejected with reason 5 on tag 201; 37 identical rejects since.

Next steps: contact ACME to confirm the engine change; short-term per-client enum override per runbook (requires four-eyes approval).

Notice what the agent didn't do. It didn't dig into order state, business rules or the broker. The evidence pointed at the message layer, so it followed the evidence and stopped. That branching is the whole point. A hardcoded script would either check everything every time, or miss the case nobody anticipated.

The design decisions that matter

A demo like this takes an afternoon. Making it something you can trust in a regulated trading environment comes down to decisions. These are the six that mattered most.

1. Deterministic where it must be, agentic where it must branch

There's a spectrum between a rigid workflow (fixed steps, AI summarises at the end) and a fully autonomous agent (vague goal, the AI decides everything). Neither end works well, so we split it deliberately.

Three zones. Code on the left: parse alert, dedupe, pre-fetch baseline, mask sensitive data. Agent in the middle: investigate, picking the next tool based on what the last one showed. Code on the right: validate schema, render report, post to Slack, audit log.
The model is used exactly where judgement is needed, and nowhere else.

Monitoring alerts are structured JSON, so parsing them is code, not AI. So are deduplication, validation, formatting and audit logging. The model's only job is the part that genuinely branches: deciding what to look at next.

2. Tools, never raw database access

It's tempting to hand the model a SQL connection and let it write its own queries. Don't.

Every data source sits behind a narrow, named tool such as get_session_rejects(session, start, end). Behind it are parameterised queries, a read-only database role on a replica, a five-second statement timeout and hard row limits. The model chooses which tool to call and with what arguments. It never writes the query, and its arguments are validated before anything runs, because those arguments may have been influenced by text the model read along the way, such as a free-text field in a client's FIX message.

This makes the agent safer, and also better. A tool that returns "37 rejects, grouped by reason" is more useful to the model than 10,000 raw log lines, and far cheaper.

3. Mask sensitive data inside the tool, not in the prompt

In block trading, information leakage is the cardinal sin. Account IDs, legal entity identifiers, order sizes and prices must not leave your environment.

Inside your Azure environment, the log store feeds raw messages to a tool function that queries, summarises and masks. Only masked data crosses the boundary to the model. Before: account, LEI, price and quantity in clear. After: account and LEI become hashed tokens, price becomes a placeholder, quantity becomes a size band, and 201=C passes through untouched.
"Please don't repeat account numbers" in a prompt is a wish. Masking in code is a control.

Two details make this work. Identifiers are replaced with consistent HMAC tokens, so the model can still tell that every reject came from one account without knowing which account. And sizes become bands rather than disappearing, so a zero quantity or an absurdly large order is still visible. The diagnosis survives; the sensitive data doesn't. The masker gets its own unit tests against real message shapes, because a missed tag is a data leak.

4. Right-size the model

The instinct is to reach for the biggest model available. For triage that's the wrong call. The task is bounded (a handful of tool calls, pattern-matching against known failure modes) and latency matters, because during an incident people are watching the Slack thread. An alert storm that triggers twenty runs in a minute also needs to stay affordable.

The investigation loop runs on Claude Sonnet 5 at medium effort, and Claude Haiku 4.5 handles the trivial job of turning a human's Slack message ("ACME rejects since 10:14") into structured fields. When the evals show accuracy falling short, the first move is to raise the effort level, not to reach for a bigger model.

5. The agent doesn't learn. You do.

This is the most common misconception I hear about agents like this one. Clicking thumbs-up on the agent's report does not make it smarter. The model's weights never change from your usage, and every run starts fresh with the same instructions and tools.

A loop: agent run, report with thumbs up or down in Slack, thumbs down asks for the real cause, weekly review of the run's full trace, fix the prompt, tool description or masking, replay 30 to 50 historical incidents as evals, merge and deploy if the score goes up. Failed incidents join the eval set. Confirmed causes feed a past-incident index the agent can search.
Improvement is an engineering loop, gated by evals.

Improvement comes from engineering discipline:

6. Humans stay in the loop

The agent is read-only by design. It has no tool that can send a FIX message, reset a sequence number or change a client's configuration. It diagnoses and recommends; a human decides and acts. Every claim must cite a specific message (session, sequence number, timestamp) so the engineer can verify in seconds rather than trust blindly, and when the evidence is thin it's instructed to say "inconclusive" rather than guess.

What it looks like in production

The architecture is less exotic than people expect.

Azure Monitor sends a webhook to an always-on listener container, which also holds an outbound Socket Mode connection to Slack. The listener puts jobs on a Service Bus queue. A worker job scales from zero, runs the agent loop, calls the Claude API, reaches systems only through a read-only masked tool layer, writes an audit log and posts the report back to the Slack thread.
Plain Python and asyncio, the Slack Bolt SDK, the Anthropic SDK and the Azure SDKs. No heavyweight agent framework needed for one focused agent.

Everything the agent touches is logged: every tool call and its arguments, which prompt version ran, tokens used. In a regulated environment, "the AI said so" is not an audit trail. "The AI called these five tools, saw these results and concluded this" is. I've written separately about how to audit AI agents; this design was built to pass that audit from day one.

Traditional versus agentic triage

Traditional triage Agent-assisted triage
Time to first diagnosis 20 to 60 minutes Under a minute
Who can do it Senior engineers who know every system Anyone on call, with the evidence laid out
Where the knowledge lives People's heads and stale wiki pages Prompts, tools and a growing incident index
Consistency Depends on who's on call and how tired they are Same investigation strategy every time
Audit trail "I checked the logs" Every query, result and conclusion recorded
How it improves Informally, and lost when people leave Measured against evals, versioned in git

The agent doesn't replace the on-call engineer. It replaces the first forty minutes of their incident: the part that was always orchestration rather than engineering.

Where this goes next

Once the tools existed, they were reusable. We extracted them into MCP servers used across the company, so tools like fix-dictionary and fix-logs became available to developers in their IDE ("why is this client getting reason 5 on tag 201?") and to other agents. These are the agents I would build on top of them next:

Build the tools once and every agent benefits.

The real shift

The traditional view treats incident triage as a skill that lives in senior engineers. The agentic view treats it as a process that can be encoded: the investigation strategy in a prompt, the access in tools, the safety in code and the quality in an eval suite.

That's the part of agents I think is most often missed. The AI isn't smarter than your best engineer. But your best engineer's approach can now run at 3 a.m., on every alert, in under a minute, and show its work.

I build AI agents that run in production, move teams onto AI workflows and automate operations through Trustflux Ltd. If you run a trading or payments platform and your incident response still depends on who happens to be on call, get in touch.