Agent Console - realtime ai chat system design

What we are gonna build

The backend (agent-server) is provided as a Docker container. You do not modify it. It speaks a documented WebSocket protocol, simulates a context-aware AI agent that streams responses, makes tool calls, retrieves context, and when chaos mode is enabled drops connections, reorders messages, injects latency spikes, and sends malformed heartbeats. Your job is to build a frontend that handles all of it gracefully.

Problem Understanding

How does a simple chat application looks like?

But things don't go as it seems, many a times we face

• ▸Network failures, reconnection, duplicate responses, replay, tool calls, partial responses, etc.

Suppose, server sends 1 2 3 4 5 but network delivers 1 3 4 2 5, like network is not bound to deliver in the manner it receives from server right? if we process these events as it then we won't understand anything, we would want that things happen in a sequence right? so we would design a system which handles this case.

we won't receive the string but a json in a format {seq,type,value}={1,TOKEN,Hello}, now seq becomes our go to way which we can use to order things and process them correctly.

now let's understand few things.

Events

now instead of just getting string we would consider everything as events like

MESSAGE_START
TOKEN
TOOL_CALL
TOOL_RESULT
MESSAGE_END

and this whole message would be a complete message instead of just string.

Event Streams

collection of many events and we will be using this as source of truth.

Reorder Buffer

We uses this when events don't come in order like server send 1 2 3 but network give us 1 3 2 so while we won't get 2 we would put the 3 in the buffer and process 2 first then pick it from buffer and process, basically it is used like storage.

Protocol Engine Design

Now, let's design the protocol engine, responsible for protocol corectness means getting the events properly and storing it so that later on can be used to create the initial state.

• ▸it would be responsible for ordering, deduplication, replay, resume, buffering.
• ▸StatenextExpectedSeq Buffer processed

Now let's talk about problems and how we are gonna solve it one - by - one

Deduplication

Suppose we were receiving event 1 2 3 and connection drops then how would backend know what to send next? it won't right? we would handle that here itself and let it send duplicates like maybe when connection re-establishes it send us 2 3 4, so for those 2 3 we would just ignore them like we would check is upcoming event sequence is equal to nextexpectedseq, is its greater we would push that to buffer, if it's smaller then we would ignore that completely and if it's equal then ofcourse we would process it.

Idempotency

It means that applying the same operation multiple times would give us same result. like if you process the same numbers again and again should give same result.

Replay

We have already discuss about it indirectly when server sends the same events then what would we do? ignore them simply.

Resume

Now, there's a catch like when the connection drops then what should we do? from where it should resume like what should be the nextexpectedseq now? is it the highestreceived or highestprocessed? ofcourse highestprocessed right, suppose there is different between two and we start asking from highestprocessed then everything would start piling up in buffer and it won't process ever.

State Reconstruction and Event Sourcing

Suppose we turned off the computer and logged in again then would be ask the server for everything again for same query? no we would store that right? so we would store the events somewhere and would reconstruct the state using those, this is also known as replay.

So that was it for protocol engine, now everything is processed and stored.

ChatStateBuilder

Now, the ChatState can be build using the ordered events we have stored in the processed. using any loop easily. then we can easily render that in UI.

Now, Message Storage in state

We can either store the message as string, which won't be of much detail, or as parts with toolcalls and results and everything. we should store the toolresults in the toolcalls itself so that it's easy for referencing like what toolcall this result even belongs to.

ACKs

one of thing is acknowledgment like once we received the the message we can say the server we got it. but then server would say yeah, i got it that you got it then client would again say that it got it that you got it so we won't do things like this, we would follow idempotency, do it once and forgets about it, doesn't matters much.

State Machines

How we are gonna store the state like connecting, resume and all? we can use bolleans but that would create problem like it may be connecting and resuming both or streaming which would lead to confusion instead of this we can use fintite state machines like conversation state = idle or whatever and connection state = disconnect or so.

Snapshots

Suppose we have millions of events coming to our way. so how are gonna replay all of them again? do we process everything at once? no right our system would hang, our ram consumption would explode, so instead we would batch it and would process it in parts like we would only process what we needed as you might have seen in chatgpt too. it keeps the latest one while the oldest ones when you tries to visit it takes some time to load usually.

Final Architecture

Implementation: Agent Console

Thanks for Reading