Letting a Human Take the Wheel: Supervisor Takeover for Athena, an Autonomous AI Interviewer
Pausing Athena was easy. The real work was letting a human enter a conversation already in progress without losing the thread.
“This is a stupid feature.”
That was my honest reaction when a request for an Athena takeover feature landed on my plate.
Our users interview dozens of domain experts per deal, on a clock, at several hundred dollars an hour. Athena is our autonomous AI interviewer: you point it at an interview guide, send a link to an expert, and it runs the whole call by itself. Takeover asks for the opposite, let a researcher drop into a live Athena call and start talking to the expert themselves.
We had just built the thing that removes the human from the call… then the first big ask was to put the human back on the call.
I was looking at it backwards, pausing Athena was easy, the real work was letting a human enter a conversation already in progress without losing the thread - then getting Athena back in without making the expert repeat themselves.
What I was wrong about
The failure mode I wasn’t taking seriously was the worst case: you build an agent, send it to a dozen experts, and find out only afterward that the results are garbage. You’ve burned expert fees, burned days in the middle of a diligence, and you still have to redo the work. All of it discovered at the end, when none of it is fixable.
Nobody is going to bet a deal on a voice agent they can’t check, for the same reason you check the work of a new analyst. Takeover gives the researcher a way to look in while the call is still salvageable. If Athena is doing fine, leave it alone. If it isn’t, step in before the hour, and the expert interview is saved.
The design that only covered half the product
Athena reaches an expert two ways: a link they open in a browser, or a bot that joins their Zoom, Teams, or Meet call and relays audio in both directions.
The obvious design worked for the meeting-bot path: when another human joins, pause Athena; when they leave, resume. It fell apart quickly:
- It covers exactly half our distribution - a browser link is for a sole interviewee, a supervisor has no way to join.
- “Detect a human, infer intent from presence” is a guess; someone joining to listen silently would pause the interview.
- There’s no handoff in it, no warning to the expert, no control over when you go live and no way back in if you leave too early.
Pausing the agent wasn’t the problem - we needed to make the supervisor a real participant in the call.
A supervisor is a peer, not a side channel
An Athena call already has two participants in the same real-time session: the agent and the expert. We added the supervisor as a third. They subscribe to both audio tracks, but publish nothing until they take control.
That asymmetry, subscribed but not publishing, is the whole design. Watching costs nothing and is invisible to the expert. Going live is one track away.
That gave us one design for both kinds of call. In the browser, the expert hears the supervisor directly. In Zoom, Teams or Meet, the bot is already relaying the session’s audio mix, so the supervisor’s voice travels through the same pipe.
What a takeover actually changes
The audio switch was the easy part; keeping the conversation intact was harder.
The agent stops replying, but it keeps listening: Our first instinct was to stop feeding Athena input while the supervisor had control, but that would have removed the expert’s answers from the transcript and left Athena blind when it returned. Instead, the expert’s turn is committed as normal; we suppress only Athena’s reply.
The supervisor needs a separate transcription stream: The existing stream is pinned to the interviewee, so it ignores the supervisor’s side of the exchange; without another stream, their questions disappear from the live transcript, the final transcript and Athena’s context. We transcribe the supervisor’s microphone separately and add it to the shared conversation under its own speaker.
That is what makes the hand-back feel coherent: Athena can see what the supervisor and expert discussed, recap it and continue without asking the same question again; if the supervisor took control but said nothing, Athena skips the recap and carries on. In either case, pausing Athena cannot mean dropping what happened while it was quiet.
Nothing waits on the supervisor
Once a supervisor takes over, Athena is silent, so if their laptop closes or their browser dies, the expert cannot be left talking into dead air; control has to return automatically. We use two liveness signals because neither covers every failure on its own:
- A periodic heartbeat from the supervisor’s client: roughly every 15 seconds, against a 45-second timeout on the other side. This catches the cases where the connection technically survives but the client is wedged.
- The session’s own peer-disconnect signal: which catches a closed laptop or a killed tab immediately, without waiting out the timeout.
If either signal drops, Athena takes the call back and keeps the interview moving; the expert does not need to know whether the supervisor clicked “hand back” or vanished halfway through. The UI needed the same bias towards recovery.
If the microphone fails, give control back: Once the supervisor goes live, the takeover button disappears; a silent microphone failure would otherwise leave them in control but unable to speak or retry, while the expert hears nothing. We hand the call straight back to Athena, show the supervisor what went wrong and return them to the same recoverable state, with the takeover button available; a dropped connection is not a deliberate handback.
We also had to deal with stale state arriving late. Reliable delivery does not guarantee ordered delivery, and one delayed frame could move the interface backwards and re-mute someone mid-sentence. Every update now carries a monotonic revision; clients ignore anything older than the latest state they have seen.
Two things that only show up in a live call
Don’t let the spoken handoff block the call: Athena introduces the supervisor before opening their microphone, so the handoff briefly depends on a text-to-speech provider; if that provider hangs, the takeover continues after a timeout. The same rule applies at the end of a call, where finalisation does not wait for the farewell to finish.
A watcher can still end the call: The supervisor may decide the interview is finished without ever taking control, so the end-call request returns immediately while Athena’s goodbye plays in the background.
These were small fixes, but they mattered more on a live call than they did in the design doc; a few seconds of unexplained silence feels much longer when an expert is waiting.
Where does it fit
Junior already had a live call page with a running transcript, question checkoff, notes and chat. It was built for human-run interviews, but the supervisor’s job was nearly identical: watch, take notes, track coverage and occasionally speak, so we reused it.
The awkward part was a single read-only flag that had come to mean two things: whether the user could edit, and whether they were recording. Those had always moved together; for a supervisor, they do not. They should be able to save notes and check off questions while the recording and consent controls stay with Athena, and splitting the flag fixed that distinction - as well as a bug that had been silently dropping supervisor notes.
As soon as an expert starts an Athena call, it appears where a human-run interview would: same page, same notes, same checkoff, same chat, with just one new control that lets the supervisor speak.
What changed
Before this, you deployed Athena, sent the link and waited; you did not know whether the call had started, how it was going or whether it needed rescuing, and by the time the results appeared, the call was over.
Now you can see which calls are live and when they are likely to finish, open one, read the transcript, check coverage and take notes without being heard. If Athena misses something important, you can ask the question yourself, then leave it to continue.
The goal is still for Athena to run the whole interview, but while people are deciding how much to trust it, they need to be able to watch the work and step in without breaking the call.
I thought takeover undermined the point of an autonomous interviewer; in practice, it is what makes handing over the call feel safe.