Support Co-Pilot

Support Co-Pilot is an experiment I built through OneNeural Labs to explore AI assistance during a live support call. A customer and a support agent share a Twilio conference, while a web dashboard shows transcription, retrieved knowledge, and suggested responses.

I designed and maintain the application across the Next.js dashboard and a unified Node backend. It is a runnable demo, with production hardening still on the roadmap.

Overview

Why I built it

I wanted to explore whether support software could offer useful information while a call was still happening. The experiment connects an ordinary phone conversation to a dashboard, so the support agent can review suggestions as they talk.

The engineering questions were concrete: how to get call audio into the transcription service, how to keep a useful window of context, and how to send updates to the browser without making the interface difficult to follow.

Conference call flow

sequenceDiagram
  participant Customer
  participant Twilio as Twilio Conference
  participant Agent
  participant API as Unified API
  participant Deepgram
  participant OpenAI
  participant Pinecone
  participant Dashboard as Agent Dashboard

  Customer->>Twilio: Dial support number
  Twilio->>API: Conference start webhook
  API->>Twilio: Dial agent into conference
  Twilio->>Agent: Ring agent's phone
  Agent-->>Twilio: Join conference
  Twilio->>API: Media stream (WebSocket, both tracks)
  Agent->>Dashboard: Open call view (SSE connect)

  loop During the call
    Twilio-->>API: μ-law audio frames
    API->>Deepgram: PCM16 stream
    Deepgram-->>API: Diarized transcript deltas
    API->>OpenAI: Prompt with transcript window
    API->>Pinecone: Query top-K KB chunks
    Pinecone-->>API: Matching policies / FAQs
    OpenAI-->>API: Suggestion text
    API-->>Dashboard: SSE: transcript, sentiment, suggestion, KB
  end

  Agent-->>Twilio: Hang up
  Twilio->>API: Conference end webhook
  API->>OpenAI: Summarization prompt
  API->>Dashboard: SSE: final summary + metrics

System architecture

flowchart LR
  subgraph Clients
    AgentPhone[Agent phone]
    CustomerPhone[Customer phone]
    Browser["Agent dashboard<br/>Next.js 15"]
  end

  subgraph TwilioLayer["Twilio"]
    Conf[Twilio Conference]
    Media[Media Streams]
  end

  subgraph Server["Unified Server"]
    Gateway[REST API]
    WS[WebSocket handler]
    SSE[SSE broadcaster]
    AudioProc[Audio processor]
    Orchestrator[AI orchestrator]
    KB[KB service]
  end

  subgraph AI["AI and Data"]
    DG[Deepgram Nova-3]
    GPT[OpenAI GPT-5 Nano]
    PC[Pinecone]
    PG[(Supabase Postgres)]
    RedisCache[(Redis)]
  end

  AgentPhone --- Conf
  CustomerPhone --- Conf
  Conf --- Media
  Media ==> WS
  Browser <-->|HTTPS| Gateway
  SSE -->|SSE| Browser

  WS --> AudioProc --> DG
  DG --> Orchestrator
  Orchestrator --> GPT
  Orchestrator --> KB --> PC
  Orchestrator --> PG
  Orchestrator --> RedisCache
  Orchestrator --> SSE

What it does

For the agent

• Live transcript. Diarized in real time, speaker-0 = Agent, speaker-1 = Customer; scrolls as the conversation evolves.
• Contextual suggestions. GPT-5 Nano continuously drafts short response suggestions grounded in the last N turns, rendered as cards the agent can dismiss, copy, or speak.
• Sentiment and flow. Running sentiment bar with "escalation risk" flags, so the agent sees tone shifts before they become objections.
• Knowledge access. Top-K matching FAQs and policies from Pinecone, linked directly to the underlying document, so the agent can cite rather than guess.
• Quick actions. One-click "send link", "escalate to tier-2", "create ticket" hooks that fire against the CRM API.

For the supervisor

• Active calls dashboard. A single view of every in-progress call with live sentiment, AHT, and agent name.
• Drop-in review. Join any live transcript read-only to coach without interrupting.
• Performance analytics. CSAT, handle time, and resolution metrics aggregated per agent and per topic.

For the business

• Faster resolution. Agents get the right answer while they are still talking to the customer.
• Consistent answers. Suggestions come from the same policy corpus every agent is reading from.
• Lower ramp time. New hires get real-time guidance instead of memorizing binders.

Real-time AI pipeline

The orchestrator is the only component that decides what goes out on SSE. Everything it sees comes from three streams: transcript deltas, sentiment scores, and KB hits. It produces one broadcast per transcript delta with any updated fields.

flowchart TB
  Audio[Audio frames]
  STT[Deepgram Nova-3<br/>speaker diarization]
  Buffer[Rolling transcript buffer<br/>last N seconds]
  Sent[Sentiment model]
  Sug[GPT-5 Nano<br/>suggestion chain]
  Search[Pinecone<br/>policy + FAQ index]
  Merge[Orchestrator merge]
  Out[SSE event]

  Audio --> STT --> Buffer
  Buffer --> Sent
  Buffer --> Sug
  Buffer --> Search
  Sent --> Merge
  Sug --> Merge
  Search --> Merge
  Merge --> Out

Latency budget

The user-visible budget is the sum of: Twilio → server, audio decode, Deepgram, orchestration, SSE broadcast, and the browser's paint. The demo uses the following latency budget:

These are design targets for the demo, rather than production benchmarks. The dashboard updates fields independently so transcription does not have to wait for a suggestion.

Repository layout

flowchart LR
  Root[copilot-assist/]
  Apps[apps/]
  Web[apps/web<br/>Next.js 15]
  API[apps/api<br/>Express + WS]
  Pkgs[packages/]
  DB[packages/database<br/>Prisma]
  Shared[packages/shared<br/>types]
  Cfg[packages/config]
  Docs[docs/<br/>product + tech + guides]

  Root --> Apps
  Root --> Pkgs
  Root --> Docs
  Apps --> Web
  Apps --> API
  Pkgs --> DB
  Pkgs --> Shared
  Pkgs --> Cfg

Nx ties it together: pnpm nx serve web and pnpm nx serve api for local work, pnpm nx run-many --target=build --all in CI. pnpm nx graph is the source of truth for dependency direction across the workspace.

Design decisions and trade-offs

• One Twilio conference, not separate agent and customer calls. Routing the audio through a conference gives me a single mixed WebSocket stream and avoids double-billing minutes. The cost is that I have less direct control over per-track audio; diarization via Deepgram is what makes this work in practice.
• Unified Node server instead of microservices. Audio ingestion, AI orchestration, and dashboard fan-out all live in one Express process with a WebSocket endpoint and an SSE endpoint. Keeping the demo in one process makes the data flow easier to follow and reduces deployment complexity.
• SSE for dashboard push, WebSocket for Twilio ingress. SSE is unidirectional, uses ordinary HTTP, auto-reconnects, and needs no client library. Twilio requires WebSocket for media streams, so that decision was made for me. Mixing the two is fine inside one process.
• GPT-5 Nano over larger models. For short suggestion strings grounded in a tight transcript window, Nano is fast enough to stay inside the 2-second budget and accurate enough with prompt-level constraints. Larger models were tested and blew the latency budget without materially better output at this task.
• Strict security level and no tool use on the LLM. The LLM never invokes anything. It drafts strings; the dashboard and the agent decide what to do with them. This is the simplest way to keep the real-time loop safe and auditable.
• Prisma + Supabase + Redis, not a bespoke store. The call tables, transcript rows, and summary records are all boring Postgres. Redis is only for live session state (SSE subscriber lists, audio buffers). Nothing clever persisted.

What I learned

• Diarization carries most of the demo. Without reliable speaker-0 vs speaker-1 tagging, every downstream feature (suggestions, sentiment, CSAT) degrades. Speaker attribution needs to remain useful throughout a call for the suggestions to have the right context.
• The latency budget is a design constraint, not a spec item. Every added service, every added round-trip, came out of the same 2-second envelope. "Unify first, split when forced" kept the demo believable.
• SSE is underused. For one-to-one server-to-client push, it is strictly simpler than WebSocket, and the auto-reconnect semantics saved me at least one bug I would have written by hand.
• Grounded suggestions feel different from open-ended ones. Retrieval gives a suggestion relevant policy or FAQ material to draw on. That also gives the person reviewing it a source to check.
• Nx was worth the setup cost for a demo. A single pnpm nx run-many --target=build --all for CI and a shared type package between apps/web and apps/api removed enough friction that adding features stayed cheap even at demo scope.

Status and roadmap

v0.1.0 is a complete, runnable demo. It is not a production contact-center product; the near-term items below are what it would take to get there:

• Post-call analytics store. Durable summaries, coaching tags, and searchable transcripts beyond the session Redis cache.
• CSAT calibration. Map live sentiment signals to historical CSAT outcomes so the "escalation risk" indicator is grounded, not just a mood ring.
• Multi-tenant hardening. Proper org-scoped KB indexes, per-tenant Redis prefixes, SSO via Clerk organizations, and audit logging.
• Agent feedback loop. One-click "this suggestion helped / didn't" that streams into a dataset for prompt tuning and model selection.
• Deployment hardening. Cloud Run autoscaling profile, Cloudflare front door, and a proper load test against 50 concurrent conferences.