Home
Trilogy AI CoE

Building a Meeting Copilot: The Vision

What if you could clone yourself for meetings? Not a creepy deepfake, but a transparent AI copilot that represents you when you can't be there—and seamlessly hands off to the real you when you can.

The Problem We're Solving

The Scenario

You have three meetings scheduled at the same time. One is a status update you don't really need to attend but should probably monitor. Another is a sales call where your presence matters but most of the talking is done by your team. The third actually requires your brain.

Common Solutions Are Bad

  • Decline two meetings: Miss information, look disengaged
  • Ask for recordings: Never actually watch them
  • Double-book and hop between: Exhausting and ineffective
  • Hire an assistant: Expensive and can't make technical decisions

What if instead you had an AI avatar that could attend meetings on your behalf—with actual video presence—and knew when to summon specialist capabilities for specific tasks?

Vision 1: One Avatar, Many Capabilities

The core model isn't "a team of bots in the meeting." It's one intelligent avatar that knows who to call.

You invite a single bot to the meeting. It joins with video presence—an actual avatar in the meeting grid, not a chat sidebar. Participants see a face, hear a voice, and interact naturally. Under the hood, this avatar is backed by a coordinator called Maestro that understands context and can summon specialist capabilities on demand.

Tempo

Someone asks about scheduling → Maestro summons Tempo, who has access to everyone's calendars and can propose times in real-time.

The Algorithm

The conversation shifts to social media strategy → Maestro summons The Algorithm, who can pull live engagement data and draft tweets on the fly.

Gatekeeper

A question comes up about recent emails → Gatekeeper gets summoned, surfaces relevant threads, and hands back.

Radar

Someone asks "what's the market rate for X?" → Radar does real-time web research and reports back.

Vision 2: The Avatar Copilot

The more ambitious vision: an avatar that represents you specifically.

What It Does

  • The avatar has your face (or a stylized representation)
  • It's loaded with your knowledge base—your articles, notes, meeting history, writing style, domain expertise
  • For many routine meetings, it can handle things autonomously. It knows your positions on common questions. It can give status updates based on your actual work.

The Key Differentiator

Here's what sets this apart from a generic bot: you can take over at any moment.

Voice Passthrough

You speak, the avatar speaks your words in real-time

Text-to-Speech

You type, the avatar vocalizes what you typed

And crucially: transparency built in. The system makes clear when it's pulling from your knowledge base versus when it's actually you. Maybe a subtle visual indicator. Maybe the avatar explicitly says "David mentioned in his notes that..." vs. "David says..."

This isn't about tricking people. It's about extending your presence honestly.

Why This Matters Now

Three things have converged to make this feasible:

01

Real-time voice AI finally works

OpenAI's Realtime API delivers speech-to-text, LLM reasoning, and text-to-speech in a single streaming pipeline. Response times are now 300-600ms—conversational speed. A year ago this required stitching together three separate systems with compounding latency.

02

Meeting integration infrastructure exists

Services like Recall.ai and LiveKit have solved the hard problems of getting AI into video calls. You can join Google Meet, Zoom, or Teams programmatically, capture audio, display video, and stream responses back. The plumbing works.

03

Agent dispatch patterns are production-ready

LiveKit's agent dispatch lets you spin up a fresh agent instance by name into an existing room. Combined with Redis for coordination state, you get clean handoffs: the current agent finishes speaking, signals exit, and the new specialist takes over within seconds. No multi-agent chaos—just sequential, controlled transitions.

The Architecture at 30,000 Feet

Here's how the single-avatar, multi-specialist model works:

This is the "summoning" pattern—Maestro summons Tempo for a calendar question the same way you'd turn to a colleague and say "hey, you handle this one." The meeting participants see one continuous avatar presence. The architecture underneath is swapping agents.

What's Working Today

Let's be honest about current state versus aspirations.

Working ✅

  • Single agent joins Google Meet via Recall.ai with avatar video
  • OpenAI Realtime API handles voice conversation at ~500ms latency
  • Avatar video renders via HeyGen LiveAvatar
  • MCP tools connect to 5 external services (X/Twitter, Google Calendar, Gmail, web search, Substack)
  • Agent dispatch: Maestro summons specialists, specialists hand back
  • Redis coordination: active agent tracking, exit signaling, handoff context

Partially Working ⚠️

  • Avatar rendering adds significant latency (500-2000ms)
  • Specialist-to-specialist handoffs (bypassing Maestro) need more testing
  • Handoff transitions have a brief gap while agents swap

Not Yet Built ❌

  • Personal knowledge base integration (the "your notes, your style" piece)
  • Human takeover UX (voice passthrough, text input)
  • Transparency signaling (AI vs. human indicator)
  • Client-side avatar rendering for lower latency

The Latency Reality Check

Here's the thing architects need to understand: cloud avatar rendering is slow.

Current end-to-end latency breakdown:

1

Speech-to-Text

100-300ms

2

LLM Reasoning

200-500ms

3

Text-to-Speech

100-300ms

4

Avatar Rendering

500-2000ms

Total: 1-3+ seconds

That's not conversational. It's awkward.

The avatar rendering is the bottleneck. Every cloud avatar service we tested (HeyGen, Hedra) has this problem. The video frames have to be generated on their servers and streamed back. Physics and network latency impose a floor.

The path forward is probably client-side avatar rendering using WebGL and something like Ready Player Me. That could get the total latency under 500ms. But it's a significant engineering lift.

For now, there's also an audio-only mode that skips avatar rendering entirely. Total latency drops to 300-800ms—genuinely conversational. The trade-off is no video presence, just a static image with an audio visualizer.

Where We're Going

This series will walk through the architecture in detail:

1

Part 2: The Audio-Video Pipeline

Deep dive into getting AI into video calls. Recall.ai integration, LiveKit room management, OpenAI Realtime API, and avatar rendering. We'll trace a single utterance from human speech to AI response and back.

2

Part 3: Multi-Agent Orchestration

How the summoning pattern works in code. LiveKit agent dispatch, Redis coordination, the Maestro-specialist handoff lifecycle, and why one-at-a-time beats a crowd.

3

Part 4: The Knowledge Layer & Human-in-the-Loop

Building the personal knowledge base, the takeover UX, and how to signal transparency between AI and human speaking.

Key Takeaways

The vision is a meeting copilot

One avatar seat with specialist capabilities behind it, plus a personal avatar that represents you

One bot, many brains

Agents swap in and out via dispatch, not multiple bots fighting for airtime

Transparency is non-negotiable

Clear signaling when AI is speaking versus when you are

Infrastructure exists

Recall.ai, LiveKit, OpenAI Realtime make the core pipeline possible today

Latency is the enemy

Cloud avatar rendering adds 0.5-2 seconds, pushing total latency to 1-3+ seconds

MCP tools are reused

The same tool infrastructure that powers chat agents powers the meeting avatar

This is Part 1 of a 4-part series on building a meeting copilot. Part 2: The Audio-Video Pipeline →

Next up: the technical deep dive into getting an AI's voice and face into a Google Meet call.