AI Meeting Assistant

The brief

You will hear it one of two ways.

“Design an AI meeting assistant. It joins calls, records them, produces a summary and action items, and emails the follow-up to attendees.”

Or the version that sounds easier and is not:

“We have Zoom recordings piling up in a bucket. Build something that turns them into meeting notes people actually read.”

The product does five things in order. It gets access to a meeting. It captures the audio. It turns the audio into text with speaker labels. It condenses that text into a summary and a list of commitments. Then it puts those commitments somewhere a human will see them, such as an email, a Slack message, or a task in Jira.

Notice that last step. Everybody wants to talk about the summarization. However, summarization will take the least of your time and the least of your risk budget.

What I’d ask first

This section is where the interview is decided. Every question below changes a box on the diagram. I would say so out loud as I ask it.

“Does the assistant join the meeting live, or do we process a recording afterwards?” This is the single biggest fork. A live bot is a participant. It needs a meeting-platform integration, a media pipeline, and streaming transcription. It is also visible in the room, and that visibility is a legal feature rather than a bug. Post-hoc processing of an uploaded file is a batch job. It needs none of that, and none of the real-time infrastructure. Assume: live bot join for the primary flow, with post-hoc upload as a secondary path. I would design for both, because the second is a strict subset of the first.

“Do users need the transcript or summary while the meeting is still happening?” This is a different question from the last one, and people conflate the two. You can join live and still summarize at the end. Live captions and mid-meeting “what did I miss” require streaming ASR, partial hypotheses, and sub-second latency budgets. End-of-meeting summaries let you run batch ASR on the complete file. Batch ASR is more accurate and much cheaper. Assume: no in-meeting surface in v1. Summary within five minutes of the meeting ending. That assumption alone deletes half the system.

“What jurisdictions are the users and their guests in?” I ask this in the first two minutes, and a surprising number of candidates never ask it at all. Twelve US states require all parties to consent to recording a confidential communication: California, Connecticut, Delaware, Florida, Illinois, Maryland, Massachusetts, Montana, New Hampshire, Oregon, Pennsylvania and Washington. California goes further. Courts have held that California’s rule reaches calls that merely touch California, even when the other party sits in a one-party state (https://www.recordinglaw.com/party-two-party-consent-states/). Pennsylvania treats a violation as a third-degree felony, with statutory civil damages on top. So consent is not a checkbox in the settings page. Consent is a data model. Assume: enterprise customers, US and EU, mixed internal and external attendees. We must be able to prove per-participant consent for every recording we hold.

“Are we identifying speakers by name, or just separating them?” These are different technologies. More importantly, they are different legal categories. Diarization answers one question: how many people spoke, and which segments belong to the same person. It gives you speaker A and speaker B. Speaker identification answers a different question: which of these segments is Priya. Speaker identification usually requires an enrolled voice profile. Plaintiffs have argued that diarization itself collects a voiceprint under Illinois’ Biometric Information Privacy Act, because diarization analyzes vocal characteristics to tell speakers apart. BIPA carries $1,000 per negligent violation and $5,000 per reckless violation. BIPA also requires written notice, written release, and a published retention schedule before collection (https://www.lewisrice.com/publications/ai-transcription-tools-give-rise-to-bipa-claims). Assume: diarize into anonymous speakers, then map speakers to names using calendar attendee lists and meeting-platform speaking events, not voice profiles. That mapping is a heuristic. It is cheap, and it sidesteps a category of liability. I will say exactly that.

“How long are these meetings, and what’s the worst case?” A 30-minute standup and a four-hour board meeting are different engineering problems. Four hours of speech is roughly 35,000–45,000 words. That fits in a modern context window, but it still summarizes badly in one shot. Assume: p50 of 30 minutes, p99 of 3 hours, hard cap at 8.

“Does it send the follow-up email automatically, or draft it?” The brief says “sends automatically.” I would push back on that in the interview. It is the most consequential product decision in the whole design, and the interviewer usually wants to see whether you notice. Sending mail to external parties is irreversible. It is attributed to the user, and it is occasionally career-damaging. Assume: draft by default, with opt-in auto-send for internal-only recipients after a delay window. I will defend this properly in the tradeoffs section.

“What’s already bought?” Meeting platform, identity provider, task tracker, data residency requirements. Assume: Zoom, Google Meet and Teams; Google Workspace and Microsoft 365 for identity, mail and calendar; Jira and Linear for tasks; EU data residency required for EU tenants.

The design

Walk the audio end to end.

 Calendar watch ──► Scheduler ──► Bot fleet (per-meeting worker)
   (Google/MS)                        │
                                      │ joins call, announces itself,
                                      │ captures per-participant audio
                                      ▼
                              Object store (raw audio, encrypted)
                                      │
                                      ▼
                        ┌──────── Job queue (durable, retryable) ────────┐
                        │                                                │
                        ▼                                                ▼
                   ASR worker                                    Consent recorder
              (batch, per-channel)                          (who was told, when, how)
                        │
                        ▼
                 Diarize + align  ──► speaker map (calendar + platform events)
                        │
                        ▼
                 Transcript store  (segments: t_start, t_end, speaker, text, conf)
                        │
          ┌─────────────┼─────────────────────┐
          ▼             ▼                     ▼
     Chunker →      Action-item          Topic / decision
   map-reduce       extractor            segmentation
    summarizer    (structured out)
          │             │                     │
          └─────────────┴──────────┬──────────┘
                                   ▼
                            Meeting record
                        (summary, decisions, items,
                         each with source citations)
                                   │
                    ┌──────────────┼───────────────┐
                    ▼              ▼               ▼
                Review UI     Task sync        Draft email
              (human gate)   (Jira/Linear)   (send only after gate)

Ingestion. A calendar watch subscription tells you that a meeting exists and who was invited. The scheduler then decides whether the meeting is eligible. Eligible means the organizer has the product, the meeting has a join link, and no one has opted out. The scheduler starts a per-meeting worker one minute before the start time. The worker joins through the platform’s bot API, announces itself in the chat and by display name, and captures audio.

Capture per participant if the platform will give it to you. Zoom and Teams can expose a separate audio stream per speaker. When you have separate streams, diarization is free and perfect, so the hardest accuracy problem in the system disappears. Say that explicitly at the whiteboard, because it turns diarization from an ML problem into an integration problem you should solve first.

Storage. Raw audio lands in object storage, encrypted with a per-tenant key, with a TTL. The transcript is the durable artifact. The audio is the expensive and sensitive part, so you want to delete it on a schedule. Default the audio retention to 30 days and the transcript retention to the customer’s policy.

Processing. Run batch ASR on the complete file, per channel where available. If you only had a single mixed channel, run diarization next. Then align speaker turns to word timings. Then map speakers to names. The calendar gives you a candidate list of names. The platform’s active-speaker events give you a time-aligned signal. A simple assignment over those two signals gets you most of the way with no voice biometrics. Everything lands in a transcript store as timestamped segments with per-segment confidence.

The AI layer. Chunk the transcript on topic boundaries rather than on fixed token counts. The boundary signals are long silences, changes in speaker-turn density, and agenda-item markers if you have them. Map each chunk to a structured intermediate: what was discussed, which decisions were made, and which commitments were made and by whom, with segment IDs as citations. Reduce the intermediates into a meeting summary. Extract action items as a strict JSON schema: owner, verb phrase, due date if stated, confidence, and the transcript segment it came from.

The citation requirement is not decoration. Citations are the mechanism that makes review fast. A human checking twelve action items wants to click each one and hear the eight seconds of audio it came from. Without citations, review means re-reading the transcript. If review is slow, nobody reviews.

Serving and the human surface. The meeting record renders as a page with the summary, the decisions, the action items, and the transcript. Each action item is an editable row with an owner dropdown and accept/reject. Task sync and email drafting run on the reviewed record, not on the raw model output.

Where the AI actually is

Genuinely needs a model:

• Transcription. A speech model. It is not an LLM, and buying beats building by a wide margin.
• Summarization. An LLM over chunks, then over the chunk summaries.
• Action-item extraction. An LLM emitting a constrained schema. “I’ll send that over by Friday” → an item; “someone should probably look at that” → not an item.
• Topic segmentation. Marginally. A cheap model or an embedding-boundary heuristic both work.

Ordinary engineering, which is most of it:

• Calendar watch subscriptions, token refresh, and reconciling three platforms’ notions of a meeting.
• The bot fleet: joining calls reliably, handling waiting rooms, being kicked, network failures mid-meeting, and cleaning up workers that outlive their meeting.
• Media capture and per-channel audio muxing.
• A durable job queue with retries, because a four-hour ASR job will fail and must not lose the audio.
• Consent capture, storage, and proof.
• Per-tenant encryption, retention schedules, deletion that actually deletes, and EU residency.
• OAuth to Gmail and Calendar under Google’s restricted-scope regime, which requires an annual third-party CASA security assessment before you can touch user mail at scale (https://developers.google.com/identity/protocols/oauth2/production-readiness/restricted-scope-verification).
• The review UI, which is where the product lives or dies.
• Jira and Linear field mapping, idempotent task creation, and not creating the same ticket twice when someone re-runs a summary.

What I would deliberately not use an LLM for:

• Deciding who the speakers are. Calendar attendees plus platform speaker events is deterministic, auditable, and free. If you ask a model to guess names from context, it produces confident wrong attributions. A wrong attribution in a meeting summary is uniquely damaging, because you have recorded in writing that someone said a thing they did not say.
• Deciding whether to send the email. That is a policy engine and a human click.
• Deduplicating action items across meetings. Use string and embedding similarity plus a rule about the same owner within a window. That is cheaper, explainable, and testable.
• Parsing dates. Resolving “next Tuesday” against the meeting’s timestamp and the user’s timezone is a library call. Let the model extract the phrase, let deterministic code resolve it, and store both.
• Redaction of sensitive content. Pattern matching plus a classifier, with the LLM nowhere in the enforcement path.

The rough 20/80 split holds here almost exactly. The AI is two model calls and a speech API. Everything else is integration, storage, permissions, scheduling and UI.

Key decisions and tradeoffs

Here is the argument for the send gate in full, because interviewers push on it.

Sending is an action tool. It is not idempotent, it is not reversible, and it is attributed to the user’s identity rather than to yours. Every other mistake in this system is embarrassing. This one is external. Consider a summary that assigns a commitment to the wrong VP and then mails it to a customer. That is a support escalation and possibly a lost account, and there is no undo.

So default to draft. Offer auto-send only where the blast radius is bounded. That means internal recipients only, and only for recurring meetings the user has already reviewed several times cleanly. Add a five-minute cancellation window. The mail sits in a queue, and a single click kills it. The delay window costs almost nothing, and it converts an irreversible action into a reversible one. That is the cheapest safety mechanism available.

What breaks

Consent breaks first, and it breaks legally rather than technically. An external guest joins from Seattle and nobody told them. Washington requires all-party consent, with civil damages starting at $100 per day. The mitigations are all product, not model. Use a visible bot with an unambiguous name. Announce the recording on join, by audio or in chat. Keep a per-meeting consent record naming who was notified and how. Add a hard rule that the notice fires again if an unrecognized participant joins. For the strictest tenants, block recording entirely unless every attendee’s domain is on an allowlist.

Diarization collapse on overlapping speech. Two people talking over each other is the most common condition in a real meeting, and it is the worst case for diarization. The symptom is that turns get attributed to the wrong person in exactly the moments that matter, because interruptions cluster around disagreements. The mitigation is per-channel capture. Where you cannot get per-channel audio, suppress speaker attribution below a confidence threshold rather than guessing. “Unattributed” is a fine answer.

Hallucinated commitments. The model reads “we could ask Sam to handle the migration” and emits “Sam will handle the migration by Friday.” This is the highest-severity content failure in the product, because it manufactures obligations. The mitigation has three parts. Require a source segment for every item. Hold the extractor to a high precision bar and accept lower recall. Make the review UI show the quote next to the item.

ASR degradation on the things your users care most about. Product names, acronyms, and non-native-accented speech are exactly where word error rate is worst. They are also exactly what makes a summary useful. The mitigation is per-tenant custom vocabulary, built from the customer’s own artifacts: attendee names from the directory, project names from Jira, product names from their docs. This is a retrieval and plumbing job, not a model job, and it improves quality more than swapping ASR vendors will.

Silent partial failure on long meetings. The three-hour recording’s ASR job times out at chunk 40 of 60. You then ship a summary of the first two hours labeled as the summary of the meeting. The mitigation is a coverage check. The pipeline records expected duration against transcribed duration, and it refuses to publish a summary with a gap. It surfaces “we couldn’t process 22 minutes” instead. Half a summary presented as a whole one is worse than an error.

The bot getting stuck. Waiting rooms, host-not-present, meetings that run 90 minutes over, and workers that never see a “meeting ended” event. The mitigation is hard wall-clock caps, a heartbeat, and a reaper. This is boring work, and it will be a meaningful share of your incident load.

Cross-tenant leakage in retrieval. The moment you add “search across my meetings” or “what did we decide about pricing,” you have built a RAG system over the most sensitive text in the company. A missing tenant filter in one query path exposes one company’s board discussion to another. The mitigation is to make tenant ID a partition key rather than a filter. Add an integration test that asserts a query from tenant A over a corpus containing tenant B returns zero rows.

Deletion that doesn’t delete. A GDPR erasure request has to reach the audio, the transcript, the derived summary, the vector index, the search index, the email drafts, the synced Jira tickets and your logs. The mitigation is to design deletion as a first-class fan-out job on day one, and to keep a registry of every store that holds meeting-derived data. Retrofitting this is a quarter of work.

How you’d evaluate it

Offline. Build a held-out set of real meetings with human-produced gold artifacts: reference transcript, reference speaker labels, and reference action items. A hundred meetings across your actual conditions beats ten thousand clean benchmark clips. Actual conditions means noisy rooms, four-way calls, accented speakers, and at least one three-hour meeting.

Measure at each stage, because end-to-end scores hide which component regressed. Measure word error rate for ASR, overall and sliced by accent, room condition and speaker count. Measure diarization error rate. Then measure the metric that actually matters downstream: speaker-attributed WER, which penalizes correct words attached to the wrong person. For action items, measure precision and recall against the gold list. Weight precision much more heavily, because a missed item is a nuisance and an invented one is a false obligation. For summaries, use a rubric-scored LLM judge calibrated against human ratings on a subset. Add a factual-consistency check that every claim traces to a cited segment.

Online. The business metric is not summary quality. It is the edit rate on action items before acceptance, together with the share of drafts sent without modification. If users accept your items untouched, the product works. If they rewrite every one, your ROUGE score is irrelevant.

Also track three more numbers: the percentage of meetings where the summary is opened at all, the time from meeting end to summary available, and the auto-send cancellation rate. A rising cancellation rate is your early warning that extraction quality has drifted.

Catching regressions. Every prompt, model version and pipeline change runs against the frozen eval set in CI with a hard quality gate. Shadow-run the new pipeline on live traffic and diff the extracted items against production before promoting. Sample and human-review a fixed number of meetings weekly forever, because the eval set ages and your customer mix drifts.

The methodology is covered properly in the sibling agentic-ai-evaluation-guide. That includes judge calibration, rubric design, drift detection, and per-stage attribution. Use it rather than reinventing the harness here.

Follow-ups they will ask

“The meeting is four hours. Walk me through the summarization concretely.” Chunk on natural boundaries: long pauses, sustained speaker changes, and agenda markers. Target a few thousand tokens per chunk, with a small overlap so a commitment spanning a boundary is not lost. Map each chunk to a structured intermediate rather than to prose: topics, decisions, commitments, and open questions, each with segment IDs. Then reduce in one pass over the intermediates, which are maybe five percent of the original volume. This beats one big call for two reasons. First, the map stage parallelizes, so wall-clock time is roughly one chunk’s latency. Second, when a user says “the summary missed the pricing discussion,” you can point at chunk 17, re-run that chunk alone, and diff the result. Single-shot summarization gives you one opaque artifact that you can only regenerate wholesale.

“Why not just use the giant context window? It fits.” Fitting and attending are different things. Recall degrades in the middle of long contexts, and a meeting’s most consequential minute is as likely to be at 1:40 as at 3:55. There is also a cost argument, because you pay for the full transcript on every retry. And there is an operational argument: with map-reduce you can cache per-chunk intermediates, so re-generating a summary with a different tone costs the reduce step only.

“How do you know ‘I’ll follow up with legal’ is an action item and ‘we should follow up with legal’ isn’t?” Partly the model, but mostly the schema and the confidence bar. The schema forces an explicit owner. If the extractor cannot ground the owner to a named attendee, it emits low confidence, and low confidence routes to review as a suggestion rather than as an item. I would tune the threshold for precision. Publish only high-confidence items as items, and show the rest in a “possible follow-ups” section the user can promote. Then I would measure the promotion rate. If users promote half the suggestions, my threshold is wrong, and I have the data to move it.

“An executive says the summary claimed she committed to something she didn’t. What do you do?” Immediately, pull the source segment for that item and play the audio. Roughly half the time the transcript is right and the attribution is wrong, which is a diarization bug rather than a summarization bug. Then classify the failure as an ASR error, a speaker misattribution, or a genuine model fabrication, because the fixes are entirely different. Structurally, this is why the send gate exists. If that summary went out auto-sent, that is the finding of the postmortem. Longer term, the item goes into the eval set as a regression case.

“A participant joins from Illinois and objects to being recorded mid-meeting. What happens?” The bot supports a stop command, and the UI supports it too. On stop, capture ends. The segments attributable to that participant are deleted rather than merely flagged, along with any derived content citing them. Mid-meeting joins re-trigger the notice. A tenant can also configure “block on unrecognized external attendee” so recording never starts at all. The Illinois detail matters beyond wiretap law. BIPA claims against transcription tools turn on whether you analyzed vocal characteristics to distinguish speakers. That is why the design avoids voice profiles and derives names from the calendar instead.

“Real-time captions are now a hard requirement. What changes?” You build a second pipeline, not a modification of the first. It needs streaming ASR with partial hypotheses and a sub-second budget, streaming diarization, and a websocket fan-out to clients. Streaming diarization is materially worse than batch, because short utterances come back unlabeled. So I would show captions without speaker labels live, then correct them in the post-meeting artifact. I would also keep the batch pipeline for the durable transcript rather than persisting the streaming output, so the artifact you store is the high-quality one. Doubling the ASR cost is the honest price, and I would say so.

“How do you handle a user who wants their meeting data deleted?” Deletion is a job, not a DELETE statement. Keep a registry of every store derived from a meeting: audio blobs, transcript rows, summary documents, vector chunks, search index entries, email drafts, cached model outputs, and logs. The erasure job fans out with per-store handlers. It is idempotent, it retries, and it records completion per store so you can prove it. Synced artifacts in Jira are the awkward case. You created a ticket in a system you do not own, so the honest answer is that the customer’s own retention policy governs it. Disclose that up front rather than pretending you can reach in.

“Two summaries of the same meeting, generated an hour apart, differ. Is that a bug?” It is a product problem whether or not it is a bug. Determinism at temperature zero is not guaranteed across providers, and it is certainly not guaranteed across model versions. So I would generate once, store the artifact, and treat regeneration as an explicit user action that produces a new version with the old one retained. Pin the model version per tenant and roll forward deliberately with a shadow diff, rather than silently inheriting whatever the API points at today.

“How do you keep the cost per meeting sane at ten thousand meetings a day?” Know the shape of the cost first. ASR is priced per audio minute and dominates at long durations. The LLM cost scales with transcript length and is a fraction of the ASR cost for a batch pipeline. Then pull the levers in order of payoff. First, skip meetings nobody opens. Most recurring standups are never read, so make summarization lazy or demote them to a cheap model. Second, route the map stage to a small model and the reduce stage to a large one, because chunk-level extraction is not the hard reasoning step. Third, cache aggressively on transcript hash. Fourth, drop the audio early. Storage on three-hour recordings at that volume is a real line item.

“The customer says it’s terrible with their industry jargon. What’s your plan?” First, measure. Get twenty of their meetings and compute WER on their vocabulary specifically. “Terrible” often means three important words are wrong, not that overall accuracy is bad. Second, add per-tenant custom vocabulary, which every serious ASR vendor supports. Populate it automatically from their directory, their Jira project names, and their docs. Third, add a post-ASR correction pass that maps near-miss transcriptions to known entities using edit distance and phonetic matching. That fixes proper nouns cheaply. Fine-tuning the acoustic model is the last resort, and it is rarely justified against those three steps.

“Where does this system fail in a way that costs someone their job?” Two places. The first is an auto-sent summary that assigns a false commitment or an unflattering quote to the wrong person and goes out externally. That is why the gate exists. The second is a compensation, layoff or legal discussion that gets recorded and summarized into a system HR did not know retains it, and is later produced in discovery. The second failure is why I would ship meeting-level sensitivity controls: a “do not record” calendar tag, organizer-only visibility, and short retention defaults. Do not assume every meeting should be captured just because it can be.

“Suppose we want it to answer questions across all my past meetings.” That is a different product bolted onto this one, and it is where the risk profile changes. You need chunk-level embeddings with hard tenant and ACL partitioning. The ACL is the difficult part, because meeting visibility is per-attendee. Retrieval must therefore filter by who is asking, not just by which company they are in. I would enforce that at the index partition level rather than in a query filter. I would also run a permission check on retrieved chunks a second time before they enter the prompt, and require citations in every answer. I would expect the first serious bug to be someone finding a meeting they were not in.

“You have one engineer for six weeks. What do you build?” Post-hoc only, one platform. Let the user upload a recording or connect Zoom cloud recordings. Use managed batch ASR, per channel where available. Do no diarization otherwise, and ship an unlabeled transcript instead. Build one map-reduce summary, one schema-constrained action-item extraction with citations, a review page, and a “copy to clipboard” button instead of email integration. No live bot, no auto-send, no cross-meeting search. That is a testable product, and it establishes whether the extraction is good enough. Everything I skipped is the 80% that takes the other six months.

Say it in one breath

A bot joins the meeting, announces itself, captures per-participant audio, and drops it in object storage. A batch pipeline then transcribes it, maps speakers to calendar attendees rather than to voiceprints, and chunks the transcript. It runs map-reduce summarization plus schema-constrained action-item extraction, and every item cites the segment it came from. A human reviews the items in a UI before anything syncs to Jira or leaves as email, because sending is irreversible and a false commitment assigned to the wrong person is the worst thing this product can do. The model work is two prompts and a speech API. The actual system is calendar integration, a bot fleet, consent records, retention and deletion, and the review surface.