Conversational AI Operator Concept

Status: deferred product concept, not an active roadmap commitment
Captured: 2026-07-19

Purpose

Explore a conversational aegi operator that can understand natural-language requests, ask follow-up questions, speak and display responses, and safely operate the app through explicit product capabilities.

The intended experience is closer to a capable in-app operator than a general chatbot. It should help an agency user complete aegi work while preserving normal permissions, validation, confirmation, and audit rules.

Example conversations:

User: Start my timer.
Assistant: Which client is this for?
User: Acme.
Assistant: Is this general client work or the Website Redesign project?
User: Website Redesign.
Assistant: Starting the Website Redesign timer for Acme.

User: Create a project for Acme.
Assistant: What should it be called?
User: Summer campaign.
Assistant: Should I use the usual defaults?
User: Yes.
Assistant: The Summer Campaign project is ready.

The same conversation should support typed input, spoken input, visible transcripts, spoken responses, and app navigation.

Product Fit

This concept fits the existing aegi operating model because the client account is the root business context. The assistant can resolve client context first, then work with client-owned projects, time entries, billing records, files, contracts, and other records without inventing a parallel ownership model.

The assistant should initially be an org-surface capability for agency staff. Portal or platform variants would require separate product decisions, permissions, prompts, tools, and surface-owned UI. They should not inherit the org assistant automatically.

The most useful assistant context would include:

• the authenticated user, organization, permissions, and current surface
• the current route and visible record identifiers
• recent or active client, project, and timer context
• exact search results returned by aegi queries
• the current conversation's unresolved fields and confirmed choices

The model should never receive unrestricted database access or a raw dump of organization data.

Core Architecture

The intelligence, speech recognition, and speaking voice can be independent:

Microphone or text input
        |
        v
Speech-to-text (for voice input)
        |
        v
Conversational model (intent, dialogue, tool selection)
        |
        +----> aegi server tools ----> Convex queries/mutations
        |
        +----> aegi client commands -> navigate/open/prefill UI
        |
        v
Text response and transcript
        |
        v
Text-to-speech (selected voice)

This modular design avoids locking the product's voice, conversational model, and application logic to one provider.

Conversation Layer

The conversation layer should:

• identify the user's intended operation
• collect required fields one question at a time
• resolve references such as "this client" or "that project" from explicit app context
• clarify ambiguous client, project, or record matches
• maintain short-lived conversation state
• produce concise spoken responses and a matching visible transcript
• call only registered aegi tools

It should not be responsible for authorization, data validation, financial truth, or final mutation rules.

Server Tools

aegi should expose a small, typed capability catalog rather than allowing the model to operate the database directly. Illustrative tools include:

• findClients
• findProjects
• getActiveTimer
• startTimer
• stopTimer
• prepareProjectDraft
• createProject
• getInvoiceSummary
• listRecentRecords

Each tool should have explicit arguments and results. The app should authenticate the caller, enforce permissions, validate organization and client scope, detect no-op or duplicate writes, and produce normal audit events.

Client Commands

Some operations should affect only the interface:

• navigate to a canonical appRoutes destination
• open a known dialog
• prefill a form with resolved values
• focus or highlight a record
• display a preview before committing an action

These should be structured commands interpreted by aegi. The model should not manipulate arbitrary selectors, generate raw URLs, or simulate uncontrolled browser clicks.

Immediate And Committed Actions

The system should distinguish two action classes:

• Immediate actions: search, navigate, open, focus, and prefill.
• Committed actions: create, update, start, stop, send, delete, charge, or otherwise mutate state.

Immediate actions can generally execute without confirmation. Committed actions need confirmation proportional to their risk. Destructive, financial, externally visible, or difficult-to-reverse operations require explicit confirmation with specific action language.

Voice Experience

The assistant can have a natural voice without using the conversational model provider's bundled voice. A separate text-to-speech engine can read the model's response in any supported or properly licensed custom voice.

For low perceived latency, the language model should stream text in stable phrases or sentences and the text-to-speech service should begin synthesizing those chunks before the entire response is complete.

The initial interaction should be push-to-talk through a microphone button or keyboard shortcut. An always-listening wake phrase such as "Hey aegi" is technically possible, but it adds microphone privacy, browser background-execution, battery, permission, and platform-support concerns. It should be evaluated later, especially if aegi gains a native desktop or mobile client.

The voice experience should support:

• interruption or barge-in while the assistant is speaking
• visible live transcription
• a text-only and silent mode
• accessible keyboard operation and labelled microphone controls
• a clear listening/speaking state that does not rely on color alone
• concise responses suited to speech rather than long chatbot prose

AI And Voice Options

Provider names, model names, previews, and prices change frequently. The following is a dated research snapshot, not a procurement decision.

Integrated Realtime Models

Integrated models accept live audio, manage the conversation, call tools, and return audio. They reduce engineering complexity but couple more of the experience to one provider.

OpenAI Realtime

gpt-realtime-2.1 supports speech-to-speech conversation, text, reasoning, and tool use. It is a strong quality baseline but may be too expensive for the intended economics.

References:

• GPT-Realtime-2.1
• Realtime API with WebRTC
• Realtime model prompting

Google Gemini Live

gemini-3.1-flash-live-preview is a low-latency audio-to-audio model with function calling through the Gemini Live API. At the time of this note, Google lists audio input at about USD 0.005 per minute and audio output at about USD 0.018 per minute, with a development free tier. It is currently a preview model, so availability, behavior, limits, and pricing may change.

References:

• Gemini API pricing
• Gemini Live API capabilities
• Gemini Live WebSocket API

Gemini Live is the leading candidate for a low-cost integrated prototype if its tool reliability, privacy terms, regional availability, and preview stability meet aegi requirements.

Managed Modular Voice Platforms

Managed platforms can supply speech recognition, voice-agent orchestration, text-to-speech, or a combination while allowing aegi to choose a separate language model.

Pricing snapshot captured on 2026-07-19:

Provider	Relevant capability	Published starting usage price
Deepgram	Voice Agent API	USD 0.075 per connected minute
Cartesia	Line voice agents	USD 0.06 per connected minute
ElevenLabs	Speech Engine	USD 0.08 per connected minute
Deepgram	Flux English speech-to-text	USD 0.0065 per audio minute
Groq	Whisper Large v3 Turbo transcription	USD 0.04 per transcribed hour

References:

• Deepgram pricing
• Cartesia pricing
• ElevenLabs API pricing
• Groq pricing

Managed prices must be compared using realistic aegi conversations. Providers differ in whether they meter connected time, user speech, generated speech, characters, tokens, silence, and bundled language-model use.

Lowest-Cost Prototype

The cheapest proof of value could use:

• browser or inexpensive speech recognition
• a small, tool-capable text model for intent and dialogue
• the browser's built-in speechSynthesis voice
• existing authenticated aegi APIs for all data and actions

This avoids most voice-generation cost. The tradeoff is inconsistent browser support, device-dependent voices, and a less distinctive experience. It is suitable for validating user behavior, not necessarily for the final product.

Open-Source Voice Layer

Open-source text-to-speech allows aegi to self-host the speaking voice and choose a voice independently from the AI provider.

Kokoro

Kokoro is a small 82-million-parameter open-weight text-to-speech model with Apache 2.0 licensed weights. It supports multiple voices and is designed to be fast and inexpensive to run.

Best fit:

• a lightweight self-hosted voice service
• choosing among preset voices
• a commercial-friendly starting point
• CPU or modest-inference experiments before provisioning larger GPU services

Reference: Kokoro-82M model card

Chatterbox

Resemble AI's Chatterbox family is MIT licensed and supports zero-shot voice cloning from a short reference recording. Chatterbox Turbo is intended for lower-latency English voice-agent use, while multilingual variants support broader language coverage. Generated speech includes a watermark.

Best fit:

• a distinctive, licensed aegi voice
• expressive or custom speech
• self-hosting on suitable GPU infrastructure
• using a short reference recording without training a new conversational model

References:

• Chatterbox repository
• Chatterbox model information

Other Projects And Licensing Cautions

• Piper is a fast local engine used by projects such as Home Assistant. Active development moved to OHF-Voice/piper1-gpl, which uses GPL-3.0; integration and distribution implications require review. Piper repository
• XTTS supports voice cloning, but the XTTS-v2 weights use the Coqui Public Model License rather than the repository's MPL software licence. Commercial use requires careful licence review. XTTS-v2 model card
• Fish Speech uses the Fish Audio Research License. It should not be treated as a permissive commercial default. Fish Speech repository
• F5-TTS code and pretrained weights have different licensing considerations, including dataset-derived restrictions. It should not be selected for commercial use based only on the code repository's licence.

"Open source" or "open weights" does not automatically mean commercially unrestricted. The code licence, model-weight licence, voice-model licence, training-data terms, and rights to the reference speaker's identity must all be checked.

Creating A Distinctive aegi Voice

A branded voice does not need to imitate a celebrity, public figure, or existing assistant. A safer and more defensible process is:

1. Choose a voice actor whose delivery fits the aegi brand.
2. Obtain explicit contractual rights for synthetic voice generation, product use, duration, territories, languages, and future model migration.
3. Record clean, varied reference material in a controlled environment.
4. Use a permissively licensed TTS model such as Chatterbox, or commission a dedicated trained voice.
5. Store reference recordings and derived voice assets as sensitive production assets.
6. Provide disclosure that the assistant voice is synthetic where appropriate.
7. Offer alternative voices, text-only use, and a silent mode.

Users should only be allowed to clone voices when they own the voice or have explicit permission. aegi should not ship an unrestricted "clone anyone" interface without consent controls, abuse prevention, and legal review.

Recommended Prototype Stack

Two sensible starting configurations are:

Minimal-Cost Validation

Input: push-to-talk browser microphone or typed text
Speech recognition: browser capability or inexpensive hosted transcription
Brain: small tool-capable hosted model
Actions: authenticated aegi server tools backed by Convex
Voice: browser speechSynthesis

This configuration answers the first product question: do agency users actually complete work through conversation?

Open-Source Voice Validation

Input: push-to-talk browser microphone or typed text
Speech recognition: Whisper/faster-whisper or a low-cost hosted service
Brain: small tool-capable hosted model
Actions: authenticated aegi server tools backed by Convex
Voice: Kokoro initially; Chatterbox Turbo for a licensed custom voice

The language model should remain replaceable behind an internal provider interface. The voice synthesizer should have a separate internal interface so provider-hosted, self-hosted, browser, and silent implementations can coexist.

Safety, Privacy, And Reliability Requirements

The difficult part is not producing a convincing demo. It is making the operator trustworthy.

Required controls include:

• enforce the authenticated user's existing permissions on every server tool
• keep organization, client, and surface boundaries explicit
• never place provider API secrets in the browser
• use short-lived browser credentials where a realtime provider requires a direct connection
• send only the minimum context required for the current operation
• clarify ambiguous names and never silently guess record identity
• require explicit confirmation for destructive, financial, externally visible, or difficult-to-reverse actions
• use idempotency keys for retried committed actions
• preserve existing no-op mutation rules
• record who requested an action, what the assistant proposed, what was confirmed, and what actually executed
• show tool failures honestly and keep the UI state recoverable
• allow the user to inspect or edit a prepared payload before committing higher-risk actions
• define conversation retention, audio retention, transcript retention, deletion, and provider-training policies
• provide clear microphone state and recording disclosure
• test multilingual names, accents, background noise, interruptions, and exact client/project matching

Financial provider truth, permission checks, and Convex business rules must remain authoritative. The model may recommend or request an action; it must not override those systems.

Suggested Delivery Sequence

This concept should remain deferred until the existing operational spine is stable and there is evidence that conversation will reduce friction.

If activated later, use this sequence:

1. Add a text-only assistant that can search and navigate but cannot mutate data.
2. Add draft and prefill commands that require the user to submit through normal UI.
3. Add a small set of low-risk committed actions such as starting or stopping a timer.
4. Add push-to-talk input and a browser-provided voice.
5. Evaluate Gemini Live against a modular speech-to-text, text-model, and TTS pipeline using representative aegi tasks.
6. Add a self-hosted Kokoro voice experiment.
7. Commission and test a licensed aegi voice with Chatterbox if usage justifies the investment.
8. Consider higher-risk tools only after permission, confirmation, audit, idempotency, and evaluation coverage are proven.
9. Consider wake-word or native-client behavior only after the push-to-talk experience is valuable.

Evaluation Criteria

Before choosing a provider or shipping committed actions, measure:

• exact client and project resolution accuracy
• correct tool selection rate
• missing-field and ambiguity handling
• false-action and duplicate-action rate
• interruption and turn-taking quality
• time to first transcript and time to first audio
• full task completion latency
• cost per completed task rather than only cost per minute or token
• performance for supported languages and accents
• user correction frequency
• confirmation comprehension
• provider outage and fallback behavior
• user preference for voice versus text

Open Product Decisions

• Is the assistant org-only, or is there eventual value in separately designed portal and platform assistants?
• Does the assistant live in the top bar, command palette, dock, or its own persistent surface?
• Which first three workflows produce enough value to justify conversation?
• Which actions may execute immediately, which require preview, and which must remain unavailable?
• Should conversations persist across sessions or remain ephemeral by default?
• What transcript and audio retention policy is acceptable for agency and client data?
• Is a distinctive branded voice worth operating GPU infrastructure?
• Should aegi offer one brand voice, multiple voices, or device-native speech by default?
• Which provider and region satisfy latency, privacy, data-processing, and commercial requirements?
• What evidence should move this concept from deferred research into the active roadmap?