Prompt Engineering for Voice: How to Stop Bots from Hallucinating on Live Calls

· Manuel · 10 min read · Innovation

If an AI text chatbot on a website hallucinates and tells a customer "Our product is 100% free forever," the user usually realizes it's an error. If an AI voice agent on a live phone call explicitly tells a CEO, "Yes, we guarantee a 500% ROI in 30 days or we will refund your 100k contract in full," you have a catastrophic legal liability on your hands.

Writing prompts for Voice LLMs requires a fundamentally stricter, adversarial approach compared to standard ChatGPT usage. You must architect "Defense-in-Depth."

Here is the exact prompt engineering methodology the GSD 500 team uses to keep voice bots surgically accurate and legally compliant.

1. The Death of the "Paragraph" Prompt

Most developers write system prompts like a novel: "You are a friendly sales person for ABC Corp. Your objective is to book a meeting. Be nice. Don't lie about prices." LLMs ignore paragraphs. As the context window fills up with the call transcript, the LLM suffers from "middle-blindness" and forgets vague instructions.

The Fix: We write Voice Prompts strictly utilizing XML markup and Boolean logic structures.

\`\`\`xml Role: SDR Tier 1 Personality: Urgent, professional, empathetic. Constraint: NEVER apologize more than once per call.

Product: BPO Automation Services Pricing: Custom quote ONLY. You CANNOT quote exact dollar amounts. \`\`\`

XML tags force the LLM to structurally categorize its own system instructions. It prevents instruction bleed.

2. The Power of "Negative Constraints"

Foundation models (like GPT-4o) are inherently people-pleasers (Helpful-Only alignment). If a prospect presses them aggressively, the bot's instinct is to agree to defuse the tension. Prospect: "Just tell me the price, is it less than 50 dollars?" Bad AI: "Yes, it is very affordable, likely in that range."

We implement heavy Negative Constraints at the very end of the prompt (the Recency Effect). Constraint List:

  • NEVER invent a price, statistic, or timeline.
  • IF asked a question you do not definitively know the answer to, YOU MUST output: "That's a great question, Manuel would actually be the best person to look at those specific metrics with you on the demo."
  • NEVER promise a specific ROI.

    • 3. Conditional State Handlers

    A voice conversation is highly dynamic. We use IF/THEN logic blocks within the prompt to guide the AI on how to handle deviations from the "Golden Path."

    Example Block: \`\`\` IF the user says "I do not have time": THEN acknowledge and pivot: "I completely understand, you're running a business. Let me just send an email with a 60-second video so you can look when you're free, is john@abc.com the best address?"

    IF the user is hostile or curses: THEN safely end the call: "I apologize for the interruption, have a great day." -> [EXECUTE HANGUP FUNCTION] \`\`\`

    4. Over-Writing for Spoken Cadence

    LLMs naturally write like Wikipedia. They use compound sentences and perfect grammar. That sounds robotic when spoken through TTS.

    We include explicit linguistic formatting rules in every prompt:

  • Rule 1: Keep responses under 25 words.
  • Rule 2: Use conversational fillers occasionally (e.g., "Got it," "Make sense," "For sure").
  • Rule 3: Ask one, and only one, question at the end of your turn to pass the conversational baton.

    • 5. Multi-Agent Shadow Grading

    Even with perfect prompting, extreme edge cases exist. For our highest-tier defense, we run a "Shadow Agent." While the main Voice Bot is talking, the live transcript is being fed silently to an incredibly fast secondary LLM (Claude Haiku). Haiku's only job is to act as a compliance auditor. It evaluates every sentence the main Voice Bot speaks against our master negative constraints list.

    If the main bot hallucinates, Haiku flags it instantly, killing the call or triggering an immediate forced correction string into the TTS engine ("Actually, forgive me, I misspoke on that last point...").

    Conclusion

    Voice AI is powerful because of its reasoning, but dangerous because of its creativity. Strong prompt engineering in a BPO environment is about putting the creative brain inside an adamantium box. You dictate the exact boundaries, and let the AI freely navigate only within that confined, compliant space. `,

    // ========================================== // SPANISH CONTENT // ========================================== contentEs: ` Si un bot de texto en un sitio web alucina y da un correo incorrecto, no pasa mucho. Si un agente de voz en una llamada en vivo le promete a un prospecto, "Garantizamos un retorno de inversión de 500% o le devolvemos sus $100k", tienes una demanda legal catastrófica.

    Escribir prompts (indicaciones) para Voz IA requiere un enfoque adversario estricto de "Defensa en Profundidad". Así es como en GSD 500 mantenemos a los bots precisos y sin alucinaciones.

    1. La Muerte del Prompt en "Párrafos"

    Los desarrolladores novatos escriben novelas en sus instrucciones: "Eres un vendedor amigable. Vende el producto. No mientas". Los LLMs olvidan esta estructura.

    Usamos marcado XML y lógica booleana estricta. Las etiquetas XML (e.g. \`\`) obligan al LLM a categorizar estructuralmente sus límites operativos.

    2. Restricciones Negativas Fuertes

    Los modelos de IA como GPT-4o están diseñados para ser hiper-complacientes. Si el cliente los presiona, su instinto es estar de acuerdo.

    Implementamos Restricciones Negativas fuertes al final del prompt:

  • NUNCA invente un precio.
  • NUNCA prometa un ROI específico.
  • SI hace una pregunta técnica que no está en la base de conocimientos, DEBE decir: "Buena pregunta, dejaré que nuestro gerente humano responda eso en la reunión."

    • 3. Manejadores de Estado Condicional

    Usamos bloques de lógica IF/THEN (SI/ENTONCES). SI el usuario insulta: ENTONCES responda "Disculpe la molestia" y EJECUTE la función de colgar la llamada.

    4. Reescribir para una Cadencia Hablada

    Los LLM escriben como la Wikipedia. Eso suena súper falso al pasarlo a audio y ser hablado. Para sonar natural, incluimos reglas lingüísticas:
  • Limita la respuesta a no más de 25 palabras.
  • Usa muletillas conversacionales (Ej: "Claro", "Entiendo", "Totalmente").
  • Haz una sola y única pregunta al finalizar cada turno para ceder la palabra.

    • 5. Auditoría con un Agente en las Sombras

    Incluso con el mejor prompt, ejecutamos un "Agente de Sombras". Mientras el bot principal habla, enviamos en secreto el texto de la llamada a un segundo modelo de IA más pequeño y ultrarrápido (Claude Haiku). Su único trabajo es auditar el cumplimiento. Si el bot principal dice una mentira o alucina, esta sombra lo detecta y corta la llamada o fuerza una disculpa.

    Controlar la creatividad de la Voz IA es poner el cerebro creativo en una caja de titanio para dejarlo maniobrar de manera segura.