Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Broken Audio After Agent Handoff During Twilio Call Using TwilioRealtimeTransportLayer and RealtimeAgent #122

Open
@John-PaulRA

Description

@John-PaulRA

Describe the bug

During a Twilio call, When a handoff is triggered from a RealtimeAgent to another agent using handoffs: [billingAgent, refundAgent], the new agent's voice becomes broken — similar to static noise or a corrupted audio stream. After this, the agent also stops responding entirely.

The issue occurs even though:

  • The agent_handoff event fires with the correct arguments.
  • The response from the new agent does appear correctly (clean audio) in the Traces dashboard.

This suggests the issue may be related to an audio format or transport inconsistency between Twilio and the OpenAI SDK after a handoff.

I was able to replicate this issue in both:

  • My original project setup
  • A minimal setup using the examples/realtime-twilio code from this repo, with agent handoff added

Debug information

  • Agents SDK version: v0.0.8
  • Node.js version: v22.13.0

Repro steps

My sample code:

import Fastify from 'fastify';
import dotenv from 'dotenv';
import fastifyFormBody from '@fastify/formbody';
import fastifyWs from '@fastify/websocket';
import { RealtimeAgent, RealtimeSession } from '@openai/agents/realtime';
import { TwilioRealtimeTransportLayer } from '@openai/agents-extensions';
import { RECOMMENDED_PROMPT_PREFIX } from '@openai/agents-core/extensions';

dotenv.config();
const { OPENAI_API_KEY } = process.env;

const fastify = Fastify();
fastify.register(fastifyFormBody);
fastify.register(fastifyWs);

const billingAgent = new RealtimeAgent({ name: 'Billing agent' });
const refundAgent = new RealtimeAgent({ 
  name: 'Refund agent',
  instructions: `${RECOMMENDED_PROMPT_PREFIX}\nYou are Sami a helpful assistant that helps users with refunds.`,
});

const agent = new RealtimeAgent({
  name: 'Triage Agent',
  instructions: 'You are a helpful and friendly triage assistant. Begin each conversation with a creative and engaging greeting. Based on the user\'s request, determine whether their issue is related to billing or refunds. If it is, use the appropriate tool to hand off the conversation to the Billing Agent or the Refund Agent so they can assist further.',
  handoffs: [billingAgent, refundAgent]
});

fastify.all('/twiml', async (request, reply) => {
  const twiml = `
    <Response>
      <Say>O.K. you can start talking!</Say>
      <Connect>
        <Stream url="wss://${request.headers.host}/media-stream" />
      </Connect>
    </Response>`.trim();
  reply.type('text/xml').send(twiml);
});

fastify.register(async (fastify) => {
  fastify.get('/media-stream', { websocket: true }, async (connection) => {
    const twilioTransportLayer = new TwilioRealtimeTransportLayer({
      twilioWebSocket: connection,
    });

    const session = new RealtimeSession(agent, {
      transport: twilioTransportLayer,
    });

    await session.connect({ apiKey: OPENAI_API_KEY });

    session.on('agent_handoff', (_, currentAgent, newAgent) => {
      console.log(`Handoff from ${currentAgent.name} to ${newAgent.name}`);
    });
  });
});

fastify.listen({ port: 5050 });

Expected behavior

  • When a handoff occurs (e.g., from Triage Agent to Refund Agent), the new agent should begin speaking clearly.
  • The new agent should continue responding as expected.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions