AnyCable x Twilio Media Streams x OpenAI Realtime

This application demonstrates how to use AnyCable to interact with voice calls via Twilio Streams and OpenAI Realtime API.

The app consists of two parts:

• A Ruby on Rails application that provides a web UI to ToDo items.
• A Go application built with AnyCable that handles Twilio Streams and OpenAI interactions in a logic-agnostic way.

The RoR app is the primary (and only) source of truth providing credentials, instructions,and tools for the Go application (i.e., Twilio and OpenAI). Check out the app/channels/twilio/media_stream_channel.rb file to see the voice UX implementation.

Requirements

• Ruby 3.3 for the application server.
• Go 1.23 to run AnyCable.
• Twilio and OpenAI accounts (see below).

Configuration

Twilio / OpenAI setup

You must obtain a phone number from Twilio as well as your account credentials. Similarly, generate an API key for OpenAI.

Then, you can store them in the local configuration files as follows:

# config/twilio.local.yml`
phone_number: <you phone number>
account_sid: <your account SID>
auth_token: <your auth token>

# config/openai.local.yml
api_key: <your key>

Alternatively, you can use the corresponding environment variables (TWILIO_PHONE_NUMBER, TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, OPENAI_API_KEY) or local credentials.

Running the app

The web app is built with Ruby on Rails. To run it, you need to install the dependencies:

bin/setup

The command above also builds the AnyCable projects stored at the cable/ directory.

Now, you can run it locally as follows:

bin/dev

Then, you can visit the app at localhost:3000. You can manage ToDo items there.

To interact with the app using a phone, you must make the realtime server (running on the :8080) by default to be accessible from the Internet as well as your Rails server. You can use ngrok for that:

# for AnyCable server
ngrok http 8080

Use the generated URL for 8080 as the TWILIO_STREAM_CALLBACK environment variable:

TWILIO_STREAM_CALLBACK=https://<your-ngrok-id>.ngrok.io bin/dev

(You can also add the url to the config/twilio.local.yml file).

Now, you can initiate a call to the provided phone number running the following Rake command:

TWILIO_STREAM_CALLBACK=https://<your-ngrok-id>.ngrok.io  \
  bin/rails "twilio:call[+12344442222]"

Then, you can interact with an AI agent and give it some instructions:

• What's on my list for today?
• What's my plans for the weekend?
• Create a "buy milk" task for tomorrow.

Initiating calls from your phone

To call your Twilio number and interact with the AI agent, you MUST configure a status callback webhook for the phone number pointing to another Ngrok tunnel (for port 3000):

ngrok http 3000

Then, use the TWILIO_STATUS_CALLBACK env var to provide the webhook URL to the application:

TWILIO_STATUS_CALLBACK=https://<your-rails-ngrok-id>.ngrok.io \
 TWILIO_STREAM_CALLBACK=https://<your-cable-ngrok-id>.ngrok.io \
 bin/dev

Calls monitoring

Go to the localhost:3000/phone_calls to see some live logs of your calls.