Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds a TTS Benchmark Suite #117

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Adds a TTS Benchmark Suite #117

wants to merge 3 commits into from

Conversation

cezarc1
Copy link

@cezarc1 cezarc1 commented Aug 11, 2024

Adds a TTS benchmark suite for 11labs, cartesia and playht.

Uses raw websocket for 11labs, cartesia. For playht using the provided python sdk w/ GRPC as I could not figure out what the grpc address was.

Each service was configured to be as fast as possible, however each service's api's are unique so I might have missed something.

TTFU - I came up with this on my own. There is TTFB but I thought this was more specific? ¯_(ツ)_/¯.

Also updates the Readme, accordingly.

Example output:

poetry run python tts_benchmark_suite.py "It's simple: Overspecialize, and you breed in weakness. It's slow death."
17:45:03 - INFO - Benchmarking Eleven Labs - websocket...
17:45:06 - INFO - Eleven Labs - websocket: (TTFU) time to first utterance: 312.93ms
17:45:06 - INFO - Eleven Labs - websocket: Average chunk latency: 230.71ms
17:45:06 - INFO - Eleven Labs - websocket: Total chunks received: 12
17:45:06 - INFO - Eleven Labs - websocket: Total processing time: 2768.55ms
17:45:06 - INFO - ----------------------------------------
17:45:06 - INFO - Benchmarking Cartesia - websocket...
17:45:09 - INFO - Cartesia - websocket: (TTFU) time to first utterance: 187.41ms
17:45:09 - INFO - Cartesia - websocket: Average chunk latency: 58.56ms
17:45:09 - INFO - Cartesia - websocket: Total chunks received: 47
17:45:09 - INFO - Cartesia - websocket: Total processing time: 2752.38ms
17:45:09 - INFO - ----------------------------------------
17:45:09 - INFO - Benchmarking PlayHT - GRPC Python SDK...
17:45:11 - INFO - PlayHT - GRPC Python SDK: (TTFU) time to first utterance: 201.89ms
17:45:11 - INFO - PlayHT - GRPC Python SDK: Average chunk latency: 16.33ms
17:45:11 - INFO - PlayHT - GRPC Python SDK: Total chunks received: 122
17:45:11 - INFO - PlayHT - GRPC Python SDK: Total processing time: 1991.93ms
17:45:11 - INFO - ----------------------------------------

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant