Rate Limit Exceeded for Big Data Analysis

[gradylippo]

Hello, I wanted to create a crew for a big data analysis task.

2024-11-01 13:34:23,459 - 15684 - llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.RateLimitError: RateLimitError: OpenAIException - Error code: 429 - {'error': {'message': 'Request too large for chatgpt-4o-latest in organization org-SQ1gR5fZgSi6WqepMuK9N7W8 on tokens per min (TPM): Limit 500000, Requested 2475027. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}} 2024-11-01 13:34:49,425 - 15684 - llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.RateLimitError: RateLimitError: OpenAIException - Error code: 429 - {'error': {'message': 'Request too large for chatgpt-4o-latest in organization org-SQ1gR5fZgSi6WqepMuK9N7W8 on tokens per min (TPM): Limit 500000, Requested 2475025. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

When I was running the crew, it finally gave out after a few minutes with the above error message. Rate limit exceeded.

Simply put, this task is pretty straightforward - the crew will have access to a directory containing multiple .csv files, in which the process is pretty straightforward:

• Given a business problem
• Given full context what does each file in the directory contain, and what does each column mean in these files
• Please try to answer the business problem based on the data provided, and write your analysis to a .md file

I use DirectoryReadTool, FileReadTool, CodeInterpreterTool, and they're working fine. I am ale to run the crew just fine.

The problem is, each .csv file contains 10000++ rows, and using CodeInterpreter usually prints out EVERYTHING, all rows, and this makes the running process extremely long, and uses a lot of tokens. It definitely does not need to print out EVERY row that it sees, right? Or is this the limitation?

So, when our files contain tens of thousands of rows, is there a way to be able to use CrewAI framework for such task?

Thank you!

Again, the code is working, and no bugs are discovered (yet), it's just that I feel CodeInterpreterTool's behavior of printing out everything makes the process quite exhaustive and expensive even for a single run.

Any improvements and critiques would be welcome.

[gradylippo]

Error:

2024-11-01 13:34:23,459 - 15684 - llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.RateLimitError: RateLimitError: OpenAIException - Error code: 429 - {'error': {'message': 'Request too large for chatgpt-4o-latest in organization org-SQ1gR5fZgSi6WqepMuK9N7W8 on tokens per min (TPM): Limit 500000, Requested 2475027. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}} 2024-11-01 13:34:49,425 - 15684 - llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.RateLimitError: RateLimitError: OpenAIException - Error code: 429 - {'error': {'message': 'Request too large for chatgpt-4o-latest in organization org-SQ1gR5fZgSi6WqepMuK9N7W8 on tokens per min (TPM): Limit 500000, Requested 2475025. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

[allina9686]

To optimize your process and reduce token usage, consider configuring CodeInterpreterTool to show only data summaries instead of printing every row, which should give you the overview you need without overwhelming output. Sampling the data instead of processing full datasets can also help; working with a smaller, representative portion will still provide meaningful insights while using fewer resources. Disabling print statements for raw data or any intermediate steps you don’t need to see can further cut down on token usage—try redirecting detailed outputs to a file if you might need them later. When writing to the .md file, focus on summarizing only key findings rather than including all details, which will keep the final output concise and easy to read. These small adjustments should make CrewAI more efficient with large datasets and prevent rate limits from interrupting your workflow.