Retrying on 429 obscures OpenAI quota limit

[jlewi]

It looks like you will get a 429 from OpenAI if you hit the quota limit.
This would normally return an openai.APIError with code "insufficent_quota" and value
"You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors."

However since we are using hashicorp's retryable client. It keeps retrying until it exhausts the retry.
It then returns a Url.Error with message
POST https://api.openai.com/v1/embeddings giving up after 5 attempt(s)

So our retryable client is swallowing the openai Error which contains more informative information.
We should probably customize the retry function to not retry in that case.

[jlewi]

I think retrying on 429 is the right thing to do because in some cases quota might replenish.

I think what we might want to do is log the 429 in case that contains additional information. Do we have to log each retry or is there someway to get the final error when the client gives up.