Refactor Databricks-AWS Qual tool to cache and process pricing info from DB website

[cindyyuanjiang]

Fixes #1139.

Changes

• Refactored DatabricksAWSPriceProvider to cache and use pricing info from DB official website instead of outdated static file
• Removed now stale databricks-premium-catalog.json file under resources
• Fixed a previous DBU cost calculation bug

Testing

spark_rapids qualification --eventlogs <my-event-logs> --platform databricks-aws --cluster <my-cluster-props>

Run the cmd above and confirm the pricing calculation is the same before and after this PR.

[amahussein]

Thanks @cindyyuanjiang !
That's good that we have finally a catalog file to use for databricks.

Have you tested the fat-mode build? We need to make sure that these changes work fine when the tools are running offline.

[amahussein]

2024 copyrights.

[cindyyuanjiang]

fixed, thanks!

[cindyyuanjiang]

Have you tested the fat-mode build? We need to make sure that these changes work fine when the tools are running offline.

Thanks @amahussein! Tested the fat-mode build successfully.

[amahussein]

LGTM
Thanks @cindyyuanjiang

[amahussein]

@cindyyuanjiang
General comment on styling: we are trying to enforce defining the return type of each function as much as possible.
This is going to hunt us back with pylint moving forward which will cause the code to fail all pylint checks. For function returning nothing, then it is recommended to define it as def foo() -> None:

[cindyyuanjiang]

@cindyyuanjiang General comment on styling: we are trying to enforce defining the return type of each function as much as possible. This is going to hunt us back with pylint moving forward which will cause the code to fail all pylint checks. For function returning nothing, then it is recommended to define it as def foo() -> None:

Thanks @amahussein! Updated function return types.

[amahussein]

Thanks @cindyyuanjiang !
LGTME