Model costs in terms of energy for self hosted models (or cloud hosted)

[vquilon]

Describe the feature or potential improvement

Hi I just wondering to know if there is a possibility to give on each trace or general summary the cost calculating by energy usage. I know its impossible because it will need the real measure of energy consumption (maybe an external API call?). But maybe it can be affordable with an approximation.

For example in this leaderbord https://huggingface.co/spaces/optimum/llm-perf-leaderboard, there is a column of Energy that the model use (Token/kWh), I know its an proximation, but, maybe it will be a good feature, if we can configure our model costs with this option, not only in a token or character dimension.

Additional information

No response

Contribute

• Yes, I can implement this and raise a PR

[marcklingen]

For self-hosted models, I'd assume that energy usage is very closely related to the metrics you'd want to measure for USD cost.

We are looking into extending the LLM unit metric (currently only TOKENS or CHARACTERS) to other units. Is kwH the metrics you'd want to track?
Summary: generation -> kwH -> cost (USD)

Alternatively, kwH could be tracked independently from cost if this is something more people are interested in.

• generation -> usage (eg tokens) -> cost (USD)
• usage (eg tokens) -> kwH

What is your main motivation to track kwH?