Problem with the estimation of carbon intensity using CarbonTracker

[Teamsusai]

In the frame of a project aiming to evaluate the environmental impact of various AI algorithms, I am using the library CarbonTracker (in its version 1.1.6) in combination with Google Colab.
After a lot of numerical experiments, it seems that the ratio between the emissions of CO2eq and consumed energy, as reported in the log file ..._carbontracker_output.log is always equal to 0.29421 kgCO2 / kWh, whatever the location of the computation (some computations were made in the US, others in Netherlands or in Taiwan).
The library is invoked in the following manner:

TRACKER = CarbonTracker(epochs=EPOCH,
epochs_before_pred=-1,
monitor_epochs=-1,
update_interval=1,
log_dir=TRACKER_PATH,
verbose=1)

Are these results corresponding to the expected behaviour?
The same experiments evaluated with the library CodeCarbon lead to a different carbon intensity with respect to the location, which turns out to be constant for a given region.

[kanding]

Hello Teamsusai,

Thanks for trying out the tool and taking the time to write an issue!

In order to estimate CO2eq emissions accurately we need to know what the carbon intensity is in the region which is executing your workload. When you run the tracker it will look at the geolocation and attempt to find a suitable 'fetcher' to get the necessary information to estimate the CO2eq emissions. ( detailed info: (https://github.com/lfwa/carbontracker/blob/master/CONTRIBUTING.md )

The energy consumption readings are naturally unaffected by this as we simply measure the hardware.

I can imagine what happens is that your workload was executed in locations from which we do not currently have a suitable fetcher. If this is the case the tracker will fall-back to a default 'average european emissions' value (which should? be reported with most verbose settings IIRC) and that would be why the reported emissions are similar

I can see why it might be deceiving, which is why I hope it was reported by the tracker with verbose=1. If it is not present at the top of the log it is something that we should look into adding. If you have access to the carbon intensity of the regions you want to measure you can try looking into adding a fetcher for that region (see above contributing link)

I am not familiar with from where CodeCarbon fetches their carbon intensities worldwide. It is something that we would've loved to do but during development of this tool we were unable to find any cheap or freely available APIs with worldwide carbon intensity values

[Teamsusai]

Hello,

Thank you very much for your reply. We confirm that the verbose parameter is set to 1 in our calculations.
It seems that we do not get any warning stating that a default value is fetched.
Instead the following logs are obtained for example:
2021-08-26 11:27:26 - carbontracker version 1.1.6
2021-08-26 11:27:26 - Only predicted and actual consumptions are multiplied by a PUE coefficient of 1.59 (Rhonda Ascierto, 2019, Uptime Institute Global Data Center Survey).
2021-08-26 11:27:26 - The following components were found: GPU with device(s) Tesla K80.
2021-08-26 11:27:26 - Monitoring thread started.
...
2021-09-09 09:58:43 - Training was interrupted before all 53 epochs were monitored.
2021-09-09 09:58:43 - Carbon intensities (gCO2/kWh) fetched every 900 s at detected location Council Bluffs, Iowa, US: [294.2060978]
2021-09-09 09:58:43 - Average carbon intensity during training was 294.21 gCO2/kWh at detected location: Council Bluffs, Iowa, US.
2021-09-09 09:58:44 - Monitoring thread ended.

Another calculation that takes place in Oregon results in the same value :
2021-09-13 10:12:12 - Carbon intensities (gCO2/kWh) fetched every 900 s at detected location The Dalles, Oregon, US: [294.2060978]

Moreover, our experiments span the following regions:
Belgium : brussels capital
Netherlands : groningen
Taiwan : new taipei
United States : district of columbia / iowa / nevada / oregon

[kanding]

It confirms that the tracker is using the default value, which is naturally not very accurate for the locations in which you are training. There are no fetchers currently for the United States since we do not have access to a free and accurate API with US real-time carbon intensity information.

It should be reported by the tracker in standard out that it could not find a suitable fetcher and used the default value, but it appears to be missing from the log file.
This is definitely an error. We will look into reporting it in the log file as well such that there is no confusion!

Thanks for reporting this.

[PedramBakh]

This issue has been addressed in Release v1.2.0. We've enhanced the Carbon Intensity Estimation notifications, ensuring users are alerted when default fallback values are used. Notifications will appear in both the log file and stdout.