Question on interchange format data reading

[rgieseke]

I was trying to read the RCMIP emissions into the PRIMAP2 interchange format as a first step:

Available from https://gitlab.com/rcmip/rcmip/-/blob/master/data/protocol/rcmip-emissions-annual-means-v5-1-0.csv
or download https://gitlab.com/rcmip/rcmip/-/raw/master/data/protocol/rcmip-emissions-annual-means-v5-1-0.csv?inline=false

The below works but i couldn't get it to read category (commented out) and entity to be read at the same time.
I build a dictionary from the CSV e.g. to map Emissions|CO2 to CO2.

Is this the right way in general? Maybe i'm also missing something conceptually.

df = pd.read_csv("rcmip-emissions-annual-means-v5-1-0.csv")
df["Entity"] = df.Unit.apply(lambda x: x.split(" ")[1].split("/")[0])
map_variables = df[["Variable", "Entity"]].set_index("Variable")["Entity"].to_dict()

file = "rcmip-emissions-annual-means-v5-1-0.csv"
coords_cols = {
    "unit": "Unit",
    "area": "Region",
    "model": "Model",
    "scenario": "Scenario",
    "entity": "Variable",
    #"category": "Variable"
}
coords_defaults = {
    "source": "RCMIP",
}
coords_terminologies = {
    "area": "RCMIP",
    "category": "RCMIP",
}
coords_value_mapping = {
    "entity": map_variables
}
meta_data = {
    "url": "https://doi.org/10.5194/gmd-13-5175-2020",
    "rights": "CC BY 4.0 International",
}
data_if = pm2.pm2io.read_wide_csv_file_if(
    file,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
    coords_terminologies=coords_terminologies,
    coords_value_mapping=coords_value_mapping,
    meta_data=meta_data,
    filter_keep={"f1": {
        "Model": "CEDS/UVA/GCP/PRIMAP",
    }}
)
data_if

[mikapfl]

Hi Robert,

Unfortunately, read_wide_csv_file_if doesn't support re-using a source column multiple times in coords_cols because behind the scenes it just does a renaming. Probably something it should support, so maybe worth opening a bug report - but I can't commit to when I will have time to fix it.

Meanwhile, you can pre-process the dataframe yourself and use convert_wide_dataframe_if directly. This does less checking of arguments etc., so error messages might be weirder, but it works with surprisingly little changes to your code since you are reading the dataframe anyway:

df = pd.read_csv("rcmip-emissions-annual-means-v5-1-0.csv")
df["Entity"] = df.Unit.apply(lambda x: x.split(" ")[1].split("/")[0])

coords_cols = {
    "unit": "Unit",
    "area": "Region",
    "model": "Model",
    "scenario": "Scenario",
    "entity": "Entity",
    "category": "Variable"
}
coords_defaults = {
    "source": "RCMIP",
}
coords_terminologies = {
    "area": "RCMIP",
    "category": "RCMIP",
}
meta_data = {
    "url": "https://doi.org/10.5194/gmd-13-5175-2020",
    "rights": "CC BY 4.0 International",
}
data_if = pm2.pm2io.convert_wide_dataframe_if(
    df,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
    coords_terminologies=coords_terminologies,
    meta_data=meta_data,
    filter_keep={"f1": {
        "Model": "CEDS/UVA/GCP/PRIMAP",
    }}
)
data_if

Of course, I would do some processing on category before using the data, if you convert it as is into the primap2 native xarray format, you might run out of memory storing all those NaNs for entity CO2 and Category Emissions|CH4.

Cheers,

Mika

[mikapfl]

Is this the right way in general? Maybe i'm also missing something conceptually.

That's one way to do it. If you already have the data in wide format, it is probably easiest to go via the interchange format because we have reading routines there. If you have data in some other format and have to roll your own reading function, I would also consider building the native xarray format directly, because it is more expressive and you can use xarray's toolbox immediately.

[rgieseke]

Thanks for the quick reply, Mika! (Maybe you could enable "Discussions" on this repo, for questions like this.)

For the record the following works

df = pd.read_csv("rcmip-emissions-annual-means-v5-1-0.csv")
df = df.drop(["Activity_Id", "Mip_Era"], axis=1)
df["Entity"] = df.Unit.apply(lambda x: x.split(" ")[1].split("/")[0])

coords_cols = {
    "unit": "Unit",
    "area": "Region",
    "model": "Model",
    "scenario": "Scenario",
    "entity": "Entity",
    "category": "Variable"
}
coords_defaults = {
    "source": "RCMIP",
}
coords_terminologies = {
    "area": "RCMIP",
    "category": "RCMIP",
    "scenario": "RCMIP"
}
meta_data = {
    "references": "doi:10.5194/gmd-13-5175-2020",
    "rights": "CC BY 4.0 International",
}
data_if = pm2.pm2io.convert_wide_dataframe_if(
    df,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
    coords_terminologies=coords_terminologies,
    meta_data=meta_data,
    filter_keep={"f1": {
        "Model": "CEDS/UVA/GCP/PRIMAP",
        "Region ": "World"
    }}
)

rcmip = pm2.pm2io.from_interchange_format(data_if)

As you noted

rcmip["CH4"].loc[{"category (RCMIP)":"Emissions|CO2"}]

and the like will be nan - but i'm not sure what you mean by processing the category?

[mikapfl]

(Maybe you could enable "Discussions" on this repo, for questions like this.)

I've never used it, why is it better than issues?

and the like will be nan - but i'm not sure what you mean by processing the category?

If you want to use this data with primap2, then it would be good to properly separate the dimensions entity and category, so Emissions|CO2 and Emissions|CH4 should both be translated to Emissions. That gives you efficient memory use and easy filtering in both dimensions without string processing (e.g. you can efficiently answer the question "give me all Emissions of Germany in 2015" without doing any category.str.split('|') calls). Of course, you can't use the RCMIP categorization then, and have to use something custom instead.

[rgieseke]

Got it, that's where we started with the RCMIP climate categories discussion, thanks!

[rgieseke]

I've never used it, why is it better than issues?

It feels a bit more light-weight than an actual 'issue' (one can also move issues to discussions) and i guess a bit like on Stackoverflow one can select an answer as "the right one".