Initial commit

[knikolla]

• Implemented library for loading rates.yaml
• Wrote simple tests for library
• Updated README with description
• Made the package installable
• Updated current rates

[larsks]

I am a little bothered by the structure of rates.yaml. We're storing different data types in the same attribute, and we're storing different types of "objects" in the same list. Here, value is a float:

- name: CPU SU Rate
  history:
    - value: 0.013
      from: 2023-06

But here it's a boolean (although this should technically be spelled false):

- name: Charge for Stopped Instances
  history:
    - value: False
      from: 2023-06
      until: 2024-02

And later on it's an integer.

Is that ever going to bite us? It certainly makes the file difficult to validate, and difficult to parse in languages with strict typing. We're cheating a bit in the code by using yaml.BaseLoader, which returns everything as strings.

I wonder if we should split the file into sections:

rates:
- name: CPU SU Rate
  history:
    - value: 0.013
      from: 2023-06


feature_flags:
- name: Charge for Stopped Instances
  history:
    - value: false
      from: 2023-06
      until: 2024-02
    - value: true
      from: 2024-03

definitions:
- name: VCPUs in CPU SU
  history:
    - value: 1
      from: 2023-06

That would allow us to iterate over a thing (like rates) and know what to expect in the value field.

Maybe the content of rates.yaml is entirely internal to this code, and rather than YAML format it's actually "nerc-rates-YAML" format, and we don't expect it to be parsed by anything else, in which case none of the above concerns are relevant.

[knikolla]

I am a little bothered by the structure of rates.yaml. We're storing different data types in the same attribute, and we're storing different types of "objects" in the same list. Here, value is a float:

- name: CPU SU Rate
  history:
    - value: 0.013
      from: 2023-06

But here it's a boolean (although this should technically be spelled false):

- name: Charge for Stopped Instances
  history:
    - value: False
      from: 2023-06
      until: 2024-02

And later on it's an integer.

Is that ever going to bite us? It certainly makes the file difficult to validate, and difficult to parse in languages with strict typing. We're cheating a bit in the code by using yaml.BaseLoader, which returns everything as strings.

I wonder if we should split the file into sections:

rates:
- name: CPU SU Rate
  history:
    - value: 0.013
      from: 2023-06


feature_flags:
- name: Charge for Stopped Instances
  history:
    - value: false
      from: 2023-06
      until: 2024-02
    - value: true
      from: 2024-03

definitions:
- name: VCPUs in CPU SU
  history:
    - value: 1
      from: 2023-06

That would allow us to iterate over a thing (like rates) and know what to expect in the value field.

Maybe the content of rates.yaml is entirely internal to this code, and rather than YAML format it's actually "nerc-rates-YAML" format, and we don't expect it to be parsed by anything else, in which case none of the above concerns are relevant.

@larsks Yeah, I was bothered by the inconsistent value types too, hence in the code I used the BaseLoader to keep everything as strings and let the downstream code accessing the specific values handle the conversion since it would know what it is trying to access.

Reading your comment, I see the ambiguity that that may lead to tools that are attempting to parse the YAML directly without using the python library, since those values appear to be floats, whereas the nerc-rates-YAML format treats everything as a string.

I tried an approach that divided things into sections initially, but I found that a better approach was to have a type field in each of the objects that defined the type stored in values (Decimal, Integer, Boolean), since the type of value is a clearer categorization of the value that is returned as opposed to the semantic meaning of the value.

I can reintroduce the type attribute and handle the conversion in the library, or I can just add " to all values and be explicit about their being strings.

Do you have a preference?

[larsks]

I can reintroduce the type attribute and handle the conversion in the library, or I can just add " to all values and be explicit about their being strings.

Do you have a preference?

I don't think a type attribute is the right way to go, because again that complicates validation (now we need a discriminator of some sort to identify expected value types).

I guess my preference in the short term would be to quote everything if it's expected to be a string, but I would combine that with validation in the code.

[larsks]

...but I would combine that with validation in the code.

E.g., like: https://github.com/larsks/nerc-rates/tree/feature/pydantic (which also drops setup.*).

The interesting bit is probably models.py (and the corresponding changes in rates.py).

[larsks]

@knikolla I've pushed changes that add support for validation using pydantic, along with specific validations for the ordering between from and until and for overlap between values in a history list.

[larsks]

It looks like typing.Self was introduced in python 3.11, and we're running ci with 3.10. Fortunately, we can use typing_extensions.Self in either case.