Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python config/library API #51

Closed
lpapp-foundry opened this issue Jun 21, 2023 · 4 comments · Fixed by #128
Closed

Python config/library API #51

lpapp-foundry opened this issue Jun 21, 2023 · 4 comments · Fixed by #128
Assignees
Milestone

Comments

@lpapp-foundry
Copy link

lpapp-foundry commented Jun 21, 2023

What

Allow users to configure the BAL library using a Python API to simplify the recreation of known states as part of automated testing.

Why

Presently, BAL can only be configured using a JSON library file. This causes several problems with the authoring of automated tests within host integrations:

  • It is difficult to single-source inputs and expected, values without authoring the JSON file in the test case and re-initializing the manager.
  • It requires additional side-car data files to accompany the tests which may not be desirable.

Notes

  • JSON should still be supported for other uses of BAL.
  • Workflows need careful consideration, including:
    • Run-time manipulation of BAL in a test case.
    • Drop-in replacement for the JSON file via declarative Python.
    • Authoring of JSON at scale via Python.
    • Can we leverage traitgen classes as part of this?

Acceptance Criteria

  • BAL has an additional settings key called "library_json", which contains the serialized database.
  • The JSON is returned from settings() and can be set through initialize()
  • Implicit interpolation variables, e.g. ${bal_library_dir} etc, are not provided. If required then they must be passed as any other user-configured "variables" entry in the library.
  • If provided,"library_json" takes precedence over "library_path".
  • A blank "library_json" reverts the library to the library stored at "library_path", if available.
@foundrytom foundrytom changed the title Add Python API for setting the data Python config/library API Jun 21, 2023
@foundrytom
Copy link
Contributor

foundrytom commented Jun 21, 2023

A suggested sketch of one way this could be used:

@pytest.fixture
def bal() -> Bal:
    return Bal.instance()

def test_when_evaluated_then_location_is_urldecoded_and_path_extracted(read, bal: Bal):
  
    resolved_value = "file:///a/path%20with%20spaces/some.%23%23%23%23.ext"
    expected_value = "/a/path with spaces/some.####.ext"

    with bal.scoped_entities() as library:

        entity = library
            .add_entity("aComplexPath")
            .locatable_content(resolved_value)

        read["file"].setValue(entity.reference)
        assert read['file'].getEvaluatedValue() == expected_value

@feltech
Copy link
Member

feltech commented May 21, 2024

There is a simpler half-way-house to consider, whereby we add some mechanism within BAL to be able to alter its in-memory JSON database at runtime as a JSON document (as opposed to an imperative API).

@feltech
Copy link
Member

feltech commented Aug 22, 2024

Riffing off past me... the settings dict could contain a serialised JSON database, as an alternative to a path to a JSON file.

This means the host could get the current state of the database (as serialised JSON) from settings(). The host could then mutate, re-serialise, and re-initialize(...) with the updated database.

Pros:

  • (Probably) trivial to implement.
  • No need to import/expose BAL internals.
  • Works in C++ too since its using the existing OpenAssetIO bindings.
  • Allows BAL database to be defined as a string value in an OpenAssetIO .toml config file, rather than in a separate file.

Cons:

  • Serialisation/deserialsation to/from JSON. The JSON has to be serialized/deserialised since InfoDictionary is a subset of JSON functionality (e.g no nested objects).
  • C++ standard library cannot natively parse JSON.
  • Need to rethink ${bal_library_dir} -type variable interpolation.

@feltech
Copy link
Member

feltech commented Sep 17, 2024

Expanded acceptance criteria based on findings during implementation.

feltech added a commit to feltech/OpenAssetIO-Manager-BAL that referenced this issue Sep 17, 2024
Closes OpenAssetIO#51.

Presently, BAL can only be configured using a JSON library file. This
causes several problems with the authoring of automated tests within
host integrations:

* It is difficult to single-source inputs and expected values without
  authoring a JSON file in the test case.
* It requires additional side-car data files to accompany the tests
  which may not be desirable.

So add a new setting, `"library_json"`. This allows hosts to query for
the current internal BAL library via `settings()`, and then mutate it
and update BAL via a re-`initialize()`.

The JSON must be passed from/to the manager in serialised form, since
the settings dictionary must be coerced to/from a (C++)
`InfoDictionary`, which is a simple key-value map, rather than an
arbitrarily nested dictionary.

In addition, calculated `"variables"` (e.g. `"bal_library_dir_url"`)
cannot be provided when the library is given as a JSON string, since
they are based on the location of the `library_path`.

If both `library_path` and `library_json` are provided, then
`library_json` takes precedence. This allows a JSON library file to be
provided initially, then the library mutated during test cases.

Recall that the OpenAssetIO `initialize` method can accept partial
settings updates, meaning the manager retains any previous settings that
are not explicitly overridden in the new settings dict. However, this
does not apply to the special `library_json` setting here. This is a bit
of an abuse for convenience. E.g. if we re-initialize with an empty
settings dict, the initialization process falls back to using the
previously-set `library_path` (if available) and resets the library to
match the file.

This compromise is somewhat justified by analogy to current behaviour
when BAL is re-initialised after entities have been published. In this
case the new entities only exist in memory, and are lost when
re-initialising, since the library JSON file is re-read and overwrites
any changes.

Signed-off-by: David Feltell <david.feltell@foundry.com>
feltech added a commit to feltech/OpenAssetIO-Manager-BAL that referenced this issue Sep 17, 2024
Closes OpenAssetIO#51.

Presently, BAL can only be configured using a JSON library file. This
causes several problems with the authoring of automated tests within
host integrations:

* It is difficult to single-source inputs and expected values without
  authoring a JSON file in the test case.
* It requires additional side-car data files to accompany the tests
  which may not be desirable.

So add a new setting, `"library_json"`. This allows hosts to query for
the current internal BAL library via `settings()`, and then mutate it
and update BAL via a re-`initialize()`.

The JSON must be passed from/to the manager in serialised form, since
the settings dictionary must be coerced to/from a (C++)
`InfoDictionary`, which is a simple key-value map, rather than an
arbitrarily nested dictionary.

In addition, calculated `"variables"` (e.g. `"bal_library_dir_url"`)
cannot be provided when the library is given as a JSON string, since
they are based on the location of the `library_path`.

If both `library_path` and `library_json` are provided, then
`library_json` takes precedence. This allows a JSON library file to be
provided initially, then the library mutated during test cases.

Recall that the OpenAssetIO `initialize` method can accept partial
settings updates, meaning the manager retains any previous settings that
are not explicitly overridden in the new settings dict. However, this
does not apply to the special `library_json` setting here. This is a bit
of an abuse for convenience. E.g. if we re-initialize with an empty
settings dict, the initialization process falls back to using the
previously-set `library_path` (if available) and resets the library to
match the file.

This compromise is somewhat justified by analogy to current behaviour
when BAL is re-initialised after entities have been published. In this
case the new entities only exist in memory, and are lost when
re-initialising, since the library JSON file is re-read and overwrites
any changes.

Signed-off-by: David Feltell <david.feltell@foundry.com>
feltech added a commit to feltech/OpenAssetIO-Manager-BAL that referenced this issue Sep 17, 2024
Closes OpenAssetIO#51.

Presently, BAL can only be configured using a JSON library file. This
causes several problems with the authoring of automated tests within
host integrations:

* It is difficult to single-source inputs and expected values without
  authoring a JSON file in the test case.
* It requires additional side-car data files to accompany the tests
  which may not be desirable.

So add a new setting, `"library_json"`. This allows hosts to query for
the current internal BAL library via `settings()`, and then mutate it
and update BAL via a re-`initialize()`.

The JSON must be passed from/to the manager in serialised form, since
the settings dictionary must be coerced to/from a (C++)
`InfoDictionary`, which is a simple key-value map, rather than an
arbitrarily nested dictionary.

In addition, calculated `"variables"` (e.g. `"bal_library_dir_url"`)
cannot be provided when the library is given as a JSON string, since
they are based on the location of the `library_path`.

If both `library_path` and `library_json` are provided, then
`library_json` takes precedence. This allows a JSON library file to be
provided initially, then the library mutated during test cases.

Recall that the OpenAssetIO `initialize` method can accept partial
settings updates, meaning the manager retains any previous settings that
are not explicitly overridden in the new settings dict. However, this
does not apply to the special `library_json` setting here. This is a bit
of an abuse for convenience. E.g. if we re-initialize with an empty
settings dict, the initialization process falls back to using the
previously-set `library_path` (if available) and resets the library to
match the file.

This compromise is somewhat justified by analogy to current behaviour
when BAL is re-initialised after entities have been published. In this
case the new entities only exist in memory, and are lost when
re-initialising, since the library JSON file is re-read and overwrites
any changes.

Signed-off-by: David Feltell <david.feltell@foundry.com>
feltech added a commit to feltech/OpenAssetIO-Manager-BAL that referenced this issue Sep 18, 2024
Closes OpenAssetIO#51.

Presently, BAL can only be configured using a JSON library file. This
causes several problems with the authoring of automated tests within
host integrations:

* It is difficult to single-source inputs and expected values without
  authoring a JSON file in the test case.
* It requires additional side-car data files to accompany the tests
  which may not be desirable.

So add a new setting, `"library_json"`. This allows hosts to query for
the current internal BAL library via `settings()`, and then mutate it
and update BAL via a re-`initialize()`.

The JSON must be passed from/to the manager in serialised form, since
the settings dictionary must be coerced to/from a (C++)
`InfoDictionary`, which is a simple key-value map, rather than an
arbitrarily nested dictionary.

In addition, calculated `"variables"` (e.g. `"bal_library_dir_url"`)
cannot be provided when the library is given as a JSON string, since
they are based on the location of the `library_path`.

If both `library_path` and `library_json` are provided, then
`library_json` takes precedence. This allows a JSON library file to be
provided initially, then the library mutated during test cases.

Recall that the OpenAssetIO `initialize` method can accept partial
settings updates, meaning the manager retains any previous settings that
are not explicitly overridden in the new settings dict. However, this
does not apply to the special `library_json` setting here. This is a bit
of an abuse for convenience. E.g. if we re-initialize with an empty
settings dict, the initialization process falls back to using the
previously-set `library_path` (if available) and resets the library to
match the file.

This compromise is somewhat justified by analogy to current behaviour
when BAL is re-initialised after entities have been published. In this
case the new entities only exist in memory, and are lost when
re-initialising, since the library JSON file is re-read and overwrites
any changes. This is similar to how the `library_json` changes are lost
by default when re-initializing.

Signed-off-by: David Feltell <david.feltell@foundry.com>
@feltech feltech self-assigned this Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants