Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added clear_resolver method to OmegaConf #770

Merged
merged 21 commits into from
Aug 5, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7ea9ae3
added clear_resolver method to OmegaConf
sugatoray Jul 19, 2021
46c1021
added tests and simplified some code
sugatoray Jul 20, 2021
9ed558d
Added test scenario for Error
sugatoray Jul 20, 2021
9812ce9
updated OmegaConf.clear_resolver method
sugatoray Jul 25, 2021
77fca7c
Updated test_omegaconf.py::test_clear_resolver
sugatoray Jul 28, 2021
ce7a05c
updated test_omegaconf.py::test_clear_resolver
sugatoray Jul 30, 2021
f62b567
Updated logic for OmegaConf.clear_resolver
sugatoray Jul 31, 2021
64ca79d
Added section for clearing resolver(s) in docs
sugatoray Jul 31, 2021
76179cd
updated docs examples for clearing resolver(s)
sugatoray Jul 31, 2021
5ec405a
Merge branch 'omry:master' into feature/769_clear_single_resolver
sugatoray Jul 31, 2021
5074cf4
cleaning up inside OmegaConf.clear_resolver
sugatoray Aug 3, 2021
cc974b6
cleaned up comments from test_omegaconf.py::test_clear_resolver
sugatoray Aug 3, 2021
324d0e1
made changes to test_omegaconf.py::test_clear_resolver
sugatoray Aug 3, 2021
8da4cbc
Update docs/source/custom_resolvers.rst
sugatoray Aug 4, 2021
2cb44f8
updated docs for OmegaConf.clear_resolvers
sugatoray Aug 4, 2021
350011b
changes to tests and docs
sugatoray Aug 4, 2021
8e51670
Update docs/source/custom_resolvers.rst
sugatoray Aug 4, 2021
1b687a4
Update docs/source/custom_resolvers.rst
sugatoray Aug 4, 2021
8d6c609
Update omegaconf/omegaconf.py
sugatoray Aug 4, 2021
1d209ca
Update docs/source/custom_resolvers.rst
sugatoray Aug 4, 2021
a42bc44
Update omegaconf/omegaconf.py
sugatoray Aug 4, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ pre-commit will verify your code lints cleanly when you commit. You can use `git
OmegaConf is compatible with Python 3.6.4 and newer. Unfortunately Mac comes with older versions.

One way to install multiple Python versions on Mac to to use pyenv.
The instructions [here](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/MAC_SETUP.md)
will provide full details. It shows how to use pyenv on mac to install multiple versions of Python and have
The instructions [here](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/MAC_SETUP.md)
will provide full details. It shows how to use pyenv on mac to install multiple versions of Python and have
pyenv make specific versions available in specific directories automatically.
This plays well with Conda, which supports a single Python version. Pyenv will provide the versions not installed by Conda (which are used when running nox).

Expand Down Expand Up @@ -49,23 +49,24 @@ Sessions defined in /home/omry/dev/omegaconf/noxfile.py:
* test_jupyter_notebook-3.8
* test_jupyter_notebook-3.9
```

To run a specific session use `-s`, for example `nox -s lint` will run linting


OmegaConf is formatted with black, to format your code automatically use `black .`

Imports are sorted using isort, use `isort .` to sort all imports prior to pushing.
Imports are sorted using isort, use `isort .` to sort all imports prior to pushing.

To build the docs execute `nox -s docs` or `make`(inside docs folder). Make gives you different options, for example, you can build the docs as html files with `make html`. Once the docs are built you can open `index.html` in the build directory to view the generated docs with your browser.

### Modifying Jupyter notebook

In order to change the Jupyter notebook you first need to open it with `jupyter notebook`.
Change the cell you want and then, execute it so the expected output is shown.
Note that the output after you execute the cell is saved as expected ouput for further
Change the cell you want and then, execute it so the expected output is shown.
Note that the output after you execute the cell is saved as expected ouput for further
testing.

In case that the in[number] of cells aren't in order you should go to the
In case that the in[number] of cells aren't in order you should go to the
kernel in the toolbar and restart it.


Expand Down
76 changes: 70 additions & 6 deletions docs/source/custom_resolvers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ Built-in resolvers

.. _oc.env:

oc.env
oc.env
^^^^^^

Access to environment variables is supported using ``oc.env``:
Expand All @@ -176,7 +176,7 @@ Input YAML file:

You can specify a default value to use in case the environment variable is not set.
In such a case, the default value is converted to a string using ``str(default)``,
unless it is ``null`` (representing Python ``None``) - in which case ``None`` is returned.
unless it is ``null`` (representing Python ``None``) - in which case ``None`` is returned.

The following example falls back to default passwords when ``DB_PASSWORD`` is not defined:

Expand Down Expand Up @@ -221,7 +221,7 @@ oc.create
... "dict_config_env": "${oc.create:${oc.env:YAML_ENV}}",
... }
... )
>>> os.environ["YAML_ENV"] = "A: 10\nb: 20\nC: ${.A}"
>>> os.environ["YAML_ENV"] = "A: 10\nb: 20\nC: ${.A}"
>>> show(cfg.plain_dict) # `make_dict` returns a Python dict
type: dict, value: {'a': 10}
>>> show(cfg.dict_config) # `oc.create` converts it to DictConfig
Expand All @@ -246,11 +246,11 @@ It takes two parameters:
.. doctest::

>>> conf = OmegaConf.create({
... "rusty_key": "${oc.deprecated:shiny_key}",
... "rusty_key": "${oc.deprecated:shiny_key}",
... "custom_msg": "${oc.deprecated:shiny_key, 'Use $NEW_KEY'}",
... "shiny_key": 10
... })
>>> # Accessing rusty_key will issue a deprecation warning
>>> # Accessing rusty_key will issue a deprecation warning
>>> # and return the new value automatically
>>> warning = "'rusty_key' is deprecated. Change your" \
... " code and config to use 'shiny_key'"
Expand Down Expand Up @@ -348,7 +348,7 @@ Another scenario where ``oc.select`` can be useful is if you want to select a mi
... "with_default": "${oc.select:missing,default value}",
... }
... )
...
...
>>> print(cfg.interpolation)
Traceback (most recent call last):
...
Expand Down Expand Up @@ -391,3 +391,67 @@ as interpolations), and they return a ``ListConfig`` that contains keys or value
>>> show(cfg.ips)
type: ListConfig, value: ['${workers.node3}', '${workers.node7}']
>>> assert cfg.ips == ["10.0.0.2", "10.0.0.9"]

.. _clearing_resolvers:

Clearing/removing resolvers
---------------------------

.. _clear_resolvers:

clear_resolvers
^^^^^^^^^^^^^^^

Use ``OmegaConf.clear_resolvers()`` to remove all resolvers except the built-in resolvers (like ``oc.env`` etc).

.. code-block:: python

def clear_resolvers() -> None

In the following example, first we register a new custom resolver ``str.lower``, and then clear all
custom resolvers.

.. doctest::

>>> # register a new resolver: str.lower
>>> OmegaConf.register_new_resolver(
... name='str.lower',
... resolver=lambda x: str(x).lower(),
... )
>>> # check if resolver exists (after adding, before removal)
>>> OmegaConf.has_resolver("str.lower")
True
>>> # clear all custom-resolvers
>>> OmegaConf.clear_resolvers()
>>> # check if resolver exists (after removal)
>>> OmegaConf.has_resolver("str.lower")
False
>>> # default resolvers are not affected
>>> OmegaConf.has_resolver("oc.env")
True
Comment on lines +407 to +431
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contract of this function is very simple, I am not convinced that it needs an example at all.
I vote to remove the example.

Copy link
Contributor Author

@sugatoray sugatoray Aug 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Granted it's down right simple 😉 . But, I believe, examples are kind of a major useful feature of the documentation of this library. So, I thought if it is made clear through an example, it will be helpful. I wanted to show that custom resolvers are removed, while default ones are not, when clear_resolvers() is used.

However, if you insist on it's (the example: starting from line-412) removal, that's okay as well. 😄


sugatoray marked this conversation as resolved.
Show resolved Hide resolved
.. _clear_resolver:

clear_resolver
^^^^^^^^^^^^^^

Use ``OmegaConf.clear_resolver()`` to remove a single resolver (including built-in resolvers).

.. code-block:: python

def clear_resolver(name: str) -> bool


``OmegaConf.clear_resolver()`` returns True if the resolver was found and removed, and False otherwise.

Here is an example.

.. doctest::

>>> OmegaConf.has_resolver("oc.env")
True
>>> # This will remove the default resolver: oc.env
>>> OmegaConf.clear_resolver("oc.env")
True
>>> OmegaConf.has_resolver("oc.env")
False
17 changes: 17 additions & 0 deletions omegaconf/omegaconf.py
Original file line number Diff line number Diff line change
Expand Up @@ -467,6 +467,23 @@ def clear_resolvers() -> None:
BaseContainer._resolvers = {}
register_default_resolvers()

@classmethod
def clear_resolver(cls, name: str) -> bool:
"""Clear(remove) any resolver only if it exists. Returns a bool: True if resolver is removed and False if not removed.

.. warning:
This method can remove deafult resolvers as well.

:param name: Name of the resolver.
:return: A bool (``True`` if resolver is removed, ``False`` if not found before removing).
"""
if cls.has_resolver(name):
BaseContainer._resolvers.pop(name)
return True
else:
# return False if resolver does not exist
return False

@staticmethod
def get_cache(conf: BaseContainer) -> Dict[str, Any]:
return conf._metadata.resolver_cache
Expand Down
39 changes: 39 additions & 0 deletions tests/test_omegaconf.py
Original file line number Diff line number Diff line change
Expand Up @@ -518,3 +518,42 @@ def test_resolve(cfg: Any, expected: Any) -> None:
def test_resolve_invalid_input() -> None:
with raises(ValueError):
OmegaConf.resolve("aaa") # type: ignore


@mark.parametrize(
("register_resolver_params", "name", "expected"),
[
param(
dict(
name="iamnew",
resolver=lambda x: str(x).lower(),
use_cache=False,
replace=False,
),
"iamnew",
dict(pre_clear=True, result=True),
id="remove-new-custom-resolver",
),
param(
dict(),
"oc.env",
dict(pre_clear=True, result=True),
id="remove-default-resolver",
),
param(
dict(),
"idonotexist",
dict(pre_clear=False, result=False),
id="remove-nonexistent-resolver",
),
],
)
def test_clear_resolver(
restore_resolvers: Any, register_resolver_params: Any, name: str, expected: Any
) -> None:
if register_resolver_params:
OmegaConf.register_new_resolver(**register_resolver_params)
assert expected["pre_clear"] == OmegaConf.has_resolver(name)

assert OmegaConf.clear_resolver(name) == expected["result"]
assert not OmegaConf.has_resolver(name)