Uniformize tests with the new API

.circleci/config.yml

@@ -23,6 +23,7 @@ jobs:
           command: |
             make install
             # pip install --editable git+https://github.com/openfisca/country-template.git@BRANCH_NAME#egg=OpenFisca-Country-Template  # use a specific branch of OpenFisca-Country-Template
+            # pip install --editable git+https://github.com/openfisca/extension-template.git@BRANCH_NAME#egg=OpenFisca-Extension-Template  # use a specific branch of OpenFisca-Extension-Template
 
       - save_cache:
           key: v1-py2-{{ .Branch }}-{{ checksum "setup.py" }}
@@ -87,6 +88,7 @@ jobs:
           command: |
             make install
             # pip install --editable git+https://github.com/openfisca/country-template.git@BRANCH_NAME#egg=OpenFisca-Country-Template  # use a specific branch of OpenFisca-Country-Template
+            # pip install --editable git+https://github.com/openfisca/extension-template.git@BRANCH_NAME#egg=OpenFisca-Extension-Template  # use a specific branch of OpenFisca-Extension-Template
 
       - save_cache:
           key: v1-py3-{{ .Branch }}-{{ checksum "setup.py" }}
@@ -99,7 +101,9 @@ jobs:
 
       - run:
           name: Run Country Template tests
-          command: openfisca-run-test /tmp/venv/openfisca_core/lib/python*/site-packages/openfisca_country_template/tests/
+          command: |
+            COUNTRY_TEMPLATE_PATH=`python -c "import openfisca_country_template; print(openfisca_country_template.CountryTaxBenefitSystem().get_package_metadata()['location'])"`
+            openfisca test $COUNTRY_TEMPLATE_PATH/openfisca_country_template/tests/
 
   deploy_python3:
     docker:

CHANGELOG.md

@@ -1,6 +1,122 @@
 # Changelog
 
-## 24.10.0 [#791](https://github.com/openfisca/openfisca-core/pull/791)
+# 25.0 [#781](https://github.com/openfisca/openfisca-core/pull/781)
+
+#### Breaking changes
+
+  - Change the syntax of OpenFisca YAML tests
+
+  For instance, a test that was using the `input_variables` keyword like:
+
+  ```yaml
+  - name: Basic income
+    period: 2016-12
+    input_variables:
+      salary: 1200
+    output_variables:
+      basic_income: 600
+  ```
+
+  becomes:
+
+  ```yaml
+  - name: Basic income
+    period: 2016-12
+    input:
+      salary: 1200
+    output:
+      basic_income: 600
+  ```
+
+  A test that was fully specifying its entities like:
+
+  ```yaml
+  name: Housing tax
+    period: 2017-01
+    households:
+      - parents: [ Alicia ]
+        children: [ Michael ]
+    persons:
+      - id: Alicia
+          birth: 1961-01-15
+      - id: Michael
+          birth: 2002-01-15
+    output_variables:
+      housing_tax:
+        2017: 1000
+  ```
+
+  becomes:
+
+  ```yaml
+  name: Housing tax
+    period: 2017-01
+    input:
+      household:
+        parents: [ Alicia ]
+        children: [ Michael ]
+      persons:
+        Alicia:
+          birth: 1961-01-15
+        Michael:
+          birth: 2002-01-15
+    output:
+      housing_tax:
+        2017: 1000
+  ```
+
+  A **migration script** is available to automatically convert tests:
+
+  ```sh
+  python openfisca_core/scripts/migrations/v24_to_25.py /path/to/tests/
+  ```
+
+  > Note for country packages using Scenarios (e.g. France, Tunisia):
+  > Tests are not using scenarios anymore. Therefore, tests cannot partially specify entities anymore. Tests using entities inference may need manual adaptation in addition to the script.
+
+  - The `Entity` constructor (usually not directly called by users) does not accept an `entities_json` parameter anymore.
+
+
+#### Deprecation
+
+  - Deprecate `openfisca-run-test`
+    - `openfisca test` should be used instead.
+
+  - Deprecate the use of the `simulation_json` parameter in the `Simulation` constructor.
+    - `SimulationBuilder(tax_benefit_system).build_from_entities(simulation_json)` should be used instead
+
+
+#### New features
+
+  - In YAML tests, allow to define expected output for a specific entity
+
+  For instance:
+
+  ```yaml
+  name: Housing tax
+    period: 2017-01
+    input:
+      ...
+    output:
+      persons:
+        Alicia:
+          salary: 3000
+  ```
+
+  - In YAML tests, allow to specify an extension to use to run the test:
+    - See [example](https://github.com/openfisca/openfisca-core/blob/25.0.0/tests/core/yaml_tests/test_with_extension.yaml)
+
+  - In YAML tests, allow the use of YAML anchors:
+    - See [example](https://github.com/openfisca/openfisca-core/blob/25.0.0/tests/core/yaml_tests/test_with_anchors.yaml)
+
+  - Introduce [`EnumArray.decode_to_str`](https://openfisca.org/doc/openfisca-python-api/enum_array.html#openfisca_core.indexed_enums.EnumArray.decode_to_str)
+
+
+#### Architecture changes
+
+  - Move the complex initialisation logics (for JSON-like inputs) to `SimulationBuilder`, away from the `Simulation` and `Entity` classes
+
+## 24.11.0 [#791](https://github.com/openfisca/openfisca-core/pull/791)
 
 - In Python, simplify getting known periods for variable in a simulation:
 

openfisca_core/entities.py

@@ -5,18 +5,11 @@
 import warnings
 import textwrap
 from os import linesep
-from datetime import date
 
 import numpy as np
-import dpath
 
-from openfisca_core.indexed_enums import Enum, EnumArray
-from openfisca_core.scenarios import iter_over_entity_members
-from openfisca_core.simulations import check_type, SituationParsingError
-from openfisca_core.holders import Holder, PeriodMismatchError
-from openfisca_core.periods import key_period_size, period as make_period
-from openfisca_core.errors import VariableNotFound
-from openfisca_core.commons import basestring_type
+from openfisca_core.indexed_enums import EnumArray
+from openfisca_core.holders import Holder
 
 ADD = 'add'
 DIVIDE = 'divide'
@@ -32,111 +25,12 @@ class Entity(object):
     doc = ""
     is_person = False
 
-    def __init__(self, simulation, entities_json = None):
+    def __init__(self, simulation):
         self.simulation = simulation
         self._holders = {}
-        if entities_json is not None:
-            self.init_from_json(entities_json)
-        else:
-            self.entities_json = None
-            self.count = 0
-            self.ids = []
-            self.step_size = 0
-
-    def init_from_json(self, entities_json):
-        """
-            Initilalises entities from a JSON dictionnary.
-
-            This method, still under experimentation, aims at replacing the initialisation from `scenario.make_json_or_python_to_attributes`
-        """
-        check_type(entities_json, dict, [self.plural])
-        self.entities_json = entities_json
-        self.count = len(entities_json)
-        self.step_size = self.count  # Related to axes.
-        self.ids = sorted(entities_json.keys())
-        for entity_id, entity_object in entities_json.items():
-            check_type(entity_object, dict, [self.plural, entity_id])
-            if not self.is_person:
-                roles_json, variables_json = self.split_variables_and_roles_json(entity_object)
-                self.init_members(roles_json, entity_id)
-            else:
-                variables_json = entity_object
-            self.init_variable_values(variables_json, entity_id)
-
-        # Due to set_input mechanism, we must bufferize all inputs, then actually set them, so that the months are set first and the years last.
-        self.finalize_variables_init()
-
-    def init_variable_values(self, entity_object, entity_id):
-        entity_index = self.ids.index(entity_id)
-        for variable_name, variable_values in entity_object.items():
-            path_in_json = [self.plural, entity_id, variable_name]
-            try:
-                self.check_variable_defined_for_entity(variable_name)
-            except ValueError as e:  # The variable is defined for another entity
-                raise SituationParsingError(path_in_json, e.args[0])
-            except VariableNotFound as e:  # The variable doesn't exist
-                raise SituationParsingError(path_in_json, e.message, code = 404)
-
-            if not isinstance(variable_values, dict):
-                raise SituationParsingError(path_in_json,
-                    "Can't deal with type: expected object. Input variables should be set for specific periods. For instance: {'salary': {'2017-01': 2000, '2017-02': 2500}}, or {'birth_date': {'ETERNITY': '1980-01-01'}}.")
-
-            holder = self.get_holder(variable_name)
-            for period_str, value in variable_values.items():
-                path_in_json.append(period_str)
-                try:
-                    period = make_period(period_str)
-                except ValueError as e:
-                    raise SituationParsingError(path_in_json, e.args[0])
-                if value is not None:
-                    array = holder.buffer.get(period)
-                    if array is None:
-                        array = holder.default_array()
-                    if holder.variable.value_type == Enum and isinstance(value, basestring_type):
-                        try:
-                            value = holder.variable.possible_values[value].index
-                        except KeyError:
-                            possible_values = [item.name for item in holder.variable.possible_values]
-                            raise SituationParsingError(path_in_json,
-                                "'{}' is not a known value for '{}'. Possible values are ['{}'].".format(
-                                    value, variable_name, "', '".join(possible_values))
-                                )
-                    try:
-                        array[entity_index] = value
-                    except (OverflowError):
-                        error_message = "Can't deal with value: '{}', it's too large for type '{}'.".format(value, holder.variable.json_type)
-                        raise SituationParsingError(path_in_json, error_message)
-                    except (ValueError, TypeError):
-                        if holder.variable.value_type == date:
-                            error_message = "Can't deal with date: '{}'.".format(value)
-                        else:
-                            error_message = "Can't deal with value: expected type {}, received '{}'.".format(holder.variable.json_type, value)
-                        raise SituationParsingError(path_in_json, error_message)
-
-                    holder.buffer[period] = array
-
-    def finalize_variables_init(self):
-        for variable_name, holder in self._holders.items():
-            periods = holder.buffer.keys()
-            # We need to handle small periods first for set_input to work
-            sorted_periods = sorted(periods, key=key_period_size)
-            for period in sorted_periods:
-                array = holder.buffer[period]
-                try:
-                    holder.set_input(period, array)
-                except PeriodMismatchError as e:
-                    # This errors happens when we try to set a variable value for a period that doesn't match its definition period
-                    # It is only raised when we consume the buffer. We thus don't know which exact key caused the error.
-                    # We do a basic research to find the culprit path
-                    culprit_path = next(
-                        dpath.search(self.entities_json, "*/{}/{}".format(e.variable_name, str(e.period)), yielded = True),
-                        None)
-                    if culprit_path:
-                        path = [self.plural] + culprit_path[0].split('/')
-                    else:
-                        path = [self.plural]  # Fallback: if we can't find the culprit, just set the error at the entities level
-
-                    raise SituationParsingError(path, e.message)
+        self.count = 0
+        self.ids = []
+        self.step_size = 0
 
     def clone(self, new_simulation):
         """
@@ -176,7 +70,7 @@ def to_json(cls):
     # Calculations
 
     def check_variable_defined_for_entity(self, variable_name):
-        variable_entity = self.simulation.tax_benefit_system.get_variable(variable_name, check_existence = True).entity
+        variable_entity = self.get_variable(variable_name, check_existence = True).entity
         if not isinstance(self, variable_entity):
             message = linesep.join([
                 "You tried to compute the variable '{0}' for the entity '{1}';".format(variable_name, self.plural),
@@ -240,12 +134,15 @@ def filled_array(self, value, dtype = None):
             warnings.simplefilter("ignore")
             return np.full(self.count, value, dtype)
 
+    def get_variable(self, variable_name, check_existence = False):
+        return self.simulation.tax_benefit_system.get_variable(variable_name, check_existence)
+
     def get_holder(self, variable_name):
         self.check_variable_defined_for_entity(variable_name)
         holder = self._holders.get(variable_name)
         if holder:
             return holder
-        variable = self.simulation.tax_benefit_system.get_variable(variable_name)
+        variable = self.get_variable(variable_name)
         self._holders[variable_name] = holder = Holder(
             entity = self,
             variable = variable,
@@ -373,76 +270,16 @@ class GroupEntity(Entity):
     flattened_roles = None
     roles_description = None
 
-    def __init__(self, simulation, entities_json = None):
-        Entity.__init__(self, simulation, entities_json)
-        if entities_json is None:
-            self.members_entity_id = None
-            self._members_role = None
-            self._members_position = None
-            self.members_legacy_role = None
+    def __init__(self, simulation):
+        Entity.__init__(self, simulation)
+        self.members_entity_id = None
+        self._members_role = None
+        self._members_position = None
+        self.members_legacy_role = None
         self.members = self.simulation.persons
         self._roles_count = None
         self._ordered_members_map = None
 
-    def split_variables_and_roles_json(self, entity_object):
-        entity_object = entity_object.copy()  # Don't mutate function input
-
-        roles_definition = {
-            role.plural or role.key: entity_object.pop(role.plural or role.key, [])
-            for role in self.roles
-            }
-        return roles_definition, entity_object
-
-    def init_from_json(self, entities_json):
-        self.members_entity_id = np.empty(
-            self.simulation.persons.count,
-            dtype = np.int32
-            )
-        self.members_role = np.empty(
-            self.simulation.persons.count,
-            dtype = object
-            )
-        self.members_legacy_role = np.empty(
-            self.simulation.persons.count,
-            dtype = np.int32
-            )
-        self._members_position = None
-
-        self.persons_to_allocate = set(self.simulation.persons.ids)
-
-        Entity.init_from_json(self, entities_json)
-
-        if self.persons_to_allocate:
-            unallocated_person = self.persons_to_allocate.pop()
-            raise SituationParsingError([self.plural],
-                '{0} has been declared in {1}, but is not a member of any {2}. All {1} must be allocated to a {2}.'.format(
-                    unallocated_person, self.simulation.persons.plural, self.key)
-                )
-
-    def init_members(self, roles_json, entity_id):
-        for role_id, role_definition in roles_json.items():
-            check_type(role_definition, list, [self.plural, entity_id, role_id])
-            for index, person_id in enumerate(role_definition):
-                check_type(person_id, basestring_type, [self.plural, entity_id, role_id, str(index)])
-                if person_id not in self.simulation.persons.ids:
-                    raise SituationParsingError([self.plural, entity_id, role_id],
-                        "Unexpected value: {0}. {0} has been declared in {1} {2}, but has not been declared in {3}.".format(
-                            person_id, entity_id, role_id, self.simulation.persons.plural)
-                        )
-                if person_id not in self.persons_to_allocate:
-                    raise SituationParsingError([self.plural, entity_id, role_id],
-                        "{} has been declared more than once in {}".format(
-                            person_id, self.plural)
-                        )
-                self.persons_to_allocate.discard(person_id)
-
-        entity_index = self.ids.index(entity_id)
-        for person_role, person_legacy_role, person_id in iter_over_entity_members(self, roles_json):
-            person_index = self.simulation.persons.ids.index(person_id)
-            self.members_entity_id[person_index] = entity_index
-            self.members_role[person_index] = person_role
-            self.members_legacy_role[person_index] = person_legacy_role
-
     @property
     def members_role(self):
         if self._members_role is None and self.members_legacy_role is not None: