Skip to content

Commit

Permalink
Uniformize tests with the new API
Browse files Browse the repository at this point in the history
  • Loading branch information
@fpagnoux
fpagnoux authored Dec 5, 2018
2 parents 6af82de + d31988b commit ee80a96
Show file tree
Hide file tree
Showing 45 changed files with 1,527 additions and 855 deletions.
6 changes: 5 additions & 1 deletion .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ jobs:
command: |
make install
# pip install --editable git+https://github.com/openfisca/country-template.git@BRANCH_NAME#egg=OpenFisca-Country-Template # use a specific branch of OpenFisca-Country-Template
# pip install --editable git+https://github.com/openfisca/extension-template.git@BRANCH_NAME#egg=OpenFisca-Extension-Template # use a specific branch of OpenFisca-Extension-Template
- save_cache:
key: v1-py2-{{ .Branch }}-{{ checksum "setup.py" }}
Expand Down Expand Up @@ -87,6 +88,7 @@ jobs:
command: |
make install
# pip install --editable git+https://github.com/openfisca/country-template.git@BRANCH_NAME#egg=OpenFisca-Country-Template # use a specific branch of OpenFisca-Country-Template
# pip install --editable git+https://github.com/openfisca/extension-template.git@BRANCH_NAME#egg=OpenFisca-Extension-Template # use a specific branch of OpenFisca-Extension-Template
- save_cache:
key: v1-py3-{{ .Branch }}-{{ checksum "setup.py" }}
Expand All @@ -99,7 +101,9 @@ jobs:

- run:
name: Run Country Template tests
command: openfisca-run-test /tmp/venv/openfisca_core/lib/python*/site-packages/openfisca_country_template/tests/
command: |
COUNTRY_TEMPLATE_PATH=`python -c "import openfisca_country_template; print(openfisca_country_template.CountryTaxBenefitSystem().get_package_metadata()['location'])"`
openfisca test $COUNTRY_TEMPLATE_PATH/openfisca_country_template/tests/
deploy_python3:
docker:
Expand Down
118 changes: 117 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,122 @@
# Changelog

## 24.10.0 [#791](https://github.com/openfisca/openfisca-core/pull/791)
# 25.0 [#781](https://github.com/openfisca/openfisca-core/pull/781)

#### Breaking changes

- Change the syntax of OpenFisca YAML tests

For instance, a test that was using the `input_variables` keyword like:

```yaml
- name: Basic income
period: 2016-12
input_variables:
salary: 1200
output_variables:
basic_income: 600
```
becomes:
```yaml
- name: Basic income
period: 2016-12
input:
salary: 1200
output:
basic_income: 600
```
A test that was fully specifying its entities like:
```yaml
name: Housing tax
period: 2017-01
households:
- parents: [ Alicia ]
children: [ Michael ]
persons:
- id: Alicia
birth: 1961-01-15
- id: Michael
birth: 2002-01-15
output_variables:
housing_tax:
2017: 1000
```
becomes:
```yaml
name: Housing tax
period: 2017-01
input:
household:
parents: [ Alicia ]
children: [ Michael ]
persons:
Alicia:
birth: 1961-01-15
Michael:
birth: 2002-01-15
output:
housing_tax:
2017: 1000
```
A **migration script** is available to automatically convert tests:
```sh
python openfisca_core/scripts/migrations/v24_to_25.py /path/to/tests/
```

> Note for country packages using Scenarios (e.g. France, Tunisia):
> Tests are not using scenarios anymore. Therefore, tests cannot partially specify entities anymore. Tests using entities inference may need manual adaptation in addition to the script.
- The `Entity` constructor (usually not directly called by users) does not accept an `entities_json` parameter anymore.


#### Deprecation

- Deprecate `openfisca-run-test`
- `openfisca test` should be used instead.

- Deprecate the use of the `simulation_json` parameter in the `Simulation` constructor.
- `SimulationBuilder(tax_benefit_system).build_from_entities(simulation_json)` should be used instead


#### New features

- In YAML tests, allow to define expected output for a specific entity

For instance:

```yaml
name: Housing tax
period: 2017-01
input:
...
output:
persons:
Alicia:
salary: 3000
```
- In YAML tests, allow to specify an extension to use to run the test:
- See [example](https://github.com/openfisca/openfisca-core/blob/25.0.0/tests/core/yaml_tests/test_with_extension.yaml)
- In YAML tests, allow the use of YAML anchors:
- See [example](https://github.com/openfisca/openfisca-core/blob/25.0.0/tests/core/yaml_tests/test_with_anchors.yaml)
- Introduce [`EnumArray.decode_to_str`](https://openfisca.org/doc/openfisca-python-api/enum_array.html#openfisca_core.indexed_enums.EnumArray.decode_to_str)


#### Architecture changes

- Move the complex initialisation logics (for JSON-like inputs) to `SimulationBuilder`, away from the `Simulation` and `Entity` classes

## 24.11.0 [#791](https://github.com/openfisca/openfisca-core/pull/791)

- In Python, simplify getting known periods for variable in a simulation:

Expand Down
197 changes: 17 additions & 180 deletions openfisca_core/entities.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,11 @@
import warnings
import textwrap
from os import linesep
from datetime import date

import numpy as np
import dpath

from openfisca_core.indexed_enums import Enum, EnumArray
from openfisca_core.scenarios import iter_over_entity_members
from openfisca_core.simulations import check_type, SituationParsingError
from openfisca_core.holders import Holder, PeriodMismatchError
from openfisca_core.periods import key_period_size, period as make_period
from openfisca_core.errors import VariableNotFound
from openfisca_core.commons import basestring_type
from openfisca_core.indexed_enums import EnumArray
from openfisca_core.holders import Holder

ADD = 'add'
DIVIDE = 'divide'
Expand All @@ -32,111 +25,12 @@ class Entity(object):
doc = ""
is_person = False

def __init__(self, simulation, entities_json = None):
def __init__(self, simulation):
self.simulation = simulation
self._holders = {}
if entities_json is not None:
self.init_from_json(entities_json)
else:
self.entities_json = None
self.count = 0
self.ids = []
self.step_size = 0

def init_from_json(self, entities_json):
"""
Initilalises entities from a JSON dictionnary.
This method, still under experimentation, aims at replacing the initialisation from `scenario.make_json_or_python_to_attributes`
"""
check_type(entities_json, dict, [self.plural])
self.entities_json = entities_json
self.count = len(entities_json)
self.step_size = self.count # Related to axes.
self.ids = sorted(entities_json.keys())
for entity_id, entity_object in entities_json.items():
check_type(entity_object, dict, [self.plural, entity_id])
if not self.is_person:
roles_json, variables_json = self.split_variables_and_roles_json(entity_object)
self.init_members(roles_json, entity_id)
else:
variables_json = entity_object
self.init_variable_values(variables_json, entity_id)

# Due to set_input mechanism, we must bufferize all inputs, then actually set them, so that the months are set first and the years last.
self.finalize_variables_init()

def init_variable_values(self, entity_object, entity_id):
entity_index = self.ids.index(entity_id)
for variable_name, variable_values in entity_object.items():
path_in_json = [self.plural, entity_id, variable_name]
try:
self.check_variable_defined_for_entity(variable_name)
except ValueError as e: # The variable is defined for another entity
raise SituationParsingError(path_in_json, e.args[0])
except VariableNotFound as e: # The variable doesn't exist
raise SituationParsingError(path_in_json, e.message, code = 404)

if not isinstance(variable_values, dict):
raise SituationParsingError(path_in_json,
"Can't deal with type: expected object. Input variables should be set for specific periods. For instance: {'salary': {'2017-01': 2000, '2017-02': 2500}}, or {'birth_date': {'ETERNITY': '1980-01-01'}}.")

holder = self.get_holder(variable_name)
for period_str, value in variable_values.items():
path_in_json.append(period_str)
try:
period = make_period(period_str)
except ValueError as e:
raise SituationParsingError(path_in_json, e.args[0])
if value is not None:
array = holder.buffer.get(period)
if array is None:
array = holder.default_array()
if holder.variable.value_type == Enum and isinstance(value, basestring_type):
try:
value = holder.variable.possible_values[value].index
except KeyError:
possible_values = [item.name for item in holder.variable.possible_values]
raise SituationParsingError(path_in_json,
"'{}' is not a known value for '{}'. Possible values are ['{}'].".format(
value, variable_name, "', '".join(possible_values))
)
try:
array[entity_index] = value
except (OverflowError):
error_message = "Can't deal with value: '{}', it's too large for type '{}'.".format(value, holder.variable.json_type)
raise SituationParsingError(path_in_json, error_message)
except (ValueError, TypeError):
if holder.variable.value_type == date:
error_message = "Can't deal with date: '{}'.".format(value)
else:
error_message = "Can't deal with value: expected type {}, received '{}'.".format(holder.variable.json_type, value)
raise SituationParsingError(path_in_json, error_message)

holder.buffer[period] = array

def finalize_variables_init(self):
for variable_name, holder in self._holders.items():
periods = holder.buffer.keys()
# We need to handle small periods first for set_input to work
sorted_periods = sorted(periods, key=key_period_size)
for period in sorted_periods:
array = holder.buffer[period]
try:
holder.set_input(period, array)
except PeriodMismatchError as e:
# This errors happens when we try to set a variable value for a period that doesn't match its definition period
# It is only raised when we consume the buffer. We thus don't know which exact key caused the error.
# We do a basic research to find the culprit path
culprit_path = next(
dpath.search(self.entities_json, "*/{}/{}".format(e.variable_name, str(e.period)), yielded = True),
None)
if culprit_path:
path = [self.plural] + culprit_path[0].split('/')
else:
path = [self.plural] # Fallback: if we can't find the culprit, just set the error at the entities level

raise SituationParsingError(path, e.message)
self.count = 0
self.ids = []
self.step_size = 0

def clone(self, new_simulation):
"""
Expand Down Expand Up @@ -176,7 +70,7 @@ def to_json(cls):
# Calculations

def check_variable_defined_for_entity(self, variable_name):
variable_entity = self.simulation.tax_benefit_system.get_variable(variable_name, check_existence = True).entity
variable_entity = self.get_variable(variable_name, check_existence = True).entity
if not isinstance(self, variable_entity):
message = linesep.join([
"You tried to compute the variable '{0}' for the entity '{1}';".format(variable_name, self.plural),
Expand Down Expand Up @@ -240,12 +134,15 @@ def filled_array(self, value, dtype = None):
warnings.simplefilter("ignore")
return np.full(self.count, value, dtype)

def get_variable(self, variable_name, check_existence = False):
return self.simulation.tax_benefit_system.get_variable(variable_name, check_existence)

def get_holder(self, variable_name):
self.check_variable_defined_for_entity(variable_name)
holder = self._holders.get(variable_name)
if holder:
return holder
variable = self.simulation.tax_benefit_system.get_variable(variable_name)
variable = self.get_variable(variable_name)
self._holders[variable_name] = holder = Holder(
entity = self,
variable = variable,
Expand Down Expand Up @@ -373,76 +270,16 @@ class GroupEntity(Entity):
flattened_roles = None
roles_description = None

def __init__(self, simulation, entities_json = None):
Entity.__init__(self, simulation, entities_json)
if entities_json is None:
self.members_entity_id = None
self._members_role = None
self._members_position = None
self.members_legacy_role = None
def __init__(self, simulation):
Entity.__init__(self, simulation)
self.members_entity_id = None
self._members_role = None
self._members_position = None
self.members_legacy_role = None
self.members = self.simulation.persons
self._roles_count = None
self._ordered_members_map = None

def split_variables_and_roles_json(self, entity_object):
entity_object = entity_object.copy() # Don't mutate function input

roles_definition = {
role.plural or role.key: entity_object.pop(role.plural or role.key, [])
for role in self.roles
}
return roles_definition, entity_object

def init_from_json(self, entities_json):
self.members_entity_id = np.empty(
self.simulation.persons.count,
dtype = np.int32
)
self.members_role = np.empty(
self.simulation.persons.count,
dtype = object
)
self.members_legacy_role = np.empty(
self.simulation.persons.count,
dtype = np.int32
)
self._members_position = None

self.persons_to_allocate = set(self.simulation.persons.ids)

Entity.init_from_json(self, entities_json)

if self.persons_to_allocate:
unallocated_person = self.persons_to_allocate.pop()
raise SituationParsingError([self.plural],
'{0} has been declared in {1}, but is not a member of any {2}. All {1} must be allocated to a {2}.'.format(
unallocated_person, self.simulation.persons.plural, self.key)
)

def init_members(self, roles_json, entity_id):
for role_id, role_definition in roles_json.items():
check_type(role_definition, list, [self.plural, entity_id, role_id])
for index, person_id in enumerate(role_definition):
check_type(person_id, basestring_type, [self.plural, entity_id, role_id, str(index)])
if person_id not in self.simulation.persons.ids:
raise SituationParsingError([self.plural, entity_id, role_id],
"Unexpected value: {0}. {0} has been declared in {1} {2}, but has not been declared in {3}.".format(
person_id, entity_id, role_id, self.simulation.persons.plural)
)
if person_id not in self.persons_to_allocate:
raise SituationParsingError([self.plural, entity_id, role_id],
"{} has been declared more than once in {}".format(
person_id, self.plural)
)
self.persons_to_allocate.discard(person_id)

entity_index = self.ids.index(entity_id)
for person_role, person_legacy_role, person_id in iter_over_entity_members(self, roles_json):
person_index = self.simulation.persons.ids.index(person_id)
self.members_entity_id[person_index] = entity_index
self.members_role[person_index] = person_role
self.members_legacy_role[person_index] = person_legacy_role

@property
def members_role(self):
if self._members_role is None and self.members_legacy_role is not None:
Expand Down
Loading

0 comments on commit ee80a96

Please sign in to comment.