How is DataStructure designed?

[hyunjimoon]

Hi @enekomartinmartinez,
Can you help me by explaining how current GetDataStructure and InlineLookupsStructure works? For instance, lookup function has Inline_ but datastructure has no Inline_ function which makes me curious about the role of inline function (which you documented as: "Dataclass for an inline lookup structure.")

I am trying to write a function that translates external data (timeseries vector) into stan function block using similar trick of linear interpolation for lookup function. So if you can briefly explain how pysd handles this it would be helpful! Thank you.

• I assume it revolves around DataStructure based on this code.
• As the above is get_xls : ae.DataStructure, I think your comment "GET XLS/DIRECT DATA/SUBSCRIPT/LOOKUPS/CONSTANS (with several test to check that everything is working ok)
  These expressions create a new class (external) whis is initialised before the stateful objects." from its function release is relevant.

[enekomartinmartinez]

There are a lot of kinds of functios that require interpolation, the equivalence is given in the following section https://pysd.readthedocs.io/en/master/structure/vensim_translation.html#supported-functions-and-features. Here a brief summary with some models with examples of each tipes

• GetDataStructure and GetLookupsStructure are used for Vensim's GET XLS/DIRECT DATA/LOOKUPS, where all the data is read from an Excel file. These create External objects which are initialized before the Statefuls -> test-models/tests/get_lookups_data_3d_xls

• DataStructure is used for regular Data definitions from Vensim (sub-type Normal, second figure). In this definition the variable's RHS is empty and the data is read from a vdf file that could come from another simulation or model (we use csv or tab files with PySD). This data is read when loading the model, data_files argument of pysd.py_backend.Model class or pysd.load function. They are defined in the final model with the class pysd.py_backend.data.TabData -> test-models/tests/data_from_other_model

• LookupsStructure is used for regular lookups where the data for the interpolation is defined in the file. This creates pysd.py_backend.lookups.HardocdedLookups object which is initialized after Externals and before statefuls (in this case initialization is just merging the different dimensions to be able to interpolate in a vectorial way) -> test-models/testt/lookups_funcnames

• InlineLookupsStructure is used for Vensim's WITH LOOKUP functions, which is a Vensim function that takes the inputs and the interpolation points. Note that this structure has a LookupStructure inside for the interpolation points definition. This is directly written in the function definition. -> test-models/tests/lookups_with_expr

Note, all the lookup functions need to be called always with an argument, except WITH LOOKUP (Inline Lookup) where the argument is provided as the first argument of the function.

Data types don't take arguments, they are interpolated using the current time of the model. Several interpolation methods can be defined as keyword (:RAW:, :INTERPOLATE:, :LOOK FORWARD:, :HOLD BACKWARD:)

There are some functions that allow shifting the time for a Data object (not implemented yet in PySD).
Lookups can also be interpolated with a different method using a Vensim function (not implemented in PySD).

For XMILE models Data types are not defined only as Lookups, they are called graphical functions (GF) in XMILE. These functions could directly take an argument to define the interpolation method (implemented but not tested in PySD as I don't have Stella).

[hyunjimoon]

During the last meeting, you recommended using xarray and commented pysd changed to this datatstructure.

What was the datastrucutre pysd was using before (e.g. pandas dataframe)?

Also, I am having difficulty matching xarray's coordinate, dimension, data variable with SD's data structure. How did you design this? Example of pysd's xarray datastructure template e.g. data from teacup model would be very helpful as I can get the sense of what coordinate, dimension, data variable could SD data have.