frozenlist/frozendict types are inconvenient when passed outside of Dagster

[kinghuang]

Problem to solve

Dagster has frozenlist/frozendict types that are immutable versions of the standard Python list and dict types.

The frozenlist/frozendict types pose difficulties when Dagster config values are passed to other systems. For example, consider the following solid that writes out a Dask dataframe to a Parquet dataset, with a solid config for partition_on.

@solid(
    config_schema={
        "partition_on": Field(list, default_value=["year", "month"], is_required=False),
    }
)
def example(context, input_df: DataFrame) -> DataFrame:
    partition_on_keys = context.solid_config.get("partition_on")
    input_df.to_parquet("s3://bucket/key", partition_on=partition_on_keys)

The list of keys to partition on (defaults to ["year", "month"]) is returned as a frozenlist from context.solid_config. When this is passed as the value of the partition_on argument in input_df.to_parquet(…), the frozenlist is pickled and sent to Dask workers.

The problem this poses is that the Dask workers now need to have the dagster module installed in order to unpickle what is otherwise a normal Python list. This adds an extra dependency on the Dask workers, or any other system that receives pickled values from a Dagster pipeline.

Intended users

• Pipeline Developer
• DevOps Engineer

User experience goal

If a solid or resource passes a config to another system, the target system should be able to unpickle the value without a complete dagster installation. Ideally, no extra modules would be required at all.

This particularly affects Dask pipelines, where most things are pickled to run on Dask workers, outside of the Dagster execution context.

Proposal

There are several possibilities.

1. Have frozenlist/frozendict pickle as their native Python types. That way they can be unpickled as list/dict without requiring any additional modules.
2. Move frozenlist/frozendict to an independent module, say dagster_collections. Systems that receive these types can install this module only instead of dagster and its complete dependency tree.
3. Provide utility functions to convert frozenlist/frozendict to list/dict. Solids and resources are then responsible for converting the Dagster types to native types before passing them to other systems. Utility functions are needed for deep conversions (e.g., a frozendict nested in a frozenlist).
4. Pass config values as list/dict to solids and resources. Not sure about the consequences of this.
5. Stop using frozenlist/frozenlist. This would have negative impacts on caching in Dagster.

I personally prefer option 1 or 2. At minimum, I don't want to install dagster completely on my Dask workers just to unpickle frozenlist/frozendict.

Option 3 might be useful to have regardless of the other options.

Other information

• Fix pickling of frozenlist #2890 fixed pickling of frozenlist.
• Add utility options to Dask DataFrame type loader/materializer #2888 no longer converts frozenlist to list when passing configs to Dask.

[yuhan]

IMO passing values as list/dict/set from outside Dagster sounds good and it may be too much to expect user inputs to be immutable objects. And we should have already guarded these cases by converting them into frozenlist/frozendict/frozenset inside the machinery for immutability.