You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Context: the current IOPointer abstraction only stores a string "pointer" to the data, or a key. An example might be features.csv or model.joblib. This means we currently can't do anything with the data, because we don't store any concept of it. If we store data, we could do many things, including the following:
Compare current values to historical ComponentRuns' values
Identify whether files have been tampered with outside of ComponentRuns
Have more fine-grained tracing (record-level)
Storing the data in its entirety may be expensive. For now we will store a hash of the data, to get us one step closer to being able to store the data. This itself may be complex.
In the future, we will incorporate an IOPointer "tag" model to store information about fine-grained tracing (i.e., PK values will be tags). This tagging is out of scope from the current project.
The text was updated successfully, but these errors were encountered:
Context: the current
IOPointer
abstraction only stores a string "pointer" to the data, or a key. An example might befeatures.csv
ormodel.joblib
. This means we currently can't do anything with the data, because we don't store any concept of it. If we store data, we could do many things, including the following:ComponentRun
s' valuesComponentRun
sStoring the data in its entirety may be expensive. For now we will store a hash of the data, to get us one step closer to being able to store the data. This itself may be complex.
Issues
IOPointer
. Change the PK to be (key, value) instead of just key. In the migration, old records should have null values. #215store.get_io_pointer
to return theIOPointer
correctly. ModifyIOPointer
client-facing abstractions but treat the value as null. #216register
decorator andlog_component_run
functions to log KV pairs for I/O. Rewriteset_dependencies_from_inputs
to reflect the newIOPointer
schema. #217trace
functions to reflect the newIOPointer
schema. #218In the future, we will incorporate an
IOPointer
"tag" model to store information about fine-grained tracing (i.e., PK values will be tags). This tagging is out of scope from the current project.The text was updated successfully, but these errors were encountered: