[PROPOSAL] Pre-processing step

[adrienaury]

Problem

Difficult to compute inter-stream coherence/identifiable rates when columns are built differently (by format or composition)

Example :

Stream A

A/real.jsonl

{"id":1,"name":"Clothilde","surname":"Renard"}
{"id":2,"name":"Andrée","surname":"Mathieu"}

A/masked.jsonl

{"id":1,"name":"John","surname":"Doe"}
{"id":2,"name":"Jade","surname":"Doe"}

Stream B

B/real.jsonl

{"id":1,"idutil":"1 - Clothilde Renard"}
{"id":2,"idutil":"2 - Andrée Mathieu"}

B/masked.jsonl

{"id":1,"idutil":"1 - John Doe"}
{"id":2,"idutil":"2 - John Doe"}

Solution

MIMO should be able to compute pre-values in the stream.

Configuration for Stream A

A/config.yaml

version: "1"
preprocess:
  - name: "idutil"
    value: "{{.id}} - {{.name}} {{.surname}}"
metrics:
  - name: "idutil"
    coherentWith: ["id"]

Configuration for Stream B

B/config.yaml

version: "1"
metrics:
  - name: "idutil"
    coherentWith: ["id"]

Command line

$ cat A/masked.jsonl | mimo --config A/config.yaml --persist localdb A/real.jsonl
$ cat B/masked.jsonl | mimo --config B/config.yaml --persist localdb B/real.jsonl

computation matrix (stored in localdb)

	1 - John Doe	2 - Jane Doe	2 - John Doe	coherent rate = 50%
1	x			1
2		x	x	2
identifiant rate = 100%	1	1	1

report.html

Field	Nil	Empty	Masked	Missed	Masking Rate	Coherent Rate	Identifiable Rate
idutil	0	0	4	0	100%	50%	100%

[adrienaury]

Another possible configuration

Configuration for Stream A

A/config.yaml

version: "1"
metrics:
  - name: "name"
    coherentWith: ["id"]
  - name: "surname"
    coherentWith: ["id"]

Configuration for Stream B

B/config.yaml

version: "1"
preprocess:
  - name: "name"
    value: '{{index (.idutil | split " ") "_2"}}'
  - name: "surname"
    value: '{{index (.idutil | split " ") "_3"}}'
metrics:
  - name: "name"
    coherentWith: ["id"]
  - name: "surname"
    coherentWith: ["id"]

Result

report.html

Field	Nil	Empty	Masked	Missed	Masking Rate	Coherent Rate	Identifiable Rate
name	0	0	4	0	100%	50%	50%
surname	0	0	4	0	100%	100%	0%

[adrienaury]

For deep nested objects/arrays, the configuration should provide a way to tell MIMO where to insert the preprocessed field.

Example

version: "1"
preprocess:
  - name: "batchs.[].accounts.[].clean_id"  # new field inserted at each account object
    value: '{{$account := Stack -1}}{{$account.id | trimSpaces}}' # access to the stack via Stack(index) function