Serialization/Deserialization of RML files from/to YAML

[lobis]

I think YAML is an overall better format for our use case compared to XML, the only read advantage of XML being that we are used to it.

We can make use of a schema to help the user correctly write / modify an RML file. (this can also be done with XML if anyone dares...).

a PoC is available at https://github.com/lobis/radiation-transport/tree/main/libraries/Simulation/Config, as you can see the amount of code needed is much smaller than what we need to TRestMetadata.

There are two ways to do this:

• Extend (rewrite) our TRestMetadata classes to support both XML and YAML files. I think this would add too much complexity to our already complex code.

• Write a parallel TRestMetadata (TRestConfiguration ?) in an experimental namespace and work to support all current configurations, translate all RML files in the framework and once it is stablished deprecate the XML files.

I like the second option the most but it is up for discussion.

[Vindaar]

My point about the choice of YAML:

YAML is a very complex data format. That means two things:

1. your external dependency is complex and large. While that might not bother you, it means it's more likely you either run into bugs or you should update your external dependency more often.
2. a complicated file format allows people to come up with "genius" ways to abuse it. My point about JSON being a subset of YAML is that people at some point might start to use more and more obscure features of the format. After multiple years of using the format you could end up at in a place where suddenly the REST YAML files aren't actually very simple anymore (which is one of the main reasons to move away from XML { aside from the obvious of XML being a boilerplate, human unfriendly mess } ).

So, what I'm simply proposing is to use the simplest file format we can get away with. Something like TOML for example:

https://toml.io/en/

But: I'm not saying the choice of YAML is necessarily bad, but I want to stress that such a choice brings with it potential problems as well. So be aware of them before making the choice and investing the time.

Regarding keeping the YAML schema in line with changes in the framework:
Ideally that should happen automatically based on some tool to generate it based from the C++ source. But that's probably insanely difficult. :/

[jgalan]

While taking the brave path of moving on with these changes, there are few considerations not to be forgotten:

The RML (through TRestMetadata methods) implements the following capabilities:

• Interpretation of mathematical expressions using TFormula.
• Implementation of FOR loops and IF conditions.
• Accepting the use of system variables through ${USER} format.
• Automatically reading parameters from InitFromConfigFile using different types and physical units.
• Propagating variable and parameter at the <globals> section to all metadata sections.
• <include file statement.

Those are some features I remember by heart. I don't say that all should be implemented in a future implementation, but if we invested our time to write those at some point it was for a reason.

= Drawbacks

• We will need a XML to YALM convertor tool to do the change everywhere!
• I would say that both approaches should co-exist at the beginning. Then, we should have a validation tool that reads the RML and the YALM file and compares the metadata members one to one to check the result is the same.
• For the auto-completion it requires additional maintenance. In my user experience I don't use auto-completion, I just go to a working example, and modify accordingly.

But more important, we should remember that TRestMetadata is a class that defines the versioning of REST. Those members should be kept at TRestMetadata. Then, I would say, a first step would be to differentiate the XML code outside TRestMetadata and move it to a new class TRestXMLMetadata. And test the differentiation adding inheritance from TRestXMLMetadata. TRestMetadata::TRestXMLMetadata

Another problem is that some inherited classes use directly xml library functions. At some point I tried to push to have all xml code encapsulated inside TRestMetadata, then derived classes just call methods at TRestMetadata, but unfortunately it is not the case. This will be some extra work to be aware of.

Probably XML is not the best, but from a personal point of view it still readable, and moving to other formats would require to change the present user habits to a new scheme.

I cannot adventure the consequences, from my point of view is a considerable work load, and we may end up on the scenario that is a question of taste and simply getting used to something that was already stablished. Most problems using XML have probably a solution, and we have full control on that. From my personal user experience is clear that I am comfortable with XML, if others cannot live with, then, we must find a way to use both formats at the same time. Then at some point, when everyone would be using the new scheme, and it is clear everything is working, do the transition. It might be a long way to get there. There was a major change in RML sometime ago, and from experience, it took us some time to adapt.

@nkx111 may have other insights.

=====

There are other features one could dream of, i.e. automatic reading of metadata members as follows:

TRestAAAMetadata::TRestMetadata

       TRestBBBMetadata *fMetadataMember;

and could be read as follows, or with YALM or TOML format.

<TRestAAAMetadata name="aaaName" />
         <parameter .... >
         <TRestBBBMetadata>
                     ....
         </TRestBBBMetadata>
</TRestAAAMetadata>

[lobis]

My point about the choice of YAML:

YAML is a very complex data format. That means two things:

1. your external dependency is complex and large. While that might not bother you, it means it's more likely you either run into bugs or you should update your external dependency more often.

Currently my solution would be to include this dependency directly on the cmake script, meaning it will clone and build the yaml-cpp library on the first build of REST. This allows the project to set a fixed version of the library and avoids the user having to install it. It takes very little time to build this library by the way, much less time than for example the google testing framework for c++. If required we could update it by changing a single line in the cmake and it would be transparent to users, hopefully its not required to update it frequently, if at all.

2. a complicated file format allows people to come up with "genius" ways to abuse it. My point about JSON being a subset of YAML is that people at some point might start to use more and more obscure features of the format. After multiple years of using the format you could end up at in a place where suddenly the REST YAML files aren't actually very simple anymore (which is one of the main reasons to move away from XML { aside from the obvious of XML being a boilerplate, human unfriendly mess } ).

I guess people should restrain from using all these freatures then as they are not supported by the framework, not sure if you raise this as a security concern. I don't think RML files are or will be very simple, they are practically a programming language already, but without all the good things about one such as linting. At this point perhaps we should be considering using python scripts as the configuration files (half joking).

Edit: I guess you mean something like this.

So, what I'm simply proposing is to use the simplest file format we can get away with. Something like TOML for example:

https://toml.io/en/

But: I'm not saying the choice of YAML is necessarily bad, but I want to stress that such a choice brings with it potential problems as well. So be aware of them before making the choice and investing the time.

Regarding keeping the YAML schema in line with changes in the framework: Ideally that should happen automatically based on some tool to generate it based from the C++ source. But that's probably insanely difficult. :/

I like your suggestion about toml, it was also in my list of possible alternatives to XML. To be honest I chose yaml because it seems very complete and many tools I use already use this format, such as the CI tools (GitLab's and GitHub's Actions) or my home's Home Assistant installation.

Taking a look into one of the TOML libraries, such as https://github.com/marzer/tomlplusplus I can see its pretty similar to https://github.com/jbeder/yaml-cpp. However I don't see much about validation through schemas, only toml-lang/toml#792.

I also found about https://github.com/crdoconnor/strictyaml which is supposed to solve these problems, there are many interesting points in the README. It has some interesting comments about toml too. The biggest drawback seems that there are no parsers in other languages except python.

Personally I disagree with implementing things such as logical expressions, loops etc in the configuration file parser. It would be interesting to find another project which faced this problem and see which solution it chose.

[jgalan]

Personally I disagree with implementing things such as logical expressions, loops etc in the configuration file parser. It would be interesting to find another project which faced this problem and see which solution it chose.

We need for loops to create TRestDetectorReadout descriptions, for example. RML is not a plain configuration file, is a dynamic mechanism to define metadata descriptions. If you disagree with that you need to find another way to do that.

[jgalan]

Wow, I didn't know issues could be moved to discussions!

[nkx111]

The simplist way to satisfy every one is to add a class to convert yaml to rml. I am not familiar with yaml, but from the first view I find it similar with xml in the *hierarchy. Then it is straight forward to do the conversion. For TOML it seems too simple to populate.

I just have a try manually:

<TRestManager name="CoBoDataAnalysis" title="Example" verboseLevel="info" >

  <TRestRun name="SJTU_Proto" >
    <parameter name="runNumber" value="auto"/>//change this value to "auto" to enable database
    <parameter name="outputFileName" value="RUN[RunNumber]_[Time]_[LastProcess].root" />

    <addMetadata name="PandaReadout_MxM" file="meta.root"/>
    <TRestDetectorGas name="Xenon-TMA 1Pct 10-10E3Vcm" pressure="10atm" electricField="1000V/cm" file="server"/>
  </TRestRun>

  <TRestProcessRunner name="Processor" verboseLevel="info">
    <parameter name="eventsToProcess" value="0" />
    <parameter name="threadNumber" value="2"/>
    
    <parameter name="outputEventStorage" value="on"/>

    <addProcess type="TRestRawMultiCoBoAsAdToSignalProcess" name="virtualDAQ" value="ON"/>
    <addProcess type="TRestRawSignalAnalysisProcess" name="sAna" value="ON">
      <observable name="BaseLineSigmaMean" value="ON" />
      <observable name="ThresholdIntegral" value="ON" />
    </addProcess>
    <addProcess type="TRestRawReadoutAnalysisProcess" name="rA" value="ON">
      <parameter name="observable" value="all"/>
    </addProcess>

  </TRestProcessRunner>

  <addTask command="Processor->RunProcess()" value="ON"/>

  <globals>
    <searchPath value="$ENV{REST_INPUTDATA}/definitions/"/>
    <parameter name="verboseLevel" value="essential" />
    <parameter name="mainDataPath" value="" />

    <parameter name="pointThreshold" value="3" />
    <parameter name="pointsOverThreshold" value="3" />
  </globals>

</TRestManager>

TRestRun:
    name: SJTU_Proto
    runNumber: auto
    outputFileName: RUN[RunNumber]_[Time]_[LastProcess].root
    addMetadata:
      - name: PandaReadout_MxM
      - file: meta.root
    TRestDetectorGas: 
        name: Xenon-TMA 1Pct 10-10E3Vcm
        pressure: 10atm
        electricField: 1000V/cm
        file: server

TRestProcessRunner:
    name: Processor
    verboseLevel: info
    eventsToProcess: 0
    threadNumber: 2
    outputEventStorage: on
    addProcess:
      - type: TRestRawMultiCoBoAsAdToSignalProcess
        name: virtualDAQ
        value: on

      - type: TRestRawSignalAnalysisProcess
        name: sAna
        value: on
        observable:
        - name: BaseLineSigmaMean
          value: on
        - name: ThresholdIntegral
          value: on

      - type: TRestRawReadoutAnalysisProcess
        name: rA
        value: on
        observable: all

addTask:
  - command: Processor->RunProcess()
    value: on

globals:
    searchPath: $ENV{REST_INPUTDATA}/definitions/
    verboseLevel: essential
    mainDataPath: ""
    pointThreshold: 3
    pointsOverThreshold: 3

@lobis is that correct in syntax? It seems the conversion is logically clear, which makes the coding easier. Note that the rml "parameter" is generalized(i.e., equivalent to the attribute of parent element), and are all set as yaml's config sections, without appearance of the "parameter" keyword.

[jgalan]

And what about FOR loop definitions?

[Vindaar]

And what about FOR loop definitions?

I honestly think adding on things like for loops etc. on a configuration format is a bad approach. Once one realizes such a thing is necessary, the configuration file approach starts to be the wrong approach, in my opinion.

In general I agree with @lobis on this. At that point you might as well use a proper scripting language for these needs. Whether that is Python, Lua or dare I say (:P), NimScript, isn't important. But it's important to use the right tool for the job.

If you don't want that for some reason, then I'd almost say writing your own custom DSL to handle this is the better solution. That way at least you can have well defined logic, that matches all your requirements. Of course that's a lot of work and I personally wouldn't recommend it.

[lobis]

In general I agree with @lobis on this. At that point you might as well use a proper scripting language for these needs. Whether that is Python, Lua or dare I say (:P), NimScript, isn't important. But it's important to use the right tool for the job.

But doesn't this just introduce the same problematic as the complex YAML definition, but on steroids? now people could write a reverse shell on the RML file, or some other crazy thing.

Writing a DSL would probably involve a lot of work but it may be worth it at least for learning purposes, seems interesting. Do you know by any chance any examples of this? I guess it should be done in C++ to avoid replicating code.

[Vindaar]

But doesn't this just introduce the same problematic as the complex YAML definition, but on steroids? now people could write a reverse shell on the RML file, or some other crazy thing.

Oh, it definitely does! But at that point you can start to consider an entirely different workflow. Ideally, one would choose a typed language and define a sane, typed API that is used to define the workflows.

Writing a DSL would probably involve a lot of work but it may be worth it at least for learning purposes, seems interesting. Do you know by any chance any examples of this? I guess it should be done in C++ to avoid replicating code.

There are many different approaches & levels of abstraction possible.

In the simplest form you write a DSL that is essentially just a rule based / pattern matching like thing. I.e. in your code you describe the patterns possible, extract the required fields / elements from that pattern as needed. And error if you have a pattern mismatch. To an extent this would be something like:

• shell: it's a DSL to write shell commands with a few neat constructs
• LatexDSL: a CT checked DSL to generate valid LaTeX from Nim code

Beyond that you get to a point where you might want a "compiler light". You start to split your input handling into a multi stage process. Input is transformed into some basic intermediate representation that makes handling saner. Then you perform the necessary transformations / checks etc. on that IR and finally generate the desired output. An example might be something like the formula syntax in Datamancer. I still need to write an up to date explanation of how it works, here are some rambling development notes and here a comparison of Datamancer to Pandas & Dplyr where the usage can be seen. The implementation of this macro is rather, uhh, complicated. Relevant code is here and here.

Depending on your use case, you might get away with a PEG based implementation though. For example with npeg there's an example of how to implement a JSON parser here.

Beyond that you get into the part where the correct solution is to write a full on compiler. That means you implement a lexer, which performs tokenization of the input, then a parser that builds some kind of abstract syntax tree and from there implement your transformations / checks / etc. There's obviously many books etc. about compiler development. A relatively simple (not extremely elegant, but easy to get the gist) is maybe the expression evaluation lib I wrote for REST.

The main point there is really the importance of lexer ⇒ parser ⇒ AST ⇒ stuff.

This is a bit rambly of course. If you want to know specifics or want better resources about specific things I mentioned, let me know and I try to dig up something that goes beyond the things floating in my head right now.

[nkx111]

Well, I suggest we just keep the current implementation of "for loop reading from rml". What to change is not TRestMetadata, but TRestDetectorReadout. We can introduce some new definition for readout channel/pixel adding, and mark this approach as recommended. An easy way to replace for loop is to define "repeat times" inside the config section:

origin rml:

<readoutModule name="module" size="((nChannels+1)*pitch-1*pitch/2,(nChannels+1)*pitch-pitch/2-pitch/4)" tolerance="1.e-4" >
  // Y-strips
  <for variable="nCh" from="1" to="nChannels-2" step="1" >
    <readoutChannel id="${nCh}" >
      <for variable="nPix" from="0" to="nChannels-1" step="1" >
        <addPixel id="${nPix}" origin="((1+${nCh})*pitch,pitch/4+${nPix}*pitch)" size="(pixelSize,pixelSize)" rotation="45" />
      </for>
      <addPixel id="nChannels" origin="((1+${nCh})*pitch-pitch/2,pitch/4+(nChannels-1)*pitch+pitch/2)" size="(pitch,pitch/2)" rotation="0" />
    </readoutChannel>
  </for>
  </readoutChannel>
</readoutModule>

without for loop:

<readoutModule name="module" size="((nChannels+1)*pitch-1*pitch/2,(nChannels+1)*pitch-pitch/2-pitch/4)" tolerance="1.e-4" >
  // Y-strips
    <readoutChannelArray repeat="nChannels-1">
      <id start="1" step="1" />
      <x  start="2*pitch" step="pitch" />
      <y  value="pitch/4"/>
      <readoutPixelArray repeat="nChannels-1" size="(pixelSize,pixelSize)" rotation="45">
        <id start="0" step="1" />
        <x  value="0" /> //relative to readout channel coordinate
        <y  start="0" step="pitch"/>
      </readoutPixelArray >
      <readoutPixel id="nChannels" origin="(-pitch/2,pitch/4+(nChannels-1)*pitch+pitch/2)" size="(pitch,pitch/2)" rotation="0" />
    </readoutChannelArray>
</readoutModule>

yaml:

readoutModule:
    name: module
    size: ((nChannels+1)*pitch-1*pitch/2,(nChannels+1)*pitch-pitch/2-pitch/4)
    tolerance: 1.e-4
    readoutChannelArray:
        repeat: nChannels-1
        id: 
            start: 1
            step: 1
        x: 
            start: 2*pitch
            step: pitch
        y: pitch/4
        readoutPixelArray: 
            repeat: nChannels-1
            id: 
                start: 1
                step: 1
            x: 0
            y: 
                start: 0
                step: pitch
            size: (pixelSize,pixelSize)
            rotation: 45
        readoutPixel:
            id: nChannels
            origin: (-pitch/2,pitch/4+(nChannels-1)*pitch+pitch/2)
            size: (pitch,pitch/2)
            rotation: 0