For some tasks it is not known in advance how many labels can be expected per instance. In particular this task-type is focusing on sequence labeling. This could be the case for example when having multiple annotators, or when having a hierarchy, like in nested NER. For example the phrase `Australian open' could be annotated as LOCderiv (Australian) and MISC (Australian open). Usually the data annotated with two layers would be represented like:
Australian B-MISC B-LOCderiv
open I-MISC O
In MaChAmp, because we accept an arbitrary number of labels, we adapt a slightly different format:
Australian B-MISC|B-LOCderiv
open I-MISC
As can be seen, multiple labels are joined with a `|' in between. The O does not have to be used if there is already another label. This task type has one important additional parameter:
threshold: The threshold which decides which labels to include; if it is set lower, it will be more likely to output multiple labels per instance. 1.0 means that it will only pick very certain candidates, and with 0.0 it ourputs all labels (default=.7)
For this task-type the default evaluation metric is multi_acc, which is the number of
tokens that get the exact correct set of labels. Note that this task does not work with
the other metrics.