Support multiple segmentations + relations between overlapping/nested segmentations

[frankier]

It seems like it would be a good roadmap item for Cineast to support multiple segmentations per file.

Some examples:

• If there was a feature for the gestures of individual people, a segment would be the person's bounding box and start/end time
• There are already scene based segmentations and fixed interval based segmentations, but we could think of these as separate segments which we constrain the latter to fit the former
• There could be segmentations based on subtitles queues
• Some features correpsondingly may be per-frame which is sort of just a very small segment
• Video captured from TV such as NewsScape can contain metadata for advert/studio/out of studio information which could be association with another segmentation
• All sorts of multimodal things e.g. music videos might have music based segmentation applied to the audio and scene based segmentation applied to the video -- but we might like to segment and search both

Generally I see that there would be at least two relations: contains and overlaps-with. I imagine doing this efficiently such that queries returning multiple related segments can be joined efficiently would need some extra support from the database.

This might also enable showing richer results in Vitrivr i.e. actually show each term of the query on a timeline and bounding boxes within the result.

[lucaro]

This is quite an interesting suggestion. In theory, you could slice a video along all axes or even have arbitrary regions in the 3d spatiotemporal object. It would probably not even require additional support in the database as long as you can efficiently describe the slices, which would be a requirement anyway. It would however require a non-trivial amount of work in both the extraction and retrieval pipelines of cineast and would immensely complicate the feature module specification and the fusion model. I'd estimate that it would be at least a master thesis worth of effort to have an initial working version. It's certainly something to keep in mind, but I don't see this being added to vitrivr in the immediate future.