Geometrical interface for raytracing models?

[sebastian-wolter-itk]

Hello everyone,

OSI is also supposed to work with raytracing sensor models.
If the simulator and the raytracing sensor model are seperated, the geometry from the simulator needs to be transferred to the sensor model.

Currently OSI only offers the way of defining a model_reference for stationary/moving/trafficLight/trafficSign/etc.
It is "implementation-specific how model_references are resolved to 3d models." as stated in the documentation.
That means you could insert "Car.obj" or just "Car" or some unique id for the asset or anything.

My first guess was to put in the actual 3d file, e.g. Car.obj

But simulators might use different 3d formats as inputs:

• IPG CarMaker: .obj / .mobj
• CARLA: .fbx
• Nvidia Omniverse: .usd
• Unity/Unreal: mix of multiple formats
• Blender: .blend / multiple formats

How should it work then?

Solution 1:
make the raytracing sensor model be compatible with multiple simulators => it needs to be able to load all these file formats.
Solution 2:
force every simulator to use a common 3d format (e.g. .gltf), => they need to convert every source 3d model to this format.
Solution 3:
provide a generic protobuf geometry interface to send just the mesh/material data (after the actual file format was parsed; so the source file format does not matter)
All 3d file formats have in common, that they provide a list of vertices and indices, normals, uvs, a topology (triangle, quad, strips, ...), list of materials and mapping which index belongs to which material. (edited: also add sensor specific properties for radar, lidar, ultrasonic)
For Animations also: weight of vertices to skeleton bones

Is there some recommended solution?

Regards,
Sebastian Wolter

[pmai]

Given the as of yet lack of convergence on both the 3d format and material data fronts, OSI could only support the bare bones model_reference approach as of now.

This means that there needs to be some extra-standard arrangement between simulator and model to properly convey the relevant data.

This can mean your solution 1, i.e. arranging for the model to support all kinds of 3d model data formats and use of model_reference to name the relevant file.

However this can also mean specifiying clear keys for the model_reference (e.g. "net.pmsf.osi3d.model.luxury_sedan_model_xx", ...) that are supplied by the different simulators, and then used by the model to look up a corresponding 3d model and material data in its own formats from its own database.

There is usually no need to tie 3d modeling/rendering of simulator and model to each other, given that they have different objectives (a 3d model to use for ray-tracing based radar simulation is unlikely to be interesting or useful to render visual output for human observers or vice-versa).

If current activities to converge on one format for 3d models and material data for ray-tracing across simulators and use cases does succeed (cf. OpenMATERIAL and related efforts), then solution 2 will become more feasible going forward. I don't see solution 3 ever becoming an interesting approach, given that this just adds yet another file format to the picture which is not going to be helpful in terms of implementation effort or performance.

[sebastian-wolter-itk]

look up a corresponding 3d model and material data in its own formats from its own database.
For phenomenological models where you have measured radar rcs values of objects, totally fine with that. This is also similar to the way it works with OpenScenario/EsMini.

But for raytracing models, don't you compare apples with pears if you substitute with some "corresponding model" (like RFPro substitutes IPG CarMaker models with their own ones)? Then the camera rendering is working based on a meshA, the radar is working on meshB and ultrasonic on meshC. Then there is no consistency. I see that a lot, that the sensor model uses just a "generic car" data and one microdoppler-pedestrian-pattern regardless of actual car type or pedestrian type or pedestrian animation.

a 3d model to use for ray-tracing based radar simulation is unlikely to be interesting or useful to render visual output for human observers or vice-versa

i would strongly disagree with that. In terms of materials, yes. Texture maps, material settings like roughness, refractive index, metalness, ... are approximated for visible light/PBR Rendering.
Radar, Lidar and Ultrasonic need different wavelength-dependend materials properties (Solution 3 was meant to deliver this).
But the source geometry is the same.

Isnt that the whole point of OpenMaterial, to define material properties specific for different sensor technologies and wavelengths, but reusing the mesh geometry in the gltf file?

given that this just adds yet another file format to the picture

sure, i would be happier, if there is something preexisting easy to use interface, which gives me low effort on integration in existing software. But from my perspective the other solutions are not practical at the moment.

• Solution 1:
  • needs access to source model file system
  • needs to support multiple file formats
  • Or create a specific sensormodel version (as you said "some extra-standard arrangement") for every simulator you want to connect
  • Or substitude models with local corresponding models.
• Solution 2:
  • only works with simulators which output .gltf (probably only Blender at the moment?).
  • I don't think GLTF is supposed to be streamed.
    • But you could use a generic file transfer interface to transfer between simulator and sensor model.
  • OpenUSD could also be an alternative maybe.
• Solution 3:
  • based on protobuf, so can be used with any prog. language you want (like OSI)
  • Easy integration in existing software
  • Works also with procedural generated in the simulator (e.g. grass)
  • Much less implementation effort to fill in the mesh data than writing a gltf exporter to a SW which does not have one.
  • At the end you set the model_reference in OSI to the pretransfered model names.

[pmai]

But for raytracing models, don't you compare apples with pears if you substitute with some "corresponding model" (like RFPro substitutes IPG CarMaker models with their own ones)? Then the camera rendering is working based on a meshA, the radar is working on meshB and ultrasonic on meshC. Then there is no consistency. I see that a lot, that the sensor model uses just a "generic car" data and one microdoppler-pedestrian-pattern regardless of actual car type or pedestrian type or pedestrian animation.

There is no consistency in any case, if you use different ray tracing engines, different modeling assumptions, different quality of geometry and material data (see below). Furthermore, where would the need for "absolute" consistency come from, given the myriad of implicit and explicit assumptions made in modeling and the general level of abstraction of the environment (just consider the handling of rain in an OSI context)?

There is a reason that tools like RFPro use different models than tools like CarMaker, where the goals of rendering fidelity differ wildly, and that is just for purely optical wavelengths.

Not that this is supposed to be a comment on either of those tools, just an observation that different use cases require different trade-offs, as has always been the case in simulation: All models are wrong, just some are useful - you have to pick your poison, and you have to validate each simulation according to the actual requirements in terms of fidelity, quality and other aspects.

And the approach outlined, where the sensor model picks the appropriate 3d model according to model_reference keys does not suggest you should just use one 3d model, but rather have as many high-accuracy and validated 3d models as you need for your scenarios and use-case. In fact this can mean that usually there will be a 1:n mapping rather than just a 1:1 or n:1 mapping from simulator models to sensor model models.

If you require that level of fidelity for a certain use case. Which is usually the big if, given that for most system-level simulation purposes it is IMHO questionable to use high-fidelity sensor models in the first place. But that is a different discussion.

a 3d model to use for ray-tracing based radar simulation is unlikely to be interesting or useful to render visual output for human observers or vice-versa

i would strongly disagree with that. In terms of materials, yes. Texture maps, material settings like roughness, refractive index, metalness, ... are approximated for visible light/PBR Rendering. Radar, Lidar and Ultrasonic need different wavelength-dependend materials properties (Solution 3 was meant to deliver this). But the source geometry is the same.

Except that it isn't the same: Why would one use the same geometry model to satisfy 4mm wavelength propagation questions vs. 750nm wavelength propagation (i.e. resolution would differ by 6 orders of magnitude)? Especially when much of the material that for visual propagation is opaque is semi-transparent for radar (and vice versa). So much of the underlying geometry does not matter for one use-case, but does for the other. E.g. the crashbar geometry underlying the bumper geometry is often of crucial importance for radar propagation (as is the recycling material mix of the bumper for that matter), but who cares to model to this depth for visual rendering?

Don't get me wrong, I'm not suggesting that there never is a case for re-use, but in general that comes with at least as many trade-offs as sensor-specific models have, so this is not a panacea. And we have not yet delved into matching the geometry to the underlying ray-tracing modeling approaches, either.

Isnt that the whole point of OpenMaterial, to define material properties specific for different sensor technologies and wavelengths, but reusing the mesh geometry in the gltf file?

That is one idea, but I would not characterize it as the whole or even a majority of the point: The major advance of OpenMaterial would be to enable the largish material databases that are needed to model reality to even become broadly exchangeable. This has huge benefits, regardless of whether you use the same or different meshes for the underlying geometry for the different sensor models.

Fixing on one standard format for the exchange of the 3d geometry would also be very beneficial in terms of tooling and model exchange, even if you still use different actual geometries for the different sensors.

• Solution 1:

  • needs access to source model file system

Which is usually not much of a problem, given that simulation, if it is to be efficient, will always be single system image (distributed simulation does not make much sense when there is massive parallelism due to the scenario workload that makes for much more efficient use of resources, especially given today's over-abundance of cores (CPU or GPU) in a single system image).

And it enables the use of standard tools and techniques for broad distribution of content, which all clusters have, without having to reinvent that badly as a speciality for OSI: You can easily distribute one simulation image containing all relevant data across a simulation cluster for a million scenario runs, and get the benefits of caching, shared memory mapped I/O etc. without much effort at all.

• needs to support multiple file formats
• Or create a specific sensormodel version (as you said "some extra-standard arrangement") for every simulator you want to connect
• Or substitude models with local corresponding models.

These sensor models will always have to be validated and fine-tuned to individual simulators to achieve the fidelity levels expected of them, otherwise one would likely not use ray-tracing in the first place, I would venture. So in practice, while this is a hassle, it is often not much of an added one to the hassle that is already there. Again not arguing that the state of affairs is great, but it is manageable for the instances that warrant the level of detail that ray-tracing is supposed to enable.

• Solution 2:

  • only works with simulators which output .gltf (probably only Blender at the moment?).

  • I don't think GLTF is supposed to be streamed.

    • But you could use a generic file transfer interface to transfer between simulator and sensor model.

  • OpenUSD could also be an alternative maybe.

• Solution 3:

  • based on protobuf, so can be used with any prog. language you want (like OSI)
  • Easy integration in existing software
  • Works also with procedural generated in the simulator (e.g. grass)
  • Much less implementation effort to fill in the mesh data than writing a gltf exporter to a SW which does not have one.
  • At the end you set the model_reference in OSI to the pretransfered model names.

Actually solution 2 and 3 are more or less basically the same: You standardize the way that simulators provide 3d models to sensor models.

Whether you do this by picking an existing file format, or invent a new one, is really not much different, except for solution 3 you first have to put in the not inconsiderable work (we are talking about person years) to standardize this from scratch, and then you start with an implementation base of 0 vs. at least 1 for solution 2.

And with solution 2 you actually make it possible to reuse the same 3d models across simulators, because they will likely support the format as input and not just as output, whereas in solution 3 you still have not solved the problem of how to get the 3d models into the simulator in the first place.

And it is likely to always be file based, because why would you try to transfer potentially gigabytes of data using a network-centric protocol that affords little in the way of centralized distribution, cacheing, etc. for generally mostly static data with much re-use? When that problem is mostly solved for files using standard mechanisms. Not to speak of the performance problems you are likely to run into using protobuf, which we have already seen for reflection-based interfaces in OSI, which are likely much less problematic than mesh-data and materials, not to speak of potentially volumetric data in the future?

Now don't get me wrong, I love OSI for what it is (even given that I am personally somewhat at fault for some of its features), but I don't think stretching it to be yet another 3d model format is a good use of resources.

Now that does not mean that if someone comes up with a concrete proposal that you might not find sufficient interest to persue this. And given that OSI like ASAM is contributor/member-driven, anything that makes some sense and is supported by a large number of contributors would be welcomed. But currently I'd think that investing in pushing something like OpenMATERIAL forward is a much more likely approach to persue.

Just my two centi-euros for what it's worth.

[sebastian-wolter-itk]

There is no consistency in any case, if you use different ray tracing engines, different modeling assumptions, different quality of geometry and material data (see below). Furthermore, where would the need for "absolute" consistency come from, given the myriad of implicit and explicit assumptions made in modeling and the general level of abstraction of the environment (just consider the handling of rain in an OSI context)?

Sure the sensor models always consist of a mixed parts of empirical, statistical, phenomenological and physical effects.
e.g. rain: simulator simulates rain, for the camera you have postprocessing effect on windshield, rain as particles in air and rain puddles as decals. If the radar model only use statistical rain, it should be at least synchronized with the camera rain effect. Otherwise you perceive two different worlds at the end.

If you mix a bounding box sensor model A and a raytracing model B, they are probably not so consistant as having both as raytracing models.

Sure there will be no 100% accuracy model at the end. You take was is available and if you can reuse the same geometry you should try that to get higher consitency between sensor technologies. It is also clear that you have to validate anyway.

Even you use embree for lidar and optix for radar with different brdf, they share the same geometry. If i move the vehicle door i see it both in radar and lidar without doing anything => so results are consistant in terms of the world change (maybe we have different understanding of the wording)

There is a reason that tools like RFPro use different models than tools like CarMaker, where the goals of rendering fidelity differ wildly, and that is just for purely optical wavelengths.

They did this, because (before MovieNX) IPG CarMaker only supported low res 3d models caused by the old 3d engine. Therefore rfpro used a more capable 3d engine, so they could use raytracing for lidar, radar, camera with high fidelity 3d models.

Except that it isn't the same: Why would one use the same geometry model to satisfy 4mm wavelength propagation questions vs. 750nm wavelength propagation (i.e. resolution would differ by 6 orders of magnitude)? Especially when much of the material that for visual propagation is opaque is semi-transparent for radar (and vice versa). So much of the underlying geometry does not matter for one use-case, but does for the other. E.g. the crashbar geometry underlying the bumper geometry is often of crucial importance for radar propagation (as is the recycling material mix of the bumper for that matter), but who cares to model to this depth for visual rendering?

In an ideal world where you have a modelled mesh for each wavelength, sure you would use that. I do not see that at the moment. So you take what is available.
Also you suggest that on a multi sensor setup you hold basically for each sensor a the a complete seperate 3d world with GB of data in memory?

In the simulator if interior or motor block is not relevant for sensor tech X you could ignore that for the raytracing (i think you can even flag visibility for each triangle seperately if wanted), but still reuse the base mesh.
If you are in the FEM world and want to make radar bumper investigation with Ansys HFSS or lens design optics simulation, sure you would directly use a CAD model. But for testing AD driving stack, the current simulation tools do not load multiple mesh model variants for each mesh in the scene at the same time in memory.

If we look at the current tools: IPG CarMaker, Vires VTD, CARLA, Gazebo, Nvidia Isaac, AiSim, ... they all use the "1 mesh but different views" approach.

Actually solution 2 and 3 are more or less basically the same: You standardize the way that simulators provide 3d models to sensor models.

from that perspective everything is the same.
For Solution 1 and 2 you have the arrangement, that you need to load the 3d models via filesystem and rebuilt the 3d scene if received the first OSI GroundTruth Message
For Solution 3 you have the arrangement, that you stream everything and rebuilt the 3d scene before receiving the first OSI Ground Truth Message

And with solution 2 you actually make it possible to reuse the same 3d models across simulators, because they will likely support the format as input and not just as output, whereas in solution 3 you still have not solved the problem of how to get the 3d models into the simulator in the first place.

But only if the simulator uses gltf as input sources. But who has that? If the simulators use different sources (normal case for simulators based on 3d engines), they need to convert all their input sources to gltf and provide that to sensor model. If they have geometry only modelled procedurally they need also export it as gltf.

I dont know if i got your "How to get 3d model in the simulator is not solved" point. The simulators already are able to import their source files. All you need to do is to grab the already parsed Mesh data from the memory.
Accessing vertex/index lists data in blender, openscenegraph, open asset importer pretty straight forward.

but I don't think stretching it to be yet another 3d model format is a good use of resources.

The intention was not to directly a proposal of a new geometry interface to be included inside OSI, it was more to figure out how is it intended to use, maybe there is some other solution to connect simulator with sensor model, which i did not think of.

Not to speak of the performance problems you are likely to run into using protobuf, which we have already seen for reflection-based interfaces in OSI, which are likely much less problematic than mesh-data and materials, not to speak of potentially volumetric data in the future?

I would appreciate if there was no for protobuf 2GB limit either. Maybe there is a captnproto fork of OSI in future.
I dont see performance problems if you send mesh data at simulation start in comparison of loading all 3d models from disk at start. Reflection data is more critical because it is send every step at runtime

But currently I'd think that investing in pushing something like OpenMATERIAL forward is a much more likely approach to persue.

I would like OpenMATERIAL more, if it was not created as gltf extension. But i think you can still mix it just parse it and mix it fbx, usd, or whatever source format you like.

The market demands physical realtime models with OSI support, they are using the tools, they are using what is available. They want to use different sensor models in combination for SiL and even HiL Testing and i just wanted to figure out how a solution could look like.

To Summarize:

• OSI keeps it open how to handle the geometry transfer (currently only offers model_reference property)
• So there are multiple ways thinkable:
  • (A) You can either combine simulator and sensor models, then you do not have the problem of using model_reference, because you just output OSI:SensorData. Probably the most common thing the existing simulators are trying to do.
  • (B) (RFPro/CoSimulation) You can run a seperate sensor model still on the same pc and need to construct the 3d scene for the raytracer by
    • (B1) loading the actual file behind the model_reference (Solution 1/2)
    • (B2) loading a substitute 3d mesh based on the model_reference name (Solution 1)
    • (B3) by prestreaming the geometry and use the same name as in model_reference (Solution 3)
  • (C) (Omniverse?) You can run a sensor model on a different pc and need to construct the 3d scene for the raytracer by
    • (C1) loading the actual file behind the model_reference (Solution 1/2) (distributed file system?)
    • (C2) loading a substitute 3d mesh based on the model_reference name (Solution 1)
    • (C3) prestreaming the geometry and use the same name as in model_reference (Solution 3)

Side discussions:

• "1 world model quality shared for every sensor tech vs. N world model qualities needed for N sensor tech"
  • The simulator tools on the market are reusing same mesh for all technologies (but with different materials)
  • Overkill of mesh quality for sensor tech which does not need it vs keeping multiple qualities in memory in parallel
  • Antenna/Bumper/Lens Effect modelling are done in special numerical tools with using CAD data (not in OSI focus anyway)
• Material Databases are needed (OpenMaterial is one step on the way)