Wrong "data type" field in sub-geometries streams datas

[plepers]

Hi,

In C4D exporter data type field seems to be set to the size (in bytes) of the underlying data type, 2 for U16 and 4 for F32, and don't take precision into account ( float or double)

Ok, specifications are not perfectly clear about this field.
But according to awd-sdk and AwayBuilder encoder implementation, this field should match one of the AWD_field_type, depending on stream precision:

Integer : 5 (AWD_FIELD_UINT16)
Floats low : 7 (AWD_FIELD_FLOAT32)
Floats hi : 8 (AWD_FIELD_FLOAT64)

The types enum :
https://github.com/awaytools/awd-sdk/blob/master/cpp-libawd/include/awd_types.h#L39

AWD-SDK stream impl:
https://github.com/awaytools/awd-sdk/blob/master/cpp-libawd/src/stream.cc#L49

AwayBuilder AS3 impl
https://github.com/awaytools/AwayBuilder/blob/master/awaybuilder-core/src/awaybuilder/utils/encoders/AWDEncoder.as#L1931

[80prozent]

Hi
Yeah. I can see why this is confusing.
If you look into the AWD2parser, you can see that the str_ftype is read for each subgeometry-stream, but the var is never used.
This is because the storage precision (float32 / float64) is set as bitflag in the header of the geometry-block.
There is no encoder that will use uint32 instead of uint16, so for now i left it as it is.
If i remember correctly, the c4d exporter has no option for storage-precision yet. It will always store Float32 and set the bitflag in the blockheader accordingly.
Its definitv something that can cause headachses in the future, so i doubt we will ever activate the the usage of this value.
I think its much more important to implement a new type of stream, that holds all vertexdata in the correct order, so we do not have to parse every value, but can send the whole junk of bytes directly to the gpu (without really parsing it). This might result in a bigger AWD-file (we have to store the normalData and tangentData into the file), but the parsing time will be reduced to a minimum.

Thanks for reminidng me about this issue.
I should update the specs to make this more clear.

[plepers]

Hi,

I think its much more important to implement a new type of stream, that holds all vertexdata in the correct order, so we do not have to parse every value

It's exactly what i try to achieve :).
I parse a legacy AWD and replace all Geometry block by namespaced "InterleavedGeometry", which hold interleaved vertex buffers and a "stride/offset" table for each attributes.

Since opengl buffer cannot hold mixed types (GL_FLOAT and GL_SHORT), i need to split interleaved buffers by type. Here i use the str_ftype field.

If you look into the AWD2parser, you can see that the str_ftype is read for each subgeometry-stream, but the var is never used.

Yep, but the parsing rely on the fact that parser exactly know the predefined type of each stream:

 id 1 : position, float, 3 component
 id 3 : uvs       , float, 2 component
 etc...

to guess the read methods, number of vertices etc...

I try to have weak dependency to those predefined types, because in generally need custom attributes (eg : vertex colors). So for now i'm just rely on 1:position to guess the number of vertices, then guess numbers of component of other streams with str_ftype, the byte length of stream and number of vertices.

Anyway, for now i fix the issue by replacing unexpected str_ftype by correct ones since "nobody need signed short or unsigned byte buffers".

---

One more word about "send the whole junk of bytes directly to the gpu "

I'm using AWD format in webgl. And to really upload awd file in GPU without copy, streams need to be aligned.

https://www.khronos.org/registry/typedarray/specs/latest/#7

The given byteOffset must be a multiple of the element size of the specific type

for my "InterleavedGeometry" i add a padding field before every streams, to ensure the stream start on a multiple of the type size.
It may be nice to add this feature in future version.

[80prozent]

Hey

I see that the valid setting of str_ftype is needed, when working with custom streams.
I will definitive add this into the next updates of the exporter.

Yep, but the parsing rely on the fact that parser exactly know the predefined type of each stream:

The thing is: none of our plugins had support for custom-streams, so the value really never was used.
All encoder can only set storage-precision globally (per AWD file). Using this setting together with the stream-type, we always know the data-type to read for the supported geometry-streams.

If someone was to do implement custom-streams (as you are doing now), he would have to use the str_ftype inside the parsing function that is exclusive for the custom-stream type anyway.

For all existing stream types, str_ftype really has no meaning.

I know this is a little inconsistent, so as mentioned before, i will change this in never versions.

I think the way you describe how you use the str_ftype is the same i would do when implementing custom streams.

I still not sure if i understand what your are doing in the end.

Are you modifing older AWD files offline, or are you modifing the c4d plugin to export custom stream ?

Thank you for the tip on the byte-offset. this might be really helpful / useful.

[plepers]

Hi,

Until now, i used to modify exporters and parser to fit my project's needs. Add custom vertex attributes, specific geometry types and other fancy stuffs.

Customizing exporter is pretty tought and we don't use the same 3D editor for each project (3DS, C4D...).
The fact is, i have to rewrite parser/encoders for every project.

Now i try to make a tool which take an fully compliant AWD file and output a modified one. (Interleaved geometries, swap Y/Z, merge geoms etc). So i can reuse / add those features across projects.

Customs vertex attributes can only be added by exporters, so i still need to customize them. But the tool should be able to parse, transform geometries containing those custom streams. That's why i try to not depends on predefined streams types.

[80prozent]

hey

Sounds very interesting.

What environment will you use to develop your tool ?
You are planning to open source it ?
Maybe even add it to the AwayTools repo ?

If you have any open source plans, i would be happy to assist you as good as i can.

Right now, i am the only active dev for all the awaytools, and its a lot that has to be done, and so little time.

I am working heavily on the FlashPro-exporter.
This is mostly about 2d stuff, but the development of this tool bring some nice additions to AWD, that can also used on 3d-objects (e.g. timeline object).
I already set up the specs or it, and making first test with actual code in js and as3, and so far it looks good (having timeline behavior for flahs-pro exported stuff, planing to add this into the 3d-exporters too).

I recently started to look into the FBX SDK, and started to work on a little cpp-application that will allow to convert FBX files into AWD. Doing this mainly because having a FBX to awd solution, will take away the pressure on tool-development for many 3d-packages (e.g. autodesk stuff).
I do not have much on this yet (only spend some h of research and experiments), but i am thinking maybe we could join the development of our tools into one good AWD-conversion tool.
Using the FBX SDK brings parsers for some other FileFormats also (obj / collada), so it should be easy to include some more conversion options into this tool.

In general i am happy about all help i can get, to make AWD workflows as painless as possible.

Anyway. if you interested in cooperation on this, just ping me on skype ("achtzig prozent").

[plepers]

Hi,

The tool is here:
https://github.com/plepers/awd-js

I started it last week for an on-going webgl project, so it's a bit rought and messy right now and i can only focus on features i need. It only parse "core features", geometries, meshes, containers...
It run on node, but could run in browser too. It's ideal for me because i can easily add "modifiers" in js directly in a grunt build chain.

FBX to AWD is definitely the ultimate solution, since almost all 3D editors can export fbx. I will help in it's development if you plan to use the C++ sdk. I didn't wrote python for years!

Pierre

[80prozent]

Hey
Cool.
I will have a look at your tool soon. Very interesting.
The next days i will not be available.
Hope i will be back online at end of week.

Cheers

Robin