Feature: USD Workflow for Maya, Houdini and Blender

[BigRoy]

Changelog Description

This is a prototype/propsosal implementation for USD in Maya, Houdini and Blender.

Houdini

• Publishes and loads USD files
• It should now also be possible to publish LOP networks that in itself also define layers with Explicit Save paths in the node graph, they will be generated as the USD instance's subset and appended with _{filename} for each of those save paths - they will turn into individual subsets, give it a go - which should solve this issue houdini: publish usd requires "flatten all layers" in the ROP #3037 (but only for layers that have an Explicit Save path configured)

Maya

• Publishes and loads USD files
  • You can export Maya native geometry via USD
  • You can export a layer from a mayaUsdProxyShape as a USD file; e.g. a look layer in which you created material assignments which can then be contributed as a look variant to the USD asset
  • You can load as a mayaUsdProxyShape but you should also be able to load into a USD stage in a mayaUsdProxyShape by selecting a prim path in the USD stage in Maya and then using the Loader to add the reference to USD.

Blender

• Publishes USD files
• I've currently only implemented very basic loading into Blender by adding it to the existing CacheModelLoader. Whether that suffices I have no idea, but it doesn't provide any variant choice options, etc.

Additional info

There is a lot to discuss and I expect this PR to take some time digest, test and brainstorm about - but it's playable and testable so I think the best first step is taking this branch and just play - and then write down questions/remarks/errors as you go and then reply with all of them in a single first comment. By doing that instead of getting way to much "knowledge beforehand" we can also somewhat lean into the what if this gets into the hands of artists for the first time. Preferably we would want you to face those issues now.

Demos

Having said that, here are some videos:

Some more prototyping:

Houdini + Maya model variants + lookdev workflow testing

This little demo shows:

1. Creating an asset with a few model variations from Maya
2. Creating an Arnold lookdev from Maya using LookdevX + Arnold and exporting that as a lookMain variant for the asset
3. Loading the asset in Houdini, rendering 1-to-1 the same as Maya and then tweaking the shader in Houdini
4. And then a lot more 'playing'/'talking' about what could be done with that.

I forgot to show that if you were to switch the model variants - the shader would actually work for all variants because the shader is assigned to that prim name /hero/geo/cube

Also not shown in the video is that you can also author the look from the beginning in Houdini - load it into Maya and edit the shader there via Attribute editor or LookdevX.

maya_houdini_usd_lookdev_testing_compressed.mp4

During step 2 I mentioned some 'issues' with the Maya USD implementation to make managing a asset look in layer trivial, those are:

• Maya USD: Unable to move /mtl from root level into the sublayered or referenced asset
• Maya USD: Unable to set the layer default prim through the UI

Houdini

houdini_usd_shot_prototype.mov

Maya

maya_usd_asset_prototypes.mov

This next video is a bit older but shows some maya USD concepts so sharing here anyway

openpype_mayausd_prototype_lets_create_a_cube.mov

Blender

blender_usd_asset_prototype.mov

Tray Publisher (note)

It would be nice if the Tray Publisher could also be able to ingest USD files and do the asset contribution stuff - that should actually work if ingesting a usd family and the attribute definitions should automatically appear - however it would require the Python process running the Tray Publisher to have access to the USD Python API which OpenPype/AYON by default currently does not. But it should be possible if you add the dependency somehow.

Asset Structure

The asset structure is based on:

• ASWF Guidelines for Structuring USD Assets
• Nvidia/Omniverse's Principles of Scalable Asset Structure in OpenUSD

And roughly looks like asset.usd:

#usda 1.0
(
    defaultPrim = "{asset_name}"
    metersPerUnit = 1
    upAxis = "Y"
)

class "__class__"
{
    class "{asset_name}"
    {
    }
}

def Xform "{asset_name}" (
    assetInfo = {
        string identifier = "{asset_name}/{asset_name}.usd"
        string name = "{asset_name}"
    }
    prepend inherits = </__class__/asset1>
    kind = "component"
    prepend payload = @./payload.usd@
)
{
}

With the payload being (a relative file to the asset, not a separate subset) containing sublayers of contributions, payload.usd:

#usda 1.0
(
    defaultPrim = "asset1"
    metersPerUnit = 1
    subLayers = [
        @\project\asset1\publish\usd\usdAsset_look\v003\test_local_asset1_usdAsset_look_v003.usd@,
        @\project\test_local\asset1\publish\usd\usdAsset_model\v032\test_local_asset1_usdAsset_model_v032.usd@
    ]
    upAxis = "Y"
)

Testing notes:

1. Launch Maya, publish and load USD files
2. Launch Houdini, publish and load USD files
3. Launch Blender, publish USD files

[moonyuet]

I think it is a good idea if traypublisher also supports the usd workflow, and users can publish the usd and preview it without the DCC.
I tested in Maya 2023.3 with some simple geometry and it is published successfully. I published highRes and LowRes Version of the model and set Low Res version as default variant selection. I can load the model withMayaProxyShapetoo. I will test again with the geometry with textured along with Add Reference to USD.(I do have some question on how this particular loader works but I will try to play around and raise it later).

[BigRoy]

I think it is a good idea if traypublisher also supports the usd workflow, and users can publish the usd and preview it without the DCC. I tested in Maya 2023.3 with some simple geometry and it is published successfully. I published highRes and LowRes Version of the model and set Low Res version as default variant selection. I can load the model withMayaProxyShapetoo. I will test again with the geometry with textured along with Add Reference to USD.(I do have some question on how this particular loader works but I will try to play around and raise it later).

Thanks for testing.

Tray Publisher support would mean we'd need to provide USD Python library bindings; which I think we should, however we want make sure to vendorize and avoid them being included for DCCs that provide their own.
USD builds have quite a few dependencies, time for ynput/ayon-core#816 :)

[MustafaJafar]

Please, excuse me. I'll slowly test this PR (mostly Houdini side as a start) because I'm new to USD.
what I have understood so far:

1. Each usd publish instance should represent one usd layer. so, something like this doesn't work in this particular workflow?
   
2. By default every published instance (layer) is part of a usd subset
3. It's doable to override layers and publish them into the same usd subset

What I have tested so far:
I was able to publish and load layers separately but I couldn't understand how they are assembled in the target usd subset.
I can't find words to explain what I've understood.
Also, here's a screen capture to show how it works on my side.

USD_workflow_houdini_test_01.mp4

Am I on the right way?

[BigRoy]

So first of all - I've pushed extra commits, particularly one that adds more tooltips to the publish attributes so that they can hopefully clarify some more.

Please, excuse me. I'll slowly test this PR (mostly Houdini side as a start) because I'm new to USD. what I have understood so far:

1. Each usd publish instance should represent one usd layer. so, something like this doesn't work in this particular workflow?
   

I can't really figure out by design what exactly you are trying to publish here - is it a model with looks but also with lights? Or is it 'just some stuff for a shot'? Or did you merely intend to use this as an example of some stuff in general?

Any way, you are totally fine to contribute whatever you want, lights, materials, etc. - and the single publish in Houdini should even be able to have layers of its own that it is currently adding (e.g. SOP import) however you must set a configured save path, e.g.:

Otherwise the output will just get some random saved name I believe (based on the SOP node's name or something - no idea?) For example by default it says:

Anyway, for now I've opted to solely use the explicit save paths as it's the most explicit and makes it much clearer that the artist actually intends that to be a sublayer or reference of its own.

If they want to SOP import but put into the main layer as (not referencing it or layering it and just add it to the editable layer) they should enable: Copy Contents in Editable Layer on the node - that way there's basically no external file reference.

Whenever an explicit save path is found it will become its own published product where the chosen filename will be added to the current subset's name, e.g. if I'm now publishing usdMain then this test.usd will be created usdMain_test publish.

2. By default every published instance (layer) is part of a usd subset

Yes, by default - but disabling "Enable" under USD Contribution means it'll just be a USD publish without doing anything else special. (When disabled note that the "Explicit Layer Save Paths" in Houdini should still be generating their own layers to your current subset, etc.

3. It's doable to override layers and publish them into the same usd subset

Hmm - not sure what you mean here. Could you elaborate? But as said, you can basically configure any USD data to your liking - even having variants, etc. but the USD contribution just means add whatever you're adding/contributing here into a layer that targets a product like the usdAsset or usdShot or alike. Whatever you are contributing is up to you - there currently also are no validators. As such, there's nothing holding you currently from e.g. contributing materials or lights in the model layer. It might not be best practice, but I'd say that as a first iteration we shouldn't limit too much except for what is absolutely necessary for this to work.

So - a validator that we should add is one that validates that whenever you're contributing to a USD Asset that it is actually contributing something INTO the default prim name which defaults to the {asset_name}/ - if you're not contributing there you're basically generating USD references that can't find the Prim path.

---

What I have tested so far: I was able to publish and load layers separately but I couldn't understand how they are assembled in the target usd subset. I can't find words to explain what I've understood. Also, here's a screen capture to show how it works on my side.
USD_workflow_houdini_test_01.mp4

Am I on the right way?

First off, it seems that you are referencing a shot - a shot should not have a default prim (I believe there's no USD structure guidelines there to have e.g. a shot within a /root prim so it allows referencing) and thus it should not be setting one.

Anyway, whenever referencing in USD you're basically saying from this USD file that I'm referencing take one top level primitive path an put that into the prim path I'm adding the reference.

So say you're referencing a USD file that defines:

/lights/light1
/geo/box1

Like, say this is our shot.usda:

#usda 1.0
(
    framesPerSecond = 24
    metersPerUnit = 1
    timeCodesPerSecond = 24
)

def Xform "lights"
{
    def SphereLight "light1" (
    )
    {
    }
}

def Xform "geo"
{
    def Cube "cube1" (
    )
    {
    }
}

Then that reference would need to pick either lights or geo to put into whatever prim path you're referencing it into - you won't get both. When referncing this file into Houdini I get this:

As such, for shots if you want to load it into Houdini you're most likely loading them as a Sublayer which means you'll get all its data (without moving it into any prim path) as a layer into your stage.

---

However having said that - you do still have another issue which might not be obvious but it's the Add as variant toggle that is causing this issue; it should have been disabled. The Add as variant option is currently designed for the Asset Structure and is not intended for Shot publishes since it says:

• Add my contribution as a variant into a variant set.

Now note that a variant set can only exist on a Prim path - as such it cannot be just a layer. Plus by enabling this option here it'll say create a variant set to the default prim (currently to {asset_name}) which for your shot structure of course does not exist.

Admittedly those USD errors even though they are trying to be clear and informational they get me every time - I just don't know where to look and I need to re-read them 10 times to understand what it is they are telling me. In your case I believe it's trying to say: "You are trying to reference a layer with defaultPrim but the default prim does not exist in the file." (And that's because the Add variant option is expecting that {asset_name}/ root prim to exist in the contribution, etc.

Anyway, it does leave your current state in a bit of an issue - since the publishing system is designed to always contribute into the last department layer state and the last target product state. You're basically always updating the latest published asset/shot file. There is currently no easy way to quickly tweak that particular subset now and republish it. You can however just publish into usdShot whatever you want, e.g. an empty USD file. So that any future contribution publishes will not also still be having that earlier added variants, etc. Or much easier for now, disable the add as variant checkbox.

(The add as variant enabled publish should actually itself also validate that whatever prim path it expects to exist that it actually exists before publishing so you can't make that faulty publish as easily)

I think now however you get to the hardest bit in the workflow (also because there's no dedicated tools to 'edit' yet) is that it always "updates" into the last publish, and then generates that as a new publish for the target product and its department layers. As such, you'll need to now somehow remove that invalid "variant contribution" so next versions won't have it when you are publishing your contribution instance again.

For now either:

• publish an empty USD file (or just tweak the last published file to be an empty USD file) so that future publishes will now add into a clean target product and department layer again.
• or test with a new asset that doesn't have these faulty product publishes.

I had though about adding an attribute there as well to "force re-init contributions" so that even if it already existed it would now just start a new and not build into an existing version - but I feel it'd just be hiding an issue we want to solve better anyway.

So, thanks for hitting this bug. Any details on how you think we can best improve the workflow let me know.

More input is welcome @MustafaJafar @moonyuet @mkolar :)

[MustafaJafar]

Many many thanks for your help!
I think I'll take some time to digest this information.

[tokejepsen]

@BigRoy you are welcome to mark this as ready to review, when its in a mergeable state.

[mkolar]

Because we're splitting OpenPype into ayon-core and individual host addons, this PR would have to be re-created to target one of those.

We're closing it down, but we'll he happy for a new PR to ynput/ayon-core or the host addon repository once it's up.