Please consider adding a non locking file mode for usdc files ( USD Memory Asset Class )

[koen-v]

The way usd handles usdc file by default locks the file from writing, this is to prevent corruption.

USD by default uses ArFilesystemAsset class to hold on to files. This class keeps the file open while the asset is in use and allows assets to be delayed read from disk. The problem with this approach is that the file may be written to by another thread or process while the file is left open. Subsequent reads will read garbage and either corrupt the asset or crash.

To solve this in our internal build, a new asset class, ArMemoryAsset, has been created. This new class will read the entire file up front and close it. All read operations are serviced using an in-memory buffer. Files are now opened with _SH_SECURE which prevent files to be opened for write while the file is being read from. This will prevent any corruption or crashes due to reads occurring after writes.

Fixes for USD

• USDC files being locked while a usdStage is open for read when using _SH_SECURE.
• USDC corruption if the file is written to while open for read when using _SH_DENYNO.

We are happy to share this code, we have a CLA in place. However, we have not implemented a nice way to choose between the options, we completely replace all usdc reading with this, which won't be appropriate in many cases. Some options we considered, try to solve it with a resolver, perhaps an env var, or setting in the pluginfo, or perhaps even adding ".usdb" as a new file extension.

The memory asset class:

/// \class ArMemoryAsset
///
/// ArAsset implementation for asset represented by a block of memory.
class ArMemoryAsset
    : public ArAsset
{
public:
    /// Constructs an ArMemoryAsset for the given \p file. 
    /// The ArMemoryAsset object takes ownership of \p file and will
    /// delete the memory on destruction.
    AR_API
    explicit ArMemoryAsset(std::shared_ptr<const char>& file, size_t fileSize);

    /// Constructs an ArMemoryAsset for the given \p file. 
    /// The ArMemoryAsset object takes ownership of \p file and will
    /// close it immediately.
    AR_API
    explicit ArMemoryAsset(FILE* file);

    /// Returns the size of the file held by this object.
    AR_API
    virtual size_t GetSize() override;

    /// Creates a read-only memory map for the file held by this object
    /// and returns a pointer to the start of the mapped contents.
    AR_API
    virtual std::shared_ptr<const char> GetBuffer() override;
    
    /// Reads \p count bytes from the file held by this object at the
    /// given \p offset into \p buffer.
    AR_API
    virtual size_t Read(void* buffer, size_t count, size_t offset) override;

    /// Returns the FILE* handle this object was created with and an offset
    /// of 0, since the asset's contents are located at the beginning of the
    /// file.
    AR_API        
    virtual std::pair<FILE*, size_t> GetFileUnsafe() override;

private:
    std::shared_ptr<const char> _buffer;
    size_t _size = 0;
};

And the change we made to the resolver ( v2 shown here)

std::shared_ptr
ArDefaultResolver::_OpenAsset(
const ArResolvedPath& resolvedPath)
{
FILE* f = ArchOpenFile(resolvedPath.GetPathString().c_str(), "rb");
if (!f) {
return nullptr;
}

return std::shared_ptr<ArAsset>(new ArMemoryAsset(f));

}

Thanks,
Koen

[spiffmon]

Thank you for filing, @koen-v , and for the details! I think we will want to implement this feature at the UsdUsdcFileFormat level, which gives us one way of selecting the behavior (SdfFileFormatArgs on the reference/subLayer assetPath), but more importantly allows us to take further advantage of having the whole layer in memory: when we Reload() or TransferContents the layer, we will be able to do the same fine-grained change-processing that we currently can do for usda format layers, rather than just throwing up our hands and saying "everything in the layer may have changed". This may have a positive impact on workflows where you actually do regenerate a layer in one app, and update a consuming app to see the changes.

[koen-v]

Hello Spiff, that sounds like a great solution, thank you very much for considering a solution to this!

Please let me know if there is anything we can do to help from our side.
Cheers,
Koen

[jilliene]

Filed as internal issue #USD-7361

[hybridherbst]

I'd also like to see this in usdview for USDZ files – for us it's quite common to regenerate the viewed files frequently, and currently one needs to close usdview, regenerate, open the file in the viewer again. Better would be a "refresh" button/shortcut, or even automatic reload :)

[spiffmon]

That's good to know, @hybridherbst ... however, as far as usdview is concerned, "File > Reopen Stage" should work for your purposes, since it first (attempts to, given python lifetime management) closes the current stage and all layers, before re-opening the root layer. Does that not work for you?

[hybridherbst]

It doesn't work since the file is locked and can't be overwritten.

Our usecase is specifically viewing monolithic/self-contained USDZ files – since the viewer locks those when opening they can't be overwritten without closing the viewer, so I can't really test the "Reopen Stage" workflow (since it's locked it's always the same content).

[spiffmon]

Ah, of course - sorry I didn't see that!

[oumad]

Any updates regarding this issue ?
This is very problematic for us when a different user updates an asset while it is staged by someone else in maya-usd or Unreal Engine, the asset appears updated for everyone except for that person who had it staged, and no stage reloading or mute/unmuting works, the only way to see the updates in that instance is to close the process and re-open it. This ofcourse doesn't happen with usda data, only the payloads which are usually in usdc.

[spiffmon]

Hi @oumad , while we agree this will be a useful and welcome feature, we are currently fully engaged with upgrading USD to handle new platforms and dependent packages without regression, general support, and a small number of features relevant to lighting and rendering workflows. We are also working on expanding our team, but it may still be some time before we can tackle this.

[oumad]

Hi @oumad , while we agree this will be a useful and welcome feature, we are currently fully engaged with upgrading USD to handle new platforms and dependent packages without regression, general support, and a small number of features relevant to lighting and rendering workflows. We are also working on expanding our team, but it may still be some time before we can tackle this.

I understand the priorities, thank you for getting back to me. Good luck with all of your challenges ! I hope there is anything I could help with.

[sunyab]

Hi all, just wanted to mention that we're actively working on this right now.

Our current plan is to add API in Sdf that allows users to specify layers that should be read fully into memory and detached from the underlying backing store. This will use new API on SdfFileFormat that requires layer data to be read into memory and new API in Ar to retrieve an "in-memory" asset from an existing ArAsset. We'll provide default implementations that should just work, but may be optimized for particular SdfFileFormat or ArAsset implementations.

The Sdf API will accept some form of pattern so that you can easily express things like "all layers should be read into memory" or "all layers except these should be read into memory." Users will also be able to specify these expressions via an environment variable. We'll also add command-line arguments to usdview to support this feature.