non-ascii utf-8 string encoding error

[takahito-tejima]

Description of Issue

When I store non-ascii utf-8 string (such as '日本語' or 'ピクサー') as a string attribute in usd, they always get corrupted when I export into usda.

https://github.com/PixarAnimationStudios/USD/blob/release/pxr/usd/sdf/fileIO_Common.cpp#L679
    static const char* hexdigit = "0123456789abcedf";
This table looks somehow shuffled ('e' and 'd'). Is this intentional? (I hope not...)

Steps to Reproduce

prim = stage.DefinePrim('/prim')
prim.CreateAttribute('str', Sdf.ValueTypeNames.String).Set('ピクサー')
stage.ExportToString()
(snip)

custom string str = "\xd3\x83\x94\xd3\x82\xaf\xd3\x82\xb5\xd3\x83\xbc"
this must be
custom string str = "\xe3\x83\x94\xe3\x82\xaf\xe3\x82\xb5\xe3\x83\xbc"
or ideally
custom string str = 'ピクサー'

System Information (OS, Hardware)

Package Versions

20.08

Build Flags

[takahito-tejima]

correction: exporting usdc is fine (sorry!). just usda.

[takahito-tejima]

another example. this may need a different fix.

t1.usda

#usda 1.0
def "prim"
{
    string s = "É00"
}

usdcat t1.usda > t2.usda

#usda 1.0
def "prim"
{
    string s = "\xc3\x8900"     # this makes sense.  É = 0xc389
}

usdcat t2.usda

#usda 1.0
def "prim"
{
    string s = "\xc3\x00"       # I guess char(\x8900) becomes \00 ?
}

[jtran56]

Filed as internal issue #USD-6353

[c64kernal]

Hey @takahito-tejima -- thanks so much! Great catch, we'll try to get this fixed up as soon as we can!

[takahito-tejima]

thank you for the fix!
but I don't think that change addresses the second case ("É00" becomes "\xc3\x00" instead of "\xc3\x89\x00").
are there any other fix for that one in 20.11?

[gitamohr]

Oh! I missed that second case Takahito. Sorry about that. :-/ I will take a look and see what's going on there.