target label paths with non-latin1 characters have inconsistent starlark semantics.

[aiuto]

If we have a source file with a non Latin1 character in the file name (e.g. λ.file)

• and it is referenced in a glob(), let's say to attribute srcs (of type label_list)
• and you evaluate attr.srcs[n].label.name
• the length of the string is 7 on linux and macos
• but it is 6 on windows

Hilarity ensues while trying to figure out how to write these strings as part of an action.
Discovered while trying to fix #10174 More details to follow.

[alandonovan]

This seems like a dup of #374.

[aiuto]

It is related to it, but it concerns a small detail - that reading the same file name on Windows and Linux can result in different strings. FIxing #374 would most likely fix this, but we could fix this discrepancy on it's own without doing all the work needed for the other issue.

Thinking about it differently, 374 could possibly be a deep starlark change. We might be able to fix this problem at the the file system API layer.

[alandonovan]

The fix to #374 requires changing the decoding applied as soon as input is obtained (from read and readdir) and the encoding applied immediately before strings are output (by print, FileWriteAction, Forge RPCs, and a few other places). It also requires changing the JVM-wide file.encoding property. It took me a week to prepare a CL to implement as an immediate breaking change, but it would be a project to put it behind a flag.

[aiuto]

I'm not sure where you are going with putting it behind a flag, but is your point that this can't be fixed without fixing #374 as well? If so, yes, this is a duplicate.

[alandonovan]

the length of the string is 7 on linux and macos but it is 6 on windows

On second thoughts, I'm surprised about the platform variation. The only parts that vary by platform in #374 are the functions used within glob to read the directory. On Unix we end up in NativePosixFiles.readdir; on Windows we end up (I assume) in java.io.WinNTFileSystem.list. It would be good to know what you get from the lowest levels of the awfully messy glob machinery when you read a directory entry named λ.

Please do provide more details.

[aiuto]

I created what should be a repo here: https://github.com/tonyaiuto/bazel/tree/master/utf8_encode
bazel build :*
more bazel-bin/*.txt

But.... I do not have access to a windows machine, so I can not be certain of it. It should match the pattern where I found the problem trying to write regression tests for ctx.actions.write(), which was why I filed this issue. I expect that on linux we will see

bazel-bin/filename.txt

|A©��.file|
starlark length: 15
n_chars: 9
n_utf8_bytes: 15
# Expected: glob targets parsed as Latin1

and on windows we will see

|A©��.file|
starlark length: 9
n_chars: 9
n_utf8_bytes: 15
# Unexpected: glob targets parsed as UTF-8

[alandonovan]

After further investigation:

On Windows, Bazel's functions for reading directory entries (e.g. JavaIoFileSystem.getDirectoryEntries) trivially decodes the UTF-16 file names correctly to Java strings, which is inconsistent with the POSIX behavior in which we treat directory entries and file contents as Latin1. If such strings are later used as file names, all is well, because the reverse trivial encoding (UTF16->UTF-16) applies. Starlark code can however observe that the string elements are UTF-16 codes, not UTF-8 codes, and the lengths differ. By contrast, if such strings are later used as file contents, they will be incorrect, because the UTF-16 codes will be treated as UTF-8 codes and the lower-8 bits written out as bytes. Non-ASCII will become garbage.

So, we have two choices. The first is to make getDirectoryEntries on Windows misinterpret file names just as it does in POSIX. I started this, but I no longer thing it is viable: we would need to do this in every place in the file system API that accepts one of Bazel's mis-encoded strings, otherwise, for example, glob() won't be able to see files with non-ASCII names because the mangled filenames it gets from the directory can't be statted. There are a large number of such places. (This is easy for POSIX, and already done, twice over in fact: once in the native file system wrappers by making explicit calls to JNI Latin1 functions instead of UTF8; and once in the JavaIoFileSystem when used on POSIX because of Java's existing -Dfile.encoding=latin1 mechanism. This mechanism doesn't exist on Windows because it isn't needed---filenames aren't bytes.) So this is a lot of work to introduce bugs to make Windows as bad as POSIX.

The other choice is to fix the behavior on POSIX. That is issue #374. We know how to do it. It's an observable change to Starlark code and external compilers, and could thus be considered a breaking change, but I will argue that if a Starlark program or external compiler relies on anything more complex than that non-ASCII string literals and file names transit through Blaze, it is probably already subtly broken in many ways.

I am not going to waste any more time on encoding issues unless it is a complete fix to #374.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[aiuto]

We need a way to turn issues into archive documents for the person who eventually wants to fix the unicode character handline

[fmeum]

@tetromino Is the plan to switch to representing strings as UTF-8 internally still alive? This seems separate from the much more complicated to solve #374, so maybe there is a way to get rid of the Latin-1 hack without also having to think about an escaping scheme for labels (#23335 (comment)).

I started to work on a PR that maps between the internal Latin-1 representation and the external world on the filesystem level, similar to what @alandonovan described above: #23443
Such a change would not preclude any later changes in this area, it would just fix issues affecting Windows users at the price of slightly higher complexity in JavaIoFileSystem. If this sounds acceptable, I could finish it and send it out for review.

CC @meteorcloudy @tjgq for their input as they have been reviewing most of my Unicode fixes so far.

[aiuto]

IIRC, we do represent strings as UTF-8 internally, we just call everything Latin-1.
That is, we read your BUILD file (or the results of readdir(1)) and get back a blob of UTF-8 encoded text. It happens to be UTF-8 because that is what people's text editors saved. Then we like to our decode stack and say it is Latin-1 so that no translation happens. Then we write those strings directly without re-encoding and everything just sort of works.

The specific of this bug is that linux, windows, and macos all encode write non-Latin-1 characters in different bit patterns that are all UTF-8 compliant. Everything works as long as the host and exec platforms are the same. If you mix them, things can break.

[tetromino]

@fmeum - the plan is there, but delayed to probably next year (so not in Bazel 8.0).

[fmeum]

Everything works as long as the host and exec platforms are the same. If you mix them, things can break.

This is not true as far as I can tell. Remote execution correctly translates between the internal encoding and UTF-8 since d91e05e, so mixing different host and exec platforms should actually not be a problem. Subtle issues arise when Bazel uses the Java (N)IO APIs to interact with the filesystem, such as in JavaIoFileSystem or WindowsFileSystem. Java uses the encoding specified by `sun.jnu.encoding` for that and this can't be influenced reliably (it's always UTF-8 on macOS and can only be forced to Latin-1 on Linux if a latin-1 locale is installed, which isn't usually the case). That's why I think that introducing a reencoding step to the filesystem layer would help squash the remaining issues. It's not too intrusive, there is already a central method for interop with Java APIs that we could extend.