Handle toUnicode cMaps that omit leading zeros in hex encoded UTF-16 (issue 18099)

[alexcat3]

Modifies partialEvaluator.readToUnicode() to handle toUnicode cMaps that omit leading zeros in hex encoded UTF-16, as seen in the PDF in [https://github.com//issues/18099](issue 18099).

In the PDF in question, the toUnicode cmap had a line to map the glyph char codes from 00 to 7F to the corresponding unicode code points. The syntax to map a range of char codes to a range of unicode code points is
<start_char_code> <end_char_code> <start_unicode_codepoint>
As the unicode code points are supposed to be given in UTF 16, the PDF's line SHOULD have probably read
<00> <7F> <0000>
Instead it omitted two leading zeros from the UTF 16 like this
<00> <7F> <00>
This confused pdf.js into mapping these character codes to the UTF-16 characters with the corresponding high bytes ( 01 becomes \u0100, 02 becomes \u0200) which ended up turning latin text in the PDF into chinese when it was copied.

I'm not sure if the specification actually allows PDFs to do this, but since there's at least one PDF in the wild that does and other PDF readers read it correctly, pdf.js should probably support this.

[calixteman]

Could you add a unit test ? something similar to:

pdf.js/test/unit/api_spec.js

Lines 3202 to 3216 in 790470c

	it("gets text content, with no extra spaces (issue 13226)", async function () {
	const loadingTask = getDocument(buildGetDocumentParams("issue13226.pdf"));
	const pdfDoc = await loadingTask.promise;
	const pdfPage = await pdfDoc.getPage(1);
	const { items } = await pdfPage.getTextContent({
	disableNormalization: true,
	});
	const text = mergeText(items);
	expect(text).toEqual(
	"Mitarbeiterinnen und Mitarbeiter arbeiten in über 100 Ländern engagiert im Dienste"
	);
	await loadingTask.destroy();
	});

You can just test that the retrieved text starts with Yýsledková listina přijímacích zkoušek.

[alexcat3]

OK, I will.

[Snuffleupagus]

When addressing the comments, please remember to squash the commits.

[alexcat3]

OK, I made the necessary changes and squashed the commits.

[timvandermeij]

The CI failed on linting. You can automatically fix that by running npx gulp lint --fix locally. Other than that this patch LGTM, with passing tests which we'll trigger once the linting issue is fixed, but I'll leave the final sign-off here to @Snuffleupagus given the familiarity with the font code. Thanks!

[calixteman]

/botio test

[moz-tools-bot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/c8ea0e506839555/output.txt

[moz-tools-bot]

From: Bot.io (Windows)

---

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/db23d9ada3a63cd/output.txt

[moz-tools-bot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.241.84.105:8877/c8ea0e506839555/output.txt

Total script time: 28.99 mins

• Unit tests: Passed
• Integration Tests: Passed
• Regression tests: FAILED

  different ref/snapshot: 19
  different first/second rendering: 2

Image differences available at: http://54.241.84.105:8877/c8ea0e506839555/reftest-analyzer.html#web=eq.log

[Snuffleupagus]

Can you please improve the commit message a bit, such that all relevant information is available on the Git command line as well without having to read GitHub?

Generally the first line of the commit message should contain a summary of the changes, with a reference to the bug/issue, and then any other relevant details below. You included a bunch of nice context in #18390 (comment) that belongs in the commit message too. My suggestion would be something like this:

Handle toUnicode cMaps that omit leading zeros in hex encoded UTF-16 (issue 18099)

In the PDF in question, the toUnicode cmap had a line to map the glyph char codes from 00 to 7F to the corresponding unicode code points. The syntax to map a range of char codes to a range of unicode code points is
<start_char_code> <end_char_code> <start_unicode_codepoint>
As the unicode code points are supposed to be given in UTF 16, the PDF's line SHOULD have probably read
<00> <7F> <0000>
Instead it omitted two leading zeros from the UTF 16 like this
<00> <7F> <00>
This confused PDF.js into mapping these character codes to the UTF-16 characters with the corresponding high bytes ( 01 becomes \u0100, 02 becomes \u0200) which ended up turning latin text in the PDF into chinese when it was copied.

I'm not sure if the specification actually allows PDFs to do this, but since there's at least one PDF in the wild that does and other PDF readers read it correctly, PDF.js should probably support this.

[moz-tools-bot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.193.163.58:8877/db23d9ada3a63cd/output.txt

Total script time: 42.46 mins

• Unit tests: Passed
• Integration Tests: Passed
• Regression tests: FAILED

  different ref/snapshot: 3

Image differences available at: http://54.193.163.58:8877/db23d9ada3a63cd/reftest-analyzer.html#web=eq.log

[alexcat3]

@Snuffleupagus I changed the commit message

[Snuffleupagus]

r=me, thank you!

[alexcat3]

Thank YOU for all your help!