utf-8 encoded german umlaut gets scrambled!

[spacepope]

Hi,

i am struggling with a huge problem getting special unicode characters (german umlaut) to get read correctly by https://github.com/litehelpers/Cordova-sqlite-evcore-extbuild-free plugin on Android.

A json gets saved to the SQLite database with the query:

// table created before with:
// CREATE TABLE IF NOT EXISTS "calls" (id PRIMARY KEY, json);

INSERT INTO "calls" SELECT json_extract(json_each.value, '$._id'), json_each.value FROM json_each(json(?));

with the following (shortened) stringifyed json as parameter:

[{"_id":"id-123456","call_description":"Bitte die Prüfung der ortsfesten Anlagen/Geräte gemäß vereinbartem Umfang durchführen"}]

Next the json is selected from the database with the query:

SELECT json FROM "calls" WHERE (json_extract(json, ?) = ?);

with parameters $._id and id-123456 which returns scrambled umlaute:

{"_id":"id-123456","call_description":"Bitte die Pr?ffffffc3??ffffffbc?fung der ortsfesten Anlagen/Ger?ffffffc3??ffffffa4?te gem?ffffffc3??ffffffa4??ffffffc3??ffffff9f? vereinbartem Umfang durchf?ffffffc3??ffffffbc?hren"}

I tried to dive in your native Android drivers implementation because this is not an issue on iOS neither on Windows.
Also when i adb pull the SQLite database to my development machine and look at the inserted columns, the umlauts are displayed correctly, so this probably is more a read than write related problem..
The SQLite DB encoding is UTF-8.

I assume that this is caused somehow by the fancy byte/string transformations done in sqlc.c lines 448ff. I can not debug it and therefore can not propose a fix or solution, neither am i a specialist on UTF-8 encodings.

Maybe this could help?
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

It would be great if you can get in touch and see if you can reproduce and hopefully fix this..
If i can supply more info you need or can help somehow, let me know..

Thanks..

[brodycj]

Closing this one as a duplicate, thanks Hannes for the contribution, further discussion will be in #2. I also raised storesafe/cordova-sqlite-storage-help#35 to track the link you gave (I saw it before, really good).

[brodycj]

Now resolved.