Latin-1 encoding in CSV presents invalid characters

[jalvesbsolus]

Bug Description

When using the flag N8N_DEFAULT_BINARY_DATA_MODE=filesystem to read a 'csv' file with the 'raw data' option set to true and 'Read As String' set to false, characters such as accents appear invalid when use latin-1 encoding.

Attention: I specify the options used for a reason - when N8N_DEFAULT_BINARY_DATA_MODE=default works correctly, the issue arises with N8N_DEFAULT_BINARY_DATA_MODE=filesystem.

I have already tried with 'Read As String' set to true (which works for UTF-8) but it doesn't work either.

To Reproduce

1. Change N8N_DEFAULT_BINARY_DATA_MODE=filesystem
2. 'Read As String' set to false and 'raw data' option set to true (only Just because these options work for the flag N8N_DEFAULT_BINARY_DATA_MODE=default)
3. import csv with latin-1
4. See the accents

Expected behavior

1. Use N8N_DEFAULT_BINARY_DATA_MODE=filesystem
2. import csv in latin-1
3. keep compatibility to utf-8 and utf-8 with bom

Finally, I use N8N_DEFAULT_BINARY_DATA_MODE=filesystem with CSV encoding latin-1, utf-8, and utf-8 with BOM (these last two already work).

Operating System

centos

n8n Version

1.39.1

Node.js Version

v18.19.1

Database

MySQL

Execution mode

own (deprecated)

[Joffcom]

Hey @jalvesbsolus,

Thanks for the report, I have put in a PR that should resolve this. Once reviewed and merged it will be available in a future release.

[jalvesbsolus]

Thanks @Joffcom .
People, can anyone follow up on this? I needed this resolution

[Joffcom]

@jalvesbsolus when you say "people can anyone follow up on this" what do you mean?

The PR is currently with our nodes team waiting to be reviewed between tasks. Once the code has been reviewed internally I will then merge the PR and when the next release goes out it will be part of that so it could be next Wednesday when we do our regular release unless we do a bug fix release before then.

[jalvesbsolus]

@Joffcom "people can anyone follow up on this" means - in this case it would be a "reviewed"

[Joffcom]

@jalvesbsolus Ah ok, In that case it will happen whenever someone is available to do it. I know I will pick up reviews between tasks and I suspect the rest of the team is the same.

[Joffcom]

@jalvesbsolus it has just been reviewed and merged so it will be in the next release.

[jalvesbsolus]

@Joffcom thanks

[janober]

Fix got released with n8n@1.43.0

[jalvesbsolus]

@janober or @Joffcom problem when updated. I got this error in login:
MaxListenersExceededWarning: Possible EventEmitter memory leak detected

FULL
(node:6) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 editorUiConnected listeners added to [Push]. Use emitter.setMaxListeners() to increase limit
(Use node --trace-warnings ... to show where the warning was created)

[Joffcom]

Hey @jalvesbsolus,

That is likely to be unrelated to this change and possibly related to your system being overloaded at the time. Are you still seeing the issue now?

[netroy]

Closing this, since the fix was released 2 weeks ago.

[souzagaabriel]

The Encoding option only appears when N8N_DEFAULT_BINARY_DATA_MODE=default . I don't know if this is the expected behavior.

[jalvesbsolus]

@Joffcom or @janober one question. How to identify the file's encoding automatically, is there any way?

I try with node "code" and lang Pyhton
`
def detectEncoding(binary_data):

encodings = ['utf-8', 'ascii', 'utf-16', 'latin1']
for encoding in encodings:
    try:
        binary_data.decode(encoding)
        return encoding
    except (UnicodeDecodeError, LookupError):
        continue

return 'Latin1'`

This is work in N8N_DEFAULT_BINARY_DATA_MODE=default but in N8N_DEFAULT_BINARY_DATA_MODE=filesystem not

[jalvesbsolus]

@souzagaabriel I don't have this problem