Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use PerlIO::utf8_strict as the default. #44

Merged
merged 1 commit into from
Aug 11, 2016

Conversation

schwern
Copy link
Contributor

@schwern schwern commented Aug 10, 2016

Leon suggested using PerlIO::utf8_strict as the default.
#42 (comment)

It seems to work.

I'm assuming there's still value in trying to use :encoding(UTF-8)?
Maybe it's faster? @Leont could you comment?

@Leont
Copy link
Contributor

Leont commented Aug 10, 2016

:utf8_strict is faster and stricter (it never recodes to replacement character U+FFFD but throw an error when decoding a invalid byte sequence), but unlike :utf8 it doesn't always work well on layers without buffer peeking (such as :stdio), but that's not an issue in our case (and just as much an issue in :encoding).

@schwern
Copy link
Contributor Author

schwern commented Aug 10, 2016

@Leont Soooo... is it ok to always use :utf8_strict?

Leon suggested using PerlIO::utf8_strict as the default.
doherty#42 (comment)

It seems to work.

I'm assuming there's still value in trying to use :encoding(UTF-8)?
Maybe it's faster?
@schwern
Copy link
Contributor Author

schwern commented Aug 10, 2016

@Leont Any idea what's happening here? https://travis-ci.org/doherty/utf8-all/jobs/151371914

@HayoBaan
Copy link
Collaborator

Any idea what's happening here? https://travis-ci.org/doherty/utf8-all/jobs/151371914

I saw this in the CPAN reports too. It's unrelated to this change. I'll have a look at this after I finish integrating PerlIO::utf8_strict.

@HayoBaan HayoBaan merged commit 6bccafd into doherty:master Aug 11, 2016
@HayoBaan
Copy link
Collaborator

I have fixed the build errors! All that was needed was a no warnings FATAL => 'utf8';. Interesting that only for some versions of Perl, the illegal \x{d800} caused this…

HayoBaan added a commit that referenced this pull request Aug 11, 2016
   - Always using PerlIO::utf8_strict as I/O conversion        [gh-44; schwern]
   - Fixed "UTF-16 surrogate 0xd800" error in some builds            [HayoBaan]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants