Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Regression and change of behaviour between 3.3.0 and 3.3.1 #520

Closed
pombredanne opened this issue Sep 5, 2024 · 3 comments
Closed
Labels
detection Related to the charset detection mechanism, chaos/mess/coherence
Milestone

Comments

@pombredanne
Copy link

pombredanne commented Sep 5, 2024

Describe the bug
The detection of encoding did change recently, and IMHO regressed (I found that in a CI failure https://dev.azure.com/nexB/commoncode/_build/results?buildId=14502&view=logs&jobId=ba20146e-138e-5341-c558-bc25972fe2bd&j=ba20146e-138e-5341-c558-bc25972fe2bd&t=18eddfd8-abe5-5f8c-405c-5d0e0bd4c25d ) where we use beautifulsoup4 that uses in turn charset_normalizer.

To Reproduce
Note that I am using bs4 UnicodeDammit to show the side effects. I added the encoding detection that to see the charset_normalizer side:

Up to 3.2.0 the behavior is stable:

$ pip install beautifulsoup4==4.12.3
$ pip install charset-normalizer==3.2.0
$ python -c "from bs4.dammit import UnicodeDammit;print(UnicodeDammit(b'/includes/webform.compon\xd2\xaants.inc/').markup)"
/includes/webform.componŇŞnts.inc/
$ python -c "import charset_normalizer as cn; print(cn.detect(b'/includes/webform.compon\xd2\xaants.inc/')['encoding'])"
windows-1250

Note the small change in 3.3.0

$ pip install charset-normalizer==3.3.0
$ python -c "from bs4.dammit import UnicodeDammit;print(UnicodeDammit(b'/includes/webform.compon\xd2\xaants.inc/').markup)"
/includes/webform.compon훩nts.inc/
$ python -c "import charset_normalizer as cn; print(cn.detect(b'/includes/webform.compon\xd2\xaants.inc/')['encoding'])"
johab

Note the big change in 3.3.1

$ pip install charset-normalizer==3.3.1
$ python -c "from bs4.dammit import UnicodeDammit;print(UnicodeDammit(b'/includes/webform.compon\xd2\xaants.inc/').markup)"
⽩湣汵摥猯睥扦潲洮捯浰潮튪湴献楮振

$ python -c "import charset_normalizer as cn; print(cn.detect(b'/includes/webform.compon\xd2\xaants.inc/')['encoding'])"
utf_16_be

Expected behavior

I would expect the behavior of 3.2.0 or 3.3.0 as correct. The 3.3.1 is not correct or if it is, then this should be IMHO an API breaking major version bump

Desktop (please complete the following information):

  • OS: Linux and Windows
  • Python version 3.8 and up
  • Package version 3.2.0 to 3.3.1
@pombredanne pombredanne added bug Something isn't working help wanted Extra attention is needed labels Sep 5, 2024
@pombredanne
Copy link
Author

pombredanne commented Sep 5, 2024

I reckon that on the surface this issue seems to be related to #391 ... but IMHO this is still a bug, as a single character or a small minority of characters should not dictate the whole encoding of the larger string that contains them.

And I would NOT expect that the behavior would change so drastically without a version bump, as this package is a dependency on pip, requests and other popular packages. The behavior of of 3.3.0 or 3.2.0 is OK, 3.3.1 should become a 4.0.0 if you do not consider these changes as a regression.

(I am assuming may be incorrectly that you use some ki9nd of semver'ish versioning scheme)

pombredanne added a commit to aboutcode-org/commoncode that referenced this issue Sep 5, 2024
charset_normalizer has stability issues

Reference: jawah/charset_normalizer#520
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@Ousret Ousret added this to the Next release milestone Sep 21, 2024
@Ousret Ousret added detection Related to the charset detection mechanism, chaos/mess/coherence and removed help wanted Extra attention is needed bug Something isn't working labels Sep 21, 2024
Ousret added a commit that referenced this issue Sep 25, 2024
Ousret added a commit that referenced this issue Sep 25, 2024
@Ousret
Copy link
Member

Ousret commented Sep 25, 2024

but IMHO this is still a bug, as a single character or a small minority of characters should not dictate the whole encoding of the larger string that contains them.

Indeed, this behavior is not ideal across minor.

And I would NOT expect that the behavior would change so drastically without a version bump...IMHO an API breaking major version bump

Don't forget that we handle an heuristic algorithm and covering all the cases hosted on all other project can be next to impossible.

ki9nd of semver'ish versioning scheme)

We follow semver as best as we can.

Nevertheless, we fixed the presented case, and it will be available in the next minor.

@Ousret Ousret closed this as completed Sep 25, 2024
@pombredanne
Copy link
Author

@Ousret Thanks! ❤️

zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 13, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 13, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 13, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
detection Related to the charset detection mechanism, chaos/mess/coherence
Development

No branches or pull requests

2 participants