gh-99540: Constant hash for _PyNone_Type to aid reproducibility #99541

yonillasky · 2022-11-16T18:58:01Z

Issue: Constant hash value for None to aid reproducibility #99540

bedevere-bot · 2022-11-16T18:58:08Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

cpython-cla-bot · 2022-11-16T18:58:31Z

All commit authors signed the Contributor License Agreement.

ethanfurman · 2022-12-10T00:13:42Z

None, like True and False, is a special object in Python, and should have either a constant hash, or one that can be set by specifying the random hash seed.

Objects/object.c

bedevere-bot · 2022-12-10T00:47:02Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

yonillasky · 2022-12-10T08:04:13Z

There is some value in making it dependent on the hash secret (PYTHONHASHSEED):

for intentional fuzzing purposes
it addresses these nebulous concerns I've seen on the discuss forum, that people might start unwittingly depending on the specific constant value of this hash

The cost of doing that is an extra global variable and a single calculation done once upon setting the hash secret. I think that is a reasonable cost - I'll try writing it that way.

EDIT: now that I did, having second thoughts - it's a larger change to CPython, and perhaps people who run their code only on platforms with ASLR disabled today will be surprised by the "non-determinism by default" we'll be introducing here.

The hash secret was introduced to negate chosen input complexity attacks on builtin hash sets/maps, which is not relevant here, as None is a singleton. The "fuzzing" use case is secondary.

I'm willing to go with either solution. Personally, I will benefit more from the PYTHONHASHSEED dependent hash, but I don't know that it's the right thing to do for general use.

bedevere-bot · 2022-12-10T08:56:31Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

netlify · 2022-12-10T08:56:31Z

✅ Deploy Preview for python-cpython-preview canceled.

Name	Link
🔨 Latest commit	`a1cbd39`
🔍 Latest deploy log	https://app.netlify.com/sites/python-cpython-preview/deploys/6394e55392d2760008ffb6e2

bedevere-bot · 2022-12-10T08:57:25Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

Python/bootstrap_hash.c

yonillasky · 2022-12-10T09:18:44Z

I'm waiting with the blurb update until we make a final decision whether to go with a hash-secret-dependent calculation or not.

yonillasky · 2022-12-10T09:19:19Z

I have made the requested changes; please review again

bedevere-bot · 2022-12-10T09:19:22Z

Thanks for making the requested changes!

@ethanfurman: please review the changes made to this pull request.

ethanfurman

Let's do a simple constant hash for None.

bedevere-bot · 2022-12-10T16:59:45Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

ethanfurman · 2022-12-10T20:28:26Z

Looks good. We'll wait a week to see if anyone else wants to comment.

methane

How 0xFCA86420 is chosen?
Is it better than random value?

ethanfurman · 2022-12-12T04:00:40Z

The constant was chosen basically at whim (that particular value amused me); it's easier to implement as a constant because if we went with a random number we would have to make it contingent on the hash seed used to randomize str and byte hashes so that different runs of a program could be consistent (a virtual necessity for bug-hunting).

If it was a truly random value (which it currently is if Python was compiled with ASLR), then program results can vary from one run to the next with no way to force them to be the same (since None's hash is not currently dependent on the PYTHONHASHSEED).

methane · 2022-12-12T04:58:35Z

it's easier to implement as a constant because if we went with a random number we would have to make it contingent on the hash seed used to randomize str and byte hashes so that different runs of a program could be consistent (a virtual necessity for bug-hunting).

I just meant constant random number, like 0xb1f921305a6be69e.

Since hashtable (dict and set) uses lower bits first, filling some lower bits can reduce hash collision against small constant ints.

But this is not a big issue because None is just one singleton.

yonillasky · 2022-12-12T18:54:41Z

I just meant constant random number, like 0xb1f921305a6be69e.

Just noting that if a 64-bit literal is used, we'd presumably need to add a static check for 32-bit builds, and add a 32-bit "fallback" value for that case.

ethanfurman · 2022-12-12T18:59:15Z

Good point. Let's just use a 32-bit number instead.

ethanfurman · 2022-12-12T19:03:25Z

Actually, unless I'm missing something, that is a 32-bit number.

ericvsmith · 2022-12-12T19:09:46Z

I think that was in response to Inada-san's suggestion of 0xb1f921305a6be69e.

ethanfurman · 2022-12-12T19:35:50Z

Ah, that's what I was missing. I wish we had a reply directly to a comment ability. :-(

* main: pythongh-99540: Constant hash for _PyNone_Type to aid reproducibility (pythonGH-99541) pythongh-100039: enhance __signature__ to work with str and callables (pythonGH-100168) pythongh-99830: asyncio: Document returns of remove_{reader,writer} (python#100302) "Compound statement" docs: Fix with-statement step indexing (python#100286) pythonGH-90043: Handle NaNs in COMPARE_OP_FLOAT_JUMP (pythonGH-100278)

* origin/main: (1306 commits) Correct CVE-2020-10735 documentation (python#100306) pythongh-100272: Fix JSON serialization of OrderedDict (pythonGH-100273) pythongh-93649: Split tracemalloc tests from _testcapimodule.c (python#99551) Docs: Use `PY_VERSION_HEX` for version comparison (python#100179) pythongh-97909: Fix markup for `PyMethodDef` members (python#100089) pythongh-99240: Reset pointer to NULL when the pointed memory is freed in argument parsing (python#99890) pythongh-99240: Reset pointer to NULL when the pointed memory is freed in argument parsing (python#99890) pythonGH-98831: Add DECREF_INPUTS(), expanding to DECREF() each stack input (python#100205) pythongh-78707: deprecate passing >1 argument to `PurePath.[is_]relative_to()` (pythonGH-94469) pythongh-99540: Constant hash for _PyNone_Type to aid reproducibility (pythonGH-99541) pythongh-100039: enhance __signature__ to work with str and callables (pythonGH-100168) pythongh-99830: asyncio: Document returns of remove_{reader,writer} (python#100302) "Compound statement" docs: Fix with-statement step indexing (python#100286) pythonGH-90043: Handle NaNs in COMPARE_OP_FLOAT_JUMP (pythonGH-100278) Improve stats presentation for calls. (pythonGH-100274) Better stats for `LOAD_ATTR` and `STORE_ATTR` (pythonGH-100295) pythongh-81057: Move the Cached Parser Dummy Name to _PyRuntimeState (python#100277) Document that zipfile's pwd parameter is a `bytes` object (python#100209) pythongh-99767: mark `PyTypeObject.tp_watched` as internal use only in table (python#100271) Fix typo in introduction.rst (python#100266) ...

Lucas-C · 2023-01-31T11:45:29Z

Thank you for this, it allowed to make my project deterministic!

I dropped a comment about it there: https://stackoverflow.com/questions/7681786/how-is-hashnone-calculated/75296606#75296606

This way, we don't need the "getPythonNone" function, which was only called once, to create the object. Beginning with Python 3.12, None will have a constant hash anyway. We use this new value. See python/cpython#99541.

gsakkis · 2024-03-01T23:10:09Z

Is this merged into 3.12? There's no mention in the changelog.

yonillasky · 2024-03-01T23:57:29Z

Is this merged into 3.12? There's no mention in the changelog.

The PR did update the blurb, and it appears in git log when I check out branch 3.12.

What makes a given PR worthy of appearing in the changelog?
This one is not part of any PEP, nor was it planned work, or something that affects any language requirements. There is no contractual obligation that the hash will stay constant in the future, or that sets will continue to have iteration stability if used with deterministic hashes.

bedevere-bot mentioned this pull request Nov 16, 2022

Constant hash value for None to aid reproducibility #99540

Closed

bedevere-bot added the awaiting review label Nov 16, 2022

yonillasky force-pushed the none-constant-hash branch from f216cd7 to 0c08cac Compare November 16, 2022 19:50

rhettinger closed this Nov 19, 2022

ethanfurman changed the title ~~gh-99540: Constant hash for _PyNone_Type~~ gh-99540: Constant hash for _PyNone_Type to aid reproducibility Dec 10, 2022

ethanfurman reopened this Dec 10, 2022

ethanfurman requested changes Dec 10, 2022

View reviewed changes

Objects/object.c Outdated Show resolved Hide resolved

bedevere-bot removed the awaiting review label Dec 10, 2022

bedevere-bot added the awaiting changes label Dec 10, 2022

yonillasky force-pushed the none-constant-hash branch from c12fc81 to 961b41f Compare December 10, 2022 08:56

yonillasky force-pushed the none-constant-hash branch from 961b41f to f154d65 Compare December 10, 2022 08:57

yonillasky commented Dec 10, 2022

View reviewed changes

Python/bootstrap_hash.c Outdated Show resolved Hide resolved

bedevere-bot removed the awaiting changes label Dec 10, 2022

bedevere-bot added the awaiting change review label Dec 10, 2022

bedevere-bot requested a review from ethanfurman December 10, 2022 09:19

ethanfurman requested changes Dec 10, 2022

View reviewed changes

bedevere-bot removed the awaiting change review label Dec 10, 2022

bedevere-bot added the awaiting changes label Dec 10, 2022

ethanfurman approved these changes Dec 10, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Dec 10, 2022

ethanfurman self-assigned this Dec 10, 2022

methane reviewed Dec 12, 2022

View reviewed changes

ethanfurman merged commit 432117c into python:main Dec 16, 2022

bedevere-bot removed the awaiting merge label Dec 16, 2022

d-torrance mentioned this pull request Jun 26, 2023

Use constant hash for Python's "None" object Macaulay2/M2#2890

Merged

This was referenced Dec 20, 2023

Fix DaskRunner GBK by bumping dask lower bound apache/beam#29802

Closed

Switch to different, stable hash algorithm in Bag dask/dask#6723

Closed

yonillasky deleted the none-constant-hash branch March 1, 2024 23:47

kshitij12345 mentioned this pull request May 25, 2024

Add support for torch FP8 dtypes Lightning-AI/lightning-thunder#445

Merged

4 tasks

flferretti mentioned this pull request Jun 14, 2024

Speed up passing different JaxSimModel with same pytree structure to JIT-compiled functions ami-iit/jaxsim#179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-99540: Constant hash for _PyNone_Type to aid reproducibility #99541

gh-99540: Constant hash for _PyNone_Type to aid reproducibility #99541

yonillasky commented Nov 16, 2022 •

edited by bedevere-bot

Loading

bedevere-bot commented Nov 16, 2022

cpython-cla-bot bot commented Nov 16, 2022 •

edited

Loading

ethanfurman commented Dec 10, 2022

bedevere-bot commented Dec 10, 2022

yonillasky commented Dec 10, 2022 •

edited

Loading

bedevere-bot commented Dec 10, 2022

netlify bot commented Dec 10, 2022 •

edited

Loading

bedevere-bot commented Dec 10, 2022

yonillasky commented Dec 10, 2022

yonillasky commented Dec 10, 2022

bedevere-bot commented Dec 10, 2022

ethanfurman left a comment

bedevere-bot commented Dec 10, 2022

ethanfurman commented Dec 10, 2022

methane left a comment

ethanfurman commented Dec 12, 2022

methane commented Dec 12, 2022

yonillasky commented Dec 12, 2022 •

edited

Loading

ethanfurman commented Dec 12, 2022

ethanfurman commented Dec 12, 2022

ericvsmith commented Dec 12, 2022

ethanfurman commented Dec 12, 2022

Lucas-C commented Jan 31, 2023

gsakkis commented Mar 1, 2024

yonillasky commented Mar 1, 2024 •

edited

Loading

gh-99540: Constant hash for _PyNone_Type to aid reproducibility #99541

gh-99540: Constant hash for _PyNone_Type to aid reproducibility #99541

Conversation

yonillasky commented Nov 16, 2022 • edited by bedevere-bot Loading

bedevere-bot commented Nov 16, 2022

cpython-cla-bot bot commented Nov 16, 2022 • edited Loading

ethanfurman commented Dec 10, 2022

bedevere-bot commented Dec 10, 2022

yonillasky commented Dec 10, 2022 • edited Loading

bedevere-bot commented Dec 10, 2022

netlify bot commented Dec 10, 2022 • edited Loading

✅ Deploy Preview for python-cpython-preview canceled.

bedevere-bot commented Dec 10, 2022

yonillasky commented Dec 10, 2022

yonillasky commented Dec 10, 2022

bedevere-bot commented Dec 10, 2022

ethanfurman left a comment

Choose a reason for hiding this comment

bedevere-bot commented Dec 10, 2022

ethanfurman commented Dec 10, 2022

methane left a comment

Choose a reason for hiding this comment

ethanfurman commented Dec 12, 2022

methane commented Dec 12, 2022

yonillasky commented Dec 12, 2022 • edited Loading

ethanfurman commented Dec 12, 2022

ethanfurman commented Dec 12, 2022

ericvsmith commented Dec 12, 2022

ethanfurman commented Dec 12, 2022

Lucas-C commented Jan 31, 2023

gsakkis commented Mar 1, 2024

yonillasky commented Mar 1, 2024 • edited Loading

yonillasky commented Nov 16, 2022 •

edited by bedevere-bot

Loading

cpython-cla-bot bot commented Nov 16, 2022 •

edited

Loading

yonillasky commented Dec 10, 2022 •

edited

Loading

netlify bot commented Dec 10, 2022 •

edited

Loading

yonillasky commented Dec 12, 2022 •

edited

Loading

yonillasky commented Mar 1, 2024 •

edited

Loading