Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-104615: don't make unsafe swaps in apply_static_swaps #104620

Merged
merged 4 commits into from
May 18, 2023

Conversation

carljm
Copy link
Member

@carljm carljm commented May 18, 2023

It's not safe to apply SWAP statically if it would reorder two instructions that store to the same location.

@carljm
Copy link
Member Author

carljm commented May 18, 2023

One thing I'm not clear on is whether we want to bump bytecode magic number for a fix like this. The old bytecode still works as well it ever did (no incompatible changes in the interpreter), so maybe no? But OTOH, nobody will see this bugfix until their pycs are invalidated.

Basically: do we bump bytecode magic number on any change to the compiler that results in different compiler output, or only when we change the interpreter in a way that makes it incompatible with previous bytecode?

@JelleZijlstra
Copy link
Member

One thing I'm not clear on is whether we want to bump bytecode magic number for a fix like this. The old bytecode still works as well it ever did (no incompatible changes in the interpreter), so maybe no? But OTOH, nobody will see this bugfix until their pycs are invalidated.

Basically: do we bump bytecode magic number on any change to the compiler that results in different compiler output, or only when we change the interpreter in a way that makes it incompatible with previous bytecode?

I was wondering a bit about that too. Several other recent fixes around comprehensions would also change the compiler output, so theoretically we should have bumped the magic number a few times. On the other hand, the only people who are really affected would be people who built from main directly after the PEP 695 change was merged (which did bump the magic number), so realistically there's not much of an issue. It's probably enough if we bump the magic number one last time before the beta is finalized.

@carljm carljm marked this pull request as ready for review May 18, 2023 20:35
@brandtbucher
Copy link
Member

brandtbucher commented May 18, 2023

My thought has mostly been that magic numbers should only be bumped if the bytecode itself is incompatible (meaning opcodes or the meanings of opargs have changed). I don't think we've bumped it for miscompilations before (but I could be wrong).

(We also bump it when changing how code objects are unmarshalled, which is arguably an abuse of the magic number and should be a marshal version bump instead.)

}
for (int idx = j + 1; idx < k; idx++) {
int store_idx = STORES_TO(block->b_instr[idx]);
if (store_idx >= 0 && (store_idx == store_j || store_idx == store_k)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would it happen that there's a STORE_FAST* between j and k?

Copy link
Member Author

@carljm carljm May 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can happen with code like a, a, b = x, y, z as in some of the added tests. This compiles to LOAD_FAST x; LOAD_FAST y; LOAD_FAST z; SWAP 3; STORE_FAST a; STORE_FAST a; STORE_FAST b. Before this PR, apply_static_swaps would optimize that (ignoring the loads) to STORE_FAST b; STORE_FAST a; STORE_FAST a (swapping the first and third store), which is invalid because it reorders the two stores to a.

k - j == n - 1 here, where n is the oparg to the SWAP. So for a SWAP 2, j and k will be adjacent, but for SWAP with oparg > 2, there will be intervening instructions. And STORE_FAST is a SWAPPABLE instruction, so those intervening instructions can be STORE_FAST.

Copy link
Member

@brandtbucher brandtbucher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix!

@carljm carljm enabled auto-merge (squash) May 18, 2023 21:00
@carljm carljm merged commit 0589c6a into python:main May 18, 2023
@carljm carljm deleted the staticswaps branch May 18, 2023 21:31
@carljm
Copy link
Member Author

carljm commented May 18, 2023

Oh, I forgot: this should be backported to 3.11, since it's a bug that exists there as well.

@carljm carljm added the needs backport to 3.11 only security fixes label May 18, 2023
@miss-islington
Copy link
Contributor

Thanks @carljm for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

@miss-islington
Copy link
Contributor

Sorry @carljm, I had trouble checking out the 3.11 backport branch.
Please retry by removing and re-adding the "needs backport to 3.11" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 0589c6a4d3d822cace42050198cb9a5e99c879ad 3.11

@carljm carljm added needs backport to 3.11 only security fixes and removed needs backport to 3.11 only security fixes labels May 18, 2023
@miss-islington
Copy link
Contributor

Thanks @carljm for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

@miss-islington
Copy link
Contributor

Sorry, @carljm, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 0589c6a4d3d822cace42050198cb9a5e99c879ad 3.11

carljm added a commit to carljm/cpython that referenced this pull request May 18, 2023
* main:
  pythongh-74690: Don't set special protocol attributes on non-protocol subclasses of protocols (python#104622)
  pythongh-104623: Update Windows installer to use SQLite 3.42.0 (python#104625)
  pythongh-104050: Add more type annotations to Argument Clinic (python#104628)
  pythongh-104629: Don't skip test_clinic if _testclinic is missing (python#104630)
  pythongh-104549: Set __module__ on TypeAliasType (python#104550)
  pythongh-104050: Improve some typing around `default`s and sentinel values (python#104626)
  pythongh-104146: Remove unused vars from Argument Clinic (python#104627)
  pythongh-104615: don't make unsafe swaps in apply_static_swaps (python#104620)
  pythonGH-104484: Add case_sensitive argument to `pathlib.PurePath.match()` (pythonGH-104565)
  pythonGH-96803: Document and test new unstable internal frame API functions (pythonGH-104211)
  pythonGH-104580: Don't cache eval breaker in interpreter (pythonGH-104581)
carljm added a commit to carljm/cpython that referenced this pull request May 19, 2023
…pythonGH-104620).

(cherry picked from commit 0589c6a)

Co-authored-by: Carl Meyer <carl@oddbird.net>
@bedevere-bot
Copy link

GH-104636 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 19, 2023
carljm added a commit that referenced this pull request May 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants