Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.10 Support #3577

Merged
merged 10 commits into from
Jun 16, 2023
Merged

Python 3.10 Support #3577

merged 10 commits into from
Jun 16, 2023

Conversation

borzacchiello
Copy link
Contributor

Your checklist for this pull request

  • I've read the guidelines for contributing to this repository
  • I made sure to follow the project's coding style
  • I've documented or updated the documentation of every function and struct this PR changes. If not so I've explained why.
  • I've added tests that prove my fix is effective or that my feature works (if possible)
  • I've updated the rizin book with the relevant information (if needed)

Detailed description

I'm adding support for Python 3.10 bytecode.

As far as I understand, the only relevant modification in 3.10 bytecode wrt 3.9 is this one, but please let me know if I am missing something:

   .. versionchanged:: 3.10
      The argument of jump, exception handling and loop instructions is now
      the instruction offset rather than the byte offset.

I also refactored the way strings are detected. Now they are added in a similar way wrt sections and symbols while running "get_sections_symbols_from_code_objects".
Before this PR, the strings were added looking into the entire binary, generating bogus strings in the middle of code in some tests (in particular in py310.pyc).
I am not sure it is the cleanest way of doing it, let me know if you have a better idea :) !

Test plan

I added some tests using the file "py310.pyc" that was already in the test repo.

Related issue

related to #3130, but adds only support for 3.10 bytecode.


Thanks,
Luca

Copy link
Member

@XVilka XVilka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good with a couple minor things

librz/bin/format/pyc/marshal.h Outdated Show resolved Hide resolved
librz/bin/format/pyc/pyc_magic.c Outdated Show resolved Hide resolved
librz/bin/format/pyc/pyc_magic.h Outdated Show resolved Hide resolved
test/db/formats/pyc Show resolved Hide resolved
librz/analysis/p/analysis_pyc.c Outdated Show resolved Hide resolved
librz/asm/arch/pyc/opcode.h Outdated Show resolved Hide resolved
librz/asm/arch/pyc/opcode.c Outdated Show resolved Hide resolved
librz/asm/arch/pyc/opcode_310.c Outdated Show resolved Hide resolved
librz/analysis/p/analysis_pyc.c Outdated Show resolved Hide resolved
librz/analysis/p/analysis_pyc.c Outdated Show resolved Hide resolved
librz/bin/format/pyc/marshal.c Outdated Show resolved Hide resolved
librz/bin/format/pyc/marshal.c Outdated Show resolved Hide resolved
librz/bin/format/pyc/marshal.c Outdated Show resolved Hide resolved
librz/bin/format/pyc/marshal.c Outdated Show resolved Hide resolved
use "magic_int_within" to check if python version
@XVilka XVilka merged commit d9a5cf3 into rizinorg:dev Jun 16, 2023
@XVilka
Copy link
Member

XVilka commented Jun 16, 2023

@borzacchiello it broke tests on big-endian platform (System Z):

[XX] db/formats/pyc pyc disasm for 2.7
RZ_NOPLUGINS=1 /home/travis/build/rizinorg/rizin/install/bin/rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -eflirt.sigdb.load.system=false -eflirt.sigdb.load.home=false -N -Qc 'pd 10
' bins/pyc/py27.pyc
-- stdout
--- expected
+++ actual
@@ -1,13 +1,10 @@
-            ;-- entry0:
-            ;-- section.module:
-            ;-- <module>:
-            0x0000001e      LOAD_CONST            CodeObject(hello_world) from hello.py ; [00] ---- section size 25 named module
-            0x00000021      MAKE_FUNCTION
-            0x00000024      STORE_NAME            hello_world
-            0x00000027      LOAD_CONST            'world'
-            0x0000002a      PRINT_ITEM
-            0x0000002b      PRINT_NEWLINE
-            0x0000002c      LOAD_NAME             hello_world
-            0x0000002f      CALL_FUNCTION         0 positional, 0 named
-            0x00000032      POP_TOP
-            0x00000033      LOAD_CONST            None
+            0x00000000      invalid
+            0x00000001      unaligned
+        ,=< 0x00000002      basr  %r0, %r10
+        |   0x00000004      invalid
+        |   0x00000005      unaligned
+        |   0x00000006      invalid
+        |   0x00000007      unaligned
+        |   0x00000008      invalid
+        |   0x00000009      unaligned
+        |   0x0000000a      invalid
-- stderr
ERROR: Undefined type in free_object (0)

https://app.travis-ci.com/github/rizinorg/rizin/jobs/604321459#L4175

@borzacchiello
Copy link
Contributor Author

@borzacchiello it broke tests on big-endian platform (System Z):

[XX] db/formats/pyc pyc disasm for 2.7
RZ_NOPLUGINS=1 /home/travis/build/rizinorg/rizin/install/bin/rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -eflirt.sigdb.load.system=false -eflirt.sigdb.load.home=false -N -Qc 'pd 10
' bins/pyc/py27.pyc
-- stdout
--- expected
+++ actual
@@ -1,13 +1,10 @@
-            ;-- entry0:
-            ;-- section.module:
-            ;-- <module>:
-            0x0000001e      LOAD_CONST            CodeObject(hello_world) from hello.py ; [00] ---- section size 25 named module
-            0x00000021      MAKE_FUNCTION
-            0x00000024      STORE_NAME            hello_world
-            0x00000027      LOAD_CONST            'world'
-            0x0000002a      PRINT_ITEM
-            0x0000002b      PRINT_NEWLINE
-            0x0000002c      LOAD_NAME             hello_world
-            0x0000002f      CALL_FUNCTION         0 positional, 0 named
-            0x00000032      POP_TOP
-            0x00000033      LOAD_CONST            None
+            0x00000000      invalid
+            0x00000001      unaligned
+        ,=< 0x00000002      basr  %r0, %r10
+        |   0x00000004      invalid
+        |   0x00000005      unaligned
+        |   0x00000006      invalid
+        |   0x00000007      unaligned
+        |   0x00000008      invalid
+        |   0x00000009      unaligned
+        |   0x0000000a      invalid
-- stderr
ERROR: Undefined type in free_object (0)

https://app.travis-ci.com/github/rizinorg/rizin/jobs/604321459#L4175

Uhm I’ll look info it this evening. I suspect that using “magic_int_within” for checking if the version is before 3.6 do not work on big-endian platforms.

@XVilka
Copy link
Member

XVilka commented Jun 16, 2023

@borzacchiello it also fails on static builds, maybe easier to reproduce locally: https://github.com/rizinorg/rizin/actions/runs/5286709397/jobs/9566390877?pr=3580#step:8:694

@borzacchiello
Copy link
Contributor Author

#3583 should fix the regressions

Crispy-fried-chicken pushed a commit to Crispy-fried-chicken/rizin that referenced this pull request Jun 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants