This repository demonstrates how to use mypyc
to compile Python functions into optimized native code, significantly improving performance.
- MyPyC Compilation for Python Functions
MyPyC
is a tool that compiles Python code into C extensions, allowing Python programs to run faster. It integrates with mypy
type annotations and is ideal for improving performance in CPU-bound tasks.
To compile a Python file using mypyc
, run the following command:
mypyc <filename>.py
This command compiles the Python file into a C extension and generates a .pyd
file (on Windows). For example, if your file is example.py
, the result will be something like:
<filename>.cp312-win_amd64.pyd
Once the .pyd
file is generated, you can import it in Python just like a regular module:
python -c "import <filename>"
The compiled functions can now be imported and used in Python, providing a performance boost over the regular Python implementation.
If you have multiple Python files to compile, you can automate the process with the included batch script.
The setup.cmd
script compiles all .py
files in the functions
folder and moves the resulting .pyd
files to the compiled_functions
folder. It also ensures that the compiled_functions
folder exists and handles the compilation automatically.
To compile and move all files in the functions
folder, run the batch script:
setup.cmd
This script will:
- Compile all
.py
files in thefunctions
folder usingmypyc
. - Move the resulting
.pyd
files to thecompiled_functions
folder.
If you'd like to use setuptools
and mypycify
to automate the compilation for Python packages, here's an example setup.py
script:
from setuptools import setup
from mypyc.build import mypycify
setup(
name="functions",
version="1.0",
packages=["functions"],
ext_modules=mypycify(
["functions"] # Compile all .py files in the functions folder
),
package_dir={
"": "./compiled_functions" # Specify the root directory for the source
},
)
To build the extension, run:
python setup.py build_ext --inplace
This will compile the .py
files and place the resulting .pyd
files in the compiled_functions
directory.
Note: This method was not used in this project due to issues during the compilation process.
MyPyC
compiles Python code into optimized C extensions, making your Python programs run faster. It works well with type annotations and is specifically designed to improve performance in CPU-bound tasks.
-
Performance Improvements:
mypyc
can provide significant speedups (ranging from 20% to 200%) for CPU-bound tasks, particularly those with computationally intensive operations.
-
Type Annotation Compatibility:
- Works seamlessly with
mypy
-annotated code, which allows for both type safety and performance optimization.
- Works seamlessly with
-
Improved Startup Time:
- Compiling Python code to C can reduce the startup time for applications, especially those with heavy initial processing or many imports.
-
Integration with the Python Ecosystem:
mypyc
compiles existing Python code into C extensions, meaning you can continue to use the vast Python ecosystem without major changes.
-
No Need to Rewrite Codebase:
- Unlike
Cython
, which requires you to rewrite large portions of your code,mypyc
allows you to improve performance incrementally by compiling existing Python files.
- Unlike
-
Limited Support for Dynamic Features:
mypyc
works best with statically typed Python code. It may not handle highly dynamic features (e.g.,exec()
,eval()
, dynamic attribute access) well.
-
Compilation Overhead:
- The need to compile Python files into C extensions adds an extra step to the build process, which can slow down development and deployment.
-
Limited Ecosystem and Adoption:
mypyc
is relatively new compared to other tools likeCython
and may not be as widely supported or documented. It may also have compatibility issues with some libraries or Python features.
-
Maintenance Challenges:
- Managing both the Python source code and its compiled C extensions requires careful maintenance to avoid issues as the codebase evolves.
-
Compatibility Issues:
- Some third-party Python libraries, especially those using dynamic features, may not work well with
mypyc
.
- Some third-party Python libraries, especially those using dynamic features, may not work well with
- Type-annotated code: If your Python code uses type hints and you want to improve its performance.
- CPU-bound tasks: For numerical computations, data processing, or other performance-critical tasks.
- Improving startup time: When startup speed is crucial, especially in larger applications.
- Dynamic code: If your code relies heavily on dynamic Python features or uses libraries that
mypyc
does not support. - Rapid iteration: If fast development cycles are more important than performance improvements.
- Non-CPU-bound tasks: If performance improvements do not justify the added complexity for non-CPU-bound code.
mypyc
is a valuable tool for improving the performance of Python code, especially when it involves CPU-bound operations and uses type annotations. While it offers many advantages, such as improved speed and seamless integration with the Python ecosystem, it also has some limitations, particularly with dynamic features and libraries.
This repository provides tools to:
-
Compile individual Python files with
mypyc
.mypyc <filename>.py
-
Import the compiled
.pyd
file into Python.python -c "import <filename>"
-
Automate the compilation process for all
.py
files in a folder using thesetup.cmd
script.setup.cmd
-
Use
setuptools
andmypycify
to automate compilation for Python packages.