Major Refactor: Add Ruff Linting, Nox, and Pre-Commit Hook

[hmd101]

Summary:

This PR refactors the project by adding Ruff Linting and formatting, integrated with Nox and an optional Pre-commit hook. It enhances both code quality and development efficiency.

Key Changes:

1. Ruff Linting:
   • We have integrated Ruff linting, “an extremely fast Python linter designed as a drop-in replacement for Flake8 (plus dozens of plugins), isort, pydocstyle, pyupgrade, autoflake, and more.”
2. Ruff Formatter:
   • Compatible with the popular Black formatter but more performant.
3. Nox Automation:
   • Nox has been introduced to automate the execution of our CI tasks, including Ruff linting and other testing steps. Nox provides a flexible way to manage and run our tasks in isolated virtual environments, improving the reproducibility and ease of running multiple environments and workflows.
4. CI Pipeline Update:
   • Our CI configuration has been updated to include Ruff linting and formatting checks. From now on, Ruff will automatically lint all new commits and pull requests to ensure they adhere to our code quality standards.
5. Optional Pre-commit Hook:
   • A pre-commit hook for Ruff is now available, ensuring developers can run lint checks locally before committing code. This helps catch and resolve linting issues early in the development process.

Next Steps & Open Questions:

• Jupyter Notebooks: Should we include Jupyter notebooks in the Ruff linting process? -> Yes!
• Are all errors raised by the linter fixed?
• Contributing file updated?

Resolves issue #51

---

Configuration Details:

1. Ruff Linting Configuration:

The current Ruff configuration includes the most commonly used linting rules. (Future PRs will likely incorporate more linting rules, as discussed here).

• Pycodestyle (E): Enforces style consistency based on PEP8.
• Pyflakes (F): Detects basic errors such as undefined variables or missing imports.
• Pyupgrade (UP): Ensures usage of modern Python syntax, making our code more future-proof.
• Flake8-simplify (SIM): Improves code readability by suggesting simplified expressions. We have chosen to ignore rule SIM105 since our team prefers explicit boolean comparisons in some cases.
• Isort (I): Ensures imports are organized and ordered consistently, improving code readability and maintainability.

Note that a couple of linting rules were ignored with #noqa <ruff rule> (ignoring a single line) and #ruff: noqa <ruff rule> (ignoring entire file):

• E501 for too long lines has been ignored several times to ensure for example that links will still work or flow charts remain readable

• I001 sorting imports has been ignored once in src/plenoptic/__init__.py as the import order matters to avoid circular dependencies and tests to fail

• F401 unused imports

  • # ruff: noqa: F401 is used in several __init__.py files to ignore unused imports. When unused imports are removed, tests fail.

• SIM105 : Checks for try-except-pass blocks that can be replaced with the contextlib.suppress context manager.

  • currently ignored globally in pyproject.toml

• F403 flags wildcard imports * , which is bad practice. (This is part of the issue Improve API #246 and notes have been added there)

  To remove wildcard (*) imports, the 2 recommended approaches are:

  1. Explicit Imports in the __init__.py File:
     • Manually specify the functions and classes to be imported.
     • Example: from .validate import remove_grad
     • This approach aligns with the flat hierarchy recommendation from the OpenSci review (see issue Improve API #246)
  2. Nested Structure with Subpackages/Modules:
     • Organize the code into subpackages and import modules explicitly.
     • Example: plenoptic.tools.validate
       • Use from . import validate within the subpackage.
     • In files like validate.py, define a list called __all__ = ["remove_grad", ...], which specifies what can be imported from the file.
       • This can live in either the module file itself or the __init__.py, and it controls the public API of that module.

2. Ruff Formatting

Currently, formatting is enforced to emulate Black in pyproject.toml:

indent-width = 4 # same as Black
[tool.ruff.format]
# Like Black, use double quotes for strings.
quote-style = "double"

# Like Black, indent with spaces, rather than tabs.
indent-style = "space"

# Like Black, respect magic trailing commas.
skip-magic-trailing-comma = false

# Like Black, automatically detect the appropriate line ending.
line-ending = "auto"

Nox & Pre-commit Hooks and their Configurations:

TL;DR:

• Both tools are complementary and contribute to better maintainability by promoting consistent coding practices and making it easier to identify and fix issues early in the development process. Both are recommended by the Scientific Python Library Development Guide.
• Nox: A task-runner automating testing across different environments after commits, ensuring code works well in varied setups specified in noxfile.py.
• Pre-commit: Ensures code quality before commits by enforcing checks like formatting and linting. For this to work, pre-commit needs to be installed with pip install pre-commit . Pre-commit checks can be ignored, by adding --no-verify to your commit message.

2. Nox Configuration:

Nox is a Python-based task runner that is used to automate various development workflows, such as testing, building, and deploying applications. It allows you to define sessions, which are Python functions in the noxfile.py decorated with @nox.session with various parameters, such as name, by which these sessions can then be executed via the command line. This easily allows to test code by running pre-defined tasks with a single command across multiple Python versions without manually setting up environments.

Example: Based on the following noxfile.py , the command nox -s tests would run the session named "tests” below. nox without any commands would run all the sessions defined in the noxfile.py .

@nox.session(name="tests", python=["3.10", "3.11", "3.12"])
def tests(session):
    # run tests
    session.install("pytest")
    session.run("pytest")

Here are the current Nox sessions available for execution:

• nox -s lint
• nox -s tests
• nox -s coverage

3. Pre-Commit Configurations:

Hook:
A pre-commit hook can be added to run Ruff linting on code before it is committed. This ensures that developers catch and fix linting issues early in the development process. It only runs when installed and can be bypassed with git commit -m <my commit message> --no-verify.

Considered Configurations: (in .pre-commit.yaml:
(Legend: ✅: included, ❓: recommended, but not included, possibly in future PR)

• check-added-large-files ✅
  • Purpose: Checks for large files added to the repository, typically to prevent accidental inclusion of large binaries or datasets.
• check-case-conflict ✅
  • Purpose: Detects potential filename conflicts due to case-insensitive filesystems (e.g., Windows) where File.txt and file.txt would be considered the same
• check-merge-conflict ✅
  • Purpose: Checks for leftover merge conflict markers (<<<<<<<, =======, >>>>>>>) in files.
• check-symlinks ❓
  • Purpose: Verifies that symbolic links are valid and do not point to non-existent targets.
• check-yaml✅
  • Purpose: Validates YAML files for syntax errors.
• debug-statements ✅
  • Purpose: Detects debug statements (e.g., print, console.log) left in code.
• end-of-file-fixer✅
  • Purpose: Ensures files have a newline at the end.
• trailing-whitespace✅
  • Purpose: Removes trailing whitespace characters from files.

CI Pipeline:

Finally, the CI configuration ci.yml has been updated to include a step each for running Ruff formatting and linting check on all new commits and pull requests. Two separate actions were chosen to signal the developer which combination of the two might be causing an error.

[codecov]

Codecov Report

Attention: Patch coverage is 87.11180% with 83 lines in your changes missing coverage. Please review.

Project coverage is 88.54%. Comparing base (b80af32) to head (5b76b6a).
Report is 11 commits behind head on main.

Files	Patch %	Lines
src/plenoptic/tools/external.py	4.76%	20 Missing ⚠️
src/plenoptic/tools/display.py	68.08%	15 Missing ⚠️
src/plenoptic/synthesize/metamer.py	93.12%	9 Missing ⚠️
src/plenoptic/synthesize/mad_competition.py	92.92%	8 Missing ⚠️
src/plenoptic/synthesize/simple_metamer.py	22.22%	7 Missing ⚠️
...e/canonical_computations/steerable_pyramid_freq.py	80.00%	5 Missing ⚠️
src/plenoptic/synthesize/synthesis.py	81.48%	5 Missing ⚠️
src/plenoptic/metric/perceptual_distance.py	80.95%	4 Missing ⚠️
src/plenoptic/synthesize/geodesic.py	92.85%	3 Missing ⚠️
...c/plenoptic/simulate/models/portilla_simoncelli.py	97.87%	2 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #266      +/-   ##
==========================================
- Coverage   88.60%   88.54%   -0.07%     
==========================================
  Files          38       38              
  Lines        3405     3386      -19     
==========================================
- Hits         3017     2998      -19     
  Misses        388      388              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[billbrod]

This looks good! I'm pushing a couple of small changes, and then I'll go ahead and merge this.

Following up from my earlier comment: we've gone ahead and added pre-commit.ci.