Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite tooling scripts in Python #211

Closed
matteodelabre opened this issue Jan 10, 2021 · 3 comments
Closed

Rewrite tooling scripts in Python #211

matteodelabre opened this issue Jan 10, 2021 · 3 comments
Assignees
@matteodelabre
Labels
tooling Set of scripts and configuration files for building the packages

Comments

@matteodelabre
Copy link
Member

matteodelabre commented Jan 10, 2021 •
edited
Loading

I’m currently rewriting the tooling scripts from Bash to Python.

Goals of the rewrite:

  • Contain it to only the tooling scripts. Package recipes should be unaffected by the change.
  • Make it easier to add features to the tooling scripts, which is quite a challenge in their current Bash form.
  • Add more sanity checks when parsing and building recipes.

If you’d like to suggest a specific feature for the rewrite, feel free to comment below. I’d like to keep the rewrite as focused as possible so that we can quickly merge it and move to other more productive things.
@matteodelabre matteodelabre added the tooling Set of scripts and configuration files for building the packages label Jan 10, 2021
@matteodelabre matteodelabre self-assigned this Jan 10, 2021
@matteodelabre matteodelabre mentioned this issue Jan 10, 2021
Closed
@Eeems
Copy link
Member

Eeems commented Jan 10, 2021

#85 #210 #8 and perhaps poke at #45

This was referenced Jan 10, 2021
Draft
Open
@matteodelabre
Copy link
Member Author

matteodelabre commented Jan 13, 2021 •
edited
Loading

In the interest of not blocking other tasks that depend on this, I’m planning on splitting this in three PRs:

@matteodelabre matteodelabre mentioned this issue Jan 13, 2021
Merged
matteodelabre added a commit that referenced this issue Jan 14, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
36a3f23 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
23bb0c3 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
f897f53 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
6a98f3f 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
1b71613 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
4c44d06 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
4216749 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
f8db03c 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 15, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
d60f01f 
(See #211.)

* Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
* Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
* Create `toltec` Python support library (in `scripts/toltec`)
* Remove superseded `opkg` Python library (in `scripts/opkg`)
* Add Python linting and formatting checks
* Update Makefile to work with the new scripts
* Update workflow files to work with the new scripts
* Update koreader to not use the `$recipedir` variable (not documented,
  should not be used)
matteodelabre added a commit that referenced this issue Jan 20, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
6d6e31a 
(See #211.)

* Rewrite to Python 3.8
    - Create `toltec` Python support library (in `scripts/toltec`)
    - Remove superseded `opkg` Python library (in `scripts/opkg`)
    - Add Python linting and formatting checks
    - Replace `scripts/repo-build` (Bash script) with
      `scripts/repo_build.py`
    - Replace `scripts/package-build` (Bash script) with
      `scripts/package_build.py`
    - Update Makefile and workflows to work with the new scripts
* Package changes
    - Update koreader to not use the `$recipedir` variable (not
      documented, not supported by the new scripts, and should not be
      used anymore)
* Documentation changes
    - Add info on how required Python version and modules
    - Document custom fields and functions support
* New features
    - Improve support for custom fields and functions in recipes
    - Parameterize "https://toltec-dev.org" in workflows, using a
      REMOTE_HTTP secret. Rename REMOTE secret to REMOTE_SSH
* Additional checks
    - Check that all custom declarations start with `_` to guard against
      typos on optional standard declarations
    - Check that timestamp fields contain a valid ISO-8601 value
    - Check that pkgver fields contain a valid version number
matteodelabre added a commit that referenced this issue Jan 22, 2021
@matteodelabre
Rewrite repo-build and package-build in Python
c855c4c 
(See #211.)

* Rewrite to Python 3.8
    - Create `toltec` Python support library (in `scripts/toltec`)
    - Remove superseded `opkg` Python library (in `scripts/opkg`)
    - Add Python linting and formatting checks
    - Replace `scripts/repo-build` (Bash script) with
      `scripts/repo_build.py`
    - Replace `scripts/package-build` (Bash script) with
      `scripts/package_build.py`
    - Update Makefile and workflows to work with the new scripts
* Package changes
    - Update koreader to not use the `$recipedir` variable (not
      documented, not supported by the new scripts, and should not be
      used anymore)
* Documentation changes
    - Add info on how required Python version and modules
    - Document custom fields and functions support
* New features
    - Improve support for custom fields and functions in recipes
    - Parameterize "https://toltec-dev.org" in workflows, using a
      REMOTE_HTTP secret. Rename REMOTE secret to REMOTE_SSH
* Additional checks
    - Check that all custom declarations start with `_` to guard against
      typos on optional standard declarations
    - Check that timestamp fields contain a valid ISO-8601 value
    - Check that pkgver fields contain a valid version number
matteodelabre added a commit that referenced this issue Jan 24, 2021
@matteodelabre
Rewrite repo-build and package-build in Python (#218)
d0d8d6d 
(See #211.)

The new build scripts are backward-compatible with existing recipes. Apart from some minor improvements on generated maintainer scripts, there are no changes in the final .ipk files.

* Rewrite to Python 3.8
    - Create `toltec` Python support library (in `scripts/toltec`)
    - Remove superseded `opkg` Python library (in `scripts/opkg`)
    - Add Python linting and formatting checks
    - Replace `scripts/repo-build` (Bash script) with `scripts/repo_build.py`
    - Replace `scripts/package-build` (Bash script) with `scripts/package_build.py`
    - Update Makefile and workflows to work with the new scripts
* Package changes
    - Update koreader to not use the `$recipedir` variable (not documented, not supported by the new scripts, and should not be used anymore)
    - Update zoneinfo-utils to fix use of an undefined variable
* Documentation changes
    - Add info on how required Python version and modules
    - Document custom fields and functions support
* New features and improvements
    - Improve support for custom fields and functions in recipes
    - Remove unnecessary generated install scripts, reduce size of generated scripts
    - Parameterize the <https://toltec-dev.org> value in workflows, using a `REMOTE_HTTP` secret.
    - Rename the `REMOTE` secret to `REMOTE_SSH` for clarity
* Additional checks
    - Check that all custom declarations start with `_` to guard against typos on optional standard declarations
    - Check that timestamp fields contain a valid ISO-8601 value
    - Check that pkgver fields contain a valid version number
    - Run all Bash scripts with `set -euo pipefail` flags enabled
matteodelabre added a commit that referenced this issue Jan 28, 2021
@matteodelabre
Rewrite repo-build-web in Python
e482b61 
This does not change the final HTML output, but only changes its
generation script to use Python (as part of the Python rewrite effort
from #211).

* Use the lightweight Jinja templating engine
* Add `Repo.make_listing()` method for generating the listing
* Call `make_listing` as part of the `repo_build.py` script
@matteodelabre matteodelabre mentioned this issue Jan 28, 2021
Merged
matteodelabre added a commit that referenced this issue Jan 30, 2021
@matteodelabre
Rewrite repo-build-web in Python (#266)
276d0dc 
This does not change the final HTML output, but only changes its
generation script to use Python (as part of the Python rewrite effort
from #211).

* Use the lightweight Jinja templating engine
* Add `Repo.make_listing()` method for generating the listing
* Call `make_listing` as part of the `repo_build.py` script
@matteodelabre
Copy link
Member Author

Closing this since almost everything has been rewritten to Python, except for the repo-check script, which is ununsed anyway since we haven’t really integrated reproducible builds into our workflow yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matteodelabre matteodelabre

Labels
tooling Set of scripts and configuration files for building the packages
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Eeems @matteodelabre