This repository has been archived by the owner on Jan 12, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Flesh out packaging/CI and fix catalog issues
With some pointers from @martindurant in [this issue](intake/intake-parquet#26) I got anonymous public access working, and caching can now be turned off when appropriate. Accessing the partitioned data is still very slow in a variety of contexts for reasons I don't understand. I also hit a snag attempting to create a consolidated external `_metadata` file to hopefully speed up access to the partitioned data so... not sure what to do there. The current Tox/pytest setup expects to find data locally, which won't work right now on GitHub. Need to set the tests up better for real world use, and less for exploring different catalog configurations. Closes #5, #6
- Loading branch information
1 parent
6bab7df
commit 7fb38ff
Showing
13 changed files
with
1,761 additions
and
445 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
custom: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=PZBZDFNKBJW5E&source=url |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
version: 2 | ||
updates: | ||
- package-ecosystem: "pip" | ||
directory: "/" | ||
target-branch: "main" | ||
labels: | ||
- "dependencies" | ||
schedule: | ||
interval: "daily" | ||
|
||
- package-ecosystem: "github-actions" | ||
directory: "/" | ||
target-branch: "main" | ||
schedule: | ||
interval: "daily" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
name: tox-pytest | ||
|
||
on: [push, pull_request] | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
python-version: ["3.8", "3.9", "3.10"] | ||
fail-fast: false | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
with: | ||
fetch-depth: 2 | ||
|
||
- name: Set up conda environment for testing | ||
uses: conda-incubator/setup-miniconda@v2.1.1 | ||
with: | ||
miniforge-variant: Mambaforge | ||
miniforge-version: latest | ||
use-mamba: true | ||
mamba-version: "*" | ||
channels: conda-forge,defaults | ||
channel-priority: true | ||
python-version: ${{ matrix.python-version }} | ||
activate-environment: python-template | ||
environment-file: environment.yml | ||
- shell: bash -l {0} | ||
run: | | ||
conda info | ||
conda list | ||
conda config --show-sources | ||
conda config --show | ||
printenv | sort | ||
- name: Run PyTest with Tox | ||
run: | | ||
conda run -n pudl-catalog tox | ||
- name: Upload test coverage report to CodeCov | ||
uses: codecov/codecov-action@v3 | ||
|
||
- name: Inform the Codemonkeys | ||
uses: 8398a7/action-slack@v3 | ||
with: | ||
status: custom | ||
fields: workflow,job,commit,repo,ref,author,took | ||
custom_payload: | | ||
{ | ||
username: 'action-slack', | ||
icon_emoji: ':octocat:', | ||
attachments: [{ | ||
color: '${{ job.status }}' === 'success' ? 'good' : '${{ job.status }}' === 'failure' ? 'danger' : 'warning', | ||
text: `${process.env.AS_WORKFLOW}\n${process.env.AS_JOB} (${process.env.AS_COMMIT}) of ${process.env.AS_REPO}@${process.env.AS_REF} by ${process.env.AS_AUTHOR} ${{ job.status }} in ${process.env.AS_TOOK}`, | ||
}] | ||
} | ||
env: | ||
GITHUB_TOKEN: ${{ github.token }} # required | ||
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} # required | ||
MATRIX_CONTEXT: ${{ toJson(matrix) }} # required | ||
if: ${{ always() && github.actor != 'dependabot[bot]' }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
name: pudl-catalog | ||
channels: | ||
- conda-forge | ||
- defaults | ||
dependencies: | ||
# Packages required for setting up the environment | ||
- pip>=21,<23 | ||
- python>=3.8,<3.11 | ||
- setuptools<62 | ||
|
||
# Packages specified in setup.py that need or benefit from binary conda packages | ||
# - geopandas>=0.9,<0.11 | ||
# - pygeos>=0.10,<0.13 # Python wrappers for the GEOS spatial libraries | ||
# - python-snappy>=0.6,<0.7 # Supports snappy compression in pyarrow/parquet | ||
|
||
# Packages not specified in setup.py that provide optional, helpful binaries: | ||
# - numba>=0.55.1,<0.56 # numba speeds up some kinds of math by 100x | ||
|
||
# Jupyter packages: | ||
- jupyterlab>=3.2,<4 | ||
- jupyter-resource-usage>=0.5,<0.7 | ||
- nbconvert>=6,<7 | ||
- nbdime>=3.1,<4 | ||
|
||
# Use pip to install the package defined by this repo for development: | ||
- pip: | ||
- --editable ./[release,tests] |
Oops, something went wrong.