Skip to content
This repository has been archived by the owner on Jan 12, 2024. It is now read-only.

Commit

Permalink
Flesh out packaging/CI and fix catalog issues
Browse files Browse the repository at this point in the history
With some pointers from @martindurant in
[this issue](intake/intake-parquet#26) I got
anonymous public access working, and caching can now be turned off when
appropriate.

Accessing the partitioned data is still very slow in a variety of
contexts for reasons I don't understand. I also hit a snag attempting to
create a consolidated external `_metadata` file to hopefully speed up
access to the partitioned data so... not sure what to do there.

The current Tox/pytest setup expects to find data locally, which won't
work right now on GitHub. Need to set the tests up better for real world
use, and less for exploring different catalog configurations.

Closes #5, #6
  • Loading branch information
zaneselvans committed Apr 19, 2022
1 parent 6bab7df commit 7fb38ff
Show file tree
Hide file tree
Showing 13 changed files with 1,761 additions and 445 deletions.
1 change: 1 addition & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
custom: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=PZBZDFNKBJW5E&source=url
15 changes: 15 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
target-branch: "main"
labels:
- "dependencies"
schedule:
interval: "daily"

- package-ecosystem: "github-actions"
directory: "/"
target-branch: "main"
schedule:
interval: "daily"
63 changes: 63 additions & 0 deletions .github/workflows/tox-pytest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: tox-pytest

on: [push, pull_request]

jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10"]
fail-fast: false

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 2

- name: Set up conda environment for testing
uses: conda-incubator/setup-miniconda@v2.1.1
with:
miniforge-variant: Mambaforge
miniforge-version: latest
use-mamba: true
mamba-version: "*"
channels: conda-forge,defaults
channel-priority: true
python-version: ${{ matrix.python-version }}
activate-environment: python-template
environment-file: environment.yml
- shell: bash -l {0}
run: |
conda info
conda list
conda config --show-sources
conda config --show
printenv | sort
- name: Run PyTest with Tox
run: |
conda run -n pudl-catalog tox
- name: Upload test coverage report to CodeCov
uses: codecov/codecov-action@v3

- name: Inform the Codemonkeys
uses: 8398a7/action-slack@v3
with:
status: custom
fields: workflow,job,commit,repo,ref,author,took
custom_payload: |
{
username: 'action-slack',
icon_emoji: ':octocat:',
attachments: [{
color: '${{ job.status }}' === 'success' ? 'good' : '${{ job.status }}' === 'failure' ? 'danger' : 'warning',
text: `${process.env.AS_WORKFLOW}\n${process.env.AS_JOB} (${process.env.AS_COMMIT}) of ${process.env.AS_REPO}@${process.env.AS_REF} by ${process.env.AS_AUTHOR} ${{ job.status }} in ${process.env.AS_TOOK}`,
}]
}
env:
GITHUB_TOKEN: ${{ github.token }} # required
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} # required
MATRIX_CONTEXT: ${{ toJson(matrix) }} # required
if: ${{ always() && github.actor != 'dependabot[bot]' }}
27 changes: 27 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: pudl-catalog
channels:
- conda-forge
- defaults
dependencies:
# Packages required for setting up the environment
- pip>=21,<23
- python>=3.8,<3.11
- setuptools<62

# Packages specified in setup.py that need or benefit from binary conda packages
# - geopandas>=0.9,<0.11
# - pygeos>=0.10,<0.13 # Python wrappers for the GEOS spatial libraries
# - python-snappy>=0.6,<0.7 # Supports snappy compression in pyarrow/parquet

# Packages not specified in setup.py that provide optional, helpful binaries:
# - numba>=0.55.1,<0.56 # numba speeds up some kinds of math by 100x

# Jupyter packages:
- jupyterlab>=3.2,<4
- jupyter-resource-usage>=0.5,<0.7
- nbconvert>=6,<7
- nbdime>=3.1,<4

# Use pip to install the package defined by this repo for development:
- pip:
- --editable ./[release,tests]
Loading

0 comments on commit 7fb38ff

Please sign in to comment.