Skip to content

Commit

Permalink
[SPARK-46495][INFRA] Merge pyspark-error to pyspark-core
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Merge `pyspark-error` to `pyspark-core`

### Why are the changes needed?
`pyspark-error` (as well as the python packaging test) is pretty fast, it was initially factored out to fix the disk space issue.
Now we can merge it back to other tests, to save one test job.

### Does this PR introduce _any_ user-facing change?
no, infra-only

### How was this patch tested?
ci
manually check the merged job https://github.com/zhengruifeng/spark/actions/runs/7312132932/job/19922676199

### Was this patch authored or co-authored using generative AI tooling?
no

Closes apache#44470 from zhengruifeng/infra_py_error.

Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
zhengruifeng authored and dongjoon-hyun committed Dec 24, 2023
1 parent 4f56958 commit bd29569
Showing 1 changed file with 3 additions and 5 deletions.
8 changes: 3 additions & 5 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -345,12 +345,10 @@ jobs:
java:
- ${{ inputs.java }}
modules:
- >-
pyspark-errors
- >-
pyspark-sql, pyspark-resource, pyspark-testing
- >-
pyspark-core, pyspark-streaming
pyspark-core, pyspark-errors, pyspark-streaming
- >-
pyspark-mllib, pyspark-ml, pyspark-ml-connect
- >-
Expand Down Expand Up @@ -436,7 +434,7 @@ jobs:
python3.9 -m pip list
pypy3 -m pip list
- name: Install Conda for pip packaging test
if: ${{ matrix.modules == 'pyspark-errors' }}
if: contains(matrix.modules, 'pyspark-errors')
run: |
curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
Expand All @@ -446,7 +444,7 @@ jobs:
env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
if [[ "$MODULES_TO_TEST" == "pyspark-errors" ]]; then
if [[ "$MODULES_TO_TEST" == *"pyspark-errors"* ]]; then
export PATH=$PATH:$HOME/miniconda/bin
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
Expand Down

0 comments on commit bd29569

Please sign in to comment.