Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nightly / fix nightly check #641

Merged
merged 26 commits into from
May 27, 2021
+384 −272
Merged

Nightly / fix nightly check #641

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
5cca709
try some changes for nightly check
devinrsmith May 14, 2021
ee6f309
An empty commit
devinrsmith May 14, 2021
4a2a992
try to find leak
devinrsmith May 17, 2021
22c9122
Revert "try to find leak"
devinrsmith May 17, 2021
b28653f
fuzzer fix
devinrsmith May 17, 2021
c453333
test serial fix
devinrsmith May 17, 2021
72cf355
try conservative parallel
devinrsmith May 17, 2021
8b5bda9
more time table fixes
devinrsmith May 17, 2021
36d32d2
test serial again
devinrsmith May 17, 2021
1050cfb
Set proper gradle properties for CI. Ensure we override project prope…
devinrsmith May 17, 2021
b5993ee
Break out jobs
devinrsmith May 18, 2021
3089bbb
Merge remote-tracking branch 'upstream/main' into fix-nightly-check
devinrsmith May 18, 2021
15ac4e7
don't fix chunker parser test wrt db.timeTable
devinrsmith May 18, 2021
7e00e56
try parallel, and remove hacks
devinrsmith May 18, 2021
295d963
lower heap size for tests
devinrsmith May 19, 2021
073657c
some changes
devinrsmith May 19, 2021
e792242
force tasks to rerun on nightly
devinrsmith May 19, 2021
80be636
Merge remote-tracking branch 'upstream/main' into fix-nightly-check
devinrsmith May 20, 2021
a70b329
Add link to ticket
devinrsmith May 20, 2021
feaf3a0
reset testParallel to original
devinrsmith May 20, 2021
def110c
Publish the reports inline for Deephaven sourced runs
devinrsmith May 20, 2021
038cb79
Fixes failing serialTest
devinrsmith May 20, 2021
0b9c2d1
Revert "reset testParallel to original"
devinrsmith May 20, 2021
912f93f
update comment
devinrsmith May 20, 2021
fd3922d
Merge remote-tracking branch 'upstream/main' into nightly/fix-nightly…
devinrsmith May 25, 2021
d1b9bcc
Fixes from review
devinrsmith May 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .github/env/Linux/gradle.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# TODO (deephaven-core#639): Customize gradle settings per-CI job
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this our "todo standard"? I've been doing TODO (deephaven/deephaven-core/issues/#).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#87 , I'm not sure exactly what @mofojed means when he says permalink will be auto added?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have this documented in our CONTRIBUTING.md guide or in our wiki anywhere... @chipkent to decide if it should be documented anywhere.
When discussing previously, we had settled on just the issue number if in the same repo, or repo and issue number if in a different repo. In this case, it would be # TODO: #639 .... These comments are on issue #87 .
Also the GitHub plugin for VSCode has a handy trigger that will automatically create issues for you, and link them back to the code: https://code.visualstudio.com/docs/editor/github#_creating-issues
I'm not seeing a similar plugin for IntelliJ.

#
# https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
org.gradle.parallel=true
org.gradle.workers.max=2
# GitHub hosted linux runner environment has 2 CPUs and 7G RAM.
#
# testParallel seems to run poorly on GH Actions, and is unable to take advantage it seems of 2
# workers. Let's bump up max heap, and target it as the lowest-common denominator.
devinrsmith marked this conversation as resolved.
Show resolved Hide resolved

org.gradle.parallel=false
org.gradle.workers.max=1
org.gradle.jvmargs=-Xmx6g

# Our CI JDKs should be pre-provisioned and invoked correctly,
Expand Down
20 changes: 8 additions & 12 deletions .github/workflows/build-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
devinrsmith marked this conversation as resolved.
Show resolved Hide resolved
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Create Dockerfile and context
uses: burrunan/gradle-cache-action@v1
Expand Down Expand Up @@ -134,9 +133,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Create Dockerfile and context
uses: burrunan/gradle-cache-action@v1
Expand Down Expand Up @@ -194,9 +192,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Create Dockerfile and context
uses: burrunan/gradle-cache-action@v1
Expand Down Expand Up @@ -260,9 +257,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Create Dockerfile and context
uses: burrunan/gradle-cache-action@v1
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/check-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Check
uses: burrunan/gradle-cache-action@v1
Expand Down
10 changes: 4 additions & 6 deletions .github/workflows/docs-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java-8.outputs.path }},${{ steps.setup-java-11.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java-8.outputs.path }},${{ steps.setup-java-11.outputs.path }}" >> gradle.properties

- name: All Javadoc
uses: burrunan/gradle-cache-action@v1
Expand Down Expand Up @@ -71,9 +70,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Generate Python Docs
uses: burrunan/gradle-cache-action@v1
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/long-check-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Check
uses: burrunan/gradle-cache-action@v1
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/nightly-benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,8 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Build
uses: burrunan/gradle-cache-action@v1
Expand Down
180 changes: 170 additions & 10 deletions .github/workflows/nightly-check-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@ on:
schedule:
# 2AM EST == 6AM UTC
- cron: '0 6 * * *'
push:
branches: [ 'nightly/**' ]

jobs:
nightly:
check:
runs-on: ubuntu-20.04
timeout-minutes: 720
steps:
- name: Checkout
uses: actions/checkout@v2
Expand All @@ -24,24 +25,183 @@ jobs:

- name: Setup gradle properties
run: |
mkdir -p $HOME/.gradle
cp .github/env/${{ runner.os }}/gradle.properties $HOME/.gradle/gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> $HOME/.gradle/gradle.properties
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Check
- name: Run gradle
uses: burrunan/gradle-cache-action@v1
timeout-minutes: 720
with:
job-id: nightly
arguments: --scan --continue nightly
job-id: check
arguments: --scan --continue --rerun-tasks check
gradle-version: wrapper

- name: Upload Test Results
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-check-ci-results
path: '**/build/test-results/*/TEST-*.xml'

- name: Upload Test Reports
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-check-ci-reports
path: '**/build/reports/tests/**'

- name: Publish Test Results
uses: scacap/action-surefire-report@v1
if: ${{ github.repository_owner == 'deephaven' }}
env:
NODE_OPTIONS: '--max_old_space_size=4096'
with:
check_name: check report
github_token: ${{ secrets.GITHUB_TOKEN }}
report_paths: '**/build/test-results/*/TEST-*.xml'

testSerial:
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2

# note: this uses Zulu and not AdoptOpenJDK or other. should make sure we build and test on the same one...
- name: Setup JDK
id: setup-java
uses: actions/setup-java@v1
with:
java-version: '8.0.292'

- name: Setup gradle properties
run: |
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Run gradle
uses: burrunan/gradle-cache-action@v1
with:
job-id: testSerial
arguments: --scan --continue --rerun-tasks testSerial
gradle-version: wrapper

- name: Upload Test Results
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-testSerial-ci-results
path: '**/build/test-results/*/TEST-*.xml'

- name: Upload Test Reports
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-testSerial-ci-reports
path: '**/build/reports/tests/**'

- name: Publish Test Results
uses: scacap/action-surefire-report@v1
if: ${{ github.repository_owner == 'deephaven' }}
env:
NODE_OPTIONS: '--max_old_space_size=4096'
with:
check_name: nightly-check-ci test report
check_name: testSerial report
github_token: ${{ secrets.GITHUB_TOKEN }}
report_paths: '**/build/test-results/*/TEST-*.xml'

testParallel:
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2

# note: this uses Zulu and not AdoptOpenJDK or other. should make sure we build and test on the same one...
- name: Setup JDK
id: setup-java
uses: actions/setup-java@v1
with:
java-version: '8.0.292'

- name: Setup gradle properties
run: |
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Run gradle
uses: burrunan/gradle-cache-action@v1
with:
job-id: testParallel
arguments: --scan --continue --rerun-tasks testParallel
gradle-version: wrapper

- name: Upload Test Results
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-testParallel-ci-results
path: '**/build/test-results/*/TEST-*.xml'

- name: Upload Test Reports
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-testParallel-ci-reports
path: '**/build/reports/tests/**'

- name: Publish Test Results
uses: scacap/action-surefire-report@v1
if: ${{ github.repository_owner == 'deephaven' }}
env:
NODE_OPTIONS: '--max_old_space_size=4096'
with:
check_name: testParallel report
github_token: ${{ secrets.GITHUB_TOKEN }}
report_paths: '**/build/test-results/*/TEST-*.xml'

testOutOfBand:
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2

# note: this uses Zulu and not AdoptOpenJDK or other. should make sure we build and test on the same one...
- name: Setup JDK
id: setup-java
uses: actions/setup-java@v1
with:
java-version: '8.0.292'

- name: Setup gradle properties
run: |
cat .github/env/${{ runner.os }}/gradle.properties >> gradle.properties
echo "org.gradle.java.installations.paths=${{ steps.setup-java.outputs.path }}" >> gradle.properties

- name: Run gradle
uses: burrunan/gradle-cache-action@v1
with:
job-id: testOutOfBand
arguments: --scan --continue --rerun-tasks testOutOfBand
gradle-version: wrapper

- name: Upload Test Results
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-testOutOfBand-ci-results
path: '**/build/test-results/*/TEST-*.xml'

- name: Upload Test Reports
uses: actions/upload-artifact@v2
if: always()
with:
name: nightly-testOutOfBand-ci-reports
path: '**/build/reports/tests/**'

- name: Publish Test Results
uses: scacap/action-surefire-report@v1
if: ${{ github.repository_owner == 'deephaven' }}
env:
NODE_OPTIONS: '--max_old_space_size=4096'
with:
check_name: testOutOfBand report
github_token: ${{ secrets.GITHUB_TOKEN }}
report_paths: '**/build/test-results/*/TEST-*.xml'
10 changes: 4 additions & 6 deletions ClientSupport/src/main/java/io/deephaven/treetable/AbstractTreeSnapshotImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import io.deephaven.base.Pair;
import io.deephaven.base.verify.Assert;
import io.deephaven.configuration.Configuration;
import io.deephaven.internal.log.LoggerFactory;
import io.deephaven.io.logger.Logger;
import io.deephaven.db.tables.Table;
import io.deephaven.db.tables.live.NotificationQueue;
Expand Down Expand Up @@ -49,13 +50,12 @@ public abstract class AbstractTreeSnapshotImpl<INFO_TYPE extends HierarchicalTab
private final TreeTableClientTableManager.ClientState clientState;
private final TreeTableClientTableManager.TreeState treeState;

protected final Logger log;
private static final Logger log = LoggerFactory.getLogger(AbstractTreeSnapshotImpl.class);

/**
* Construct a new query that will create a flat snapshot of the tree table using a flat viewport beginning at the specified rows
* and columns, applying the specified sorts and filters if required to fetch tables
*
* @param baseTable The ID of the base table. This will be used as a key to maintain this client state.
* @param baseTable The ID of the base table. This will be used as a key to maintain this client state.
* @param tablesByKey The tables within the tree for which viewports are being tracked, separated by table key.
* @param firstRow The first row of the flat viewport
* @param lastRow The last row of the flat viewport
Expand All @@ -65,8 +65,7 @@ public abstract class AbstractTreeSnapshotImpl<INFO_TYPE extends HierarchicalTab
* @param client The CLIENT_TYPE instance
* @param includedOps The set of operations the client has performed before submitting this TSQ.
*/
AbstractTreeSnapshotImpl(Logger log,
int baseTableId,
AbstractTreeSnapshotImpl(int baseTableId,
HierarchicalTable baseTable,
Map<Object, TableDetails> tablesByKey,
long firstRow, long lastRow,
Expand All @@ -81,7 +80,6 @@ public abstract class AbstractTreeSnapshotImpl<INFO_TYPE extends HierarchicalTab
this.client = client;
this.baseTable = baseTable;
this.baseTableId = baseTableId;
this.log = log;
this.includedOps = includedOps;

firstViewportRow = firstRow;
Expand Down
Loading