Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AJ-1094: Initial Project Setup and Hello World CLI #1

Merged
merged 46 commits into from
Jul 20, 2023
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
e877db0
terra-java-project-template
snf2ye Jun 20, 2023
ccd077c
Remove example repo code
snf2ye Jun 21, 2023
7194f92
rename javatemplate to javapfb
snf2ye Jun 20, 2023
67b3e75
Update GHA's
snf2ye Jun 21, 2023
1008712
Hello world CLI
snf2ye Jun 21, 2023
8d530f6
Another round of template deletions
snf2ye Jun 22, 2023
7fd15a5
rename service to lib directory
snf2ye Jun 22, 2023
d0cc905
update github workflows
snf2ye Jun 23, 2023
ffbbc44
Try slightly different jfrog import
snf2ye Jun 23, 2023
8324e8f
Update LICENSE
snf2ye Jul 10, 2023
2fb38b5
Update release version
snf2ye Jul 10, 2023
faf6fa1
Update gradle version
snf2ye Jul 10, 2023
b909cd9
Remove spring; rename lib to library
snf2ye Jul 11, 2023
2e041ac
rework build.gradle
snf2ye Jul 13, 2023
41b8068
Update build/test GHA: Remove python check; add best test and sonar s…
snf2ye Jul 14, 2023
dac7e43
update sonar configuration
snf2ye Jul 14, 2023
834ce97
info
snf2ye Jul 14, 2023
460bece
Update readme
snf2ye Jul 14, 2023
fc87d7f
switch back to sonarqube command
snf2ye Jul 14, 2023
b693257
same exact setup as data catalog
snf2ye Jul 14, 2023
d494492
update project key
snf2ye Jul 14, 2023
86bc394
Update library/build.gradle
TomConner Jul 14, 2023
81b7874
switch to sonar command and upgrade to v4
snf2ye Jul 14, 2023
22927ac
Dependency cleanup
snf2ye Jul 17, 2023
7b4294a
Run sonar both CLI and library
snf2ye Jul 18, 2023
6b41911
Remove spotbugs: Sonar is sufficient to meet to appsec requirement
snf2ye Jul 18, 2023
6fc95f3
Just run sonar in library for now
snf2ye Jul 18, 2023
93ce6cf
ignore publish action; will fix with AJ-1095
snf2ye Jul 18, 2023
7371bb2
Update main method so it's not empty
snf2ye Jul 18, 2023
397ae31
update fatjar task to just be named jar
snf2ye Jul 18, 2023
9dc57a0
Update readme
snf2ye Jul 18, 2023
f519144
stub out some library code
snf2ye Jul 18, 2023
cda1397
rework package naming/structure
snf2ye Jul 18, 2023
c07d89c
update to new setup method of picocli
snf2ye Jul 18, 2023
e66e9ab
set up help and version message for CLI
snf2ye Jul 19, 2023
fb48d59
Add some test coverage for CLI
snf2ye Jul 19, 2023
79ad761
bare minimum description for library
snf2ye Jul 19, 2023
41b05de
update cli readme
snf2ye Jul 19, 2023
4870da5
Try running sonar for both CLI and library
snf2ye Jul 19, 2023
e90b6cc
Remove unneeded gradle conventions file
snf2ye Jul 19, 2023
b18f455
separate step for cli and library sonar scans
snf2ye Jul 19, 2023
82642ac
upgrade gradle to 8.2.1
snf2ye Jul 20, 2023
2017fad
Remove cli test command
snf2ye Jul 20, 2023
f484b35
Set srcclr scope to runtime
snf2ye Jul 20, 2023
0aecde2
Add a little more context to our source clear and sonar cloud runs.
snf2ye Jul 20, 2023
868ab14
Only need test action in common definitions; don't need jfrog def
snf2ye Jul 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
name: Build and Test

on:
push:
branches: [ main ]
paths-ignore: [ '*.md' ]
pull_request:
branches: [ '**' ]

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Set up JDK
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
cache: 'gradle'

- name: Build all projects without running tests
run: ./gradlew --build-cache build -x test

source-clear:
needs: [ build ]
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Set up JDK
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
cache: 'gradle'

- name: SourceClear scan
env:
SRCCLR_API_TOKEN: ${{ secrets.SRCCLR_API_TOKEN }}
run: ./gradlew --build-cache srcclr

unit-tests-and-sonarqube:
needs: [ build ]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
# Needed by sonar to get the git history for the branch the PR will be merged into.
with:
fetch-depth: 0
- name: Set up JDK
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
cache: 'gradle'
- name: Test with coverage
run: ./gradlew --build-cache test jacocoTestReport
- name: SonarQube scan
run: ./gradlew --build-cache :library:sonar --info
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

notify-slack:
needs: [ build, unit-tests-and-sonarqube, source-clear ]
runs-on: ubuntu-latest

if: failure() && github.ref == 'refs/heads/main'

steps:
- name: Notify slack on failure
uses: broadinstitute/action-slack@v3.8.0
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
with:
channel: '#dsp-analysis-journeys-alerts'
status: failure
author_name: Build on dev
fields: job,message
text: 'Build failed :sadpanda:'
username: 'Java-PFB GitHub Action'

dispatch-tag:
needs: [ build, unit-tests-and-sonarqube, source-clear ]
runs-on: ubuntu-latest

if: success() && github.ref == 'refs/heads/main'

steps:
- name: Fire off tag action
uses: broadinstitute/workflow-dispatch@v1
with:
workflow: Tag
token: ${{ secrets.BROADBOT_TOKEN }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we will want to tag on every push to main - but I'm totally fine leaving this as-is and addressing in the future, once we have a better idea of what we DO want

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok! I'm inclined to tag on every push to main, but agreed - let's leave as-is and we'll address it in the future once we've had a chance to discuss!

50 changes: 50 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Publish and deploy
on: create

env:
SERVICE_NAME: ${{ github.event.repository.name }}
GOOGLE_PROJECT: broad-dsp-gcr-public

jobs:
publish-job:
if: startsWith(github.ref, 'refs/tags/')
permissions:
contents: 'read'
id-token: 'write'
runs-on: ubuntu-latest
outputs:
tag: ${{ steps.tag.outputs.tag }}
steps:
- name: Enable publish with AJ-1095
run: echo "TODO"
# - uses: actions/checkout@v3
# - name: Set up JDK
# uses: actions/setup-java@v3
# with:
# java-version: '17'
# distribution: 'temurin'
# cache: 'gradle'

# - name: Parse tag
# id: tag
# run: echo "tag=$(git describe --tags)" >> $GITHUB_OUTPUT
#
# - name: Publish to Artifactory
# run: ./gradlew --build-cache :client:artifactoryPublish
# env:
# ARTIFACTORY_USERNAME: ${{ secrets.ARTIFACTORY_USERNAME }}
# ARTIFACTORY_PASSWORD: ${{ secrets.ARTIFACTORY_PASSWORD }}
# ARTIFACTORY_REPO_KEY: "libs-release-local"
#
# - name: Notify slack on failure
# uses: broadinstitute/action-slack@v3.8.0
# if: failure()
# env:
# SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
# with:
# channel: '#dsp-analysis-journeys-alerts'
# status: failure
# author_name: Publish to dev
# fields: job
# text: 'Publish failed :sadpanda:'
# username: 'Java-PFB GitHub Action'
22 changes: 22 additions & 0 deletions .github/workflows/tag.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: Tag
on: workflow_dispatch

jobs:
tag-job:
runs-on: ubuntu-latest
steps:
- name: Checkout current code
uses: actions/checkout@v3
with:
token: ${{ secrets.BROADBOT_TOKEN }} # this allows the push to succeed later
- name: Bump the tag to a new version
# https://github.com/DataBiosphere/github-actions/tree/master/actions/bumper
uses: databiosphere/github-actions/actions/bumper@bumper-0.0.6
id: tag
env:
GITHUB_TOKEN: ${{ secrets.BROADBOT_TOKEN }}
HOTFIX_BRANCHES: hotfix.*
DEFAULT_BUMP: minor
RELEASE_BRANCHES: main
VERSION_FILE_PATH: settings.gradle
VERSION_LINE_MATCH: "^\\s*gradle.ext.releaseVersion\\s*=\\s*'.*'"
38 changes: 38 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
HELP.md
.gradle
build/
!gradle/wrapper/gradle-wrapper.jar
!**/src/main/**/build/
!**/src/test/**/build/
bootrun.log

### STS ###
.apt_generated
.classpath
.factorypath
.project
.settings
.springBeans
.sts4-cache
bin/
!**/src/main/**/bin/
!**/src/test/**/bin/

# Emacs backup files #
*.*~

### IntelliJ IDEA ###
.idea/
*.iml

### VS Code ###
.vscode/

# Mac directory metadata
.DS_Store

# PyEnv environment files
.env/

# Ignore generated credentials from google-github-actions/auth
gha-creds-*.json
29 changes: 29 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
BSD 3-Clause License

Copyright (c) 2023, Broad Institute
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
37 changes: 36 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,36 @@
# java-pfb
# Java-PFB

A java implementation of the [pyPFB](https://github.com/uc-cdis/pypfb) library that includes a CLI and a java library.

The CLI is a wrapper around the library. See the [CLI README](cli/README.md) for more information.

## Running SourceClear locally

[SourceClear](https://srcclr.github.io) is a static analysis tool that scans a project's Java
dependencies for known vulnerabilities. If you get a build failure due a SourceClear error and want
to debug the problem locally, you need to get the API token from vault before running the gradle
task.

```shell
export SRCCLR_API_TOKEN=$(vault read -field=api_token secret/secops/ci/srcclr/gradle-agent)
./gradlew srcclr
```

## Running SonarQube locally

[SonarQube](https://www.sonarqube.org) is a static analysis code that scans code for a wide
range of issues, including maintainability and possible bugs. If you get a build failure due to
SonarQube and want to debug the problem locally, you need to get the the sonar token from vault
before runing the gradle task.

```shell
export SONAR_TOKEN=$(vault read -field=sonar_token secret/secops/ci/sonarcloud/java-pfb)
./gradlew sonar
```

Unlike SourceClear, running this task produces no output unless your project has errors. To always
generate a report, run using `--info`:

```shell
./gradlew sonar --info
```
25 changes: 25 additions & 0 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
plugins {
id 'groovy-gradle-plugin'
}

repositories {
mavenCentral()
maven {
url 'https://broadinstitute.jfrog.io/broadinstitute/libs-release-local/'
}
maven {
url 'https://broadinstitute.jfrog.io/broadinstitute/libs-snapshot-local/'
}
gradlePluginPortal()
}

dependencies {
implementation 'com.diffplug.spotless:spotless-plugin-gradle:6.11.0'
implementation 'com.srcclr.gradle:com.srcclr.gradle.gradle.plugin:3.1.12'
implementation 'org.sonarqube:org.sonarqube.gradle.plugin:4.2.1.3168'
Comment on lines +10 to +12
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will these be included in, or required by, the final jar we publish to artifactory? I haven't tried it myself to see what the results are. If these ARE included or required, can that be prevented, potentially by choosing a configuration other than implementation? We want the published jar to be as small and as dependency-free as possible; and these are only used by build tasks, not at runtime, correct?

I'm fine fixing this in a follow-on PR if you want to defer this work.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point, I will make sure this is covered with AJ-1095

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file (buildSrc/build.gradle) is different than other gradle project configuration files. It's (normally) only used to configure plugins, and its "dependencies" block only affects the plugins loaded by gradle, not compile or runtime dependencies used when running gradle tasks.

So these dependencies have no effect on the jar published to artifactory. They're here so when a plugin is used elsewhere in the project you don't need to specify the version. It provides a way to make sure that the same plugin versions are used in all subprojects.

With that in mind, the repositories block should only include those that are needed to load plugins. In my project I use:

repositories {
    maven {
        url 'https://broadinstitute.jfrog.io/artifactory/plugins-snapshot'
    }
    gradlePluginPortal()
}

The artifactory configuration is needed for the terra-test-runner plugin which is locally published. If you aren't using that plugin then gradlePluginPortal() should be sufficient.

Also I'd remove the test block below. I don't think it has any effect, and it should go in one of the plugin subprojects instead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pshapiro4broad thanks for this context, I think I understand the buildSrc setup a little better now.

implementation 'info.picocli:picocli:4.7.4'
}

test {
useJUnitPlatform()
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
plugins {
// Apply the common convention plugin for shared build configuration between library and application projects.
id 'bio.terra.pfb.java-common-conventions'
id 'application'
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
plugins {
id 'idea'
id 'jacoco'
id 'java'
id 'org.sonarqube'
id 'com.diffplug.spotless'
}

boolean isCiServer = System.getenv().containsKey("CI")

java {
toolchain {
languageVersion = JavaLanguageVersion.of(17)
}
}

repositories {
maven {
// Terra proxy for maven central
url 'https://broadinstitute.jfrog.io/broadinstitute/maven-central/'
}
mavenCentral()
maven {
url 'https://broadinstitute.jfrog.io/broadinstitute/libs-release/'
}
maven {
url 'https://broadinstitute.jfrog.io/broadinstitute/libs-snapshot-local/'
}
}

dependencies {
testImplementation 'org.hamcrest:hamcrest:2.2'

testImplementation 'org.junit.jupiter:junit-jupiter-api:5.8.1'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.8.1'
}

tasks.named('test') {
useJUnitPlatform()
}

version = gradle.releaseVersion
group = 'bio.terra'

spotless {
java {
targetExclude "${buildDir}/**"
googleJavaFormat()
}
}

// Run spotless check when running in github actions, otherwise run spotless apply.
compileJava {
if (isCiServer) {
dependsOn(spotlessCheck)
} else {
dependsOn(spotlessApply)
}
}

jacocoTestReport {
reports {
// sonarqube requires XML coverage output to upload coverage data
xml.required = true
}
}

sonar {
properties {
property "sonar.projectKey", "DataBiosphere_java-pfb"
property "sonar.projectName", "java-pfb"
property "sonar.organization", "broad-databiosphere"
property "sonar.host.url", "https://sonarcloud.io"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
plugins {
id 'bio.terra.pfb.java-common-conventions'
id 'java-library'
}
Loading