Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gradle: create a separate sub build for all connectors #11833

Merged
merged 3 commits into from
Apr 8, 2022
Merged

gradle: create a separate sub build for all connectors #11833

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dabcf59
create a separate sub build for all connectors
alafanechere Apr 8, 2022
9f7bb34
rename ALL_CONNECTORS to CONNECTORS
alafanechere Apr 8, 2022
c612d9f
keep ALL_CONNECTORS
alafanechere Apr 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,14 @@ sourceControl {

rootProject.name = 'airbyte'

// SUB_BUILD is an enum of <blank>, PLATFORM, CONNECTORS_BASE, OCTAVIA_CLI. Blank is equivalent to all.
// SUB_BUILD is an enum of <blank>, PLATFORM, CONNECTORS_BASE, ALL_CONNECTORS and OCTAVIA_CLI. Blank is equivalent to all.
if (!System.getenv().containsKey("SUB_BUILD")) {
println("Building all of Airbyte.")
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment just above here is out of date now. Add ALL_CONNECTORS to the list. It's minor but I think just CONNECTORS would be fine too and maybe more intuitive.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use ALL_CONNECTORS to distinguish it from CONNECTORS_BASE.

def subBuild = System.getenv().get("SUB_BUILD")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this is getting used in more places, it might be nice to move it up a level in visibility, so the below doesn't have to repeat itself much.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please clarify? This sub-build is only defined in this settings file and is already at top level of the file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant you could have move the variable definition def subBuild = System.getenv().get("SUB_BUILD") to higher up in the file and then used it below instead of System.getenv().containsKey("SUB_BUILD") around line 79 below. Not a huge deal.

println("Building Airbyte Sub Build: " + subBuild)
if (subBuild != "PLATFORM" && subBuild != "CONNECTORS_BASE" && subBuild != "OCTAVIA_CLI") {
throw new IllegalArgumentException(String.format("%s is invalid. Must be unset or PLATFORM or CONNECTORS_BASE or OCTAVIA_CLI", subBuild))
if (subBuild != "PLATFORM" && subBuild != "CONNECTORS_BASE" && subBuild != "ALL_CONNECTORS" && subBuild != "OCTAVIA_CLI") {
throw new IllegalArgumentException(String.format("%s is invalid. Must be unset or PLATFORM or CONNECTORS_BASE, ALL_CONNECTORS or OCTAVIA_CLI", subBuild))
}
}

Expand Down Expand Up @@ -76,7 +76,7 @@ if (!System.getenv().containsKey("SUB_BUILD") || System.getenv().get("SUB_BUILD"
}

// connectors base
if (!System.getenv().containsKey("SUB_BUILD") || System.getenv().get("SUB_BUILD") == "CONNECTORS_BASE") {
if (!System.getenv().containsKey("SUB_BUILD") || System.getenv().get("SUB_BUILD") == "CONNECTORS_BASE" || System.getenv().get("SUB_BUILD") == "ALL_CONNECTORS") {
include ':airbyte-cdk:python'
include ':airbyte-integrations:bases:airbyte-protocol'
include ':airbyte-integrations:bases:base'
Expand Down Expand Up @@ -115,7 +115,7 @@ if (!System.getenv().containsKey("SUB_BUILD") || System.getenv().get("SUB_BUILD"
}

// connectors
if (!System.getenv().containsKey("SUB_BUILD")) {
if (!System.getenv().containsKey("SUB_BUILD") || System.getenv().get("SUB_BUILD") == "ALL_CONNECTORS") {
// include all connector projects
def integrationsPath = rootDir.toPath().resolve('airbyte-integrations/connectors')
println integrationsPath
Expand Down
2 changes: 1 addition & 1 deletion tools/bin/ci_integration_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ all_integration_tests=$(./gradlew integrationTest --dry-run | grep 'integrationT
run() {
if [[ "$connector" == "all" ]] ; then
echo "Running: ./gradlew --no-daemon --scan integrationTest"
./gradlew --no-daemon --scan integrationTest
SUB_BUILD=ALL_CONNECTORS ./gradlew --no-daemon --scan integrationTest
else
if [[ "$connector" == *"base-normalization"* ]]; then
selected_integration_test="base-normalization"
Expand Down