Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BWC tests neural search #515

Merged
merged 47 commits into from
Jan 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
6a570bd
Reformatting test package
vibrantvarun Dec 7, 2023
b5a0fe4
rebasing
vibrantvarun Dec 7, 2023
724bd36
Initial commit of BWC Test
vibrantvarun Dec 7, 2023
a9357f2
Text Search
vibrantvarun Dec 19, 2023
8c0f22d
Fixing bug
vibrantvarun Dec 19, 2023
26eb401
Text Search bug fix
vibrantvarun Dec 20, 2023
7926189
Adding windows platform in bwc
vibrantvarun Dec 20, 2023
1f880dd
Adding windows platform in bwc
vibrantvarun Dec 20, 2023
a92e37c
Rolling Upgrade tests
vibrantvarun Dec 20, 2023
723fc13
Bux Fix in rolling upgrade
vibrantvarun Dec 20, 2023
975eaea
Bug Fix Rolling Upgrade
vibrantvarun Dec 21, 2023
5482a4e
Fixing Flaky tests
vibrantvarun Dec 21, 2023
38bd10b
Updating BWC version to latest
vibrantvarun Dec 21, 2023
b794610
Fixing bwc test
vibrantvarun Dec 21, 2023
74d9aef
Semantic Search
vibrantvarun Dec 26, 2023
ba17401
Bug Fix
vibrantvarun Dec 26, 2023
eccaf2a
Debugging
vibrantvarun Dec 26, 2023
e400e06
Bug Fix
vibrantvarun Dec 26, 2023
038cd1a
Increase memory in nodes
vibrantvarun Dec 26, 2023
6bb2dd3
Removing extra logging
vibrantvarun Dec 27, 2023
a5d17e1
Cleaning up
vibrantvarun Dec 27, 2023
f84bb56
Updating Pipeline Configuration
vibrantvarun Dec 27, 2023
d6d7f9b
Remove KNN delete models
vibrantvarun Dec 27, 2023
20ec55f
Remove unecessary KNN code
vibrantvarun Dec 27, 2023
4a7ea91
Addressing comments of naveen
vibrantvarun Dec 27, 2023
563caf8
Addressing comments of naveen
vibrantvarun Dec 27, 2023
88823b3
Addressing comments of Naveen
vibrantvarun Dec 27, 2023
de02df2
Addressing martin comments
vibrantvarun Dec 28, 2023
d7c6151
Addressing comments of martin
vibrantvarun Dec 28, 2023
820917b
Apply formatting.xml in all lines
vibrantvarun Dec 28, 2023
69a41bc
Removing extra spacess from formatting.gradle
vibrantvarun Dec 28, 2023
2a71941
Addressing martin comment
vibrantvarun Dec 28, 2023
3cd927b
Addressing Jack comments
vibrantvarun Dec 28, 2023
3a3d175
Addressing Jack comments
vibrantvarun Dec 29, 2023
24dab4f
Merge branch 'main' into formatting-1
vibrantvarun Dec 29, 2023
cba8193
Addressing Jack's comments
vibrantvarun Dec 29, 2023
8721079
Fixing Test cases
vibrantvarun Dec 29, 2023
e371d58
Addressing comments of Navneet
vibrantvarun Jan 2, 2024
2dcb631
Addressing comments of Navneet
vibrantvarun Jan 2, 2024
2ace98f
Addressing comments of Navneet
vibrantvarun Jan 2, 2024
ef84f99
Addressing comments of Navneet
vibrantvarun Jan 2, 2024
1ebe6a4
Removing extra parameter from createPipelineProcessor
vibrantvarun Jan 2, 2024
7b3498e
Fixing bug
vibrantvarun Jan 2, 2024
7f3e9e4
Increasing number of shards
vibrantvarun Jan 2, 2024
136bb81
Bug fix of load model id
vibrantvarun Jan 2, 2024
96cc6bb
Changing names of tests to TextEmbeddingPRocessor
vibrantvarun Jan 2, 2024
faa8dec
Updating indexes and replicas
vibrantvarun Jan 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/draft-release-notes-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ categories:
- 'Maintenance'
- title: 'Refactoring'
labels:
- 'Refactoring'
- 'Refactoring'
38 changes: 19 additions & 19 deletions .github/workflows/add-untriaged.yml
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
name: Apply 'untriaged' label during issue lifecycle
on:
issues:
types: [opened, reopened, transferred]
jobs:
apply-label:
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@v6
with:
script: |
github.rest.issues.addLabels({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
labels: ['untriaged']
})
name: Apply 'untriaged' label during issue lifecycle

on:
issues:
types: [opened, reopened, transferred]

jobs:
apply-label:
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@v6
with:
script: |
github.rest.issues.addLabels({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
labels: ['untriaged']
})
65 changes: 65 additions & 0 deletions .github/workflows/backwards_compatibility_tests_workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: Backwards Compatibility Tests NeuralSearch
on:
push:
branches:
- "*"
- "feature/**"
pull_request:
branches:
- "*"
- "feature/**"

jobs:
Restart-Upgrade-BWCTests-NeuralSearch:
strategy:
matrix:
java: [ 11, 17, 21 ]
os: [ubuntu-latest,windows-latest]
bwc_version : ["2.9.0","2.10.0","2.11.0","2.12.0-SNAPSHOT"]
opensearch_version : [ "3.0.0-SNAPSHOT" ]

name: NeuralSearch Restart-Upgrade BWC Tests
runs-on: ${{ matrix.os }}
env:
BWC_VERSION_RESTART_UPGRADE: ${{ matrix.bwc_version }}

steps:
- name: Checkout NeuralSearch
uses: actions/checkout@v1

- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

- name: Run NeuralSearch Restart-Upgrade BWC Tests from BWCVersion-${{ matrix.bwc_version }} to OpenSearch Version-${{ matrix.opensearch_version }} on ${{matrix.os}}
run: |
echo "Running restart-upgrade backwards compatibility tests ..."
./gradlew :qa:restart-upgrade:testAgainstNewCluster -D'tests.bwc.version=${{ matrix.bwc_version }}'

Rolling-Upgrade-BWCTests-NeuralSearch:
strategy:
matrix:
java: [ 11, 17, 21 ]
os: [ubuntu-latest,windows-latest]
bwc_version: [ "2.12.0-SNAPSHOT" ]
opensearch_version: [ "3.0.0-SNAPSHOT" ]

name: NeuralSearch Rolling-Upgrade BWC Tests
runs-on: ${{ matrix.os }}
env:
BWC_VERSION_ROLLING_UPGRADE: ${{ matrix.bwc_version }}

steps:
- name: Checkout NeuralSearch
uses: actions/checkout@v1

- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

- name: Run NeuralSearch Rolling-Upgrade BWC Tests from BWCVersion-${{ matrix.bwc_version }} to OpenSearch Version-${{ matrix.opensearch_version }} on ${{matrix.os}}
run: |
echo "Running rolling-upgrade backwards compatibility tests ..."
./gradlew :qa:rolling-upgrade:testRollingUpgrade -D'tests.bwc.version=${{ matrix.bwc_version }}'
6 changes: 3 additions & 3 deletions .github/workflows/delete_backport_branch.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
name: Delete merged branch of the backport PRs
on:
on:
pull_request:
types:
- closed

jobs:
delete-branch:
runs-on: ubuntu-latest
Expand All @@ -12,4 +12,4 @@ jobs:
- name: Delete merged branch
uses: SvanBoxel/delete-merged-branch@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ jobs:
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Fail if there were link errors
run: exit ${{ steps.lychee.outputs.exit_code }}
run: exit ${{ steps.lychee.outputs.exit_code }}
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Bug Fixes
- Fixing multiple issues reported in #497 ([#524](https://github.com/opensearch-project/neural-search/pull/524))
### Infrastructure
- BWC tests for Neural Search ([#515](https://github.com/opensearch-project/neural-search/pull/515))
### Documentation
### Maintenance
### Refactoring
vibrantvarun marked this conversation as resolved.
Show resolved Hide resolved
- Added spotless check in the build ([#515](https://github.com/opensearch-project/neural-search/pull/515))
14 changes: 7 additions & 7 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ git clone https://github.com/[your username]/neural-search.git

#### JDK 11

OpenSearch builds using Java 11 at a minimum. This means you must have a JDK 11 installed with the environment variable
OpenSearch builds using Java 11 at a minimum. This means you must have a JDK 11 installed with the environment variable
`JAVA_HOME` referencing the path to Java home for your JDK 11 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-11`.

One easy way to get Java 11 on *nix is to use [sdkman](https://sdkman.io/).
Expand Down Expand Up @@ -83,10 +83,10 @@ Please follow these formatting guidelines:

## Build

OpenSearch neural-search uses a [Gradle](https://docs.gradle.org/6.6.1/userguide/userguide.html) wrapper for its build.
OpenSearch neural-search uses a [Gradle](https://docs.gradle.org/6.6.1/userguide/userguide.html) wrapper for its build.
Run `gradlew` on Unix systems.

Build OpenSearch neural-search using `gradlew build`
Build OpenSearch neural-search using `gradlew build`

```
./gradlew build
Expand Down Expand Up @@ -221,8 +221,8 @@ See [CONTRIBUTING](CONTRIBUTING.md).

## Backports

The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the
original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow
run successfully on the PR. For example, if a PR on main needs to be backported to `2.x` branch, add a label
`backport 2.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the
original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow
run successfully on the PR. For example, if a PR on main needs to be backported to `2.x` branch, add a label
`backport 2.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
merged to main, the workflow will create a backport PR to the `2.x` branch.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)

## OpenSearch Neural Search
**OpenSearch Neural Search** is an OpenSearch plugin that adds dense neural retrieval into the OpenSearch ecosystem.
**OpenSearch Neural Search** is an OpenSearch plugin that adds dense neural retrieval into the OpenSearch ecosystem.
The plugin provides the capability for indexing documents and doing neural search on the indexed documents.

## Project Resources
Expand Down
40 changes: 30 additions & 10 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,14 @@ import org.opensearch.gradle.test.RestIntegTestTask
import java.util.concurrent.Callable

apply plugin: 'java'
apply plugin: 'java-test-fixtures'
apply plugin: 'idea'
apply plugin: 'opensearch.opensearchplugin'
apply plugin: 'opensearch.pluginzip'
apply plugin: 'jacoco'
apply plugin: "com.diffplug.spotless"
apply plugin: 'io.freefair.lombok'
apply from: 'gradle/formatting.gradle'

def pluginName = 'opensearch-neural-search'
def pluginDescription = 'A plugin that adds dense neural retrieval into the OpenSearch ecosytem'
Expand Down Expand Up @@ -123,6 +125,25 @@ allprojects {
version = "${opensearch_build}"
targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_11

apply from: rootProject.file('repositories.gradle').absoluteFile
plugins.withId('java') {
sourceCompatibility = targetCompatibility = "11"
}

afterEvaluate {
project.dependencyLicenses.enabled = false
project.thirdPartyAudit.enabled = false
project.loggerUsageCheck.enabled = false
project.forbiddenApis.ignoreFailures = false
project.forbiddenPatterns {
setEnabled(false)
}
project.testingConventions.enabled = false
project.validateNebulaPom.enabled = false
project.licenseFile = rootProject.file('LICENSE.txt')
project.noticeFile = rootProject.file('NOTICE.txt')
}
}

repositories {
Expand All @@ -144,6 +165,7 @@ dependencies {
zipArchive group: 'org.opensearch.plugin', name:'opensearch-ml-plugin', version: "${opensearch_build}"
compileOnly fileTree(dir: knnJarDirectory, include: '*.jar')
api group: 'org.opensearch', name:'opensearch-ml-client', version: "${opensearch_build}"
testFixturesImplementation "org.opensearch.test:framework:${opensearch_version}"
implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.14.0'
// ml-common excluded reflection for runtime so we need to add it by ourselves.
// https://github.com/opensearch-project/ml-commons/commit/464bfe34c66d7a729a00dd457f03587ea4e504d9
Expand All @@ -154,6 +176,10 @@ dependencies {
runtimeOnly group: 'org.apache.commons', name: 'commons-text', version: '1.10.0'
runtimeOnly group: 'com.google.code.gson', name: 'gson', version: '2.10.1'
runtimeOnly group: 'org.json', name: 'json', version: '20231013'
testFixturesImplementation "org.opensearch:common-utils:${version}"
testFixturesImplementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.14.0'
testFixturesCompileOnly group: 'com.google.guava', name: 'guava', version:'32.0.1-jre'
testFixturesCompileOnly fileTree(dir: knnJarDirectory, include: '*.jar')
}

// In order to add the jar to the classpath, we need to unzip the
Expand All @@ -173,6 +199,9 @@ compileJava {
compileTestJava {
options.compilerArgs.addAll(["-processor", 'lombok.launch.AnnotationProcessorHider$AnnotationProcessor'])
}
compileTestFixturesJava {
options.compilerArgs.addAll(["-processor", 'lombok.launch.AnnotationProcessorHider$AnnotationProcessor'])
}

def opensearch_tmp_dir = rootProject.file('build/private/opensearch_tmp').absoluteFile
opensearch_tmp_dir.mkdirs()
Expand Down Expand Up @@ -284,16 +313,6 @@ run {
useCluster testClusters.integTest
}

spotless {
java {
removeUnusedImports()
importOrder 'java', 'javax', 'org', 'com'
eclipse().configFile rootProject.file('formatterConfig.xml')
trimTrailingWhitespace()
endWithNewline()
}
}

jacocoTestReport {
dependsOn integTest, test
reports {
Expand All @@ -302,6 +321,7 @@ jacocoTestReport {
}
}

check.dependsOn spotlessCheck
vibrantvarun marked this conversation as resolved.
Show resolved Hide resolved
check.dependsOn jacocoTestCoverageVerification
jacocoTestCoverageVerification.dependsOn jacocoTestReport

Expand Down
File renamed without changes.
4 changes: 4 additions & 0 deletions formatter/license-header.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
6 changes: 6 additions & 0 deletions gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
# SPDX-License-Identifier: Apache-2.0
#

# The BWC version here should always be the latest opensearch version set in
# https://github.com/opensearch-project/OpenSearch/blob/main/libs/core/src/main/java/org/opensearch/Version.java .
# Wired compatibility of OpenSearch works like 3.x version is compatible with 2.(latest-major) version.
# Therefore, to run rolling-upgrade BWC Test on local machine the BWC version here should be set 2.(latest-major).
systemProp.bwc.version=2.12.0-SNAPSHOT
martin-gaievski marked this conversation as resolved.
Show resolved Hide resolved
vibrantvarun marked this conversation as resolved.
Show resolved Hide resolved

# For fixing Spotless check with Java 17
org.gradle.jvmargs=--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
Expand Down
32 changes: 32 additions & 0 deletions gradle/formatting.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
allprojects {
project.apply plugin: "com.diffplug.spotless"
spotless {
java {
// Normally this isn't necessary, but we have Java sources in
// non-standard places
target '**/*.java'

removeUnusedImports()
eclipse().configFile rootProject.file('formatter/formatterConfig.xml')
trimTrailingWhitespace()
endWithNewline();

custom 'Refuse wildcard imports', {
// Wildcard imports can't be resolved; fail the build
if (it =~ /\s+import .*\*;/) {
throw new AssertionError("Do not use wildcard imports. 'spotlessApply' cannot resolve this issue.")
}
}
}
format 'misc', {
target '*.md', '**/*.gradle', '**/*.json', '**/*.yaml', '**/*.yml', '**/*.svg'

trimTrailingWhitespace()
endWithNewline()
}
format("license", {
licenseHeaderFile("${rootProject.file("formatter/license-header.txt")}", "package ");
target("src/*/java/**/*.java","qa/*/java/**/*.java")
})
}
}
Loading
Loading