Skip to content

Commit

Permalink
Merge develop into main (#142)
Browse files Browse the repository at this point in the history
* Support construct AggregationResponseParser during Aggregator build stage (#108)

* Support construct AggregationResponseParser during Aggregator build stage

* modify the doc

Signed-off-by: penghuo <penghuo@gmail.com>

* Impl stddev and variance function in SQL and PPL (#115)

* impl variance frontend and backend

* Support construct AggregationResponseParser during Aggregator build stage

* add var and varp for PPL

Signed-off-by: penghuo <penghuo@gmail.com>

* add UT

Signed-off-by: penghuo <penghuo@gmail.com>

* fix UT

Signed-off-by: penghuo <penghuo@gmail.com>

* fix doc format

Signed-off-by: penghuo <penghuo@gmail.com>

* fix doc format

Signed-off-by: penghuo <penghuo@gmail.com>

* fix the doc

Signed-off-by: penghuo <penghuo@gmail.com>

* add stddev_samp and stddev_pop

Signed-off-by: penghuo <penghuo@gmail.com>

* fix UT coverage

* address comments

Signed-off-by: penghuo <penghuo@gmail.com>

* Fix the aggregation filter missing in named aggregators (#123)

* Take the condition expression as property to the named aggregator when wrapping the delegated aggregator

Signed-off-by: chloe-zh <chloezh1102@gmail.com>

* update

Signed-off-by: chloe-zh <chloezh1102@gmail.com>

* Added test case where filtered agg is not pushed down

Signed-off-by: chloe-zh <chloezh1102@gmail.com>

* update

Signed-off-by: chloe-zh <chloezh1102@gmail.com>

* update

Signed-off-by: chloe-zh <chloezh1102@gmail.com>

* Rename gradle tasks in the manner of Opensearch (#133)

Signed-off-by: Zhongnan Su <szhongna@amazon.com>

* add testing-library/user-event to pass CI (#141)

Signed-off-by: David Cui <davidcui@amazon.com>

* Deprecated legacy settings instead of removing (#140)

* Deprecated cursor enabling and fetch size setting

Signed-off-by: penghuo <penghuo@gmail.com>

* update doc

Signed-off-by: penghuo <penghuo@gmail.com>

* add more deprecated settings

Signed-off-by: penghuo <penghuo@gmail.com>

* Add Codecov to SQL  (#138)

* add codecov to CI workflow

Signed-off-by: David Cui <davidcui@amazon.com>

* add codecov token
Signed-off-by: David Cui <davidcui@amazon.com>

* Add Docs and link checker workflow (#139)

* save progress on doc renaming

* update rest of documentation links
Signed-off-by: David Cui <davidcui@amazon.com>

* add workbench developer guide and update readme
Signed-off-by: David Cui <davidcui@amazon.com>

* Bump glob-parent from 5.1.1 to 5.1.2 in /workbench

Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2.
- [Release notes](https://github.com/gulpjs/glob-parent/releases)
- [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md)
- [Commits](gulpjs/glob-parent@v5.1.1...v5.1.2)

---
updated-dependencies:
- dependency-name: glob-parent
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* add testing-library
Signed-off-by: David Cui <davidcui@amazon.com>

* rename workbench artifact name to kebab case
Signed-off-by: David Cui <davidcui@amazon.com>

* standardize maintainers list based on team roles, fix workbench name in dev guide
Signed-off-by: David Cui <davidcui@amazon.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add links to exclude from link checker
Signed-off-by: David Cui <davidcui@amazon.com>

* fix localhost exclusion flag
Signed-off-by: David Cui <davidcui@amazon.com>

* add all exclusion links, revert localhost exclusion away from regex
Signed-off-by: David Cui <davidcui@amazon.com>

* fix family.zzz exclusion, remove stale link from odbc sign_installers
Signed-off-by: David Cui <davidcui@amazon.com>

* fix lychee workflow after local testing and replace old sign_installers url
Signed-off-by: David Cui <davidcui@amazon.com>

* remove accidental duplicate exclude flag
Signed-off-by: David Cui <davidcui@amazon.com>

* remove sign installers as it is out of date
Signed-off-by: David Cui <davidcui@amazon.com>

Co-authored-by: Peng Huo <penghuo@gmail.com>
Co-authored-by: Chloe <chloezh1102@gmail.com>
Co-authored-by: Zhongnan Su <szhongna@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
5 people authored Jun 26, 2021
1 parent 8cd976f commit 254510e
Show file tree
Hide file tree
Showing 81 changed files with 2,478 additions and 508 deletions.
11 changes: 11 additions & 0 deletions .codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
codecov:
require_ci_to_pass: yes

coverage:
precision: 2
round: down
status:
project:
default:
target: 99% # the required coverage value
threshold: 1% # the leniency in hitting the target
23 changes: 23 additions & 0 deletions .github/workflows/link-checker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Link Checker
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
linkchecker:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: lychee Link Checker
id: lychee
uses: lycheeverse/lychee-action@master
with:
args: --accept=200,403,429 "**/*.html" "**/*.md" "**/*.txt" --exclude "http://localhost*" "https://localhost" "https://odfe-node1:9200/" "https://community.tableau.com/docs/DOC-17978" ".*family.zzz" "https://pypi.python.org/pypi/opensearch-sql-cli/" "opensearch*" ".*@amazon.com" ".*email.com" "git@github.com" "http://timestamp.verisign.com/scripts/timstamp.dll"
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Fail if there were link errors
run: exit ${{ steps.lychee.outputs.exit_code }}
6 changes: 6 additions & 0 deletions .github/workflows/sql-test-and-build-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ jobs:
mkdir -p opensearch-sql-builds
cp -r ./plugin/build/distributions/*.zip opensearch-sql-builds/
# This step uses the codecov-action Github action: https://github.com/codecov/codecov-action
- name: Upload SQL Coverage Report
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}

- name: Upload Artifacts
uses: actions/upload-artifact@v1
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sql-workbench-release-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
- 'v*'

env:
PLUGIN_NAME: queryWorkbenchDashboards
PLUGIN_NAME: query-workbench-dashboards
OPENSEARCH_VERSION: '1.0'
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-rc1

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: SQL Workbench Test and Build
on: [pull_request, push]

env:
PLUGIN_NAME: queryWorkbenchDashboards
PLUGIN_NAME: query-workbench-dashboards
OPENSEARCH_VERSION: '1.0'
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-rc1

Expand Down
7 changes: 7 additions & 0 deletions ADMINS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
## Admins

| Admin | GitHub ID | Affiliation |
| --------------- | --------------------------------------- | ----------- |
| Henri Yandell | [hyandell](https://github.com/hyandell) | Amazon |

[This document](https://github.com/opensearch-project/.github/blob/main/ADMINS.md) explains what admins do in this repo. and how they should be doing it. If you're interested in becoming a maintainer, see [MAINTAINERS](MAINTAINERS.md). If you're interested in contributing, see [CONTRIBUTING](CONTRIBUTING.md).
107 changes: 3 additions & 104 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,105 +1,4 @@
# Contributing Guidelines
## Contributing to this project

Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
documentation, we greatly value feedback and contributions from our community.

Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
information to effectively respond to your bug report or contribution.


## Reporting Bugs/Feature Requests

We welcome you to use the GitHub issue tracker to report bugs or suggest features.

When filing an issue, please check [existing open](https://github.com/opensearch-project/sql/issues), or [recently closed](https://github.com/opensearch-project/sql/issues?q=is%3Aissue+is%3Aclosed), issues to make sure somebody else hasn't already
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:

* A reproducible test case or series of steps
* The version of our code being used
* Any modifications you've made relevant to the bug
* Anything unusual about your environment or deployment

## Sign your work
OpenSearch is an open source product released under the Apache 2.0 license (see either [the Apache site](https://www.apache.org/licenses/LICENSE-2.0) or the [LICENSE.txt file](./LICENSE.txt)). The Apache 2.0 license allows you to freely use, modify, distribute, and sell your own products that include Apache 2.0 licensed software.

We respect intellectual property rights of others and we want to make sure all incoming contributions are correctly attributed and licensed. A Developer Certificate of Origin (DCO) is a lightweight mechanism to do that.

The DCO is a declaration attached to every contribution made by every developer. In the commit message of the contribution, the developer simply adds a `Signed-off-by` statement and thereby agrees to the DCO, which you can find below or at [DeveloperCertificate.org](http://developercertificate.org/).

```
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the
best of my knowledge, is covered under an appropriate open
source license and I have the right under that license to
submit that work with modifications, whether created in whole
or in part by me, under the same open source license (unless
I am permitted to submit under a different license), as
Indicated in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including
all personal information I submit with it, including my
sign-off) is maintained indefinitely and may be redistributed
consistent with this project or the open source license(s)
involved.
```
We require that every contribution to OpenSearch is signed with a Developer Certificate of Origin. Additionally, please use your real name. We do not accept anonymous contributors nor those utilizing pseudonyms.

Each commit must include a DCO which looks like this

```
Signed-off-by: Jane Smith <jane.smith@email.com>
```
You may type this line on your own when writing your commit messages. However, if your user.name and user.email are set in your git configs, you can use `-s` or `– – signoff` to add the `Signed-off-by` line to the end of the commit message.

## Contributing via Pull Requests
Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:

1. You are working against the latest source on the *develop* branch.
2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
3. You open an issue to discuss any significant work - we would hate for your time to be wasted.

To send us a pull request, please:

1. Fork the repository.
2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
3. Ensure local tests pass; please add unit tests for all the new code paths introduced by your change.
4. Commit to your fork using clear commit messages.
5. Send us a pull request, answering any default questions in the pull request interface.
6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.

GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).

## Developing

For instructions for development in details, please refer to [Developer Guide](./DEVELOPER_GUIDE.rst).

## Finding contributions to work on
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/opensearch-project/sql/issues?q=label%3A%22help+wanted%22) issues is a great place to start.


## Code of Conduct

This project has adopted an [Open Source Code of Conduct](./CODE_OF_CONDUCT.md).


## Security issue notifications

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue.


## Licensing

See the [LICENSE](./LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
OpenSearch is a community project that is built and maintained by people just like **you**.
[This document](https://github.com/opensearch-project/.github/blob/main/CONTRIBUTING.md) explains how you can contribute to this and related projects.
4 changes: 2 additions & 2 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@
| --------------- | --------- | ----------- |
| Anirudha (Ani) Jadhav | [anirudha](https://github.com/anirudha) | Amazon |
| Peng Huo | [penghuo](https://github.com/penghuo) | Amazon |
| Chloe | [chloe-zh](https://github.com/chloe-zh) | Amazon |
| Chen Dai | [dai-chen](https://github.com/dai-chen) | Amazon |
| Harold Wang | [harold-wang](https://github.com/harold-wang) | Amazon |
| Nick Knize | [nknize](https://github.com/nknize) | Amazon |
| Charlotte Henkle | [CEHENKLE](https://github.com/CEHENKLE) | Amazon |
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

[![Test and Build Workflow](https://github.com/opendistro-for-elasticsearch/sql/workflows/Java%20CI/badge.svg)](https://github.com/opendistro-for-elasticsearch/sql/actions)
[![codecov](https://codecov.io/gh/opendistro-for-elasticsearch/sql/branch/develop/graph/badge.svg)](https://codecov.io/gh/opendistro-for-elasticsearch/sql)
[![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://docs-beta.opensearch.org/docs/sql/endpoints/)
[![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://docs-beta.opensearch.org/search-plugins/sql/endpoints/)
[![Chat](https://img.shields.io/badge/chat-on%20forums-blue)](https://discuss.opendistrocommunity.dev/c/sql/)
![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)

Expand Down
1 change: 1 addition & 0 deletions RELEASING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This project follows the [OpenSearch release process](https://github.com/opensearch-project/.github/blob/main/RELEASING.md).
3 changes: 3 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Reporting a Vulnerability

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/) or directly via email to aws-security@amazon.com. Please do **not** create a public GitHub issue.
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,18 @@ public enum Key {
/**
* Legacy Common Settings.
*/
QUERY_SIZE_LIMIT("opendistro.query.size_limit");
QUERY_SIZE_LIMIT("opendistro.query.size_limit"),

/**
* Deprecated Settings.
*/
SQL_NEW_ENGINE_ENABLED("opendistro.sql.engine.new.enabled"),
QUERY_ANALYSIS_ENABLED("opendistro.sql.query.analysis.enabled"),
QUERY_ANALYSIS_SEMANTIC_SUGGESTION("opendistro.sql.query.analysis.semantic.suggestion"),
QUERY_ANALYSIS_SEMANTIC_THRESHOLD("opendistro.sql.query.analysis.semantic.threshold"),
QUERY_RESPONSE_FORMAT("opendistro.sql.query.response.format"),
SQL_CURSOR_ENABLED("opendistro.sql.cursor.enabled"),
SQL_CURSOR_FETCH_SIZE("opendistro.sql.cursor.fetch_size");

@Getter
private final String keyValue;
Expand Down
1 change: 1 addition & 0 deletions core/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ dependencies {
compile group: 'org.springframework', name: 'spring-beans', version: '5.2.5.RELEASE'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.10'
compile group: 'com.facebook.presto', name: 'presto-matching', version: '0.240'
compile group: 'org.apache.commons', name: 'commons-math3', version: '3.6.1'
compile project(':common')

testImplementation('org.junit.jupiter:junit-jupiter:5.6.2')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,8 @@ public Expression visitNot(Not node, AnalysisContext context) {

@Override
public Expression visitAggregateFunction(AggregateFunction node, AnalysisContext context) {
Optional<BuiltinFunctionName> builtinFunctionName = BuiltinFunctionName.of(node.getFuncName());
Optional<BuiltinFunctionName> builtinFunctionName =
BuiltinFunctionName.ofAggregation(node.getFuncName());
if (builtinFunctionName.isPresent()) {
Expression arg = node.getField().accept(this, context);
Aggregator aggregator = (Aggregator) repository.compile(
Expand Down
16 changes: 16 additions & 0 deletions core/src/main/java/org/opensearch/sql/expression/DSL.java
Original file line number Diff line number Diff line change
Expand Up @@ -500,6 +500,22 @@ public Aggregator count(Expression... expressions) {
return aggregate(BuiltinFunctionName.COUNT, expressions);
}

public Aggregator varSamp(Expression... expressions) {
return aggregate(BuiltinFunctionName.VARSAMP, expressions);
}

public Aggregator varPop(Expression... expressions) {
return aggregate(BuiltinFunctionName.VARPOP, expressions);
}

public Aggregator stddevSamp(Expression... expressions) {
return aggregate(BuiltinFunctionName.STDDEV_SAMP, expressions);
}

public Aggregator stddevPop(Expression... expressions) {
return aggregate(BuiltinFunctionName.STDDEV_POP, expressions);
}

public RankingWindowFunction rowNumber() {
return (RankingWindowFunction) repository.compile(
BuiltinFunctionName.ROW_NUMBER.getName(), Collections.emptyList());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@
import static org.opensearch.sql.data.type.ExprCoreType.STRING;
import static org.opensearch.sql.data.type.ExprCoreType.TIME;
import static org.opensearch.sql.data.type.ExprCoreType.TIMESTAMP;
import static org.opensearch.sql.expression.aggregation.StdDevAggregator.stddevPopulation;
import static org.opensearch.sql.expression.aggregation.StdDevAggregator.stddevSample;
import static org.opensearch.sql.expression.aggregation.VarianceAggregator.variancePopulation;
import static org.opensearch.sql.expression.aggregation.VarianceAggregator.varianceSample;

import com.google.common.collect.ImmutableMap;
import java.util.Collections;
Expand Down Expand Up @@ -68,6 +72,10 @@ public static void register(BuiltinFunctionRepository repository) {
repository.register(count());
repository.register(min());
repository.register(max());
repository.register(varSamp());
repository.register(varPop());
repository.register(stddevSamp());
repository.register(stddevPop());
}

private static FunctionResolver avg() {
Expand Down Expand Up @@ -159,4 +167,48 @@ private static FunctionResolver max() {
.build()
);
}

private static FunctionResolver varSamp() {
FunctionName functionName = BuiltinFunctionName.VARSAMP.getName();
return new FunctionResolver(
functionName,
new ImmutableMap.Builder<FunctionSignature, FunctionBuilder>()
.put(new FunctionSignature(functionName, Collections.singletonList(DOUBLE)),
arguments -> varianceSample(arguments, DOUBLE))
.build()
);
}

private static FunctionResolver varPop() {
FunctionName functionName = BuiltinFunctionName.VARPOP.getName();
return new FunctionResolver(
functionName,
new ImmutableMap.Builder<FunctionSignature, FunctionBuilder>()
.put(new FunctionSignature(functionName, Collections.singletonList(DOUBLE)),
arguments -> variancePopulation(arguments, DOUBLE))
.build()
);
}

private static FunctionResolver stddevSamp() {
FunctionName functionName = BuiltinFunctionName.STDDEV_SAMP.getName();
return new FunctionResolver(
functionName,
new ImmutableMap.Builder<FunctionSignature, FunctionBuilder>()
.put(new FunctionSignature(functionName, Collections.singletonList(DOUBLE)),
arguments -> stddevSample(arguments, DOUBLE))
.build()
);
}

private static FunctionResolver stddevPop() {
FunctionName functionName = BuiltinFunctionName.STDDEV_POP.getName();
return new FunctionResolver(
functionName,
new ImmutableMap.Builder<FunctionSignature, FunctionBuilder>()
.put(new FunctionSignature(functionName, Collections.singletonList(DOUBLE)),
arguments -> stddevPopulation(arguments, DOUBLE))
.build()
);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ public class NamedAggregator extends Aggregator<AggregationState> {

/**
* NamedAggregator.
* The aggregator properties {@link #condition} is inherited by named aggregator
* to avoid errors introduced by the property inconsistency.
*
* @param name name
* @param delegated delegated
Expand All @@ -64,6 +66,7 @@ public NamedAggregator(
super(delegated.getFunctionName(), delegated.getArguments(), delegated.returnType);
this.name = name;
this.delegated = delegated;
this.condition = delegated.condition;
}

@Override
Expand Down
Loading

0 comments on commit 254510e

Please sign in to comment.