Skip to content

Commit

Permalink
Feature branch sync 06/29/2023 (#2918)
Browse files Browse the repository at this point in the history
* add search model group permission to ml_read_access role (#2855)

* add search model group permission to ml_read_access role

Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com>

* IntegrationTest spotless (#2863)

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Format everything (#2866)

* Use boucycastle PEM reader instead of reg expression (#2864)

Use BouncyCastle PEMReader instead of
regular expression to read and parse private key pem files.

Signed-off-by: Andrey Pleskach <ples@aiven.io>

* Adding field level security test cases for FlatFields (#2876)

Signed-off-by: Peter Nied <petern@amazon.com>

* Update snappy to 1.1.10.1 and guava to 32.0.1-jre (#2886)

* Update snappy to 1.1.10.1 and guava to 32.0.1-jre

Signed-off-by: Craig Perkins <cwperx@amazon.com>

* Upgrade kafka to 3.5.0

Signed-off-by: Craig Perkins <cwperx@amazon.com>

* Force snappy

Signed-off-by: Craig Perkins <cwperx@amazon.com>

* Add runtime dependency on org.scala-lang.modules:scala-java8-compat_3:1.0.2 to fix issue with KafkaSinkTest

Signed-off-by: Craig Perkins <cwperx@amazon.com>

---------

Signed-off-by: Craig Perkins <cwperx@amazon.com>

* Role permissions order tool and workflow (#2733)

* Check Permissions Order tool and workflow

Adds a NodeJS tool that can inspect yaml role definitions, check if they are in alphabetical order, correct them if required.

Signed-off-by: Peter Nied <peternied@hotmail.com>

* Apply fixes to roles.yml files

Signed-off-by: Peter Nied <peternied@hotmail.com>

* Fixing busted test, adding findArrayInJson for response bodies

Signed-off-by: Peter Nied <petern@amazon.com>

---------

Signed-off-by: Peter Nied <peternied@hotmail.com>
Signed-off-by: Peter Nied <petern@amazon.com>

* Misc changes (#2902)

Moved isStatic and isReserved methods to the
SecurityDynamicConfiguration class

Signed-off-by: Andrey Pleskach <ples@aiven.io>

* Update triaging guidelines (#2899)

* Update triaging guidelines

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>

* fix cluster perm classification for msearch template (#2892)

* fix cluster perm classification for msearch template

Signed-off-by: Derek Ho <dxho@amazon.com>

* move test to unit test file

Signed-off-by: Derek Ho <dxho@amazon.com>

* fully revert integration test file

Signed-off-by: Derek Ho <dxho@amazon.com>

* Update src/test/java/org/opensearch/security/privileges/PrivilegesEvaluatorUnitTest.java

Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>

* spotless

Signed-off-by: Derek Ho <dxho@amazon.com>

---------

Signed-off-by: Derek Ho <dxho@amazon.com>
Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>
Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>

* [Doc] Add architecture document (#2869)

* Add initial architecture document

Signed-off-by: Peter Nied <petern@amazon.com>

* [Enhancement] Parallel test jobs for CI (#2861)

* Split multiple tests into separate gradle tasks.
  * Tasks are configured in "splitTestConfig" map in build.gradle file.
    Map allows to use all patterns from TestFilter like:
    includeTestsMatching, excludeTestsMatching, includeTest etc.
  * Tasks are automatically generated from "splitTestConfig" map.
  * Two new Gradle tasks: listTasksAsJSON and listTasksAsParam to
    output task names to console. First one outputs them as a JSON
    and second - in gradlew "-x <TASK>" format to use in CLI.
  * Patterns included in tasks are automatically excluded from main
    "test" task but at the same time generated tasks are dependencies
    for "test". Running "gradlew test" will run whole suite at once.
* CI pipeline has been configured to accomodate all changes.
  * New 'master' task to generate list of jobs to run in parallel.
  * Updated matrix strategy to include task name to start.

Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>

* Bump BouncyCastle from jdk15on to jdk15to18 (#2901)

jdk15to18 contains fix for
 - CVE-2023-33201 - Medium
   Severity Vulnerability

Signed-off-by: Andrey Pleskach <ples@aiven.io>

* Spotless Apply

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Bhavana Ramaram <rbhavna@amazon.com>
Signed-off-by: Stephen Crawford <steecraw@amazon.com>
Signed-off-by: Andrey Pleskach <ples@aiven.io>
Signed-off-by: Peter Nied <petern@amazon.com>
Signed-off-by: Craig Perkins <cwperx@amazon.com>
Signed-off-by: Peter Nied <peternied@hotmail.com>
Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>
Signed-off-by: Derek Ho <dxho@amazon.com>
Signed-off-by: Pawel Gudel <pawel.gudel@eliatra.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: Bhavana Ramaram <rbhavna@amazon.com>
Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>
Co-authored-by: Andrey Pleskach <ples@aiven.io>
Co-authored-by: Peter Nied <petern@amazon.com>
Co-authored-by: Craig Perkins <cwperx@amazon.com>
Co-authored-by: Derek Ho <derek01778@gmail.com>
Co-authored-by: pawel-gudel-eliatra <136344230+pawel-gudel-eliatra@users.noreply.github.com>
  • Loading branch information
8 people authored Jun 29, 2023
1 parent 26244e9 commit 748a711
Show file tree
Hide file tree
Showing 282 changed files with 23,462 additions and 17,558 deletions.
44 changes: 28 additions & 16 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,34 @@ env:
GRADLE_OPTS: -Dhttp.keepAlive=false

jobs:
build:
name: build
generate-test-list:
runs-on: ubuntu-latest
outputs:
separateTestsNames: ${{ steps.set-matrix.outputs.separateTestsNames }}
steps:
- name: Set up JDK for build and test
uses: actions/setup-java@v2
with:
distribution: temurin # Temurin is a distribution of adoptium
java-version: 17

- name: Checkout security
uses: actions/checkout@v2

- name: Generate list of tasks
id: set-matrix
run: |
echo "separateTestsNames=$(./gradlew listTasksAsJSON -q --console=plain | tail -n 1)" >> $GITHUB_OUTPUT
test:
name: test
needs: generate-test-list
strategy:
fail-fast: false
matrix:
gradle_task: ${{ fromJson(needs.generate-test-list.outputs.separateTestsNames) }}
platform: [windows-latest, ubuntu-latest]
jdk: [11, 17]
platform: ["ubuntu-latest", "windows-latest"]
runs-on: ${{ matrix.platform }}

steps:
Expand All @@ -29,12 +50,8 @@ jobs:
uses: gradle/gradle-build-action@v2
with:
arguments: |
build test -Dbuild.snapshot=false
-x integrationTest
-x spotlessCheck
-x checkstyleMain
-x checkstyleTest
-x spotbugsMain
${{ matrix.gradle_task }} -Dbuild.snapshot=false
-x test
- name: Coverage
uses: codecov/codecov-action@v1
Expand All @@ -59,7 +76,7 @@ jobs:
fail-fast: false
matrix:
jdk: [17]
platform: ["ubuntu-latest", "windows-latest"]
platform: [ubuntu-latest, windows-latest]
runs-on: ${{ matrix.platform }}

steps:
Expand All @@ -78,18 +95,13 @@ jobs:
with:
arguments: |
integrationTest -Dbuild.snapshot=false
-x spotlessCheck
-x checkstyleMain
-x checkstyleTest
-x spotbugsMain
backward-compatibility:

strategy:
fail-fast: false
matrix:
jdk: [11, 17]
platform: ["ubuntu-latest", "windows-latest"]
platform: [ubuntu-latest, windows-latest]
runs-on: ${{ matrix.platform }}

steps:
Expand Down
23 changes: 23 additions & 0 deletions .github/workflows/code-hygiene.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,26 @@ jobs:
- uses: gradle/gradle-build-action@v2
with:
arguments: spotbugsMain

check-permissions-order:
runs-on: ubuntu-latest
name: Check permissions orders
steps:
- uses: actions/checkout@v2
- run: npm install yaml

- name: Check permissions order
run: |
exclude_pattern="(^|/)roles_invalidxcontent.yml($|/)
(^|/)invalid_config/config.yml($|/)"
# Set pattern to exclude certain files
set -e
exit_code=0
for file in $(find . -name '*.yml' | grep -Ev "$exclude_pattern"); do
if ! node check-permissions-order.js "$file" --slient; then
exit_code=1
echo "Error: $file requires changes. Run the following command to fix:"
echo "node check-permissions-order.js $file --fix"
fi
done
exit $exit_code
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,7 @@ out/
build/
gradle-build/
.gradle/

# nodejs
node_modules/
package-lock.json
131 changes: 131 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# OpenSearch Security Plugin Architecture

OpenSearch’s core systems do not include security features, these features are added by installing the Security Plugin. The Security Plugin extends OpenSearch to provide authentication, authorization, end to end Encryption, audit logging, and management interfaces.

## Components

The Security Plugin is packaged into a standard plugin zip file used by OpenSearch which can be installed by using the plugin tool. The security configuration is accessible on disk for modification before the node has been turned on. After node startup, the admin tools or API endpoints can be used for dynamic changes.

```mermaid
graph TD
subgraph OpenSearch Node
subgraph File System
cfg[Security Configuration files]
adm[Admin Tools]
end
subgraph Indices
idx(Index 1..n)
secIdx[Security Index]
end
subgraph Plugins
pgns(Plugins 1..n)
sec[Security Plugin]
end
sec -- bootstrap security config --> cfg
sec -- refresh security config from cluster --> secIdx
adm -- backup/restore security config --> sec
end
```

### Security Plugin

The runtime of the Security Plugin uses extension points to insert itself into the path actions. Several security management actions are registered in OpenSearch so they can be changed through REST API actions.

### Security Configuration

The security configuration is stored in an system index that is replicated to all nodes. When a change has been made to the configuration, the Security Plugin is reloaded to cleanly initialize its components with the new settings.

#### Configuration Files

When starting up with no security index detected in the cluster, the Security Plugin will attempt to load configuration files from disk into a new security index. The configuration files can be manually modified or sourced from a backup of a security index created using the admin tools.

### Admin Tools

For OpenSearch nodes to join a cluster, they need to have the same security configuration. Complete security configurations will include SSL settings and certificate files. The admin tools allow users to manage these settings and other features.

## Flows

### Authentication / Authorization

The Security Plugin supports multiple authentication backends including an internal identity provider which works with HTTP basic authentication as well as support [external providers](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/) such as OpenId Connect (OIDC) and SAML.

Authorization is governed by roles declared in the security configuration. Roles control resource access by referencing the transport action name and/or index names in combination with OpenSearch action names.

Users are assigned roles via the role mappings. These mappings include backend role assignments from authentication providers as well as internal roles defined in the Security Plugin.

```mermaid
sequenceDiagram
title Basic Authorization flow
autonumber
participant C as Client
participant O as OpenSearch
participant SP as Security Plugin
participant RH as Request Handler
participant AL as Audit Log
C->>O: Request
O->>SP: Request Received
activate SP
SP->>SP: Authenticate user via internal/external auth providers
SP->>SP: Resolve Authorization for user
SP-->>O: Allow/Deny request
SP->>AL: Update Audit Log asynchronously
deactivate SP
O->>RH: Request continues to request handler
RH-->>O: Result
O->>C: Response
```

#### Multiple Authorization Provider flow

Based on the order within the Security Plugin's configuration authentication providers are iterated through to discover which provider can authenticate the user.

```mermaid
sequenceDiagram
title Multiple Authorization Provider flow
autonumber
participant C as Client
participant SP as Security Plugin
participant IAP as Internal Auth Provider
participant EAP as External Auth Provider*
participant SC as Security Configuration
C->>SP: Incoming request
SP->>IAP: Attempt to authenticate internally
IAP-->>SP: Internal user result
loop for each External Auth Provider
SP->>EAP: Attempt to authenticate
EAP-->>SP: External user result
end
SP->>SC: Check Authorization rules
SC->>SC: Match user roles & permissions
SC-->>SP: Authorization result
SP-->>C: Response
```

#### Rest vs Transport flow

OpenSearch treats external REST requests differently than internal transport requests. While REST requests allow for client-to-node communication and make use of API routes, transport requests are more structured and are used to communicate between nodes.

```mermaid
sequenceDiagram
title Rest vs Transport Flow
autonumber
participant C as Client
participant O as OpenSearch
participant SP as Security Plugin (Rest Filter & Security Interceptor)
participant AH as Action Handler
C->>O: Request
O->>SP: REST Request Received
SP->>SP: If using client cert, Authenticate
SP-->>O: Continue request
O->>SP: Transport Request Received
SP->>SP: Authenticate user via internal/external auth providers
SP->>SP: Resolve Authorization for user
SP-->>O: Allow/Deny request
O->>AH: Send transport request to action handler
AH-->>O: Result
O->>C: Response
```
5 changes: 2 additions & 3 deletions TRIAGING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,8 @@ Meetings are lightly structured as follows:
1. Announcements: If there are any announcements to be made they will happen at the start of the meeting.
2. Review of new issues: The meetings always start with reviewing all untriaged [issues](https://github.com/search?q=label%3Auntriaged+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues&ref=advsearch&s=created&o=desc) for the security and security-dashboards repositories.
3. Untriaged items: Review any [issues](https://github.com/search?q=-label%3Atriaged+is%3Aopen++is%3Aissue+repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues) that might have had the 'untriaged' label removed but require additional triage discussion.
4. Open discussion: Next, we open the floor in case anyone wants to highlight an issue.
5. Backlog discussion: Then, we review issues from the [backlogs](https://github.com/search?q=label%3A%22sprint+backlog%22+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues&ref=advsearch&s=created&o=desc) of the security and security-dashboards repositories.
6. Least recent discussed issue: Finally, to close out the meeting we will [review the oldest](https://github.com/search?q=+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=issues&ref=advsearch&s=updated&o=asc) issues from both repositories, security and security-dashboards, to help identify issues that have languished.
4. Pull request discussion: Then, we review the status of outstanding [pull requests](https://github.com/search?q=+is%3Aopen++repo%3Aopensearch-project%2Fsecurity+repo%3Aopensearch-project%2Fsecurity-dashboards-plugin&type=pullrequests&ref=advsearch) from the security and security-dashboards repositories.
5. Open discussion: Finally, we open the floor in case anyone wants to highlight an issue.

There is no specific ordering within each category.

Expand Down
Loading

0 comments on commit 748a711

Please sign in to comment.