Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature/Identity] Adds Basic Auth mechanism via Internal IdP #4798

Conversation

DarshitChanpura
Copy link
Member

@DarshitChanpura DarshitChanpura commented Oct 14, 2022

Description

Adds a REST request wrapper to intercept all incoming REST requests and authenticate them via Internal IdP (HTTP only for this PR)

NOTE: This PR does not reject the request if no credentials were present and will proceed as normal. This is intentional.

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@DarshitChanpura DarshitChanpura force-pushed the basic-auth-via-internal-idp branch from 5d3277f to 42e8115 Compare October 14, 2022 23:32
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@DarshitChanpura
Copy link
Member Author

A whole bunch of tests are failing because of addition of REST request wrapper (literally a breaking change 😅). Looking into possible solutions to fix those now.

@JsonProperty(value = "primary_principal")
public StringPrincipal getPrimaryPrincipal() {
public StringPrincipal getPrincipal() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous version implied that a subject could have more than one principal (which is true), this one does not imply this. Is this intentional or was this change made since it should only ever be interacting with a specific principal at a time?

@JsonProperty(value = "primary_principal")
public StringPrincipal getPrimaryPrincipal() {
public StringPrincipal getPrincipal() {
return primaryPrincipal;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May want to rename this to be consistent if you are renaming the function call.

} else if (!isAuthenticationSuccessful) {
throw new AuthenticationException("Authentication finally failed");
}
} catch (Throwable e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a dead throw?

* @return true if authentication was successful, false otherwise
* @throws Exception generic exception rethrown in case authenticate method throws an exception
*/
private boolean checkAndAuthenticateRequest(RestRequest request, RestChannel channel, NodeClient client) throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider splitting this into separate formatting and a authenticated checks--may be easier to provide meaningful exception messages.

* @return true if authentication was successful, false otherwise
* @throws IOException when an exception is raised writing response to channel
*/
private boolean authenticate(RestRequest request, RestChannel channel, NodeClient client) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above comment; this method authenticates from within the checkAndAuthenticateRequest -- why not split the two if you already have the separate authenticate method?

@github-actions
Copy link
Contributor

github-actions bot commented Nov 1, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 1, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 1, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@DarshitChanpura DarshitChanpura force-pushed the basic-auth-via-internal-idp branch from ce25681 to 61f658d Compare November 3, 2022 20:12
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@DarshitChanpura
Copy link
Member Author

Gradle check failing due to unrelated issue. Check logs from this build and log from CI build a day ago

Looks like an issue with openJDK installation

@DarshitChanpura
Copy link
Member Author

Local run of `./gradlew :server:assemble` is successful:
➜  OpenSearch git:(basic-auth-via-internal-idp) ./gradlew :server:assemble

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.5.1
  OS Info               : Mac OS X 12.6.1 (x86_64)
  Runtime JDK Version   : 11 (Amazon Corretto JDK)
  Runtime java.home     : /Library/Java/JavaVirtualMachines/amazon-corretto-11.jdk/Contents/Home
  Gradle JDK Version    : 11 (Amazon Corretto JDK)
  Gradle java.home      : /Library/Java/JavaVirtualMachines/amazon-corretto-11.jdk/Contents/Home
  Random Testing Seed   : D442BC7730404DBF
  In FIPS 140 mode      : false
=======================================

> Task :server:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL in 2m 15s
35 actionable tasks: 12 executed, 6 from cache, 17 up-to-date
➜  OpenSearch git:(basic-auth-via-internal-idp)   

@DarshitChanpura DarshitChanpura force-pushed the basic-auth-via-internal-idp branch from 61f658d to 080d2be Compare November 9, 2022 16:03
@github-actions
Copy link
Contributor

github-actions bot commented Nov 9, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 9, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Nov 9, 2022

Gradle Check (Jenkins) Run Completed with:

@DarshitChanpura
Copy link
Member Author

DarshitChanpura commented Nov 9, 2022

Seeing too_long_http_line_exception error on the logs:

 java.lang.AssertionError: Failure at [search/30_limits:98]: the error message was expected to match the provided regex but didn't
    Expected: The length of regex \[1110\] used in the Regexp Query request has exceeded the allowed maximum of \[1000\]\. This maximum can be set by changing the \[index.max_regex_length\] index level setting\.
         but: was "{root_cause=[{type=too_long_http_line_exception, reason=An HTTP line is larger than 4096 bytes.}], type=too_long_http_line_exception, reason=An HTTP line is larger than 4096 bytes.}"
        at __randomizedtesting.SeedInfo.seed([7700579BEDAE21BE:FF54684143524C46]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:459)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:432)
        at java.****/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
.....

https://netty.io/4.1/api/io/netty/handler/codec/http/TooLongHttpLineException.html

Possibly related: elastic/elasticsearch#1174

try {
// support other type of header tokens
headerToken = new HttpHeaderToken(authHeader.get());
subject = Identity.getAuthManager().getSubject();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've seen this call in Shiro documentation, but I'm not quite following how this resolves to the correct subject here.

According to Shiro docs:

In almost all environments, you can obtain the currently executing Subject by using org.apache.shiro.SecurityUtils:

Subject currentUser = SecurityUtils.getSubject();

The getSubject() call in a standalone application might return a Subject based on user data in an application-specific location, and in a server environment (e.g. web app), it acquires the Subject based on user data associated with current thread or incoming request.

https://shiro.apache.org/subject.html

Will this always resolve to the correct Subject?

Copy link
Member Author

@DarshitChanpura DarshitChanpura Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.. Shiro has its own resource store and it fetches correct subject via ThreadContext.

This is how shiro puts resource into its ThreadContext store:

https://github.com/apache/shiro/blob/main/core/src/main/java/org/apache/shiro/util/ThreadContext.java#L153

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to shiro source, the DefaultSecurityManager uses ThreadContext under the hood: https://github.com/apache/shiro/blob/main/core/src/main/java/org/apache/shiro/SecurityUtils.java#LL54C9-L54C54

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call to get subject might have to do some weird stuff such as inspecting the call stack or thread context, but the pattern should be abstracted for use cases like this. Does that seem viable to you, or is there a way you think we should consider reframing the calls?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the ask here

Signed-off-by: Craig Perkins <cwperx@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@DarshitChanpura DarshitChanpura force-pushed the basic-auth-via-internal-idp branch from 7b7fb72 to c14ab0c Compare November 22, 2022 16:57
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@DarshitChanpura
Copy link
Member Author

@cwperks @peternied Can you please review and merge this is there is nothing else?

Copy link
Member

@peternied peternied left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many build errors, please revert the broad chances to the namespace under server/src/... until these are resolved

@@ -18,7 +18,7 @@ dependencies {
implementation "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
implementation "com.fasterxml.jackson.core:jackson-databind:${versions.jackson_databind}"

implementation 'org.apache.shiro:shiro-core:1.9.1'
api 'org.apache.shiro:shiro-core:1.9.1'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this change is needed, Could you provide the why of switching from implementation to api dependency type?

// Malformed AuthHeader strings
if (decodedUserNamePassword.length != 2) return null;

logger.info("Logging in as: " + decodedUserNamePassword[0]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about creating an issue and finding/implementing a better version of this?

🤞 I would rather pull a small library in for this kind of logic

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will modify the logic to be similar to one that Craig posted above. However, this seems like a good avenue to explore libraries that provide support for this.

@peternied
Copy link
Member

There are many build errors, please revert the broad chances to the namespace under server/src/... until these are resolved

🤦‍♂️ My build was not clean, I'll review more deeply...

@cwperks
Copy link
Member

cwperks commented Nov 30, 2022

@DarshitChanpura Thank you for adding tests. Can you please answer @peternied's question on why the change from api to implementation for the shiro dependency is needed.

I am ok with accepting and merging this change and addressing some of the other comments as follow-ups. From what I can see this is what is remaining:

  • How to reject a request without bringing down the cluster - Resolved by skipping dispatchRequest when the request is not authenticated.
  • Add same scenario tests for username/password combos as security plugin to ensure it works with the same set of rules
  • I'd be curious to see if we can follow a similar RestWrapper pattern for basic and bearer auth handling. In the current model, core has an extension point called getRestHandlerWrapper (
    /**
    * Returns a function used to wrap each rest request before handling the request.
    * The returned {@link UnaryOperator} is called for every incoming rest request and receives
    * the original rest handler as it's input. This allows adding arbitrary functionality around
    * rest request handlers to do for instance logging or authentication.
    * A simple example of how to only allow GET request is here:
    * <pre>
    * {@code
    * UnaryOperator<RestHandler> getRestHandlerWrapper(ThreadContext threadContext) {
    * return originalHandler -> (RestHandler) (request, channel, client) -> {
    * if (request.method() != Method.GET) {
    * throw new IllegalStateException("only GET requests are allowed");
    * }
    * originalHandler.handleRequest(request, channel, client);
    * };
    * }
    * }
    * </pre>
    *
    * Note: Only one installed plugin may implement a rest wrapper.
    */
    default UnaryOperator<RestHandler> getRestHandlerWrapper(ThreadContext threadContext) {
    return null;
    }
    ) that is only allowed to be registered once. It is the extension point used by the security plugin to filter requests and apply authcz on the rest request. The registered wrapper is then applied to the restHandler here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/rest/RestController.java#L207

As far as I can see, there is nothing preventing us from applying multiple wrappers. @peternied @DarshitChanpura What do you think about creating a wrapper in the sandbox/lib/opensearch-authn and applying the wrapper first. i.e.

protected void registerHandler(RestRequest.Method method, String path, RestHandler handler) {
        if (handler instanceof BaseRestHandler) {
            usageService.addRestHandler((BaseRestHandler) handler);
        }
        UnaryOperator<RestHandler> authHandlerWrapper = ...; // import the handler from opensearch-authn
        RestHandler wrappedHandler = authHandlerWrapper.apply(handler);
        registerHandlerNoWrap(method, path, handlerWrapper.apply(wrappedHandler));
    }

and we can handle the rest filtering there as its done in the plug-in model? I'm not positive if the location of the authenticate call in this PR is the appropriate place to put the call and by creating a wrapper we know that it will behave similarly to how its working now.

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@DarshitChanpura
Copy link
Member Author

Can you please answer @peternied's question on why the change from api to implementation for the shiro dependency is needed.

I did. Check this out: #4798 (comment)

  • How to reject a request without bringing down the cluster

I will look into this as a follow-up to this PR.

  • Add same scenario tests for username/password combos as security plugin to ensure it works with the same set of rules

Done. I have modified handleBasicAuth method to follow same rules as security plugin

@peternied @DarshitChanpura What do you think about creating a wrapper in the sandbox/lib/opensearch-authn and applying the wrapper first... we can handle the rest filtering there as its done in the plug-in model

I like this idea. I have an abandoned branch where I experimented with a "filter" for rest requests. It needs some modifications but is ready to go. Check this out: https://github.com/DarshitChanpura/OpenSearch/blob/basic-auth-with-security-rest-filter/server/src/main/java/org/opensearch/rest/SecurityRestFilter.java

I'm not positive if the location of the authenticate call in this PR is the appropriate place to put the call and by creating a wrapper we know that it will behave similarly to how its working now.

I'm positive that this is in the correct place, as dispatchRequest checks for content validity and circuitBreaker configuration amongst other things before proceeding to hand it over to the request handler. And since authentication should happen before this and after request validation (which happens in tryAllHandlers method), I believe that this check is in correct place.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Member

@cwperks cwperks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @DarshitChanpura! This is a good step forward for security in core.

…ests with no headers to pass-through for time being

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
… and adds a sample user in example users yml

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@stephen-crawford
Copy link
Contributor

This seems like it is in a good spot. Once this is merged, I will take the bearer auth stuff, and close the draft I have open to make a new PR to introduce bearer auth on top of this. That will give us bearer & basic auth and then we can go from there.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

Gradle Check (Jenkins) Run Completed with:

@peternied
Copy link
Member

DC I know there are failures, I'm going to merge this change as is and you can keep working on them separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants