Intermittent SAML and 2FA Push Notification Timeouts from Okta

[sdann]

Posting to see if any other users are having similar issues. Since 9/24/20 several of our users are getting HTTP timeouts waiting for responses from Okta. The behavior is inconsistent, but it takes 3 forms:

1. Initial SAML Authn never responds. No MFA notification.

   getting creds via SAML: Failed to authenticate with okta. If your credentials have changed, use 'aws-okta add': &url.Error{Op:"Post", URL:"https://convoy.okta.com/api/v1/authn", Err:(*http.httpError)(0xc0002fc0a0)}

2. MFA prompt, but Push notification never shows up on mobile device. Eventual error:

   getting creds via SAML: Post "https://[company].okta.com/api/v1/authn/factors/[id]/verify?rememberDevice=true": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

3. MFA notification arrives on phone and I complete it but aws-okta doesn’t seem to register it and it fails with timeout.

   getting creds via SAML: Failed authn verification for okta. Err: Post "https://[company].okta.com/api/v1/authn/factors/[id]/verify": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Right now, my assumption is this is a network level issue on Okta's servers, but it's only affecting aws-okta. Okta GUI authentication works fine.

[nickatsegment]

Hm, haven't heard anything from our users at Segment.

It might be the case that the undocumented API we're using has changed in a subtle way that we're not hitting, like #293 but we're not hitting the same codepaths somehow. Your only recourse is to compare HTTP flows between the two and reverse engineer it :(

Could also be network gremlins, as you suggest

[brandt]

Several folks here have been running into this problem intermittently.

#299 does appear to fix it.

[sdann]

I can confirm that HTTP2 over TLS is present in all of our failure cases, while HTTP1 over TLS is negotiated in all of my success repos.

In version 1.0.4, the aws-okta HTTP client offers that it can speak h2 or h1 in the Client Hello message.

Even though the Get implementation explicitly requests HTTP/1.1.

aws-okta/lib/okta.go

Line 594 in 98c40a4

Proto: "HTTP/1.1",

The Okta server Server Hello message picks h2.

The fix in #299 to explicitly set h1 in the TLS configuration matches the current code intent, in addition to fixing the issue.

[Chippiewill]

I started noticing this problem yesterday and after poking some debug prints into aws-okta I noticed that it was always sending a Content-Length of 0. This turns out to be a bug in the way it's calculated. It's currently based on the length of an object that's never initialized.

https://github.com/segmentio/aws-okta/blob/master/lib/okta.go#L599

PR raised: #300

[ngodec]

Noticing this a lot today, intermittently across different AWS profiles and Okta instances.
Present in aws-okta versions: 0.26.3, 1.0.1, 1.0.4

[nickatsegment]

How consistent is this for folks? I.e., what percentage of auth attempts timeout? I haven't been able to repro personally, but have had a handful of users at Segment say they are affected.

[Chippiewill]

For me personally I've had it be super consistent for an hour (100%) and then stop entirely for hours although there's been a bit of inconsistentency on which bit of the flow times out (first request vs MFA push etc). Similar reports from many of my colleagues (but equally some colleagues entirely unaffected). On any occasion it's broken I've had saml2aws work entirely fine (which is using the same underlying API) and even manually stepped through the sign-in flow fine with curl.

[nickatsegment]

Having some trouble with our CI publishing pipeline, but the tag v1.0.5 is there. That's enough for Homebrew. If somebody else could submit a PR there, that'd be great (we don't use it).

I'm working on getting the binaries and packages published to our GH Releases.

Update: it's just the linux binary that's failed to publish. The RPMs and DEBs are live on packagecloud.

[nickatsegment]

Big shoutout to @Chippiewill and @mvallaly-rally for their PRs.

[Ngibb]

Thanks for getting this fixed so fast.

[richadams]

I raised Homebrew/homebrew-core#61790 to bump the version in Homebrew.

[sdann]

Amazing community effort to get this bug root caused and fixed quickly. @nickatsegment, even with partial deprecation, you've got a large and caring userbase.

[seanorama]

Great work! Any progress on getting the assets pushed in to the 1.0.5 github release?

[nickatsegment]

@seanorama See #301 (comment). TLDR: no.