bulk audit event export api #46399

fspmarshall · 2024-09-09T19:05:34Z

This PR implements a new api for exporting very large numbers of audit events in bulk very quickly. This change is intended to support ongoing work related to #46193. The problem, in short, is that some teleport clusters have so much activity that our existing APIs for exporting events aren't performant enough. In particular, reading audit events via Athena is very slow. The new API implemented here bypasses Athena in favor of reading audit events directly from the parquet files in s3. Bypassing Athena loses us the ability to exactly time-order events. The export api is intended to prefer yielding events in time order, but makes no guarantees and may yield events out of order, especially in clusters that are under high load or when exporting historical data.

Use of the export api is a two-step process. First, chunks of events for a given date are listed, then calls are made to stream the contents of each chunk, with per-chunk streaming being resumable via a cursor. The api was designed this way for a few reasons: it maps well to the athena backend, it makes no assumptions about ordering abilities of the backend more granular than date, it makes it easy to poll past dates for changes in order to catch "straggler" events, and it provides an easy and powerful means of resumable parallelization since each chunk maintains an independent cursor state.

This PR also adds a hidden test utility to the loadtest subcommand for performing bulk export of events starting from a target date.

Performance

The per-chunk streaming API has a real-world throughput of over 200k events per minute, compared to a best-case 20k observed when querying through Athena (typically closer to 10-15k). When listing chunks and streaming them synchronously, actual throughput drops to about 150k per minute if chunks are small and numerous. The exact limits of export via parallel streaming of chunks haven't been determined.

api/proto/teleport/auditlog/v1/auditlog.proto

lib/events/api.go

lib/events/athena/consumer.go

lib/events/test/suite.go

rosstimothy · 2024-09-12T13:57:04Z

tool/tctl/common/loadtest_command.go

+					Cursor: cursor,
+				})
+
+			Events:


Woah. Four labeled for loops 🤯.

timothyb89

Seems pretty reasonable, just a few nits

api/proto/teleport/auditlog/v1/auditlog.proto

lib/events/multilog.go

lib/events/test/suite.go

public-teleport-github-review-bot · 2024-09-19T19:51:57Z

@fspmarshall See the table below for backport results.

Branch	Result
branch/v14	Failed
branch/v15	Failed
branch/v16	Failed

* Displaying mode and controls to additional participants * Moving SessionControlsInfoBroadcast over to kube/proxy * Transitioning to consistent proxy-emitted mode+controls * Moving message broadcast so new participant wont see it * Possible unit test fix (cant seem to test locally) * Fixed unit test * Adding a line break before messaging the participant * Linter errors * Emitting audit event and controls message for additional parties, i.e. not the session initiator * Revert "Emitting audit event and controls message for additional parties, i.e. not the session initiator" This reverts commit b66ad27. * Add User Tasks resource - protos (#46059) * Add User Integration Tasks resource - protos * add account id * move state to task instead of instance * rename from user integration task to user task * add instance id * Add notice to web UI that users arent equal to MAU (#46686) This adds a dismissible notice to the Users page for usage based billing users that notifies them that the user count here isn't an accurate reflection of MAU * Clarify TLS requirements in the Jira guide (#46484) Closes #45654 - Indicate that certificates for the Jira web server cannot be self signed. - Remove references to Caddy and a `Certificate` resource, which were left over from an attempted change to this guide that was not fully completed. * Remove TXT record validation of custom DNS zones in VNet (#46709) * Remove TXT record validation from custom DNS zones * Remove mentions of TXT records from docs * Outline in the RFD why domain verification was dropped * Update rfd/0163-vnet.md Co-authored-by: Nic Klaassen <nic@goteleport.com> --------- Co-authored-by: Nic Klaassen <nic@goteleport.com> * docs: mention the --days flag when executing an audit log query (#45764) * Update access-monitoring.mdx Include the default date range in the CLI example. This range is otherwise unclear and is hidden in the tctl audit help menu. * Update access-monitoring.mdx * Update docs/pages/admin-guides/access-controls/access-monitoring.mdx Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com> --------- Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com> * Remove access-graph path resolution, proxy `/enterprise` requests (#46541) * Remove `access-graph` path from tsconfig * Proxy /enterprise requested in dev * update e ref (#46726) * fix: tolerate mismatched key PEM headers (#46725) * fix: tolerate mismatched key PEM headers Issue #43381 introduced a regression where we now fail to parse PKCS8 encoded RSA private keys within an "RSA PRIVATE KEY" PEM block in some cases. This format is somewhat non-standard, usually PKCS8 data should be in a "PRIVATE KEY" PEM block. However, certain versions of OpenSSL and possibly even Teleport in specific cases have generated private keys in this format. This commit updates ParsePrivateKey and ParsePublicKey to be more tolerant of PKCS8, PKCS1, or PKIX key data no matter which PEM header is used. Fixes #46710 changelog: fixed regression in private key parser to handle mismatched PEM headers * fix typo in comment Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> --------- Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Use dynamic base path for favicon images (#46719) * Add Datadog Incident Management plugin support (#46271) * Implement datadog plugin * Add unit tests * Add fallback recipient config * Rename to Datadog Incident Management * Update tests * Datadog Incident Management * Update tctl resource plugin command * Typos * Lint * Exclude api changes for now * Set channel size * Address feedback - Add PluginShutdownTimeout const - Support api endpoint configuration - Add additional godocs/comments * Comment about datadog client package * Document Datadog API types * Only post resolution message when the AR is resolved * Fix lint * Unused function --------- Co-authored-by: hugoShaka <hugo.hervieux@goteleport.com> * Add AutoUpdate Client/Cache implementation (#46661) * Add AutoUpdate Client/Cache implementation * CR changes * Add permission for proxy to access resources * Rename all occurrences auto update to camelcase * Remove auto update client wrapper * Drop AutoUpdateServiceClient helper Rename comments for consistency * User Tasks: services and clients implementation (#46131) This PR adds the implementation for the User Tasks: - services (backend+cache) - clients (API + tctl) - light validation to set up the path for later PRs * expanding testplan for host user creation (#46729) * Fix operator docs reference generator bug (#46732) In the reference page for one Kubernetes operator resource, some Markdown links are malformed. The issue is that some fields of custom resource definitions used by the operator consist of arrays of anonymous objects with fields that are also objects. When creating docs based on these fields, the operator resource docs generator creates a malformed link reference. This change modifies the generator to replace any spaces with hyphens before outputting link references, causing the resulting internal links to work correctly. This change also does some light refactoring to remove an unnecessary `switch` statement. * [auto] Update AMI IDs for 16.4.0 (#46746) Co-authored-by: GitHub <noreply@github.com> * Remove deprecated HTTP RemoteCluster endpoints (#46756) * Remove deprecated HTTP RemoteCluster endpoints * Remove redundant test * Add `tbot` helm chart to `version.mk` (#46763) * Remove LockConfiguration.LockName (#46772) Cleans up the deprecated config option now that gravitational/teleport.e#5034 has been merged. * adding a reference to to the host user guide (#46765) * Replace more Logrus usage with Slog (#46757) * Remove logrus from lib/auth/machineid * Switch authclient.Config.Log and TunnelAuthDialerConfig.Log to Slog * Add *slog.Logger to auth.Server * Remove logrus usage in `lib/auth/access.go` * Replace logrus with slog in lib/auth/accountrecovery.go * Replace logrus with slog in `lib/auth/apiserver.go` * Add missing logger to auth.Server * Fix test * Update AWS roles ARNs displayed on `tsh app login` for AWS console apps (#44983) * feat(tsh): list aws console logins from server * chore(services): remove unified resources change This is being covered on another PR. * test(tsh): solve TestAzure flakiness by waiting using app servers are ready * fix(tsh): apps with logins were fallingback into using aws arns * refactor(client): use GetEnrichedResources * chore(client): rename function * refactor(tsh): directly resource lisiting for apps and reuse cluster client * chore(client): reset client changes * refactor(tsh): reuse cluster client for fetching allowed logins * chore(tsh): remove unused function param * refactor(tsh): update getApp retry with login * refactor(tsh): use a single function to grab profile and cluste client * refactor(tsh): perform retry with login at caller site * fix(tsh): close auth client * test(tsh): fix test failing due to login misconfiguration * test(tsh): fix lint errors * test(tsh): remove unused imports * bulk audit event export api (#46399) * Reverting back to using the emitSessionJoin boolean * Nits and removing a debug log --------- Co-authored-by: Marco Dinis <marco.dinis@goteleport.com> Co-authored-by: Michael <michael.myers@goteleport.com> Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com> Co-authored-by: Nic Klaassen <nic@goteleport.com> Co-authored-by: Dan Johns <117299936+djohns7@users.noreply.github.com> Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com> Co-authored-by: Ryan Clark <ryan.clark@goteleport.com> Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> Co-authored-by: Bernard Kim <bernard@goteleport.com> Co-authored-by: hugoShaka <hugo.hervieux@goteleport.com> Co-authored-by: Vadym Popov <vadym.popov@goteleport.com> Co-authored-by: Erik Tate <erik.tate@goteleport.com> Co-authored-by: teleport-post-release-automation[bot] <128860004+teleport-post-release-automation[bot]@users.noreply.github.com> Co-authored-by: GitHub <noreply@github.com> Co-authored-by: Noah Stride <noah.stride@goteleport.com> Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Gabriel Corado <gabriel.oliveira@goteleport.com> Co-authored-by: Forrest <30576607+fspmarshall@users.noreply.github.com>

) * Displaying mode and controls to additional participants * Moving SessionControlsInfoBroadcast over to kube/proxy * Transitioning to consistent proxy-emitted mode+controls * Moving message broadcast so new participant wont see it * Possible unit test fix (cant seem to test locally) * Fixed unit test * Adding a line break before messaging the participant * Linter errors * Emitting audit event and controls message for additional parties, i.e. not the session initiator * Revert "Emitting audit event and controls message for additional parties, i.e. not the session initiator" This reverts commit b66ad27. * Add User Tasks resource - protos (#46059) * Add User Integration Tasks resource - protos * add account id * move state to task instead of instance * rename from user integration task to user task * add instance id * Add notice to web UI that users arent equal to MAU (#46686) This adds a dismissible notice to the Users page for usage based billing users that notifies them that the user count here isn't an accurate reflection of MAU * Clarify TLS requirements in the Jira guide (#46484) Closes #45654 - Indicate that certificates for the Jira web server cannot be self signed. - Remove references to Caddy and a `Certificate` resource, which were left over from an attempted change to this guide that was not fully completed. * Remove TXT record validation of custom DNS zones in VNet (#46709) * Remove TXT record validation from custom DNS zones * Remove mentions of TXT records from docs * Outline in the RFD why domain verification was dropped * Update rfd/0163-vnet.md --------- * docs: mention the --days flag when executing an audit log query (#45764) * Update access-monitoring.mdx Include the default date range in the CLI example. This range is otherwise unclear and is hidden in the tctl audit help menu. * Update access-monitoring.mdx * Update docs/pages/admin-guides/access-controls/access-monitoring.mdx --------- * Remove access-graph path resolution, proxy `/enterprise` requests (#46541) * Remove `access-graph` path from tsconfig * Proxy /enterprise requested in dev * update e ref (#46726) * fix: tolerate mismatched key PEM headers (#46725) * fix: tolerate mismatched key PEM headers Issue #43381 introduced a regression where we now fail to parse PKCS8 encoded RSA private keys within an "RSA PRIVATE KEY" PEM block in some cases. This format is somewhat non-standard, usually PKCS8 data should be in a "PRIVATE KEY" PEM block. However, certain versions of OpenSSL and possibly even Teleport in specific cases have generated private keys in this format. This commit updates ParsePrivateKey and ParsePublicKey to be more tolerant of PKCS8, PKCS1, or PKIX key data no matter which PEM header is used. Fixes #46710 changelog: fixed regression in private key parser to handle mismatched PEM headers * fix typo in comment --------- * Use dynamic base path for favicon images (#46719) * Add Datadog Incident Management plugin support (#46271) * Implement datadog plugin * Add unit tests * Add fallback recipient config * Rename to Datadog Incident Management * Update tests * Datadog Incident Management * Update tctl resource plugin command * Typos * Lint * Exclude api changes for now * Set channel size * Address feedback - Add PluginShutdownTimeout const - Support api endpoint configuration - Add additional godocs/comments * Comment about datadog client package * Document Datadog API types * Only post resolution message when the AR is resolved * Fix lint * Unused function --------- * Add AutoUpdate Client/Cache implementation (#46661) * Add AutoUpdate Client/Cache implementation * CR changes * Add permission for proxy to access resources * Rename all occurrences auto update to camelcase * Remove auto update client wrapper * Drop AutoUpdateServiceClient helper Rename comments for consistency * User Tasks: services and clients implementation (#46131) This PR adds the implementation for the User Tasks: - services (backend+cache) - clients (API + tctl) - light validation to set up the path for later PRs * expanding testplan for host user creation (#46729) * Fix operator docs reference generator bug (#46732) In the reference page for one Kubernetes operator resource, some Markdown links are malformed. The issue is that some fields of custom resource definitions used by the operator consist of arrays of anonymous objects with fields that are also objects. When creating docs based on these fields, the operator resource docs generator creates a malformed link reference. This change modifies the generator to replace any spaces with hyphens before outputting link references, causing the resulting internal links to work correctly. This change also does some light refactoring to remove an unnecessary `switch` statement. * [auto] Update AMI IDs for 16.4.0 (#46746) * Remove deprecated HTTP RemoteCluster endpoints (#46756) * Remove deprecated HTTP RemoteCluster endpoints * Remove redundant test * Add `tbot` helm chart to `version.mk` (#46763) * Remove LockConfiguration.LockName (#46772) Cleans up the deprecated config option now that gravitational/teleport.e#5034 has been merged. * adding a reference to to the host user guide (#46765) * Replace more Logrus usage with Slog (#46757) * Remove logrus from lib/auth/machineid * Switch authclient.Config.Log and TunnelAuthDialerConfig.Log to Slog * Add *slog.Logger to auth.Server * Remove logrus usage in `lib/auth/access.go` * Replace logrus with slog in lib/auth/accountrecovery.go * Replace logrus with slog in `lib/auth/apiserver.go` * Add missing logger to auth.Server * Fix test * Update AWS roles ARNs displayed on `tsh app login` for AWS console apps (#44983) * feat(tsh): list aws console logins from server * chore(services): remove unified resources change This is being covered on another PR. * test(tsh): solve TestAzure flakiness by waiting using app servers are ready * fix(tsh): apps with logins were fallingback into using aws arns * refactor(client): use GetEnrichedResources * chore(client): rename function * refactor(tsh): directly resource lisiting for apps and reuse cluster client * chore(client): reset client changes * refactor(tsh): reuse cluster client for fetching allowed logins * chore(tsh): remove unused function param * refactor(tsh): update getApp retry with login * refactor(tsh): use a single function to grab profile and cluste client * refactor(tsh): perform retry with login at caller site * fix(tsh): close auth client * test(tsh): fix test failing due to login misconfiguration * test(tsh): fix lint errors * test(tsh): remove unused imports * bulk audit event export api (#46399) * Reverting back to using the emitSessionJoin boolean * Nits and removing a debug log --------- Co-authored-by: Marco Dinis <marco.dinis@goteleport.com> Co-authored-by: Michael <michael.myers@goteleport.com> Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com> Co-authored-by: Nic Klaassen <nic@goteleport.com> Co-authored-by: Dan Johns <117299936+djohns7@users.noreply.github.com> Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com> Co-authored-by: Ryan Clark <ryan.clark@goteleport.com> Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> Co-authored-by: Bernard Kim <bernard@goteleport.com> Co-authored-by: hugoShaka <hugo.hervieux@goteleport.com> Co-authored-by: Vadym Popov <vadym.popov@goteleport.com> Co-authored-by: Erik Tate <erik.tate@goteleport.com> Co-authored-by: teleport-post-release-automation[bot] <128860004+teleport-post-release-automation[bot]@users.noreply.github.com> Co-authored-by: GitHub <noreply@github.com> Co-authored-by: Noah Stride <noah.stride@goteleport.com> Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Gabriel Corado <gabriel.oliveira@goteleport.com> Co-authored-by: Forrest <30576607+fspmarshall@users.noreply.github.com>

fspmarshall added backport/branch/v14 no-changelog Indicates that a PR does not require a changelog entry backport/branch/v15 backport/branch/v16 labels Sep 9, 2024

github-actions bot requested review from fheinecke and timothyb89 September 9, 2024 19:06

github-actions bot added audit-log Issues related to Teleports Audit Log size/lg tctl tctl - Teleport admin tool labels Sep 9, 2024

rosstimothy reviewed Sep 12, 2024

View reviewed changes

fspmarshall mentioned this pull request Sep 16, 2024

add event exporter helper #46651

Merged

fspmarshall force-pushed the fspmarshall/bulk-event-export-api branch from 9d34d52 to 330fdb5 Compare September 17, 2024 15:25

timothyb89 approved these changes Sep 17, 2024

View reviewed changes

api/proto/teleport/auditlog/v1/auditlog.proto Show resolved Hide resolved

lib/events/multilog.go Outdated Show resolved Hide resolved

lib/events/multilog.go Outdated Show resolved Hide resolved

rosstimothy approved these changes Sep 18, 2024

View reviewed changes

lib/events/test/suite.go Outdated Show resolved Hide resolved

lib/events/test/suite.go Outdated Show resolved Hide resolved

public-teleport-github-review-bot bot removed the request for review from fheinecke September 18, 2024 14:32

fspmarshall force-pushed the fspmarshall/bulk-event-export-api branch from 330fdb5 to 36cf634 Compare September 18, 2024 17:07

fspmarshall enabled auto-merge September 18, 2024 17:08

fspmarshall force-pushed the fspmarshall/bulk-event-export-api branch from 36cf634 to 6449ac8 Compare September 18, 2024 17:25

bulk audit event export api

ada64d4

fspmarshall force-pushed the fspmarshall/bulk-event-export-api branch from 6449ac8 to ada64d4 Compare September 19, 2024 19:15

fspmarshall added this pull request to the merge queue Sep 19, 2024

Merged via the queue into master with commit a99c5ed Sep 19, 2024
42 checks passed

fspmarshall deleted the fspmarshall/bulk-event-export-api branch September 19, 2024 19:49

fspmarshall added a commit that referenced this pull request Sep 19, 2024

bulk audit event export api (#46399)

11e165a

fspmarshall added a commit that referenced this pull request Sep 19, 2024

bulk audit event export api (#46399)

2aebc3a

fspmarshall added a commit that referenced this pull request Sep 19, 2024

bulk audit event export api (#46399)

3e48d74

This was referenced Sep 19, 2024

[v14] bulk audit event export api #46782

Merged

[v15] bulk audit event export api #46783

Merged

fspmarshall mentioned this pull request Sep 19, 2024

[v16] bulk audit event export api #46784

Merged

mvbrock pushed a commit that referenced this pull request Sep 19, 2024

bulk audit event export api (#46399)

48fae88

fspmarshall added a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399)

f1269a7

fspmarshall added a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399)

594961e

fspmarshall added a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399)

b984833

fspmarshall added a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399)

3aa047f

fspmarshall added a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399)

106d2ba

github-merge-queue bot pushed a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399) (#46784)

32d7be7

github-merge-queue bot pushed a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399) (#46783)

92d8b5d

github-merge-queue bot pushed a commit that referenced this pull request Sep 26, 2024

bulk audit event export api (#46399) (#46782)

3489e12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bulk audit event export api #46399

bulk audit event export api #46399

fspmarshall commented Sep 9, 2024

rosstimothy Sep 12, 2024

timothyb89 left a comment

public-teleport-github-review-bot bot commented Sep 19, 2024

bulk audit event export api #46399

bulk audit event export api #46399

Conversation

fspmarshall commented Sep 9, 2024

Performance

rosstimothy Sep 12, 2024

Choose a reason for hiding this comment

timothyb89 left a comment

Choose a reason for hiding this comment

public-teleport-github-review-bot bot commented Sep 19, 2024