feat(discovery): Podman platform #1394

andrewazores · 2023-03-09T23:53:59Z

Welcome to Cryostat! 👋

Before contributing, make sure you have:

Read the contributing guidelines
Linked a relevant issue which this PR resolves
Linked any other relevant issues, PR's, or documentation, if any
Resolved all conflicts, if any
Rebased your branch PR on top of the latest upstream main branch
Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
Signed the last commit: git commit --amend --signoff

Related to cryostatio/cryostat#492
Related to cryostatio/cryostat#421

Description of the change:

I revisited an old, old branch (June 2021 - unfortunately the original commit dates are lost now that this has been rebased and signed) where I was experimenting with talking to the Podman HTTP API from within Cryostat. At the time the Discovery tree querying work was under way, but there was not yet any notion of a persistent database for this data, and no developed system for discovery plugins other than the runtime detection of the single type platform strategy to pick up and use. At that time the original goal of getting a Podman HTTP API discovery query working was just to enable that discovery tree structure development for a contributor who was unable to run crc or other heftier systems that can provide a more interesting and non-flat deployment structure as compared to JDP's flat list.
Today we have much more sophistication around discovery, persistence, and modularity, and so I got the Podman HTTP API communication working and wrapped it up into a built-in discovery plugin.

Motivation for the change:

This may be helpful because it is an additional example of a discovery plugin implementation, but it also removes dependency on JDP which may relieve some work from upstream JMC-core refactoring. JDP is probably only used in toy deployments of Cryostat like the upstream development smoketest.sh, so dropping it is unlikely to cause anyone any real issues so long as we have a reasonable development cycle replacement. Querying the Podman API periodically seems to do just as well for this purpose, and as an added bonus also is a little closer to other container runtimes in that we could configure multiple pods along with multiple containers within the pods and emulate something more like an OpenShift deployment (more complex topology).

In this PR the JDP platform client is still present, still named Default, and still has the lowest priority of the built-in plugins. The system will still choose the highest priority available built-in client to run, alongside Custom Targets and the plugin API, so in smoketest.sh/run.sh the Podman setup will be active rather than JDP. In integration tests the Podman socket is not active and mounted into the container, so the system continues to fall back on the old JDP discovery built-in.

How to manually test:

Run CRYOSTAT_IMAGE=quay.io... sh smoketest.sh...
List targets, view topology if you have that PR checked out, etc.

andrewazores · 2023-03-09T23:57:32Z

andrewazores · 2023-03-10T00:18:49Z

This + cryostatio/cryostat-web#891 =

Of course, the two quarkus-test applications are also within the pod, but they are self-publishing as discovery plugins rather than being configured for discovery by podman container label.

github-actions · 2023-03-10T00:39:16Z

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1394-0769036a15b56f049b987b01c835e351798c3809 sh smoketest.sh

maxcao13 · 2023-03-10T03:55:37Z

src/main/java/io/cryostat/net/HttpServer.java

@@ -82,7 +82,10 @@ public class HttpServer extends AbstractVerticle {
                                        .setPort(netConf.getInternalWebServerPort())
                                        .addWebSocketSubProtocol("*")
                                        .setCompressionSupported(true)
-                                        .setLogActivity(true)));
+                                        .setLogActivity(true)
+                                        .setTcpFastOpen(true)


Just curious what is reason for enabling these tcp features now vs. before?

maxcao13 · 2023-03-10T18:42:10Z

Running the smoketest with the podman socket open results in both the Podman Realm and the JDP realm visible. Does this make sense?

maxcao13 · 2023-03-10T18:52:14Z

Hmm... weird but it seems like jvmIds are different for two equivalent JVMs in the two realms.

andrewazores · 2023-03-10T19:20:09Z

Running the smoketest with the podman socket open results in both the Podman Realm and the JDP realm visible. Does this make sense?

Ah right, I forgot to mention. Since the built-in discovery only loads one of these plugins at runtime (plus custom targets, always), it's the Podman realm that is up to date. The JDP realm you're seeing is old data in the database from the last time you ran it and JDP was loaded.

I'm not sure what I want to do about that. Maybe the builtins that are not activated should have their realms deleted automatically?

maxcao13 · 2023-03-10T19:21:15Z

Oh, that makes a lot of sense then. Then the jvmID thing is not an issue.

Realms being deleted automatically if not loaded makes sense. We probably will never activate/deactive any built-in discovery platforms over the lifecycle after startup is complete so any of that data that isn't loaded will always be stale in a future run.

andrewazores · 2023-03-10T20:00:31Z

I'm not sure what I want to do about that. Maybe the builtins that are not activated should have their realms deleted automatically?

External plugins that go through the API have the ping-pruning mechanism that will result in their realms being deleted if the plugins becomes unreachable too, so they do potentially suffer the same "problem". And just in general, if a target disappears, that context is already lost for any archived recordings that came from it, which is why we have separate archived recording metadata that we store. So deleting the builtin plugins realms' doesn't seem too much different.

On a technical level it will take some refactoring, but I don't think it should be too hard to do.

andrewazores · 2023-03-10T21:24:16Z

Okay, last set of commits:

Allows multiple platform strategies to be selected. The highest priority available strategy continues to provide the sole AuthManager implementation. All selected strategies provide PlatformClients, which are what the internal implementation calls a built-in discovery plugin still. Custom Targets is special cased and always enabled. JDP happens to have its availability method hardcoded to returning true. The Podman client checks for the existence of the Podman API Unix socket. These may all be available and publishing to the discovery database at once, along with any external discovery plugins (like -agents). The previous variables for selecting a specific auth manager, a specific platform strategy, and for disabling built-ins (except custom targets) are all still there and work the same way.
Deregisters built-in plugins if they are unavailable on startup. For example, CRYOSTAT_PLATFORM=io.cryostat.platform.internal.PodmanPlatformStrategy sh smoketest.sh will cause the internal JDP built-in plugin to be deregistered and its discovery realm deleted from the tree. A subsequent rerun with no CRYOSTAT_PLATFORM setting will allow JDP to be selected alongside Podman again and it will re-register again as a new built-in discovery plugin.

github-actions · 2023-03-10T21:38:04Z

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1394-f22338525b586e3dbd25d88bab066bd56ee76e07 sh smoketest.sh

github-actions · 2023-03-13T15:46:07Z

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1394-5dc644aa0c47fa00f04b80b53c576093d7d93e5b sh smoketest.sh

andrewazores · 2023-03-13T20:16:03Z

Maybe CRYOSTAT_PLATFORM should support a comma-separated list instead of just a single platform now, too.

maxcao13 · 2023-03-13T20:26:32Z

Running smoketest just like this: sh smoketest.sh runs both the podman client discovery platform and the JDP discovery platform.

But running it with the env var CRYOSTAT_PLATFORM=io.cryostat.platform.internal.PodmanPlatformStrategy sh smoketest.sh removes the stale JDP built-in targets from the database and runs the podman discovery correctly on its own.

andrewazores · 2023-03-13T22:17:26Z

Running smoketest just like this: sh smoketest.sh runs both the podman client discovery platform and the JDP discovery platform.

But running it with the env var CRYOSTAT_PLATFORM=io.cryostat.platform.internal.PodmanPlatformStrategy sh smoketest.sh removes the stale JDP built-in targets from the database and runs the podman discovery correctly on its own.

Yep, that sounds like it's working as expected.

github-actions · 2023-03-13T22:32:07Z

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1394-e365bf4510bf54617aea1a722e6208b3ad75aa47 sh smoketest.sh

maxcao13

Looks good to me!

github-actions · 2023-03-14T15:08:16Z

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1394-dfae9a139d156928527eb75c6633f6490319bfef sh smoketest.sh

ebaron

Dependency addition looks fine

Signed-off-by: Andrew Azores <aazores@redhat.com>

allow multiple available strategies to be selected. the highest priority available platform provides the authmanager if none explicitly specified. set of selected strategies can be mapped to set of available platform clients, allowing multiple enabled built-in plugins at once Signed-off-by: Andrew Azores <aazores@redhat.com>

Signed-off-by: Andrew Azores <aazores@redhat.com>

github-actions · 2023-03-14T19:33:41Z

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1394-ffca3e1107cbd7a8333262cd22d7d063720d0006 sh smoketest.sh

andrewazores added the feat New feature or request label Mar 9, 2023

github-actions bot added the needs-triage Needs thorough attention from code reviewers label Mar 9, 2023

mergify bot added the safe-to-test label Mar 9, 2023

andrewazores removed the needs-triage Needs thorough attention from code reviewers label Mar 9, 2023

andrewazores force-pushed the podman-api branch from e1f7d2c to 1fe20c3 Compare March 10, 2023 00:16

andrewazores requested review from maxcao13 and tthvo March 10, 2023 00:30

andrewazores marked this pull request as ready for review March 10, 2023 00:30

andrewazores requested a review from ebaron as a code owner March 10, 2023 00:30

maxcao13 reviewed Mar 10, 2023

View reviewed changes

andrewazores force-pushed the podman-api branch 2 times, most recently from 2d3a028 to 5dc644a Compare March 13, 2023 15:30

andrewazores force-pushed the podman-api branch from 5dc644a to e365bf4 Compare March 13, 2023 22:16

maxcao13 previously approved these changes Mar 13, 2023

View reviewed changes

andrewazores dismissed maxcao13’s stale review via dfae9a1 March 14, 2023 14:51

ebaron previously approved these changes Mar 14, 2023

View reviewed changes

andrewazores added 22 commits March 14, 2023 15:01

tmp

f3b2d6c

rebase fixes

84120c5

working on experimental podman handler

46cc55c

update netty native version

d93f1cc

fixup! working on experimental podman handler

0a4aa54

add comment with vertx deps BOM note

b0fe060

add note about systemctl start podman.socket

adf5d4e

fixup! update netty native version

977e243

fixup netty version

7f78617

Signed-off-by: Andrew Azores <aazores@redhat.com>

add security opt flag for mounting podman socket

247218e

cleanup UID lookup, run on worker thread, set API version to 2.3

6ec374e

add Podman built-in discovery plugin

e66ac7b

Signed-off-by: Andrew Azores <aazores@redhat.com>

use request query param to filter for containers with specific label

20ee03e

cleanup

e89d66a

remove obsolete podman API testing endpoints

a6c2c14

improve JMX URL handling, add label to smoketest targets

3b2ebc1

perform fewer COW updates

f542ac7

Signed-off-by: Andrew Azores <aazores@redhat.com>

fix ignored podless containers

5e85751

rebase cleanup

43469d4

deregister builtin plugins if they are no longer available/selected

571b2f2

Signed-off-by: Andrew Azores <aazores@redhat.com>

CRYOSTAT_PLATFORM takes comma-separated list

ffca3e1

Signed-off-by: Andrew Azores <aazores@redhat.com>

andrewazores dismissed ebaron’s stale review via ffca3e1 March 14, 2023 19:01

andrewazores force-pushed the podman-api branch from 9ad065e to ffca3e1 Compare March 14, 2023 19:01

andrewazores merged commit d675fef into cryostatio:main Mar 14, 2023

andrewazores deleted the podman-api branch March 14, 2023 19:38

andrewazores mentioned this pull request Jun 1, 2023

feat(podman): use labels to specify JMX port and optionally host #1514

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(discovery): Podman platform #1394

feat(discovery): Podman platform #1394

andrewazores commented Mar 9, 2023 •

edited

Loading

andrewazores commented Mar 9, 2023

andrewazores commented Mar 10, 2023 •

edited

Loading

github-actions bot commented Mar 10, 2023

maxcao13 Mar 10, 2023 •

edited

Loading

maxcao13 commented Mar 10, 2023

maxcao13 commented Mar 10, 2023 •

edited

Loading

andrewazores commented Mar 10, 2023

maxcao13 commented Mar 10, 2023 •

edited

Loading

andrewazores commented Mar 10, 2023 •

edited

Loading

andrewazores commented Mar 10, 2023

github-actions bot commented Mar 10, 2023

github-actions bot commented Mar 13, 2023

andrewazores commented Mar 13, 2023

maxcao13 commented Mar 13, 2023 •

edited

Loading

andrewazores commented Mar 13, 2023

github-actions bot commented Mar 13, 2023

maxcao13 left a comment

github-actions bot commented Mar 14, 2023

ebaron left a comment

github-actions bot commented Mar 14, 2023

feat(discovery): Podman platform #1394

feat(discovery): Podman platform #1394

Conversation

andrewazores commented Mar 9, 2023 • edited Loading

Welcome to Cryostat! 👋

Before contributing, make sure you have:

Description of the change:

Motivation for the change:

How to manually test:

andrewazores commented Mar 9, 2023

andrewazores commented Mar 10, 2023 • edited Loading

github-actions bot commented Mar 10, 2023

maxcao13 Mar 10, 2023 • edited Loading

Choose a reason for hiding this comment

maxcao13 commented Mar 10, 2023

maxcao13 commented Mar 10, 2023 • edited Loading

andrewazores commented Mar 10, 2023

maxcao13 commented Mar 10, 2023 • edited Loading

andrewazores commented Mar 10, 2023 • edited Loading

andrewazores commented Mar 10, 2023

github-actions bot commented Mar 10, 2023

github-actions bot commented Mar 13, 2023

andrewazores commented Mar 13, 2023

maxcao13 commented Mar 13, 2023 • edited Loading

andrewazores commented Mar 13, 2023

github-actions bot commented Mar 13, 2023

maxcao13 left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2023

ebaron left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2023

andrewazores commented Mar 9, 2023 •

edited

Loading

andrewazores commented Mar 10, 2023 •

edited

Loading

maxcao13 Mar 10, 2023 •

edited

Loading

maxcao13 commented Mar 10, 2023 •

edited

Loading

maxcao13 commented Mar 10, 2023 •

edited

Loading

andrewazores commented Mar 10, 2023 •

edited

Loading

maxcao13 commented Mar 13, 2023 •

edited

Loading