[Task] Add support for new event type jdk.ObjectAllocationSample

[gunnarmorling]

JDK 16 adds this (see https://bugs.openjdk.java.net/browse/JDK-8257602), essentially superseding the earlier TLAB event types. Would be great to have support for it here.

[andrewazores]

@maxcao13 or @tthvo would you like to take a look at this?

[tthvo]

Great sure! I will look into it. Any guide on where to start?

[andrewazores]

You'll need a sample application running JDK16+. [0] Collect a JFR file from that application and load it into jfr-datasource, and then see if the ObjectAllocationSample event shows up in query results. You can also use JMC or the jfr command-line tool to verify the recording contains the event. If the recording contains the events but jfr-datasource doesn't return it in queries/searches then you'll need to do some work in https://github.com/cryostatio/jfr-datasource/blob/main/src/main/java/io/cryostat/jfr/datasource/events/RecordingService.java , I think.

[0] https://github.com/andrewazores/vertx-fib-demo/blob/678d12362f1c2867f526813c7f491134bb532d61/build.gradle#L17

[tthvo]

Great I will check it out and start on this.

[tthvo]

I did a check on a jfr for vertx-fib-demo:0.8.0 and found:

Jfr Summary

[...hiddden]

jdk.ObjectAllocationSample {
  startTime = 22:01:25.094
  objectClass = java.util.TreeMap (classLoader = bootstrap)
  weight = 99.5 kB
  eventThread = "RMI TCP Connection(97)-10.0.2.100" (javaThreadId = 86)
  stackTrace = [
    javax.management.openmbean.CompositeDataSupport.makeMap(String[], Object[]) line: 136
    javax.management.openmbean.CompositeDataSupport.<init>(CompositeType, String[], Object[]) line: 119
    com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$TabularMapping.toNonNullOpenValue(Object) line: 768
    com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$NonNullMXBeanMapping.toOpenValue(Object) line: 142
    com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeMapping.toNonNullOpenValue(Object) line: 841
    ...
  ]
}

[...hiddden]

Source: summary.txt

Load Jfr and query/search

$ http GET :8080/search | jq  -c '.[] | select(contains("jdk.ObjectAllocationSample"))'
"jdk.ObjectAllocationSample.eventThread"
"jdk.ObjectAllocationSample.stackTrace"
"jdk.ObjectAllocationSample.objectClass"
"jdk.ObjectAllocationSample.weight"

Looks like Datasource supports this type event. Any further work on this issue or maybe I miss anything?

[andrewazores]

Great, thanks for checking.

The next step would be to add it to the Grafana dashboard - at the time this issue was filed, the dashboard definition was in this project and only later was it moved to our customized Grafana image.

[tthvo]

Great thanks! I will look into how to do that and reference this issue :D

[tthvo]

Among the 4 fields above, I think we add "jdk.ObjectAllocationSample.weight" to Grafana Dashboard?

Just not sure how to visualize the rest :\

[andrewazores]

The thread name and stack trace definitely don't seem like they will map well to the timeseries charts on the dashboard.

Though, I don't think charting the weights alone as a single data set is useful. I think we would either want one big chart with multiple series on it - time on the X, weight on the Y, and each series labelled by its objectClass. Or else a whole separate dashboard with dynamic charts, one for each objectClass, each with weight vs time. I'm not sure if either of these is well-supported by the dashboard definition JSON.

[tthvo]

Good idea :D I will see if we could how to define the graph as such. The first one probably sounds better.

[andrewazores]

It sounds better, but it might be extremely difficult or impossible to read if there are very many different objectClass samples...

[tthvo]

Right...How about...since Grafana allows a closer/bigger view when clicking on the panel, I guess we could follow the first option and somehow add a filter (not sure if supported)?

[andrewazores]

If that can be done then sure, that sounds like it could be good.

[tthvo]

Unfortunately, per-panel-filter is not possible at the moment: grafana/grafana#8408

My idea now is:

• Create a panel in main dashboard with jdk.ObjectAllocationSample.weight using multiple series (need to modify jfr-datasource to support categorizing/fitlers).
• Create another dashboard with each panel for each class type with ad-hoc filter for entire dashboard based on object type.
• Create a dashboard link on top-left corner of the original panel (total weight) that links to this new dashboard: https://grafana.com/docs/grafana/latest/dashboards/manage-dashboard-links/

Until above grafana feature is compeleted, not sure we can go with option 1 completely. Any thoughts @andrewazores?