Skip to content

Commit

Permalink
fix(archives): add target-specific archive API endpoints (#1047)
Browse files Browse the repository at this point in the history 
* Add target-specific RecordingDeleteHandler

* Add target-specific RecordingGetHandler

* Add target-specific ReportGetHandler

* fixup! Add target-specific ReportGetHandler

* fixup! Add target-specific ReportGetHandler

* Add target-specific RecordingUploadPostHandler

* Start updating unit tests

* Apply spotless

* Continue updating unit tests

* fixup! Add target-specific RecordingDeleteHandler

* fixup! fixup! Add target-specific RecordingDeleteHandler

* Update notifications with correct archived recording and report download URLs

* Update tests

* Apply spotless

* Minor fixes

* fixup! Minor fixes

* Fix unit tests

* Add beta JWT and non-JWT version of the RecordingGetHandler and ReportGetHandler

* Apply spotless

* fixup! Add beta JWT and non-JWT version of the RecordingGetHandler and ReportGetHandler

* Create temporary subdirectory when retrieving cached report path

* fixup! Create temporary subdirectory when retrieving cached report path

* Refactoring

* refactor(recording-archive): clean up overloadded deleteRecording method

* fix(archives): fix tmp dirs not checked for existence

* tests(archived-reports): update report-cache tests

* tests(recording-archive): fix tests for RecordingArchiveHelper

* fixed concurrency bug where viewing report would timeout

* tests(archives): add tests for case that recording-path resolution fails

* tests(archives): use valid jmx uri for tests

* deprecated handlers and disabled unit tests

* fix integration tests (disabled RecordingMetadata test since PR971 revamps it anyways)

* added tests for new handlers, deprecated old v2 jwt handlers and its tests

* cleaned up after rebase

* spotless:applied

* extracted jvmId as a module, added wip sourceTarget validation

* testing commit

* fixed all? hanging issues, commiting to refactor

* refactored and fixed graphql queries which failed exceptionally because of getJvmId on non-reachable targets -> instead just return empty lists, changed ArchivedRecordingInfo to have non encoded serviceuri field now that the subdirectories are not no longer encoded targetIds, removed debugging prints

* testing commit

* commit before removing prints, reverting executeConnectedTask useCache, fixing tests

* fixed unit tests, still have to fix deprecation and integration tests

* before rebase

* rebased ontop of content negotiation -> fixed deprecation, fixed its and uts, fixed bugs

* updated docs

* fixed spotbugs

* fixup! removed print

* fixed as per comments

* fixed failing tests

* spotless:applied

* fixed archives migration

* rebase cleanup

* some fixes

* cleaned up exception printing and some comments

* spotless:applied

* fixed null sourceTarget propagating into getJvmId from getReportCachedPath

* rebase cleanup #5

* revert 'async' vertx.blocking for report generation

Co-authored-by: Thuan Vo <thvo@redhat.com>
Co-authored-by: Max Cao <macao@redhat.com>
  • Loading branch information
@maxcao13
3 people authored Sep 27, 2022
1 parent 1c0fe9f commit 525b62d
Show file tree
Hide file tree
Showing 62 changed files with 3,754 additions and 573 deletions.
193 changes: 192 additions & 1 deletion docs/HTTP_API.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@
**DEPRECATED**: Endpoints treating the archived recording storage as
uncategorized storage, where files are not associated with a particular
target application, are deprecated and will be removed in a future release.
See [`RecordingDeleteHandler`](#RecordingDeleteHandler-1).
###### request
`DELETE /api/v1/recordings/:recordingName`
Expand Down Expand Up @@ -313,6 +314,7 @@
**DEPRECATED**: Endpoints treating the archived recording storage as
uncategorized storage, where files are not associated with a particular
target application, are deprecated and will be removed in a future release.
See [`RecordingGetHandler`](#RecordingGetHandler-2).
###### request
`GET /api/v1/recordings/:recordingName`
Expand Down Expand Up @@ -446,6 +448,7 @@
**DEPRECATED**: Endpoints treating the archived recording storage as
uncategorized storage, where files are not associated with a particular
target application, are deprecated and will be removed in a future release.
See [`RecordingUploadPostHandler`](#RecordingUploadPostHandler-1).
###### request
`POST /api/v1/recordings/:recordingName/upload`
Expand Down Expand Up @@ -489,6 +492,7 @@
**DEPRECATED**: Endpoints treating the archived recording storage as
uncategorized storage, where files are not associated with a particular
target application, are deprecated and will be removed in a future release.
See [`RecordingGetHandler`](#RecordingGetHandler-2).
###### request
`GET /api/v1/reports/:recordingName`
Expand Down Expand Up @@ -1947,6 +1951,7 @@ The handler-specific descriptions below describe how each handler populates the
**DEPRECATED**: Endpoints treating the archived recording storage as
uncategorized storage, where files are not associated with a particular
target application, are deprecated and will be removed in a future release.
See [`RecordingGetWithJwtHandler`](#RecordingGetWithJwtHandler).
###### request
`GET /api/v2.1/recordings/:recordingName?token=:jwt`
Expand Down Expand Up @@ -1979,6 +1984,7 @@ The handler-specific descriptions below describe how each handler populates the
**DEPRECATED**: Endpoints treating the archived recording storage as
uncategorized storage, where files are not associated with a particular
target application, are deprecated and will be removed in a future release.
See [`ReportGetWithJwtHandler`](#ReportGetWithJwtHandler).
###### request
`GET /api/v2.1/reports/:recordingName?token=:jwt`
Expand Down Expand Up @@ -2399,7 +2405,13 @@ The handler-specific descriptions below describe how each handler populates the
| **Recordings in Target JVMs** | |
| Create metadata labels for a recording in a target JVM | [`TargetRecordingMetadataLabelsPostHandler`](#TargetRecordingMetadataLabelsPostHandler) |
| **Recordings in archive** | |
| Delete a recording from archive | [`RecordingDeleteHandler`](#RecordingDeleteHandler-1) |
| Download a recording in archive | [`RecordingGetHandler`](#RecordingGetHandler-2) |
| Download a recording in archive using JWT | [`RecordingGetWithJwtHandler`](#RecordingGetWithJwtHandler) |
| Download a report of a recording in archive | [`ReportGetHandler`](#ReportGetHandler-3) |
| Download a report of a recording in archive using JWT | [`ReportGetWithJwtHandler`](#ReportGetWithJwtHandler) |
| Create metadata labels for a recording | [`RecordingMetadataLabelsPostHandler`](#RecordingMetadataLabelsPostHandler) |
| Upload a recording from archive to the Grafana datasource | [`RecordingUploadPostHandler`](#RecordingUploadPostHandler-1) |
### Miscellaneous
* #### `JvmIdGetHandler`
Expand Down Expand Up @@ -2455,6 +2467,185 @@ The handler-specific descriptions below describe how each handler populates the
```
### Recordings in Archives
* #### `RecordingDeleteHandler`
##### synopsis
Delete a recording from archive. This does not affect any recordings in any target JVM's JFR buffer.
##### request
`DELETE /api/beta/recordings/:sourceTarget/:recordingName`
`sourceTarget` - The target JVM from which Cryostat saved the recording. Must be in the form of a service:rmi:jmx:// JMX Service URL and should use percent-encoding. If a recording was re-uploaded to archives, this field should be set to `uploads`.
`recordingName` - The name of the recording to delete. Should use percent-encoding.
##### response
`200` - The result is null. The request was processed successfully and the
recording was deleted.
`401` - User authentication failed. The reason is an error message.
There will be an `X-WWW-Authenticate: $SCHEME` header that indicates
the authentication scheme that is used.
`404` - `recordingName` could not be found for the given `sourceTarget` or `sourceTarget` is invalid. The body is an error message.
##### example
```
$ curl -X DELETE http://localhost:8181/api/beta/recordings/service%3Ajmx%3Armi%3A%2F%2F%2Fjndi%2Frmi%3A%2F%2Fcryostat%3A9091%2Fjmxrmi/localhost_foo_20200910T214559Z.jfr
{"meta":{"type":"text/plain","status":"OK"},"data":{"result":null}}
```
* #### `RecordingGetHandler`
##### synopsis
Returns a recording that was saved to archive, as an octet stream
##### request
`GET /api/beta/recordings/:sourceTarget/:recordingName`
`sourceTarget` - The target JVM from which Cryostat saved the recording. Must be in the form of a service:rmi:jmx:// JMX Service URL and should use percent-encoding. If a recording was re-uploaded to archives, this field should be set to `uploads`.
`recordingName` - The name of the recording to download. Should use percent-encoding.
##### response
`200` - The result is the recording file.
`401` - User authentication failed. The reason is an error message.
There will be an `X-WWW-Authenticate: $SCHEME` header that indicates
the authentication scheme that is used.
`404` - `recordingName` could not be found for the given `sourceTarget` or `sourceTarget` is invalid. The body is an error message.
##### example
```
$ curl http://localhost:8181/api/beta/recordings/service%3Ajmx%3Armi%3A%2F%2F%2Fjndi%2Frmi%3A%2F%2Fcryostat%3A9091%2Fjmxrmi/localhost_foo_20200910T214559Z.jfr --output foo.jfr
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 391k 100 391k 0 0 64.7M 0 --:--:-- --:--:-- --:--:-- 76.5M
```
* #### `RecordingGetWithJwtHandler`
##### synopsis
Returns a recording that was saved to archive, as an octet stream with JWT auth.
##### request
`GET /api/beta/recordings/:sourceTarget/:recordingName/jwt`
`sourceTarget` - The target JVM from which Cryostat saved the recording. Must be in the form of a service:rmi:jmx:// JMX Service URL and should use percent-encoding. If a recording was re-uploaded to archives, this field should be set to `uploads`.
`recordingName` - The name of the recording to download. Should use percent-encoding.
`jwt` - The JSON Web Token providing authorization for this request. See [`AuthTokenPostHandler`](#AuthTokenPostHandler)
##### response
`200` - The result is the recording file.
`401` - User authentication failed. The reason is an error message.
There will be an `X-WWW-Authenticate: $SCHEME` header that indicates
the authentication scheme that is used.
`404` - `recordingName` could not be found for the given `sourceTarget` or `sourceTarget` is invalid. The body is an error message.
##### example
```
$ curl http://localhost:8181/api/beta/recordings/service%3Ajmx%3Armi%3A%2F%2F%2Fjndi%2Frmi%3A%2F%2Fcryostat%3A9091%2Fjmxrmi/localhost_foo_20200910T214559Z.jfr?token=(trimmed) --output foo.jfr
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 391k 100 391k 0 0 64.7M 0 --:--:-- --:--:-- --:--:-- 76.5M
```
* #### `ReportGetHandler`
##### synopsis
Returns the report of a recording that was saved to archive.
##### request
`GET /api/beta/reports/:sourceTarget/:recordingName`
`sourceTarget` - The target JVM from which Cryostat saved the recording. Must be in the form of a service:rmi:jmx:// JMX Service URL and should use percent-encoding. If a recording was re-uploaded to archives, this field should be set to `uploads`.
`recordingName` - The name of the recording to get the report for. Should use percent-encoding.
##### response
`200` - The body is the requested report as an HTML document.
`401` - User authentication failed. The reason is an error message.
There will be an `X-WWW-Authenticate: $SCHEME` header that indicates
the authentication scheme that is used.
`404` - `recordingName` could not be found for the given `sourceTarget` or `sourceTarget` is invalid. The body is an error message.
##### example
```
$ curl localhost:8181/api/beta/reports/service%3Ajmx%3Armi%3A%2F%2F%2Fjndi%2Frmi%3A%2F%2Fcryostat%3A9091%2Fjmxrmi/localhost_foo_20200911T144545Z.jfr? --output report.html
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 116k 100 116k 0 0 134k 0 --:--:-- --:--:-- --:--:-- 134k
```
* #### `ReportGetWithJwtHandler`
##### synopsis
Returns the report of a recording that was saved to archive with JWT auth.
##### request
`GET /api/beta/reports/:sourceTarget/:recordingName/jwt`
`sourceTarget` - The target JVM from which Cryostat saved the recording. Must be in the form of a service:rmi:jmx:// JMX Service URL and should use percent-encoding. If a recording was re-uploaded to archives, this field should be set to `uploads`.
`recordingName` - The name of the recording to get the report for. Should use percent-encoding.
`jwt` - The JSON Web Token providing authorization for this request. See [`AuthTokenPostHandler`](#AuthTokenPostHandler)
##### response
`200` - The body is the requested report as an HTML document.
`401` - User authentication failed. The reason is an error message.
There will be an `X-WWW-Authenticate: $SCHEME` header that indicates
the authentication scheme that is used.
`404` - `recordingName` could not be found for the given `sourceTarget` or `sourceTarget` is invalid. The body is an error message.
##### example
```
$ curl localhost:8181/api/beta/reports/service%3Ajmx%3Armi%3A%2F%2F%2Fjndi%2Frmi%3A%2F%2Fcryostat%3A9091%2Fjmxrmi/localhost_foo_20200911T144545Z.jfr?token=(trimmed) --output report.html
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 116k 100 116k 0 0 134k 0 --:--:-- --:--:-- --:--:-- 134k
```
* #### `RecordingUploadPostHandler`
##### synopsis
Uploads a recording that was saved to archive to the Grafana datasource that Cryostat is configured with (determined by the environment variable `GRAFANA_DATASOURCE_URL`).
##### request
`POST /api/beta/recordings/:sourceTarget/:recordingName/upload`
`sourceTarget` - The target JVM from which Cryostat saved the recording. Must be in the form of a service:rmi:jmx:// JMX Service URL and should use percent-encoding. If a recording was re-uploaded to archives, this field should be set to `uploads`.
`recordingName` - The name of the saved recording to upload. Should use percent-encoding.
##### response
`200` - The body is the body of the response that Cryostat got
after sending the upload request to the Grafana datasource server.
`401` - User authentication failed. The body is an error message.
There will be an `X-WWW-Authenticate: $SCHEME` header that indicates
the authentication scheme that is used.
`404` - `recordingName` could not be found for the given `sourceTarget` or `sourceTarget` is invalid. The body is an error message.
`501` - The Grafana datasource URL is malformed.
The body is an error message.
`502` - JMX connection failed. This is generally because the target
application has SSL enabled over JMX, but Cryostat does not trust the
certificate.
`512` - Cryostat received an invalid response from the
Grafana datasource after sending the upload request.
The body is an error message.
##### example
```
$ curl -X POST localhost:8181/api/beta/recordings/service%3Ajmx%3Armi%3A%2F%2F%2Fjndi%2Frmi%3A%2F%2Fcryostat%3A9091%2Fjmxrmi/localhost_foo_20200911T144545Z.jfr/upload
Uploaded: file-uploads/555f4dab-240b-486b-b336-2d0e5f43e7cd
Loaded: file-uploads/555f4dab-240b-486b-b336-2d0e5f43e7cd
```
* #### `RecordingMetadataLabelsPostHandler`
##### synopsis
Expand All @@ -2468,7 +2659,7 @@ The handler-specific descriptions below describe how each handler populates the
`sourceTarget` - The target JVM from which Cryostat saved the recording.
in the form of a `service:rmi:jmx://` JMX Service URL, or `hostname:port`.
Should use percent-encoding. If a recording was re-uploaded to archives, this field should be
set to `unlabelled`.
set to `uploads`.
`recordingName` - The name of the recording to attach labels to.
Expand Down
31 changes: 21 additions & 10 deletions src/main/java/io/cryostat/net/reports/ArchivedRecordingReportCache.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,19 +72,26 @@ class ArchivedRecordingReportCache {
}

Future<Path> get(String recordingName, String filter) {
return this.get(null, recordingName, filter);
}

Future<Path> get(String sourceTarget, String recordingName, String filter) {
CompletableFuture<Path> f = new CompletableFuture<>();
Path dest = recordingArchiveHelper.getCachedReportPath(recordingName);
/* NOTE: This is just a temporary solution: If a request includes a filter,
* the report is never cached and just constructed on demand.
*/
if (fs.isReadable(dest) && fs.isRegularFile(dest) && filter.isBlank()) {
f.complete(dest);
return f;
}
Path dest = null;

try {
dest = recordingArchiveHelper.getCachedReportPath(sourceTarget, recordingName).get();
/* NOTE: This is just a temporary solution: If a request includes a filter,
* the report is never cached and just constructed on demand.
*/
if (fs.isReadable(dest) && fs.isRegularFile(dest) && filter.isBlank()) {
f.complete(dest);
return f;
}

logger.trace("Archived report cache miss for {}", recordingName);
Path archivedRecording = recordingArchiveHelper.getRecordingPath(recordingName).get();
Path archivedRecording =
recordingArchiveHelper.getRecordingPath(sourceTarget, recordingName).get();
Path saveFile =
reportGeneratorServiceProvider
.get()
Expand All @@ -104,6 +111,10 @@ Future<Path> get(String recordingName, String filter) {
}

boolean delete(String recordingName) {
return recordingArchiveHelper.deleteReport(recordingName);
return this.delete(null, recordingName);
}

boolean delete(String sourceTarget, String recordingName) {
return recordingArchiveHelper.deleteReport(sourceTarget, recordingName);
}
}
3 changes: 2 additions & 1 deletion src/main/java/io/cryostat/net/reports/RemoteReportGenerator.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ public CompletableFuture<Path> exec(Path recording, Path destination, String fil
HttpMimeType.OCTET_STREAM.mime());

var f = new CompletableFuture<Path>();

this.http
.postAbs(String.format("%s/report", reportGenerator))
.timeout(TimeUnit.SECONDS.toMillis(generationTimeoutSeconds))
Expand Down Expand Up @@ -122,7 +123,7 @@ public CompletableFuture<Path> exec(Path recording, Path destination, String fil
}
f.complete(destination);
logger.info(
"Report response for {} success",
"Report response for {}" + " success",
recording);
});
});
Expand Down
8 changes: 8 additions & 0 deletions src/main/java/io/cryostat/net/reports/ReportService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,18 @@ public Future<Path> get(String recordingName, String filter) {
return archivedCache.get(recordingName, filter);
}

public Future<Path> get(String sourceTarget, String recordingName, String filter) {
return archivedCache.get(sourceTarget, recordingName, filter);
}

public boolean delete(String recordingName) {
return archivedCache.delete(recordingName);
}

public boolean delete(String sourceTarget, String recordingName) {
return archivedCache.delete(sourceTarget, recordingName);
}

public Future<String> get(
ConnectionDescriptor connectionDescriptor, String recordingName, String filter) {
return activeCache.get(connectionDescriptor, recordingName, filter);
Expand Down
8 changes: 4 additions & 4 deletions src/main/java/io/cryostat/net/web/WebServer.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -322,9 +322,9 @@ URI getHostUri() throws SocketException, UnknownHostException, URISyntaxExceptio
}

// FIXME this has an implicit dependency on the RecordingGetHandler path
public String getArchivedDownloadURL(String recordingName)
public String getArchivedDownloadURL(String sourceTarget, String recordingName)
throws UnknownHostException, URISyntaxException, SocketException {
return getAssetDownloadURL(ApiVersion.V1, "recordings", recordingName);
return getAssetDownloadURL(ApiVersion.BETA, "recordings", sourceTarget, recordingName);
}

// FIXME this has a an implicit dependency on the TargetRecordingGetHandler path
Expand All @@ -335,9 +335,9 @@ public String getDownloadURL(JFRConnection connection, String recordingName)
}

// FIXME this has a an implicit dependency on the ReportGetHandler path
public String getArchivedReportURL(String recordingName)
public String getArchivedReportURL(String sourceTarget, String recordingName)
throws SocketException, UnknownHostException, URISyntaxException {
return getAssetDownloadURL(ApiVersion.V1, "reports", recordingName);
return getAssetDownloadURL(ApiVersion.BETA, "reports", sourceTarget, recordingName);
}

// FIXME this has a an implicit dependency on the TargetReportGetHandler path
Expand Down
Loading

0 comments on commit 525b62d

Please sign in to comment.