Feature / HTTP file download (#376)

* Update existing REST mappings for conciseness * Move gateway HTTP proxy tests into package structure * Remove gw test for base gRPC protocol (depends on HTTP/2) * Update platform test and config helpers to handle multiple config files (platform + gw) * Add unit test config for the gateway to the test library * Add POST method support to the HTTP test client used for testing the gateway * Add some unit tests for gRPC-Web protocol in the gateway * Update existing REST mappings for conciseness * Add GW rest proxy test for platform-info and list-tenants * Add platform-info and list-tenants to REST API mapping * Include content type headers in REST proxy test * Add a test case for the REST API to create, search for and retrieve a FLOW * Remove un-used test code * Fix REST get-method validators for metadata API * Fix REST get-method API mapping for metadata API * Add tests for some error cases in the REST API * Make REST API proxy check for correct content type and accept headers, rather than just assuming JSON * Refactor RestApiTranslator - extract the body field using descriptors without an explicit third type argument * Remove embedded channel logic in core router for REST routes * Simplify REST API proxy builder - executor and channel params no longer needed * Do not error on missing content type in gRPC-Web proxy * Updates in RestApiTranslator * Update usage of RestApiTranslator for new constructors * Use HTTP/2 flow control in rest proxy builder * gRPC helper methods to encode / decode LPMs * Working version of REST API proxy and translator * Remove gateway dependency on gRPC implementation libraries * Use protobuf-style descriptor in rest API method class * Allow body = * in rest API field mapping * Read REST API mapping from proto files * Add download file methods into proto file for data service definition * Update shipped gateway configs to include REST mapping for the data service * Add implementation of file download methods in the data service API impl * Clean up one warning * Add some unit tests for file download * Fix a bug in the validation constants * Add validators for file download methods * Support single-segment wildcard in REST API matcher * Test fixes * Validator fix * Update content type checking to allow for downloads * Fix handling of content headers for GET requests * Include response body in REST API method class * Working HTTP download - needs cleaning up * Clean up LPM decoding * Clean up LPM decoding * Move data API test helpers to the test lib * Add a unit test for downloading a larger file * Handle misalignment between gRPC LPM and HTTP DATA frame in REST download streams * Extra assertions in REST download tests * Increase aggregation limit in HTTP test client for gateway tests * Handle trailers and EOS in rest API proxy for download streams * Do not use HTTP headers in route matching (just use method and URI) * Do not use HTTP headers in route matching (just use method and URI) * Flag to indicate which REST API methods are download endpoints * Handle content headers for download endpoints in the REST API proxy * Include content headers in unit tests for downloads * Fix check for download endpoints in RestApiMethod * Fix some warnings * Fix some warnings * Include gateway config when starting platform for end-to-end tests * Fix some test warnings
finos · Jun 13, 2023 · f3da052 · f3da052
1 parent 7053814
commit f3da052
Show file tree

Hide file tree

Showing 50 changed files with 1,987 additions and 882 deletions.
diff --git a/dev/config/trac-devlocal-gateway.yaml b/dev/config/trac-devlocal-gateway.yaml
@@ -49,7 +49,7 @@ services:
       port: 8081
 
   data:
-    protocols: [GRPC_WEB]
+    protocols: [GRPC_WEB, REST]
     target:
       scheme: http
       host: localhost

diff --git a/dist/template/etc/trac-gateway.yaml b/dist/template/etc/trac-gateway.yaml
@@ -50,7 +50,7 @@ services:
       port: 8081
 
   data:
-    protocols: [GRPC_WEB]
+    protocols: [GRPC_WEB, REST]
     target:
       scheme: http
       host: localhost

diff --git a/examples/rest_calls/readme.txt b/examples/rest_calls/readme.txt
@@ -5,12 +5,12 @@ localhost:8080. You should also have a test tenant created, called ACME_CORP.
 
 To load the sample flow into TRAC, you can use this API call:
 
-POST http://localhost:8080/tracdap-meta/api/v1/ACME_CORP/create-object
+POST http://localhost:8080/trac-meta/api/v1/ACME_CORP/create-object
 
 You should get a response with the object ID of the new object. To run the
 example search, you can use this call:
 
-POST http://localhost:8080/tracdap-meta/api/v1/ACME_CORP/search
+POST http://localhost:8080/trac-meta/api/v1/ACME_CORP/search
 
 If you create objects and versions of objects with different tag attributes
 you will be able to experiment with search parameters.

diff --git a/tracdap-api/tracdap-services/src/main/proto/tracdap/api/data.proto b/tracdap-api/tracdap-services/src/main/proto/tracdap/api/data.proto
@@ -461,6 +461,32 @@ service TracDataApi {
    */
   rpc readSmallFile (FileReadRequest) returns (FileReadResponse);
 
+  /**
+   * Download a file as a data stream
+   *
+   * This method is intended for use by browsers and other HTTP clients
+   * to download a file using an HTTP GET request.
+   */
+  rpc downloadFile (DownloadRequest) returns (stream DownloadResponse) {
+    option (google.api.http) = {
+      get: "/{tenant}/FILE/{objectId}/versions/{objectVersion}/*"
+      response_body: "content"
+    };
+  }
+
+  /**
+   * Download the latest version of a file as a data stream
+   *
+   * This method is intended for use by browsers and other HTTP clients
+   * to download a file using an HTTP GET request.
+   */
+  rpc downloadLatestFile (DownloadRequest) returns (stream DownloadResponse) {
+    option (google.api.http) = {
+      get: "/{tenant}/FILE/{objectId}/versions/latest/*"
+      response_body: "content"
+    };
+  }
+
 }
 
 
@@ -764,3 +790,22 @@ message FileReadResponse {
    */
   bytes content = 1000;
 }
+
+
+message DownloadRequest {
+
+  string tenant = 1;
+
+  metadata.ObjectType objectType = 2;
+  string objectId = 3;
+  optional int32 objectVersion = 4;
+};
+
+
+message DownloadResponse {
+
+  optional string contentType = 1;
+  optional uint64 contentLength = 2;
+
+  bytes content = 1000;
+};
diff --git a/tracdap-libs/tracdap-lib-test/build.gradle b/tracdap-libs/tracdap-lib-test/build.gradle
@@ -34,6 +34,16 @@ dependencies {
     api project(':tracdap-lib-db')
     api project(':tracdap-lib-data')
 
+    // Tests need the ability to spin up instances of the TRAC services to test against
+    api project(':tracdap-svc-meta')
+    api project(':tracdap-svc-data')
+    api project(':tracdap-svc-orch')
+    api project(':tracdap-gateway')
+
+    // Give tests access to the deploy tools to set up services for integration
+    api project(':deploy-metadb')
+    api project(':secret-tool')
+
     implementation group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: "$junit_version"
     implementation group: 'org.junit.jupiter', name: 'junit-jupiter-params', version: "$junit_version"
 
@@ -43,26 +53,19 @@ dependencies {
     implementation group: 'io.netty', name: 'netty-handler-proxy', version: "$netty_version"
     implementation group: 'io.grpc', name: 'grpc-netty', version: "$grpc_version"
 
-    // Use Flyway to deploy schemas in test cases
-    implementation group: 'org.flywaydb', name: 'flyway-core', version: "$flyway_version"
+    // Apache Arrow data framework
+    implementation group: 'org.apache.arrow', name: 'arrow-vector', version: "$arrow_version"
+    implementation group: 'org.apache.arrow', name: 'arrow-memory-netty', version: "$arrow_version"
 
     // Jackson uses runtime class resolution
     // Make sure classes needed for config parsing are always on the runtime classpath for tests
-    api group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: "$jackson_version"
-    api group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: "$jackson_databind_version"
-    api group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: "$jackson_version"
+    implementation group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: "$jackson_version"
+    implementation group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: "$jackson_databind_version"
+    implementation group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: "$jackson_version"
+    implementation group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-csv', version: "$jackson_version"
 
-    // Tests need the ability to spin up instances of the TRAC services to test against
-
-    api project(':tracdap-svc-meta')
-    api project(':tracdap-svc-data')
-    api project(':tracdap-svc-orch')
-    api project(':tracdap-gateway')
-
-    // Give tests access to the deploy tools to set up services for integration
-
-    api project(':deploy-metadb')
-    api project(':secret-tool')
+    // Use Flyway to deploy schemas in test cases
+    implementation group: 'org.flywaydb', name: 'flyway-core', version: "$flyway_version"
 }
 
 

diff --git a/tracdap-libs/tracdap-lib-test/src/main/java/org/finos/tracdap/test/config/ConfigHelpers.java b/tracdap-libs/tracdap-lib-test/src/main/java/org/finos/tracdap/test/config/ConfigHelpers.java
@@ -21,6 +21,8 @@
 import java.io.IOException;
 import java.net.URL;
 import java.nio.file.*;
+import java.util.ArrayList;
+import java.util.List;
 import java.util.Map;
 import java.util.jar.JarFile;
 
@@ -29,13 +31,15 @@ public class ConfigHelpers {
 
     private static final String BACKSLASH = "/";
 
-    public static URL prepareConfig(
-            String rootConfigFile, Path targetDir,
+    public static List<URL> prepareConfig(
+            List<String> rootConfigFiles, Path targetDir,
             Map<String, String> substitutions)
             throws Exception {
 
         // URL of config file resource in JAR or on file system
 
+        var rootConfigFile = rootConfigFiles.get(0);
+
         var rootConfigUrl = Files.exists(Paths.get(rootConfigFile))
             ? Paths.get(rootConfigFile)
                 .toAbsolutePath()
@@ -87,14 +91,20 @@ public static URL prepareConfig(
             ConfigHelpers.copyConfigDir(sourceDir, targetConfigDir);
         }
 
-        // Apply config substitutions
+        var targetRootUrls = new ArrayList<URL>(rootConfigFiles.size());
+
+        for (var configRootEntry : rootConfigFiles) {
 
-        var targetRootFile = targetDir.resolve(rootConfigFile);
-        ConfigHelpers.setConfigVars(targetRootFile, substitutions);
+            // Apply config substitutions
+            var targetRootFile = targetDir.resolve(configRootEntry);
+            ConfigHelpers.setConfigVars(targetRootFile, substitutions);
 
-        // URL of the root file in the target location
+            // URL of the root file in the target location
+            var targetRootUrl = targetRootFile.toUri().toURL();
+            targetRootUrls.add(targetRootUrl);
+        }
 
-        return targetRootFile.toUri().toURL();
+        return targetRootUrls;
     }
 
     public static void copyConfigFromJar(String jarPath, String rootConfigPath, Path targetDir) throws IOException {
@@ -108,9 +118,10 @@ public static void copyConfigFromJar(String jarPath, String rootConfigPath, Path
             if (!name.startsWith(rootConfigPath))
                 continue;
 
-            if (entry.isDirectory())
-                Files.createDirectory(targetDir.resolve(name));
-
+            if (entry.isDirectory()) {
+                if (!Files.exists(targetDir.resolve(name)))
+                    Files.createDirectory(targetDir.resolve(name));
+            }
             else {
                 try (var stream = jar.getInputStream(entry)) {
 

diff --git a/...cdap/svc/data/api/DataApiTestHelpers.java → ...tracdap/test/data/DataApiTestHelpers.java b/...cdap/svc/data/api/DataApiTestHelpers.java → ...tracdap/test/data/DataApiTestHelpers.java
@@ -1,5 +1,5 @@
 /*
- * Copyright 2022 Accenture Global Solutions Limited
+ * Copyright 2023 Accenture Global Solutions Limited
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-package org.finos.tracdap.svc.data.api;
+package org.finos.tracdap.test.data;
 
 import org.finos.tracdap.api.FileReadRequest;
 import org.finos.tracdap.common.data.IExecutionContext;
@@ -57,9 +57,9 @@
 import java.util.stream.Stream;
 
 
-class DataApiTestHelpers {
+public class DataApiTestHelpers {
 
-    static <TReq, TResp>
+    public static <TReq, TResp>
     void serverStreaming(
             BiConsumer<TReq, StreamObserver<TResp>> grpcMethod,
             TReq request, Flow.Subscriber<TResp> response) {
@@ -68,7 +68,7 @@ void serverStreaming(
         grpcMethod.accept(request, responseGrpc);
     }
 
-    static<TReq, TResp>
+    public static<TReq, TResp>
     CompletionStage<List<TResp>> serverStreaming(
             BiConsumer<TReq, StreamObserver<TResp>> grpcMethod,
             TReq request, IExecutionContext execCtx){
@@ -86,7 +86,7 @@ CompletionStage<List<TResp>> serverStreaming(
         return collectList;
     }
 
-    static <TReq, TResp>
+    public static <TReq, TResp>
     CompletionStage<Void> serverStreamingDiscard(
             BiConsumer<TReq, StreamObserver<TResp>> grpcMethod,
             TReq request, IExecutionContext execCtx) {
@@ -103,7 +103,7 @@ CompletionStage<Void> serverStreamingDiscard(
         return discard;
     }
 
-    static <TReq, TResp>
+    public static <TReq, TResp>
     CompletableFuture<TResp> clientStreaming(
             Function<StreamObserver<TResp>, StreamObserver<TReq>> grpcMethod,
             Flow.Publisher<TReq> requestPublisher) {
@@ -119,15 +119,15 @@ CompletableFuture<TResp> clientStreaming(
         return response;
     }
 
-    static <TReq, TResp>
+    public static <TReq, TResp>
     CompletableFuture<TResp> clientStreaming(
             Function<StreamObserver<TResp>, StreamObserver<TReq>> grpcMethod,
             TReq request) {
 
         return clientStreaming(grpcMethod, Flows.publish(Stream.of(request)));
     }
 
-    static FileReadRequest readRequest(String tenant, TagHeader fileId) {
+    public static FileReadRequest readRequest(String tenant, TagHeader fileId) {
 
         var fileSelector = MetadataUtil.selectorFor(fileId);