Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Playback and Record for Azure Storage Blob #4971

Merged
merged 42 commits into from
Aug 23, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
4a16876
Initial recording for Spock unit tests
alzimmermsft Aug 6, 2019
3688f6c
Updated client construction to use playback client
alzimmermsft Aug 6, 2019
fe71531
Ignore sleeps in playback
alzimmermsft Aug 6, 2019
4fadfdc
Tests cleanup after each run
alzimmermsft Aug 7, 2019
12908e6
Merge branch 'master' into AzStorage_Blob_PlaybackRecords
alzimmermsft Aug 7, 2019
b8816fa
Each test keeps track of its increment count instead of the global
alzimmermsft Aug 7, 2019
62e7831
Initial testing records (WIP)
alzimmermsft Aug 7, 2019
c5c83f5
Partial investigation into record and playback issues
alzimmermsft Aug 12, 2019
749efbd
Fixing more playback and record edge cases, added new pipeline except…
alzimmermsft Aug 13, 2019
3f2d4e1
More cleaning up of edge cases
alzimmermsft Aug 13, 2019
b533ce2
More edge cases fixed
alzimmermsft Aug 13, 2019
3896b73
Octet-stream response types work properly
alzimmermsft Aug 14, 2019
f7d466d
Merged TestCommon into APISpec, removed JUnit dependency in blobs tes…
alzimmermsft Aug 15, 2019
ef18122
Added redaction for sensitive information in UserDelegationKey responses
alzimmermsft Aug 16, 2019
04c7d8c
Merge branch 'master' into AzStorage_Blob_PlaybackRecords
alzimmermsft Aug 16, 2019
1908db5
Added @Requires annotations to live tests only
alzimmermsft Aug 16, 2019
8f66bad
Added playback records and added test ignore for live tests only
alzimmermsft Aug 16, 2019
6b54be5
Merged in master and updated tests
alzimmermsft Aug 16, 2019
b1b2e85
Set dummy values for playback in pipeline
alzimmermsft Aug 16, 2019
850589b
Merge branch 'master' into AzStorage_Blob_PlaybackRecords
alzimmermsft Aug 19, 2019
950b062
Updating failing test records
alzimmermsft Aug 19, 2019
862f2fb
Fixed playback issue with AAD
alzimmermsft Aug 19, 2019
008eef7
Cleaning up failing File tests
alzimmermsft Aug 19, 2019
bb6579e
Updating playback records
alzimmermsft Aug 19, 2019
7625493
Updated tests.yml for storage
alzimmermsft Aug 19, 2019
5fbb5c4
Fixed tests and playback, added more testing configurations
alzimmermsft Aug 19, 2019
7bf3802
Merging in master
alzimmermsft Aug 22, 2019
e410753
Removing files that were deleted before merge
alzimmermsft Aug 22, 2019
da2b952
Fixed checkstyle rule issues
alzimmermsft Aug 22, 2019
7f43a00
Merge branch 'master' into AzStorage_Blob_PlaybackRecords
alzimmermsft Aug 22, 2019
c6410c1
Updating playback records
alzimmermsft Aug 22, 2019
77d1bc2
Responses to comments
alzimmermsft Aug 22, 2019
a77c07f
Fixed spacing issue
alzimmermsft Aug 22, 2019
d85e310
Fix unit test issues
alzimmermsft Aug 22, 2019
2dda271
Variable name change to try to fix test
alzimmermsft Aug 23, 2019
f50ed33
Added SLF4J testing dependency and ignoring tests that fail live run
alzimmermsft Aug 23, 2019
c6dee32
Merge branch 'master' into AzStorage_Blob_PlaybackRecords
alzimmermsft Aug 23, 2019
9705b68
Ignore more failing tests
alzimmermsft Aug 23, 2019
8b90d62
Minor change to attempt to fix CI being stuck
alzimmermsft Aug 23, 2019
54dd638
Merged in master and updated playback records
alzimmermsft Aug 23, 2019
43219c4
Fixed not enough bytes issue
alzimmermsft Aug 23, 2019
be14558
Fixed test issue
alzimmermsft Aug 23, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,14 @@
import com.azure.core.http.HttpRequest;
import com.azure.core.http.HttpResponse;
import com.azure.core.http.ProxyOptions;
import com.azure.core.implementation.http.UrlBuilder;
import com.azure.core.test.models.NetworkCallRecord;
import com.azure.core.test.models.RecordedData;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.azure.core.util.logging.ClientLogger;
import reactor.core.Exceptions;
import reactor.core.publisher.Mono;

import java.net.URI;
import java.io.ByteArrayOutputStream;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
Expand All @@ -25,14 +26,14 @@
* HTTP client that plays back {@link NetworkCallRecord NetworkCallRecords}.
*/
public final class PlaybackClient implements HttpClient {
private final Logger logger = LoggerFactory.getLogger(PlaybackClient.class);
private final ClientLogger logger = new ClientLogger(PlaybackClient.class);
private final AtomicInteger count = new AtomicInteger(0);
private final Map<String, String> textReplacementRules;
private final RecordedData recordedData;

/**
* Creates a PlaybackClient that replays network calls from {@code recordedData} and replaces
* {@link NetworkCallRecord#response() response text} for any rules specified in {@code textReplacementRules}.
* Creates a PlaybackClient that replays network calls from {@code recordedData} and replaces {@link
* NetworkCallRecord#response() response text} for any rules specified in {@code textReplacementRules}.
*
* @param recordedData The data to playback.
* @param textReplacementRules A set of rules to replace text in network call responses.
Expand Down Expand Up @@ -88,14 +89,16 @@ private Mono<HttpResponse> playbackHttpResponse(final HttpRequest request) {
count.incrementAndGet();

if (networkCallRecord == null) {
if (logger.isWarnEnabled()) {
logger.warn("NOT FOUND - Method: {} URL: {}", incomingMethod, incomingUrl);
logger.warn("Records requested: {}.", count);
}
logger.warning("NOT FOUND - Method: {} URL: {}", incomingMethod, incomingUrl);
logger.warning("Records requested: {}.", count);

return Mono.error(new IllegalStateException("==> Unexpected request: " + incomingMethod + " " + incomingUrl));
}

if (networkCallRecord.exception() != null) {
throw logger.logExceptionAsWarning(Exceptions.propagate(networkCallRecord.exception().get()));
}

int recordStatusCode = Integer.parseInt(networkCallRecord.response().get("StatusCode"));
HttpHeaders headers = new HttpHeaders();

Expand All @@ -112,7 +115,7 @@ private Mono<HttpResponse> playbackHttpResponse(final HttpRequest request) {
}

String rawBody = networkCallRecord.response().get("Body");
byte[] bytes = new byte[0];
byte[] bytes = null;

if (rawBody != null) {
for (Map.Entry<String, String> rule : textReplacementRules.entrySet()) {
Expand All @@ -121,7 +124,20 @@ private Mono<HttpResponse> playbackHttpResponse(final HttpRequest request) {
}
}

bytes = rawBody.getBytes(StandardCharsets.UTF_8);
String contentType = networkCallRecord.response().get("Content-Type");

// octet-stream's are written to disk using Arrays.toString() which creates an output such as "[12, -1]".
if (contentType != null && contentType.equalsIgnoreCase("application/octet-stream")) {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
for (String piece : rawBody.substring(1, rawBody.length() - 1).split(", ")) {
outputStream.write(Byte.parseByte(piece));
}

bytes = outputStream.toByteArray();
} else {
bytes = rawBody.getBytes(StandardCharsets.UTF_8);
}

if (bytes.length > 0) {
headers.put("Content-Length", String.valueOf(bytes.length));
}
Expand All @@ -141,7 +157,12 @@ private String applyReplacementRule(String text) {
}

private static String removeHost(String url) {
URI uri = URI.create(url);
return String.format("%s?%s", uri.getPath(), uri.getQuery());
UrlBuilder urlBuilder = UrlBuilder.parse(url);

if (urlBuilder.query().containsKey("sig")) {
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved
urlBuilder.setQueryParameter("sig", "REDACTED");
}

return String.format("%s%s", urlBuilder.path(), urlBuilder.queryString());
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

package com.azure.core.test.models;

import com.fasterxml.jackson.annotation.JsonProperty;

import java.net.UnknownHostException;

/**
* This class represents a caught throwable during a network call. It is used to serialize exceptions that were thrown
* during the pipeline and deserialize them back into their actual throwable class when running in playback mode.
*/
public class NetworkCallError {
@JsonProperty("Throwable")
private Throwable throwable;

@JsonProperty("ClassName")
private String className;

/**
* Empty constructor used by deserialization.
*/
public NetworkCallError() {
}

/**
* Constructs the class setting the throwable and its class name.
*
* @param throwable Throwable thrown during a network call.
*/
public NetworkCallError(Throwable throwable) {
this.throwable = throwable;
this.className = throwable.getClass().getName();
}

/**
* @return the thrown throwable as the class it was thrown as by converting is using its class name.
*/
public Throwable get() {
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved
switch (className) {
case "java.lang.NullPointerException":
return new NullPointerException(throwable.getMessage());

case "java.lang.IndexOutOfBoundsException":
return new IndexOutOfBoundsException(throwable.getMessage());

case "java.net.UnknownHostException":
return new UnknownHostException(throwable.getMessage());

default:
return throwable;
}
}

/**
* Sets the throwable that was thrown during a network call.
*
* @param throwable Throwable that was thrown.
*/
public void throwable(Throwable throwable) {
this.throwable = throwable;
}

/**
* Sets the name of the class of the throwable. This is used during deserialization the construct the throwable
* as the actual class that was thrown.
*
* @param className Class name of the throwable.
*/
public void className(String className) {
this.className = className;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ public class NetworkCallRecord {
@JsonProperty("Response")
private Map<String, String> response;

@JsonProperty("Exception")
private NetworkCallError exception;

/**
* Gets the HTTP method for with this network call
*
Expand Down Expand Up @@ -96,4 +99,22 @@ public Map<String, String> response() {
public void response(Map<String, String> response) {
this.response = response;
}

/**
* Gets the throwable thrown during evaluation of the network call.
*
* @return Throwable thrown during the network call.
*/
public NetworkCallError exception() {
return exception;
}

/**
* Sets the throwable thrown during evaluation of the network call.
*
* @param exception Throwable thrown during the network call.
*/
public void exception(NetworkCallError exception) {
this.exception = exception;
}
}
Loading