Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Storage] Stress tests in pipeline? #21030

Merged
merged 23 commits into from
May 18, 2021
Merged

[Storage] Stress tests in pipeline? #21030

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
e4a73b7
draft.
kasobol-msft Apr 28, 2021
d7eb205
Merge remote-tracking branch 'upstream/master' into resource-constrai…
kasobol-msft Apr 28, 2021
5f9a986
does this work ?
kasobol-msft Apr 28, 2021
40c3bd4
try this.
kasobol-msft Apr 29, 2021
be5a803
well, here is verbosity...
kasobol-msft Apr 29, 2021
fbb7011
Merge remote-tracking branch 'upstream/master' into resource-constrai…
kasobol-msft May 17, 2021
326ad61
disable tests temporarily
kasobol-msft May 17, 2021
a2d29c8
try this.
kasobol-msft May 17, 2021
478309b
ugh..
kasobol-msft May 17, 2021
4c7ba2b
right java ?
kasobol-msft May 17, 2021
d4c4810
right java ?
kasobol-msft May 17, 2021
fac73ea
try this.
kasobol-msft May 18, 2021
3dea100
try this.
kasobol-msft May 18, 2021
eff14bb
comment out hanging case.
kasobol-msft May 18, 2021
02aa1ce
make sure at least iteration completes
kasobol-msft May 18, 2021
bbecc1c
undo maven way...
kasobol-msft May 18, 2021
ab92b86
undo disable tests.
kasobol-msft May 18, 2021
0c8adab
refactor.
kasobol-msft May 18, 2021
15b0db1
this normally completes faster now.
kasobol-msft May 18, 2021
ebd3e2c
this isn't necessary.
kasobol-msft May 18, 2021
b5afc84
Revert "this isn't necessary."
kasobol-msft May 18, 2021
ac33658
this isn't necessary
kasobol-msft May 18, 2021
1964409
add assertion to make sure we don't regress "at least one operation c…
kasobol-msft May 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions sdk/storage/azure-storage-perf/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,73 @@
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>storage-stress-scenarios</id>
<activation>
<property>
<name>storage-stress-scenarios</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version> <!-- {x-version-update;org.codehaus.mojo:exec-maven-plugin;external_dependency} -->
<executions>
<execution>
<id>uploadoutputstream</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>java</executable>
<arguments>
<argument>-Xms700m</argument>
<argument>-Xmx700m</argument>
<argument>-jar</argument>
<argument>${project.basedir}/target/${project.artifactId}-${project.version}-jar-with-dependencies.jar</argument>
<argument>uploadoutputstream</argument>
<argument>--warmup</argument>
<argument>0</argument>
<argument>--duration</argument>
<argument>120</argument>
<argument>--size</argument>
<argument>1048576000</argument>
<argument>--sync</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>uploadblob</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>java</executable>
<arguments>
<argument>-Xms50m</argument>
<argument>-Xmx50m</argument>
<argument>-jar</argument>
<argument>${project.basedir}/target/${project.artifactId}-${project.version}-jar-with-dependencies.jar</argument>
<argument>uploadblob</argument>
<argument>--warmup</argument>
<argument>0</argument>
<argument>--duration</argument>
<argument>120</argument>
<argument>--size</argument>
<argument>1048576000</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
2 changes: 1 addition & 1 deletion .../azure-storage-perf/src/main/java/com/azure/storage/blob/perf/UploadOutputStreamTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ public UploadOutputStreamTest(PerfStressOptions options) {
@Override
public void run() {
try {
BlobOutputStream blobOutputStream = blockBlobClient.getBlobOutputStream();
BlobOutputStream blobOutputStream = blockBlobClient.getBlobOutputStream(true);
writeBytesToOutputStream(blobOutputStream, options.getSize());
blobOutputStream.close();
} catch (IOException e) {
Expand Down
16 changes: 16 additions & 0 deletions sdk/storage/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ stages:
- name: azure-storage-queue
groupId: com.azure
safeName: azurestoragequeue
AdditionalModules:
- name: perf-test-core
groupId: com.azure
TestMode: 'LIVE'
TimeoutInMinutes: 240
Location: canadacentral
Expand All @@ -47,6 +50,19 @@ stages:
Path: sdk/storage/platform-matrix.json
Selection: sparse
GenerateVMJobs: true
PostSteps:
- task: Maven@3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to run this task if the test run fails? Also, if this step fails do we want to fail the job?

Copy link
Contributor Author

@kasobol-msft kasobol-msft Apr 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can live with this not running if tests fail. We want to have green build. I'll check the later (but yes, we want to fail a job)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

displayName: 'Run Stress Scenarios'
inputs:
mavenPomFile: sdk/storage/azure-storage-perf/pom.xml
options: -Dstorage-stress-scenarios
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd either add $(DefaultOptions) or --no-transfer-progress to options to reduce the verbosity of this step.

mavenOptions: '$(MemoryOptions) $(LoggingOptions)'
javaHomeOption: 'JDKVersion'
jdkVersionOption: $(JavaTestVersion)
jdkArchitectureOption: 'x64'
goals: verify
env:
STORAGE_CONNECTION_STRING: $(AZURE_STORAGE_BLOB_CONNECTION_STRING)
EnvVars:
AZURE_TENANT_ID: $(aad-azure-sdk-test-tenant-id)
AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id)
Expand Down