Automate Cloudshell testing

[catalinaperalta]

We need to automate the way we run tests. The steps to run tests in CloudShell for the identity component are as follows:

Initial Setup:

PLEASE NOTE: This setup is Go specific since CloudShell is running an older version of Go by default and we need to include the current version that has module support. Other languages will have a different set up.
Also, if running CloudShell with a linked storage account steps 1-4 only need to be done the first time this test is run. Step 5 still needs to be run since the environment variables are not persisted every session.

1. Open https://shell.azure.com/
2. cd $HOME
3. Download Golang version 1.13.1
   Run this command: wget https://dl.google.com/go/go1.13.1.linux-amd64.tar.gz
4. Extract contents from tar.gz
   Run this command: tar -xvf go1.13.1.linux-amd64.tar.gz
5. Set environment variables:
   Run the following commands in order:
   $env:GOROOT="$HOME/go"
   $env:GOPATH=$HOME
   $env:PATH="$env:GOPATH/bin:$env:GOROOT/bin:$env:PATH"
6. Confirm you are running go version 1.13 or above
   Run this command: go version
   Expected output: go version go1.13.1 linux/amd64

Run the test:

7. Clone the Go SDK repo.
   Run this command: git clone https://github.com/Azure/azure-sdk-for-go.git
8. Change directory to the azidentity package.
   Run this command: cd azure-sdk-for-go/sdk/azidentity
9. Run the CloudShell test:
   Run this command: go test -run TestManagedIdentityCredential_GetTokenInCloudShellLive
   Expected output: PASS

[joshfree]

@AlexGhiondea - can you please help get this to the Vendor team

@kurtzeborn - can you please include this on the EngSys backlog as a template for future integration into devOps (for all languages)

[jianghaolu]

Java:

Initial setup

1. Create a key vault with vault name {vault name}. Add a secret named "secret" with any value.
2. Open https://shell.azure.com/
3. cd $HOME

Run the test:

4. Clone the Java SDK repo.
   Run this command: git clone https://github.com/Azure/azure-sdk-for-java.git --depth 1
5. Change directory to the root directory.
   Run this command: cd azure-sdk-for-java
6. Set environment variable

export AZURE_VAULT_URL=https://{vault name}.vault.azure.net

7. Run the CloudShell test:
   Run following commands sequentially:

mvn clean install -Dgpg.skip -DskipTests  -f common/perf-test-core/pom.xml 
mvn clean install -Dgpg.skip -DskipTests -f eng/code-quality-reports/pom.xml
mvn clean install -Dgpg.skip -DskipTests  -f sdk/core/azure-core/pom.xml 
mvn clean install -Dgpg.skip -DskipTests  -f sdk/core/azure-core-http-netty/pom.xml 
mvn clean install -Dgpg.skip -DskipTests  -f sdk/identity/azure-identity/pom.xml 
mvn clean install -Dgpg.skip -DskipTests  -f sdk/keyvault/pom.xml 

mvn test -Dtest=ManagedIdentityCredentialLiveTest#testMSIEndpointWithSystemAssigned* -f sdk/e2e/pom.xml -Dgpg.skip -am -DfailIfNoTests=false

Expected output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.azure.e2e.identity.ManagedIdentityCredentialLiveTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.906 s - in com.azure.e2e.identity.ManagedIdentityCredentialLiveTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[chlowell]

Python instructions are here.

[XuGuang-Yao]

Hi @jianghaolu ,

I am following above steps to do java E2E test. My suggestion as below. Please correct me if I miss anything.

1.Commands with .skip fails with below error.

So we need to inclose Dgpg.skip with quotation marks.
mvn test -Dtest=ManagedIdentityCredentialLiveTest#testMSIEndpointWithSystemAssigned* -pl sdk/e2e -f pom.client.xml -"Dgpg.skip" -am -DfailIfNoTests=false

[XuGuang-Yao]

Hi @catalinaperalta ,

I am following above steps to do Go E2E test. My suggestion as below. Please correct me if I miss anything.

1.Test fails when running last command "go test".

I tried to specify the test name by running
go test -run TestManagedIdentityCredential_GetTokenInCloudShellLive
and it passed.

Did you miss specifying the test case name? If it is by design, could you please add more detials to guide to resolve this error?

[jianghaolu]

@XuGuang-Yao I believe on some shells the symbol "." needs to be escaped like "." or "`.". You may be running on such a terminal. I did not test it in PowerShell as most of our customers run this in CMD or Bash.

[sophiajt]

For JavaScript:

Testing azure-identity in Azure Cloud Shell

Open Azure Cloud Shell

https://shell.azure.com/

Create an Azure Key Vault

set environment variables to simplify copy-pasting

• RESOURCE_GROUP
  • name of an Azure resource group
  • must be unique in the Azure subscription
  • e.g. 'cloudshell-identity-test'
• KEY_VAULT_NAME
  • 3-24 alphanumeric characters
  • must begin with a letter
  • must be globally unique

create a resource group

az group create -n $RESOURCE_GROUP --location westus2

create the Key Vault

az keyvault create -g $RESOURCE_GROUP -n $KEY_VAULT_NAME --sku standard

Run the tests (from inside Cloudshell)

Build the webapp

git clone https://github.com/azure/azure-sdk-for-js --single-branch --branch master --depth 1

cd azure-sdk-for-js/sdk/identity/identity/test/manual-integration/Cloudshell

Install the requirements:

npm install

npm install typescript

Build the job:

node_modules/typescript/bin/tsc -p .

Run test

The tests expect the vault's URI in an environment variable. Replace <put key vault name here> with the name of the keyvault you created earlier :

export KEY_VAULT_NAME=<put key vault name here>

node index

Deactivate

deactivate

Verify success

az keyvault secret show -n "secret-name-cloudshell" --vault-name "$($KEY_VAULT_NAME)"

Delete Azure resources

After running tests, delete the resources provisioned earlier:

az group delete -n $RESOURCE_GROUP -y --no-wait

[XuGuang-Yao]

Hi @jonathandturner,

I am following above steps to do JS E2E test. My suggestion as below. Please correct me if I miss anything.

Bolow error occurred after running command nmp install. It may be caused by the known keytar issue on linux. Could you please help to change the test step to resolve the error?

[sophiajt]

Yes, all of these issues seem to be related to keytar. We'll have to figure out what is happening there. These steps largely worked before we added it but it may mean there are additional steps we need to take now that we've added keytar.

[catalinaperalta]

@XuGuang-Yao Yes you are correct to include the test name when running this test. Thanks for catching that, I've also updated the instructions to include that.

[JosueJoshua]

@jonathandturner - we are following the step of E2E Javascript test.
It will return an error when we execute npm install.

And then, it return an error(Permission denied) when we try to fix it by execute the command apt install libsecret-1-0.Could you help to check and resolve the error?

[v-jiaodi]

Hi @jianghaolu, I am following above steps to do java E2E test. Bolow error occurred after running command

mvn test -Dtest=ManagedIdentityCredentialLiveTest#testMSIEndpointWithSystemAssigned*  -f sdk/e2e/pom.xml -"Dgpg.skip" -am -DfailIfNoTests=false

The plugin com.azure.tools:codesnippet-maven-plugin:1.0.0-beta.2 requires Maven version 3.6.3.

I suggest that the version of Maven should be updated to 3.6.3 on CloudShell, or do you have any suggestions?

[chlowell]

@g2vinay for Java SDK issues

[v-jiaodi]

Hi @jntrnr, I am following above steps to do JavaScript E2E test. Bolow error occurred after running command node index with @azure/identity version 2.0.1. There is no problem using this version in the local environment. Could you help to check and resolve the error?

[v-jiaodi]

Hi @catalinaperalta, @chlowell follow the above steps to test for go. When run command go test -run TestManagedIdentityCredential_GetTokenInCloudShellLive, get error as follow:

Through investigation, after clone the code, switch to azure-sdk-for-go/sdk/samples/azidentity/manual-tests/managed-identity/general and run go run main.go. It works normally. For more details, please refer to Azure Arc testing.

For Azure Cloudshell testing, can we refer to Azure Arc testing to update testing instructions?

[v-jiaodi]

@joshfree, @billwert, @g2vinay For Cloudshell testing for Java, when run mvn clean install -Dgpg.skip -DskipTests -f sdk/core/azure-core/pom.xml ,get error message : No space left on device.

Can you help check this issue?

[billwert]

Will take a look. @v-jiaodi is this a clean VM? Do you have a link to a pipeline run?

[v-jiaodi]

@joshfree, @billwert, @g2vinay For Cloudshell testing for Java, when run mvn clean install -Dgpg.skip -DskipTests -f sdk/core/azure-core/pom.xml ,get error message : No space left on device.

Can you help check this issue?

@billwert It cannot be reproduced now.

[github-actions]

Hi @catalinaperalta, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[v-xuto]

@joshfree This issue has been closed. Do we need to continue testing this Automate Cloudshell testing?