Updating test-resources for communication #15551

minnieliu · 2020-09-30T00:36:27Z

No description provided.

damirault · 2020-09-30T21:23:12Z

/azp run net - communication - tests

azure-pipelines · 2020-09-30T21:23:23Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-09-30T21:57:27Z

/azp run net - communication - tests

damirault · 2020-09-30T21:57:27Z

/azp run net - communication - tests

azure-pipelines · 2020-09-30T21:57:37Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2020-09-30T21:57:37Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-09-30T23:41:49Z

/azp run net - communication - tests

azure-pipelines · 2020-09-30T23:41:59Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-09-30T23:59:45Z

/azp run net - communication - tests

azure-pipelines · 2020-09-30T23:59:55Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-10-01T00:02:04Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T00:02:13Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-10-01T00:47:13Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T00:47:23Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-10-01T05:26:27Z

/azp run net - communication - tests

minnieliu · 2020-10-01T06:33:21Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T06:33:31Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-10-01T16:46:23Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T16:46:33Z

Azure Pipelines failed to run 1 pipeline(s).

minnieliu · 2020-10-01T16:54:00Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T16:54:11Z

Azure Pipelines successfully started running 1 pipeline(s).

…nto automateLiveTests

minnieliu · 2020-10-01T17:22:30Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T17:22:40Z

Azure Pipelines successfully started running 1 pipeline(s).

juancamilor · 2020-10-01T17:29:10Z

sdk/communication/Azure.Communication.Chat/tests/ChatClients/ChatClientsTest.cs

@@ -147,6 +148,8 @@ public void E2E_ThreadCreateUpdateGetDelete_MemberAddUpdateRemove_MessageGetSend
            chatClient.DeleteChatThread(threadId);
            #endregion Snippet:Azure_Communication_Chat_Tests_E2E_DeleteChatThread

+            Thread.Sleep(500);


@minnieliu These are synchronous calls, we should not be adding Thread.Sleep on our tests. Any reason we need this ?

This test is flakey in the pipeline, see this: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=559498&view=logs&j=fb683405-d979-52da-6de9-2541dff429a6&t=47b32f90-f4d1-5232-a550-dd1152acf48a

In Java, a 500ms sleep was added to the tests as a temporary solution. There is already a bug created on the server side and they will be looking into the delay https://skype.visualstudio.com/SPOOL/_workitems/edit/2271329/. Once this is fixed on server, the sleep can be removed. Let me know what you think

The thread sleep here is not doing much actually. This value was taken before :
var readReceiptsCount = readReceipts.Count();
Probably what we are seeing here is a case where the read receipt is sent, and downstream on MFE it gets stored, but due to distributed storage when we do a get it is probably not there yet (we will confirm that soon). I suggest we just remove the thread sleeps and comment the readreciept assertion.

juancamilor · 2020-10-01T17:31:38Z

sdk/communication/Azure.Communication.Chat/tests/ChatClients/ChatClientsTest.cs

@@ -253,7 +258,7 @@ public async Task E2E_ThreadCreateUpdateGetDelete_MemberAddUpdateRemove_MessageG
            Assert.AreEqual(3, chatThreadMembersAfterOneAddedCount);
            Assert.AreEqual(2, chatThreadMembersAfterOneDeletedCount);
            Assert.AreEqual((int)HttpStatusCode.OK, typingNotificationResponse.Status);
-            Assert.AreEqual(1, readReceiptsCount);
+            //Assert.AreEqual(1, readReceiptsCount);


Instead of commenting this out, we should look into why is not passing (if that is the case)

Yup, thanks for catching this. I commented it out to see if pipeline tests would pass without it (it does) and missed removing this comment. Fixing in next commit

juancamilor · 2020-10-01T17:47:11Z

sdk/communication/tests.yml

+    EnvVars:
+      AZURE_COMMUNICATION_TEST_MODE: Record


Does this mean we will run in 'Record' mode on CI ? My understanding is that it should run on Playback mode
@JoshLove-msft is that right ?

This variable is newly introduced and only used in the service.projects file (added in this PR) as a condition to skip SMS tests in the live test automation since we dont have a good solution to automating generation of phone number resources yet. In addition, I believe that tests.yml files are specifically used for the live test automation pipeline and not the build pipeline.

I had a different understanding of Recorded vs Live tests, but I may be wrong.
I will talk to the azure folks today to clarify how this should work.

minnieliu · 2020-10-01T18:07:04Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T18:07:14Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-10-01T21:33:07Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T21:33:18Z

Azure Pipelines successfully started running 1 pipeline(s).

juancamilor · 2020-10-01T21:35:14Z

sdk/communication/Azure.Communication.Chat/tests/ChatClients/ChatClientsTest.cs

@@ -5,6 +5,7 @@
 using System.Collections.Generic;
 using System.Linq;
 using System.Net;
+using System.Threading;


This one is not needed anymore

Thanks for catching this, I will remove :)

minnieliu · 2020-10-01T21:41:17Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T21:41:27Z

Azure Pipelines successfully started running 1 pipeline(s).

minnieliu · 2020-10-01T22:05:27Z

/azp run net - communication - tests

azure-pipelines · 2020-10-01T22:05:37Z

Azure Pipelines successfully started running 1 pipeline(s).

* Updating test-resources.json to automate live tests * Update spacing * Remove discrepency * Run against the public cloud * Skip SMS automation tests for now * Skip SMS automation tests for now * Remove the test csproj * Update service projects to remove sms automated tests * Update service projects to remove sms automated tests * Update service projects to remove sms automated tests * Update service projects to remove sms automated tests * Testing chat fix * Adding condition to remove sms * Updating condition to remove sms * Adding sleep to failing chat tests * Testing chat fix * Comment out flakey assert * Clean up * Adding TODOs for tests Co-authored-by: Minnie Liu <peiliu@microsoft.com>

Minnie Liu added 2 commits September 23, 2020 17:25

Updating test-resources.json to automate live tests

13d8912

Update spacing

fbe4cc1

minnieliu requested review from DominikMe and RezaJooyandeh September 30, 2020 00:36

RezaJooyandeh approved these changes Sep 30, 2020

View reviewed changes

minnieliu requested a review from damirault September 30, 2020 18:22

Remove discrepency

444fd16

Run against the public cloud

1c56f58

minnieliu requested a review from danieljurek as a code owner September 30, 2020 21:52

damirault approved these changes Sep 30, 2020

View reviewed changes

Skip SMS automation tests for now

b564fd6

Skip SMS automation tests for now

2efb8d5

Merge changes

4e33ccb

Remove the test csproj

e17e758

Update service projects to remove sms automated tests

5e85b68

Testing chat fix

db78634

Adding condition to remove sms

ae865b0

Updating condition to remove sms

a959d65

Minnie Liu added 2 commits October 1, 2020 10:09

Merge branch 'master' of https://github.com/Azure/azure-sdk-for-net i…

81ef3fc

…nto automateLiveTests

Adding sleep to failing chat tests

0070220

juancamilor self-requested a review October 1, 2020 17:27

juancamilor reviewed Oct 1, 2020

View reviewed changes

Testing chat fix

e2bc828

Comment out flakey assert

6dd939b

juancamilor approved these changes Oct 1, 2020

View reviewed changes

Clean up

55dcc00

Adding TODOs for tests

32b6136

minnieliu merged commit c5f4741 into Azure:master Oct 1, 2020

Updating test-resources for communication #15551

Updating test-resources for communication #15551

Conversation

minnieliu commented Sep 30, 2020

damirault commented Sep 30, 2020

azure-pipelines bot commented Sep 30, 2020

minnieliu commented Sep 30, 2020

damirault commented Sep 30, 2020

azure-pipelines bot commented Sep 30, 2020

azure-pipelines bot commented Sep 30, 2020

minnieliu commented Sep 30, 2020

azure-pipelines bot commented Sep 30, 2020

minnieliu commented Sep 30, 2020

azure-pipelines bot commented Sep 30, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

juancamilor Oct 1, 2020

Choose a reason for hiding this comment

minnieliu Oct 1, 2020 • edited Loading

Choose a reason for hiding this comment

juancamilor Oct 1, 2020

Choose a reason for hiding this comment

juancamilor Oct 1, 2020

Choose a reason for hiding this comment

minnieliu Oct 1, 2020 • edited Loading

Choose a reason for hiding this comment

juancamilor Oct 1, 2020

Choose a reason for hiding this comment

minnieliu Oct 1, 2020 • edited Loading

Choose a reason for hiding this comment

juancamilor Oct 1, 2020

Choose a reason for hiding this comment

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

juancamilor Oct 1, 2020

Choose a reason for hiding this comment

minnieliu Oct 1, 2020

Choose a reason for hiding this comment

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

minnieliu Oct 1, 2020 •

edited

Loading

minnieliu Oct 1, 2020 •

edited

Loading

minnieliu Oct 1, 2020 •

edited

Loading