Added a test execution example #117
Conversation
| @@ -75,7 +75,7 @@ __Description:__ Identifies the composition from which an artifact declared by [ | |||
|
|
|||
| ### ENVIRONMENT | |||
| __Required in:__ None | |||
| __Optional in:__ [EiffelArtifactCreatedEvent](../eiffel-vocabulary/EiffelArtifactCreatedEvent.md), | |||
| __Optional in:__ [EiffelArtifactCreatedEvent](../eiffel-vocabulary/EiffelArtifactCreatedEvent.md), [EiffelTestSuiteStartedEvent](../eiffel-vocabulary/EiffelTestSuiteStartedEvent.md), | |||
There was a problem hiding this comment.
Do we really want this? What does it mean, conceptually, that a test suite is executed in a particular environment? What happens if the test cases also reference an environment? What if they reference another one?
There was a problem hiding this comment.
Yes I think we want it that way. It relates to the discussion of what a test suite really is, which we haven't finalized yet as far as I'm aware.
The scenario I have in mind is that the execution of a TERC is divided into several steps:
- For each recipe in the TERC, which might have different test environment needs, we will provision and deploy a test environment. This will render an EiffelEnvironmentDefinedEvent.
- The test cases in each recipe, forming a test suite, is then sent to a test executor together with a reference to the created environment to test on.
- The executor initiates its test execution by sending an EiffelTestSuiteStarted event referencing the given environment, and then executes each test case in the given test suite.
Of course it would be awkward if the test case referenced an orthogonal environment to what its test suite did, but that could be handled by rules and guidelines.
I see the need to discuss this matter in front of a whiteboard, and I look forward to get that possibility in one of the coming weeks.
There was a problem hiding this comment.
Sorry for late reply, I've been out of the office...
- Yes, makes sense.
- Wait, last time we talked about this we said that we should let TERC and TestSuite map 1:1. So if you're using TERC events, you should wrap all test case executions following that in a single TestSuite (hence the TestSuite has a TERC link type). Now you're suggesting that each recipe within the TERC corresponds to a separate TestSuite - there I think is the basic problem.
- I think this pretty much just follows from the above problem.
I'll try to schedule a whiteboard session so we can hash this out.
usage-examples/test-execution.md
Outdated
| ## Introduction | ||
| Using Eiffel events to describe the execution of a test activity can be done in wide variety of ways. On one edge, a test activity can be described with Eiffel activity events only, and on the other edge a test activity can be expanded to send a big set of Eiffel events revealing the inner details of the test activity. Which way to choose depends on the complexity of your test execution system and on the need to describe and maybe visualize the nitty-gritty details of the test activity. | ||
|
|
||
| The Eiffel protocol provides several events to use within a test activity. This example shows how to make use of most of the Eiffel events related to test activities, with the purpose to present how they all are related. |
There was a problem hiding this comment.
I think we need to explain the context a bit for TERC to make sense to people. I've been thinking of doing this in a screen cast, but haven't had time yet. This would also be a good place.
E.g. "To understand the example provided here, it is important to explain a few concepts. While Eiffel does not make any assumptions about the underlying infrastructure and/or testing methodology, it does encourage separation of concerns. In this view, it is not the responsibility of the activity orchestrator (e.g. the CI server) to determine the test scope, nor is it the responsibility of the test executor. Instead, the EiffelTestExecutionRecipeCollectionEvent makes it possible for a third actor to determine the test scope, which is then handed off to a test executor."
There was a problem hiding this comment.
👍
And for the screen cast, I'd be happy to review the manuscript before you publish it :)
usage-examples/test-execution.md
Outdated
| [EiffelTestSuiteFinishedEvents](../eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md) report that a complete test suite is executed. | ||
|
|
||
| ### IV2A, IV2B | ||
| [EiffelIssueVerifiedEvents](../eiffel-vocabulary/EiffelTestCaseFinishedEvent.md) can be used to override the verdict of a test case reported in a [EiffelTestCaseFinishedEvents](../eiffel-vocabulary/EiffelTestCaseFinishedEvent.md). These events can also be used to connect one or several test case executions to one or several external issues that by these test case executions are considered verified. |
There was a problem hiding this comment.
I don't think "override" is the correct word here. Rather, what IssueVerified does is allowing you to separate test executions from issue (e.g. requirement) verification. So an IssueVerified may be issued based on a host of test executions, some of which may have a positive verdict and some may not.
There was a problem hiding this comment.
Ok. To be rephrased.
|
@d-stahl-ericsson , I've updated some after our discussion the other week. I took the liberty to not handle it exactly as we agreed... let's have another chat around that when time allows :) |
usage-examples/test-execution.md
Outdated
| @@ -6,6 +6,8 @@ Using Eiffel events to describe the execution of a test activity can be done in | |||
|
|
|||
| The Eiffel protocol provides several events to use within a test activity. This example shows how to make use of most of the Eiffel events related to test activities, with the purpose to present how they all are related. | |||
|
|
|||
| To understand the example provided here, it is important to explain a few concepts. While Eiffel does not make any assumptions about the underlying infrastructure and/or testing methodology, it does encourage separation of concerns. In this view, it is not the responsibility of the activity orchestrator (e.g. the CI server) to determine the contents of the test scope, nor is it the responsibility of the test executor. Instead, the [EiffelTestExecutionRecipeCollectionCreatedEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionCreatedEvent.md) makes it possible for a third actor to determine the test scope, which is then handed off to a test executor. And a forth actor is used to provision and deploy a test environment, based on the environment needs described in the [EiffelTestExecutionRecipeCollectionCreatedEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionCreatedEvent.md), for the test to be executed towards, which is also then handed off to the test executor. | |||
There was a problem hiding this comment.
Typo: forth
I'm having trouble with the syntax of the last sentence. How about "Then a fourth actor may be used to set up and provide a test environment, based on the needs described in the EiffelTestExecutionRecipeCollectionCreatedEvent, which the test executor may use to run the tests.
There was a problem hiding this comment.
ok. To be rephrased.
usage-examples/test-execution.md
Outdated
|
|
||
| ### EDef1, EDef2 | ||
| For each test environment set up and/or created an [EiffelEnvironmentDefinedEvents](../eiffel-vocabulary/EiffelEnvironmentDefinedEvent.md) event SHALL be sent. The environments could be static and thereby reused between a lot of test activities, or they could be created in runtime for every test activity. The latter is the typical case when testing in a cloud environment. | ||
|
|
||
| ### TSS1, TSS2 | ||
| The [EiffelTestSuiteStartedEvents](../eiffel-vocabulary/EiffelTestSuiteStartedEvent.md) signal that the execution of a set of test cases have started. There may be several such sets defined as "recipes" in the [EiffelTestExecutionRecipeCollectionEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionEvent.md) event. Each of those sets could have their own constraints on the environment in which they should be executed, defined by the _constraints_ parameter in the _recipe_ object in the [EiffelTestExecutionRecipeCollectionEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionEvent.md). The environment used when executing the test suite SHALL be defined using [EiffelEnvironmentDefinedEvents](../eiffel-vocabulary/EiffelEnvironmentDefinedEvent.md) | ||
| The [EiffelTestSuiteStartedEvents](../eiffel-vocabulary/EiffelTestSuiteStartedEvent.md) signal that the execution of a set of test cases have started. There may be several such sets defined as "recipes" in the [EiffelTestExecutionRecipeCollectionCreatedEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionCreatedEvent.md) event. Each of those sets could have their own constraints on the environment in which they should be executed, defined by the _constraints_ parameter in the _recipe_ object in the [EiffelTestExecutionRecipeCollectionCreatedEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionCreatedEvent.md). The environment used when executing the test suite SHALL be defined using [EiffelEnvironmentDefinedEvents](../eiffel-vocabulary/EiffelEnvironmentDefinedEvent.md) |
There was a problem hiding this comment.
Typo: signal -> signals
Typo: have started -> has started
We're using bold rather than italics to reference property names, and using full paths. I.e. data.batches.recipes.constraints rather than constraints.
usage-examples/test-execution.md
Outdated
| @@ -32,13 +34,13 @@ The [EiffelTestSuiteStartedEvents](../eiffel-vocabulary/EiffelTestSuiteStartedEv | |||
| [EiffelTestSuiteFinishedEvents](../eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md) report that a complete test suite is executed. | |||
|
|
|||
| ### IV2A, IV2B | |||
| [EiffelIssueVerifiedEvents](../eiffel-vocabulary/EiffelTestCaseFinishedEvent.md) can be used to override the verdict of a test case reported in a [EiffelTestCaseFinishedEvents](../eiffel-vocabulary/EiffelTestCaseFinishedEvent.md). These events can also be used to connect one or several test case executions to one or several external issues that by these test case executions are considered verified. | |||
| [EiffelIssueVerifiedEvents](../eiffel-vocabulary/EiffelTestCaseFinishedEvent.md) can be used to declare that the execution of a test case has verified a certain issue (e.g. a requirement or a trouble report) successfully or not. | |||
There was a problem hiding this comment.
I think "bug report" is more universally understood than "trouble report".
usage-examples/test-execution.md
Outdated
| The execution of each test case is surrounded by [EiffelTestCaseStartedEvents](../eiffel-vocabulary/EiffelTestCaseStartedEvent.md) and [EiffelTestCaseFinishedEvents](../eiffel-vocabulary/EiffelTestCaseFinishedEvent.md). When a complete suite of test cases is executed, a [EiffelTestSuiteFinishedEvent](../eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md) is issued. | ||
| * When all test suites corresponding to the recipes in the [EiffelTestExecutionRecipeCollectionEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionEvent.md) are finished, the test activity is considered finished and an [EiffelActivityFinishedEvents](../eiffel-vocabulary/EiffelActivityFinishedEvent) is sent to declare that. | ||
| * When all test suites corresponding to the recipes in the [EiffelTestExecutionRecipeCollectionCreatedEvent](../eiffel-vocabulary/EiffelTestExecutionRecipeCollectionCreatedEvent.md) are finished, the test activity is considered finished and an [EiffelActivityFinishedEvents](../eiffel-vocabulary/EiffelActivityFinishedEvent) is sent to declare that. |
There was a problem hiding this comment.
Perhaps "to communicate this fact" rather than "to declare that"?
|
I think this would be a great addition if updated according to the changes to test events. |
|
Should this be closed? |
|
No, it should be updated |
e-backmark-ericsson
force-pushed
the
master
branch
from
e5364a5
to
f4077bc
Compare
|
Will it be updated? :) |
|
Oh, but it was... On August 30th |
|
It's time to get this PR done... Anyone up for a review? |
|
Happy to! Are you planning to update the sequence diagram according to our previous conversation, or would you like a review right now? |
|
The sequence diagram is now updated to reflect our discussions yesterday. Please review :) |
|
There's one unresolved issue here. We had a conversation above about link from TestSuiteStarted to EnvironmentDefined. What is your thinking there? There is currently no such link, but there is one TestCaseStarted -> EnvironmentDefined, and that's also what the event graph in the example shows. I still think we can't have both, we have to pick one. Even if you were to argue to changing to TestSuiteFinished (and I admit to being skeptical), I would prefer that as a separate PR (because then we're changing event definitions). On a more technical note, following #176 this PR conflicts on this particular point. The new usage example also needs to be linked from README.md. For the example itself, some suggestions:
|
Added ArtifactCreated/CompositionDefined and ConfidenceLevelModified events and the links to/from them, for completeness. Updated link styles to reflect whether a link is optional or mandatory.
The sequence diagram is updated where the test executor and the test manager are switched. The logic is also changed so that the CI server now calls the test manager to get the TERC and then calls the test executor.
Added a link in README.md to test-execution.md Rephrased some in test-execution.md according to review comments.
e-backmark-ericsson
force-pushed
the
master
branch
from
200f771
to
ae4d2ad
Compare
|
I agree to keep the change regarding linking EnvironmentDefined to TestSuiteStarted or not as a separate issue. I've removed it from this PR now. I added link from the README file and updated the text to match your suggestion. Regarding the CAUSE links in the events picture I added them there to show that the ActT and ArtC events not just pop up by no reason, but rather due to some reason which is not specified in this picture. I think it could be good to have them there, but if there is a strong reason not to they can very well be removed. |
|
Created Issue #184 for this. |
To describe how events for a test activity can be linked.