Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README documentation #78

Merged
merged 5 commits into from
Nov 15, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 48 additions & 55 deletions doc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@

This plugin collects and publishes [Robot
Framework](http://robotframework.org/ "Robot Framework Home")
test results in Jenkins.
test results in Jenkins. The plugin **doesn't** run Robot Framework, it
only reports the results.

Preconditions: all "suggested plugins" are "installed" in Jenkins.

[Issue
Tracker](https://issues.jenkins-ci.org/issues/?jql=project+%3D+JENKINS+AND+component+%3D+robot-plugin)

### Current features
## Current features

- Test suite and case details with trend graphs
- Environment variable expansion for build paths in configuration
Expand All @@ -24,17 +26,16 @@ Tracker](https://issues.jenkins-ci.org/issues/?jql=project+%3D+JENKINS+AND+compo
- Trend graph of passed tests over builds on project page
- Marking build failed/unstable/successful according to pass
thresholds given by user
- Option of evaluating only critical tests against the pass thresholds
- Listview column to show overall passed/failed tests in project
listing and duration trend
- Configurable archiving of arbitrary Robot related artifacts per
build 
build
- Token macros for usage with e.g. email-ext plugin
- Test visibility in radiator views

### Configuration
## Configuration

#### Basic configuration of testing task
### Basic configuration of testing task

1. Configure your project
2. Select: Build -\> Add build step -\> execute shell/Execute Windows
Expand All @@ -58,69 +59,61 @@ Tracker](https://issues.jenkins-ci.org/issues/?jql=project+%3D+JENKINS+AND+compo
7. If your output files are named differently than default (i.e.
output.xml, report.html and log.html) specify the filenames by
pressing the "Advanced..." button and writing the names in relevant
fields. The fields support wildcards `*` and `?`.
fields. The fields support wildcards `*` and `?`.
8. If you have other artifacts such as screenshots that you want to
persist for viewing later in the logs you must configure them under
"Advanced... -\> Other files to copy". The field accepts comma
separated list of files and supports wildcards `*` and `?`.
9. Set thresholds and optionally disable thresholds for critical tests
only to count every test in the pass percentage.

#### Pipeline configuration
### Pipeline configuration

1. Run your tests in a similar way as you would normally by using the `sh` step.
2. Call the plugin with either the general `step` or directly by calling `robot` (only from
version 2.0 onwards): `robot outputPath: '.', logFileName: 'log.html', outputFileName:
'output.xml', reportFileName: 'report.hml', passThreshold: 100, unstableThreshold: 75.0`
version 2.0 onwards):

```groovy
robot(outputPath: ".",
passThreshold: 90.0,
unstableThreshold: 70.0,
disableArchiveOutput: true,
outputFileName: "output.xml",
logFileName: 'log.html',
reportFileName: 'report.html',
countSkippedTests: true, // Optional, defaults to false
otherFiles: 'screenshot-*.png'
)
```

#### Configuring direct links to log files
**NOTE!** `onlyCritical` parameter has been deprecated and will be removed in the future. It has no effect on the results.

##### Version 1.3.0
### Configuring direct links to log files

#### Version 1.3.0+

Links are automatically generated to whatever files are configured to
"Report html" and "Log html" in the plugin configuration. Links to log
and report HTMLs are now only in the summary sections in the middle of
the page and the sidebar link from previous versions have been removed.

##### Version 1.2.3 and earlier

From version 1.2 it's possible to configure a direct link from both
side panel and project/build page summaries to chosen file persisted with
the build. If no file is configured the link won't show up. Steps to do
this are as follows:

1. Go to project configuration
2. Configure the name of the file to be linked to Log/Report link field
3. Make sure that you are saving the actual file you want to link with
the build by checking the filenames/masks configured in the fields
behind "Advanced..." button. By default these are output.xml,
report.html and log.html which are the Robot Framework default names
4. Save the configuration and run a build. The links are persisted per
build in order to withstand changes in file naming. Thus the links
won't appear in builds that exist already.
5. Links should appear in the project/build summaries and in build
side panel.

NOTE: After configuration change etc. the link will appear also in
project side panel, but for now it doesn't appear there right after the
build. Conversely, the project view side panel link won't go away until
after configuration change/resave in project. We're working on a
solution to this.

![sidepanel link](images/sidepanel.png)
*side panel link*

![summary link](images/robot_4_table.png)
![summary link](images/robot_summary_table.png)
*Summary link*

#### Configuring Robot overall pass/fail to show in the project list
### Configuring Robot overall pass/fail to show in the project list

The list view column showing the overall pass/fail and duration trend should be visible
by default in the project list. If it's not visible, you can add it by following these steps:

1. Go to Jenkins front page
2. Select the + sign in the top tabs of the project listing to create a
new view or go to
[http://YOURJENKINSHOST/newView](http://yourjenkinshost/newView)
3. Name your view and proceed creating a list view
4. Choose project you want to include to the view. By default the
4. Choose project you want to include to the view. By default, the
"Robot pass/fail" column will appear to the column listing.
5. Save view and select it from the top tabs in Jenkins project listing
6. (Optional) If you want to have the view on by default go to "Manage
Expand All @@ -131,7 +124,7 @@ solution to this.
![List view](images/robot_view_column.png)
*List view column in action*

#### Using token macros in e-mail report
### Using token macros in e-mail report

Prerequisites: token-macro plugin and email-ext plugin installed.

Expand All @@ -144,19 +137,19 @@ Prerequisites: token-macro plugin and email-ext plugin installed.

- `${ROBOT_FAILEDCASES}` - Expands to list of failed Robot cases. Each
case on its own line.
- `${ROBOT_PASSPERCENTAGE, onlyCritical}` - Expands to pass percentage
of tests. (passed / total \* 100 %),
onlyCritical - True if only critical tests should be calculated in
percentage. Defaults to false.
- `${ROBOT_PASSRATIO, onlyCritical}` - Expands to build result in 'passed
/ total' format.
onlyCritical - True if only critical tests should be calculated in
percentage. Defaults to false.
- `${ROBOT_FAILED}` - Expands to count of failed cases.
- `${ROBOT_PASSPERCENTAGE, countSkippedTests}` - Expands to pass percentage
of tests. (passed / total \* 100 %).
`countSkippedTests` is an optional parameter that can be used to include skipped tests in the total count.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add: By default, false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

- `${ROBOT_PASSRATIO}` - Expands to build result in 'passed
/ total' format. This total includes skipped test cases.
- `${ROBOT_PASSED}` - Expands to count of passed cases.
- `${ROBOT_REPORTLINK}` - If logfile link is configured in the Robot
plugin this link will point to that file for the build. Else show
link to Robot reports directory for the build.
- `${ROBOT_TOTAL}` - Expands to total count of test cases. This total includes skipped test cases.

#### Displaying test numbers in build radiator views etc.
### Displaying test numbers in build radiator views etc.

> :warning: If the Robot plugin marks the build as failure the tests will not show
up. This is because only unstable and successful builds are considered
Expand All @@ -165,7 +158,7 @@ to have test results per Jenkins.
>This means that in order to see the test results in other views you must
set your unstable threshold so that the build never goes to failure.

#### Log File Not Showing Properly
### Log File Not Showing Properly

![](images/log_fail_open.png)

Expand All @@ -178,7 +171,7 @@ and [here](https://content-security-policy.com/) how to change you CSP settings
but be aware that **Changing CSP settings will potentially expose you Jenkins instance for
security vulnerabilities**.

### Robot Framework 4.x+ compatibility
## Robot Framework 4.x+ compatibility

:heavy_exclamation_mark: As we're preparing to drop support for RF 3.x, the `onlyCritical` flag has been deprecated and no
longer has any effect. It will be removed in the future, but for now it's available for the transition period.
Expand All @@ -187,15 +180,15 @@ tests in the pass percentage calculation.

The plugin still supports RF 3.x, but no longer takes criticality into account. If you want to use RF 3.x with criticality, please downgrade to the last major version (4.0.0)

### Overall Screenshots
## Overall Screenshots


##### Project view
#### Project view

![Build page overview](images/build_page.png)


##### Detailed build view
#### Detailed build view

![Detailed results view](images/detailed.png)

Expand Down
Binary file removed doc/images/robot_4_table.png
Binary file not shown.
Binary file added doc/images/robot_summary_table.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading