Merge pull request #110 from Webperf-se/pa11y

Pa11y

.github/workflows/regression-test-pa11y.yml

@@ -0,0 +1,32 @@
+name: "Regression Test - Accessibility (Pa11y) Test"
+on:
+  push:
+    paths-ignore:
+      - '**.md'
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out repository code
+      uses: actions/checkout@v2
+      if: ${{ success() }}
+    - name: Setup python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x' # Version range or exact version of a Python version to use, using SemVer's version range syntax
+        architecture: 'x64' # optional x64 or x86. Defaults to x64 if not specified
+    - name: Setup dependencies using pip
+      run: pip install -r requirements.txt
+      if: ${{ success() }}
+    - name: Setup node
+      uses: actions/setup-node@v2
+    - name: Setup Pa11y CI npm package
+      run: npm install -g pa11y-ci
+    - name: Setup config (using SAMPLE-config.py as config.py)
+      run: python .github/workflows/verify_result.py -c false
+      if: ${{ success() }}
+    - name: Accessibility (Pa11y)
+      run: |
+         python default.py -t 18 -r -u https://webperf.se/ -o .github/workflows/testresult-18.json
+         python .github/workflows/verify_result.py -t 18
+      if: ${{ success() }}

README.md

@@ -35,17 +35,18 @@ but you can run this project in many ways and what you choose depends on your ne
 Add a interesting ingress here.
 [Read more about our tests](./docs/tests/README.md) or go directly to a specific test below.
 
-* [Accessibility](./docs/tests/google-lighthouse-a11y.md)
-* [Website performance](./docs/tests/google-lighthouse-performance.md)
-* [Best practice on Web](./docs/tests/google-lighthouse-best-practice.md)
-* [Progressive web apps](./docs/tests/google-lighthouse-pwa.md)
-* [SEO best practise](./docs/tests/google-lighthouse-seo.md)
+* [Accessibility (Pa11y)](./docs/tests/pa11y.md)
+* [Accessibility (Lighthouse)](./docs/tests/google-lighthouse-a11y.md)
+* [Website performance (SiteSpeed)](./docs/tests/sitespeed.md)
+* [Website performance (Lighthouse)](./docs/tests/google-lighthouse-performance.md)
+* [Best practice on Web (Lighthouse)](./docs/tests/google-lighthouse-best-practice.md)
+* [Progressive web apps (Lighthouse)](./docs/tests/google-lighthouse-pwa.md)
+* [SEO best practise (Lighthouse)](./docs/tests/google-lighthouse-seo.md)
 * [Validate 404 page (by default checks for Swedish text, though)](./docs/tests/page-not-found.md)
-* [Validate HTML](./docs/tests/html.md)
-* [Validate CSS](./docs/tests/css.md)
-* [Security, data-protecting & Integrity](./docs/tests/webbkoll.md)
-* [Frontend quality](./docs/tests/yellowlab.md)
-* [Website performance](./docs/tests/sitespeed.md)
+* [Validate HTML (W3C)](./docs/tests/html.md)
+* [Validate CSS (W3C)](./docs/tests/css.md)
+* [Security, data-protecting & Integrity (Webbkoll)](./docs/tests/webbkoll.md)
+* [Frontend quality (YellowLab Tools)](./docs/tests/yellowlab.md)
 * [Energy Efficiency](./docs/tests/energy-efficiency.md)
 * [Standard files](./docs/tests/standard.md)
 * [HTTP and Network](./docs/tests/http.md)

default.py

@@ -9,7 +9,7 @@
 
 
 def validate_test_type(test_type):
-    if test_type != utils.TEST_HTML and test_type != utils.TEST_PAGE_NOT_FOUND and test_type != utils.TEST_CSS and test_type != utils.TEST_WEBBKOLL and test_type != utils.TEST_GOOGLE_LIGHTHOUSE and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_PWA and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_A11Y and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_SEO and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_BEST_PRACTICE and test_type != utils.TEST_STANDARD_FILES and test_type != utils.TEST_YELLOW_LAB_TOOLS and test_type != utils.TEST_HTTP and test_type != utils.TEST_ENERGY_EFFICIENCY and test_type != utils.TEST_TRACKING and test_type != utils.TEST_SITESPEED:
+    if test_type != utils.TEST_HTML and test_type != utils.TEST_PAGE_NOT_FOUND and test_type != utils.TEST_CSS and test_type != utils.TEST_WEBBKOLL and test_type != utils.TEST_GOOGLE_LIGHTHOUSE and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_PWA and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_A11Y and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_SEO and test_type != utils.TEST_GOOGLE_LIGHTHOUSE_BEST_PRACTICE and test_type != utils.TEST_STANDARD_FILES and test_type != utils.TEST_YELLOW_LAB_TOOLS and test_type != utils.TEST_PA11Y and test_type != utils.TEST_HTTP and test_type != utils.TEST_ENERGY_EFFICIENCY and test_type != utils.TEST_TRACKING and test_type != utils.TEST_SITESPEED:
         print(_('TEXT_TEST_VALID_ARGUMENTS'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_GOOGLE_LIGHTHOUSE'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_PAGE_NOT_FOUND'))
@@ -22,6 +22,7 @@ def validate_test_type(test_type):
         print(_('TEXT_TEST_VALID_ARGUMENTS_GOOGLE_LIGHTHOUSE_A11Y'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_SITESPEED'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_YELLOW_LAB_TOOLS'))
+        print(_('TEXT_TEST_VALID_ARGUMENTS_PA11Y'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_WEBBKOLL'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_HTTP'))
         print(_('TEXT_TEST_VALID_ARGUMENTS_ENERGY_EFFICIENCY'))

docs/tests/README.md

@@ -3,21 +3,23 @@
 Add a small ingress here
 
 ## List of Tests
-* [Google Lighthouse accessibility with Axe](google-lighthouse-a11y.md)
-* [Google Lighthouse performance](google-lighthouse-performance.md)
-* [Google Lighthouse best practice](google-lighthouse-best-practice.md)
-* [Google Lighthouse progressive web apps](google-lighthouse-pwa.md)
-* [Google Lighthouse SEO](google-lighthouse-seo.md)
-* [Testing the 404 page and status code (by default checks for Swedish text, though)](page-not-found.md)
-* [Validating the HTML code against W3C](html.md)
-* [Validating the CSS code against W3C](css.md)
-* [Users’ integrity test against Webbkoll, provided by Dataskydd.net](webbkoll.md)
-* [Frontend quality against Yellow Lab Tools](yellowlab.md)
-* [Website performance with Sitespeed.io](sitespeed.md)
-* [Energy efficiency](energy-efficiency.md)
-* [Standard files (checks for robots.txt, security.txt and more)](standard.md)
-* [HTTP and Network test (checks HTTP version, TLS version and more)](http.md)
-* [Tracking & Integrity test (checks GDPR compliance, tracking and more)](tracking.md)
+* [Accessibility (Pa11y)](pa11y.md)
+* [Accessibility (Lighthouse)](google-lighthouse-a11y.md)
+* [Website performance (SiteSpeed)](sitespeed.md)
+* [Website performance (Lighthouse)](google-lighthouse-performance.md)
+* [Best practice on Web (Lighthouse)](google-lighthouse-best-practice.md)
+* [Progressive web apps (Lighthouse)](google-lighthouse-pwa.md)
+* [SEO best practise (Lighthouse)](google-lighthouse-seo.md)
+* [Validate 404 page (by default checks for Swedish text, though)](page-not-found.md)
+* [Validate HTML (W3C)](html.md)
+* [Validate CSS (W3C)](css.md)
+* [Security, data-protecting & Integrity (Webbkoll)](webbkoll.md)
+* [Frontend quality (YellowLab Tools)](yellowlab.md)
+* [Energy Efficiency](energy-efficiency.md)
+* [Standard files](standard.md)
+* [HTTP and Network](http.md)
+* [Tracking & Integrity](tracking.md)
+
 
 
 ## Code Quality

docs/tests/pa11y.md

@@ -0,0 +1,47 @@
+# Accessibility (Pa11y)
+[![Regression Test - Accessibility (Pa11y) Test](https://github.com/Webperf-se/webperf_core/actions/workflows/regression-test-pa11y.yml/badge.svg)](https://github.com/Webperf-se/webperf_core/actions/workflows/regression-test-pa11y.yml)
+
+Add small description of what this test is.
+
+## What is being tested?
+
+It test accessibility on the specified url.
+Please note that automated accessibility test generally only find 20-30% of all errors.
+Even if this test is not finding anything you should still do a manuall check once in a while.
+
+## How are rating being calculated?
+
+We are rating the url based on:
+- If Pa11y finds 0 errors you get 5.0 in rating
+- If Pa11y finds just 1 error you get 4.0 in rating
+- If Pa11y finds 2-3 errors you get 3.0 in rating
+- If Pa11y finds 4-7 errors you get 2.0 in rating
+- Else you will get 1.0 in rating
+
+## Read more
+
+* https://github.com/pa11y/pa11y-ci
+* https://github.com/pa11y/pa11y
+
+## How to setup?
+
+This section has not been written yet.
+
+### Prerequirements
+
+* Fork this repository
+
+### Setup with GitHub Actions
+
+Read more on the [general page for github actions](../getting-started-github-actions.md).
+
+### Setup Locally
+
+* Follow [general local setup steps for this repository](../getting-started-local.md)
+* Download and install Node.js
+* Install Pa11y CI npm package ( `npm install -g pa11y-ci` )
+
+## FAQ
+
+No frequently asked questions yet :)
+

locales/en/LC_MESSAGES/a11y_pa11y.po

@@ -1,13 +1,13 @@
 # English
-# Copyright (C) 2020 WebPerf
+# Copyright (C) 2022 WebPerf
 # FIRST AUTHOR <mattias@webperf.se>, 2020.
 #
 msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "POT-Creation-Date: 2020-12-12 10:45+0200\n"
-"PO-Revision-Date: 2021-06-21 17:15+0200\n"
-"Last-Translator: Marcus <m@webperf.se>\n"
+"PO-Revision-Date: 2022-02-18 17:15+0200\n"
+"Last-Translator: Mattias <mattias@webperf.se>\n"
 "Language-Team: English <team@webperf.se>\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -18,19 +18,25 @@ msgid "TEXT_RUNNING_TEST"
 msgstr "## Test: 18 - Accessibility (Pa11y)\n"
 
 msgid "TEXT_REVIEW_A11Y_VERY_GOOD"
-msgstr "- The website do not have any apparent issues with accessibility!!\n"
+msgstr "- The website do not have any apparent issues with accessibility!!"
 
 msgid "TEXT_REVIEW_A11Y_IS_GOOD"
-msgstr "- The website can be more accessible, but is rather good!\n"
+msgstr "- The website can be more accessible, but is rather good!"
 
 msgid "TEXT_REVIEW_A11Y_IS_OK"
-msgstr "- The accessibility is about average, but needs to get better.\n"
+msgstr "- The accessibility is about average, but needs to get better."
 
 msgid "TEXT_REVIEW_A11Y_IS_BAD"
-msgstr "- The website is quite bad at accessibility and sucks for disabled people!\n"
+msgstr "- The website is quite bad at accessibility and sucks for disabled people!"
 
 msgid "TEXT_REVIEW_A11Y_IS_VERY_BAD"
-msgstr "- The accessibility is apparently really bad! Probably both for disabled people and everyone of us when we need some accessibility!\n"
+msgstr "- The accessibility is apparently really bad! Probably both for disabled people and everyone of us when we need some accessibility!"
 
 msgid "TEXT_REVIEW_A11Y_NUMBER_OF_PROBLEMS"
-msgstr "- Number of problems with accessibility: {}\n"
+msgstr "- Number of problems with accessibility: {}"
+
+msgid "TEXT_REVIEW_A11Y_TOO_MANY_PROBLEMS"
+msgstr "- Info: Too many unique problems to show them all.\n"
+
+msgid "TEXT_REVIEW_A11Y_PROBLEMS"
+msgstr "\nProblems:\n"

locales/en/LC_MESSAGES/webperf-core.po

@@ -81,6 +81,9 @@ msgstr "-t 5\t: Best Practice (Google Lighthouse)"
 msgid "TEXT_TEST_VALID_ARGUMENTS_YELLOW_LAB_TOOLS"
 msgstr "-t 17\t: Quality on frontend (Yellow Lab Tools)"
 
+msgid "TEXT_TEST_VALID_ARGUMENTS_PA11Y"
+msgstr "-t 18\t: Accessibility (Pa11y)"
+
 #: default.py:133
 msgid "TEXT_TEST_VALID_ARGUMENTS_PAGE_NOT_FOUND"
 msgstr "-t 2\t: 404 (Page not Found)"

locales/sv/LC_MESSAGES/a11y_pa11y.po

@@ -1,13 +1,13 @@
 # Swedish
-# Copyright (C) 2020 WebPerf
+# Copyright (C) 2022 WebPerf
 # FIRST AUTHOR <mattias@webperf.se>, 2020.
 #
 msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "POT-Creation-Date: 2020-12-12 10:45+0200\n"
-"PO-Revision-Date: 2021-06-21 17:15+0200\n"
-"Last-Translator: Marcus <m@webperf.se>\n"
+"PO-Revision-Date: 2022-02-18 17:15+0200\n"
+"Last-Translator: Mattias <mattias@webperf.se>\n"
 "Language-Team: Swedish <team@webperf.se>\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -18,19 +18,25 @@ msgid "TEXT_RUNNING_TEST"
 msgstr "## Test: 18 - Tillgänglighet (Pa11y)\n"
 
 msgid "TEXT_REVIEW_A11Y_VERY_GOOD"
-msgstr "- Webbplatsen har inga uppenbara fel inom tillgänglighet! Bra, men gör manuella undersökningar för säkerhets skull!\n"
+msgstr "- Webbplatsen har inga uppenbara fel inom tillgänglighet! Bra, men gör manuella undersökningar för säkerhets skull!"
 
 msgid "TEXT_REVIEW_A11Y_IS_GOOD"
-msgstr "- Webbplatsen kan bli mer tillgänglig, men är ganska ok.\n"
+msgstr "- Webbplatsen kan bli mer tillgänglig, men är ganska ok."
 
 msgid "TEXT_REVIEW_A11Y_IS_OK"
-msgstr "- Genomsnittlig tillgänglighet men och bli bättre gentemot automatiska tester.\n"
+msgstr "- Genomsnittlig tillgänglighet men och bli bättre gentemot automatiska tester."
 
 msgid "TEXT_REVIEW_A11Y_IS_BAD"
-msgstr "- Webbplatsen är dålig på tillgänglighet för funktions&shy;nedsatta personer.\n"
+msgstr "- Webbplatsen är dålig på tillgänglighet för funktions&shy;nedsatta personer."
 
 msgid "TEXT_REVIEW_A11Y_IS_VERY_BAD"
-msgstr "- Väldigt dålig tillgänglighet!\n"
+msgstr "- Väldigt dålig tillgänglighet!"
 
 msgid "TEXT_REVIEW_A11Y_NUMBER_OF_PROBLEMS"
-msgstr "- Antal tillgänglighets&shy;problem: {} st\n"
+msgstr "- Antal tillgänglighets&shy;problem: {} st"
+
+msgid "TEXT_REVIEW_A11Y_TOO_MANY_PROBLEMS"
+msgstr "- Info: För många unika problem för att lista alla.\n"
+
+msgid "TEXT_REVIEW_A11Y_PROBLEMS"
+msgstr "\nProblem:\n"

locales/sv/LC_MESSAGES/webperf-core.po

@@ -72,6 +72,9 @@ msgstr "-t 5\t: God praxis (Google Lighthouse)"
 msgid "TEXT_TEST_VALID_ARGUMENTS_YELLOW_LAB_TOOLS"
 msgstr "-t 17\t: Kvalitet på frontend (Yellow Lab Tools)"
 
+msgid "TEXT_TEST_VALID_ARGUMENTS_PA11Y"
+msgstr "-t 18\t: Tillgänglighet (Pa11y)"
+
 msgid "TEXT_TEST_VALID_ARGUMENTS_PAGE_NOT_FOUND"
 msgstr "-t 2\t: 404 (sida finns inte)"