EPIC: Centralize and update test documentation #1558

smlambert · 2020-01-15T17:11:31Z

Since test doc lives all over the place it is hard to find and navigate, we will do a reorg/rewrite based on feedback received through slack, etc.

Based on comments captured below in this issue, it will be tackled in parts

Tutorials / Learning-oriented (longer learning session)
- Add new platform support
  - reference: https://github.com/adoptium/aqa-tests/wiki/Adding-AQAvit-support-for-a-New-Platform
- Incorporate vendor tests
  - example: See how smoke tests are incorporated using vendor test repos from the build pipeline code
  - provide another simple example of laying down test material in a repository separate from the main ones that are directly part of "AQAvit", and the minimum requirements for what is currently needed to execute those tests (playlist.xml, build.xml files and testcases)
- Run tests in a Github action
- Use TRSS to triage a release
- ... other tutorials on commonly conducted activities
How-to Guides / Task-oriented (quick lessons)
- How to verify a release (with testenv.properties) (see also https://adoptium.net/docs/aqavit-verification/)
- How to match the right aqa-tests release branch with a particular JDK
- Rerun a failing test in a Grinder job
  - reference: https://github.com/adoptium/aqa-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins#rerun-a-failing-openjdk-regression-test to show how to rerun a failing testcase (different from rerunning a failing test target)
  - Also show the case for rerunning a failing test target
- ... other quick how-to's based on lightning talks
Reference / Information-oriented (detailed encyclopedic knowledge)
- TKG features reference
- Jenkins features reference - Can start with https://github.com/adoptium/aqa-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins#reference
- TRSS features reference
- Terminology reference (top-level targets down to individual test cases)
- Jenkins label schema (refer to smlambert branch addressing Standardize and document a labelling scheme for Jenkins nodes infrastructure#93)
Explanation / Understanding-oriented ('why' things are designed a certain way, 'why' AQAvit exists)
- AQA Definition (Manifesto) - Add Manifesto.md document #4419
- Scope of AQAvit (AQAvit verification, Cloud compatibility, Developer support)
- Why the '3 layer cake' architecture (via Centralize and update test documentation - Part 1 Explanation section #4686)

Noting that this EPIC specifically targets the content that needs to be created, updated and centralized. There will be a separate issue to cover how to display the final set of docs (with some prototyping happening under https://github.com/smlambert/smlambert.github.io and viewable at https://smlambert.github.io/

smlambert · 2020-01-15T17:15:42Z

Test-related documentation (some of which is not relevant to most consumers of AQA, but listed for completeness:

https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/master/README.md (and all its subdirs)
https://github.com/eclipse/openj9-systemtest/blob/master/README.md
and
https://github.com/eclipse/openj9/blob/master/test/README.md
https://github.com/eclipse/openj9/tree/master/test/docs
and
https://github.com/AdoptOpenJDK/openjdk-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins
https://github.com/AdoptOpenJDK/openjdk-tests/wiki/Guidance-for-Creating-OpenJDK-Test-Defects
https://github.com/AdoptOpenJDK/openjdk-tests/wiki/Graduated-Testing-&-Test-Numbers

and probably some of the most used / useful guides:
https://github.com/eclipse/openj9/wiki/Reproducing-Test-Failures-Locally

smlambert · 2022-06-30T02:00:22Z

Use https://diataxis.fr/ as a guidance for this work.

Tutorials / Learning-oriented
How-to Guides / Task-oriented
Reference / Information-oriented
Explanation / Understanding-oriented

Hopefully within the next weeks, I will be able to outline some content and categorize it according to the above areas

smlambert · 2022-07-07T01:21:15Z

Do not forget to add a section for the labeling schema we use (and also so that as it is extended, it continues to make sense and is not extended in non-intuitive ways), original notes here: https://github.com/smlambert/openjdk-infrastructure/blob/labels/docs/jenkinslabels.md

smlambert · 2022-10-06T23:47:21Z

C4Dynamic
    title The 3 layers of AQAvit 

Boundary(b, "AQAvit") {

System_Boundary(b2, "Monitoring, Aggregation & Reporting") {
    Component(c4, "TRSS", "Test Results Summary Service", "Stores and represents aggregate test results, allows for advanced search and reporting capabilities.")
    }

    System_Boundary(b1, "CI Layer") {
      Component(c3, "Jenkins", "CI Server", "Multiplex/parallelize test jobs across multiple nodes and platforms.")
    }


    Container_Boundary(b01, "Test Nodes") {
      Component(c01, "TKG", "Test Kit Gen", "Provides base level functionality for standardizing the many disparate test frameworks.")

   }


}

    BiRel(c4, c3, "TRSS polls CI for console output", "Jenkins API")
    BiRel(c01, c3, "Jenkins in contact with Jenkins agent on test node")
   
    UpdateRelStyle(c01, c3, $textColor="red", $offsetY="-20", $offsetX="10") 
    UpdateRelStyle(c4, c3, $textColor="red", $offsetY="-40", $offsetX="10")
    
   UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")

smlambert · 2023-02-21T20:21:19Z

smlambert · 2023-02-22T14:36:26Z

smlambert · 2023-03-10T13:27:18Z

Other docs to include:

AQA definition to be updated from AdoptOpenJDK Quality Assurance v1 to Adoptium Quality Assurance Definition
Etiquette for handling PRs (if contributor asks for a review from a particular reviewer because they know the code well, we should respect that request, within reason. Also unless there is something breaking the world, it is better to take time with reviews, it is not a race)
Additional Guidance for code reviewers (how to determine what additional testing is required, its is not all automated PR testing, by choice and for good reason)
Terminology reference (what does 'test target', 'dynamic agent', 'dynamic compilation', 'dynamic parallelization', 'TAP', etc refer to?)
Scope of AQAvit, various ways it can and should be used:
- AQAvit verification (9 top-level targets currently forming the definition of that)
- Cloud Compatibility Program (9 top-level targets + 1 more, still being defined)
- Development support (all top-level targets in the grid, including any and all dev level testing)
- Vendor-specific proprietary test support (and and all top-level targets in the grid, plus overlaid vendor tests)
Jenkins label schema (including all existing/known labels in use, by the project and downstream/vendor use)

Ndacyayisenga-droid · 2023-07-26T17:15:39Z

Hi @smlambert. Can we work on this together?

smlambert · 2023-07-26T17:33:15Z

Hi @smlambert. Can we work on this together?

Happily @Ndacyayisenga-droid !

llxia · 2023-09-12T12:48:51Z

It would be nice to have docs about the AQAvit release and add the usage of testenv.properties in the AQA repo (https://adoptium.net/docs/aqavit-verification/ ).

smlambert · 2023-09-12T12:50:20Z

re: #1558 (comment) - agree, added it to the description above (in the "how-to" section).

smlambert · 2024-11-20T18:27:09Z

Update docs regarding container based and dynamic agent tests riffing off of this diagram created during a meeting with AQAvit committers

sxa · 2024-11-22T15:56:18Z

If not explicitly included in any of the points above, #5770 (comment) has a short comment thread on how to improve usability of the custom_target value for the tests.

smlambert added the docs label Jan 15, 2020

This was referenced Jan 15, 2020

Update openjdk-test and TSC docs once this repo is populated adoptium/TKG#1

Closed

Overhaul of AQA test documentation #1556

Closed

karianna modified the milestone: January 2020 Jan 16, 2020

smlambert mentioned this issue Jan 22, 2020

AQAvit Meeting - Jan 22, 2020 Agenda and Minutes #1571

Closed

smlambert mentioned this issue Apr 14, 2020

Feedback on test reproduction quirks for the test team eclipse-openj9/openj9#4548

Closed

This was referenced Jul 10, 2021

Update README.md #2726

Closed

Minor Enhancement in README.md #2725

Closed

smlambert self-assigned this Mar 29, 2022

smlambert added this to Adoptium 1Q 2023 Plan Feb 22, 2023

smlambert mentioned this issue Feb 23, 2023

AQAvit Meeting: Feb 22, 2023 #4321

Closed

smlambert moved this to Todo in Adoptium 1Q 2023 Plan Feb 23, 2023

smlambert moved this from Todo to In Progress in Adoptium 1Q 2023 Plan Feb 23, 2023

smlambert changed the title ~~Centralize and update test documentation~~ EPIC: Centralize and update test documentation Mar 13, 2023

smlambert added this to Adoptium 2Q 2023 Plan Mar 13, 2023

smlambert mentioned this issue Mar 13, 2023

Add Manifesto.md document #4419

Merged

smlambert moved this to In Progress in Adoptium 2Q 2023 Plan Mar 13, 2023

smlambert removed this from Adoptium 1Q 2023 Plan Mar 27, 2023

smlambert removed this from Adoptium 2Q 2023 Plan Jul 12, 2023

smlambert added this to Adoptium 3Q 2023 Plan Jul 12, 2023

smlambert moved this to Todo in Adoptium 3Q 2023 Plan Jul 12, 2023

smlambert assigned Ndacyayisenga-droid Jul 26, 2023

This was referenced Jul 26, 2023

Centralize and update test documentation - Part 1 Explanation section #4686

Merged

Centralize and update test documentation -Jenkins label schema #4689

Merged

Centralize and update test documentation - Jenkins features reference #4690

Merged

smlambert moved this from Todo to In Progress in Adoptium 3Q 2023 Plan Sep 5, 2023

smlambert removed this from Adoptium 3Q 2023 Plan Nov 20, 2023

smlambert added this to 2024 1Q Adoptium Plan Nov 20, 2023

smlambert moved this to In Progress in 2024 1Q Adoptium Plan Nov 20, 2023

smlambert unassigned Ndacyayisenga-droid Feb 26, 2024

smlambert added this to 2024 2Q Adoptium Plan Apr 2, 2024

smlambert moved this to In Progress in 2024 2Q Adoptium Plan Apr 2, 2024

smlambert removed this from 2024 1Q Adoptium Plan Apr 2, 2024

This was referenced Jul 30, 2024

documentation? How to contribute first testsuite.. smlambert/smlambert.github.io#1

Closed

add howto contribution iteration 1 #5471

Merged

howto jenkins job doc #5485

Open

smlambert added this to Adoptium Backlog Oct 2, 2024

github-project-automation bot moved this to Todo in Adoptium Backlog Oct 2, 2024

smlambert removed this from 2024 2Q Adoptium Plan Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EPIC: Centralize and update test documentation #1558

EPIC: Centralize and update test documentation #1558

smlambert commented Jan 15, 2020 •

edited

Loading

smlambert commented Jan 15, 2020

smlambert commented Jun 30, 2022

smlambert commented Jul 7, 2022

smlambert commented Oct 6, 2022

smlambert commented Feb 21, 2023

smlambert commented Feb 22, 2023 •

edited

Loading

smlambert commented Mar 10, 2023 •

edited

Loading

Ndacyayisenga-droid commented Jul 26, 2023

smlambert commented Jul 26, 2023

llxia commented Sep 12, 2023

smlambert commented Sep 12, 2023

smlambert commented Nov 20, 2024

sxa commented Nov 22, 2024

EPIC: Centralize and update test documentation #1558

EPIC: Centralize and update test documentation #1558

Comments

smlambert commented Jan 15, 2020 • edited Loading

smlambert commented Jan 15, 2020

smlambert commented Jun 30, 2022

smlambert commented Jul 7, 2022

smlambert commented Oct 6, 2022

smlambert commented Feb 21, 2023

smlambert commented Feb 22, 2023 • edited Loading

smlambert commented Mar 10, 2023 • edited Loading

Ndacyayisenga-droid commented Jul 26, 2023

smlambert commented Jul 26, 2023

llxia commented Sep 12, 2023

smlambert commented Sep 12, 2023

smlambert commented Nov 20, 2024

sxa commented Nov 22, 2024

smlambert commented Jan 15, 2020 •

edited

Loading

smlambert commented Feb 22, 2023 •

edited

Loading

smlambert commented Mar 10, 2023 •

edited

Loading