Skip to content

Creating a OpenGL and OpenGL ES Submission Package

Jump to bottom
Piers Daniell edited this page Aug 28, 2020 · 10 revisions

Creating a Submission Package

Obtaining the sources

The CTS releases are git tags of form <API prefix>-<API major>.<API minor>.<CTS major>.<CTS minor> where

  • <API prefix> is either opengl-es-cts for OpenGL ES releases or opengl-cts for OpenGL releases;
  • <API major> and <API minor> match OpenGL (ES) API version;
  • <CTS major>.<CTS minor> is the CTS version.

To clone the repository and check out a specific release, run:

git clone https://github.com/KhronosGroup/VK-GL-CTS.git -b <release>

If you have existing clone of the repository you can check out specific release with:

git checkout <release>

Building and installing the CTS

Refer to Building the tests for detailed instructions. The essential steps are:

  • Download zlib, libpng glslang, spirv-headers, and spirv-tools by running
	python external/fetch_sources.py
  • For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0 download Khronos Confidential Conformance Test Suite by running
	python external/fetch_kc_cts.py
  • Build the conformance tests for Linux/Windows:
	cmake <path to openglcts> -DDEQP_TARGET=<platform target> -DGLCTS_GTF_TARGET=<GTF target> -G"<Generator Name>"
	cmake --build .

Available values for GLCTS_GTF_TARGET are gles2, gles3, gles31, gles32, and gl. The default value is gles32.

The binaries and data files will be placed to

	<builddir>/external/openglcts/modules
  • Build the conformance tests for Android:
	python external/openglcts/scripts/build_android.py

The script sets GLCTS_GTF_TARGET to gles32 by default. A different GLCTS_GTF_TARGETcan be selected by supplying --glcts-gtf-target, e.g.:

       python external/openglcts/scripts/build_android.py --glcts-gtf-target=gles31

Available values for --glcts-gtf-target are gles32, gles31, gles3 and gles2.

Run one of the following commands to install the binary on your device

	python android/scripts/install.py

or

	adb install --abi <ABI name> android/openglcts/bin/dEQP-debug.apk /data/local/tmp/dEQP-debug.apk

Running the CTS

This section provides a short summary on how to launch a conformance run. For detailed instructions refer to Running the tests.

  • Linux
	./cts-runner --type=<submission type>
  • Windows
	cts-runner.exe --type=<submission type>

Available <submission type> are es2, es3, es31, es32, gl30, gl31, gl32, gl33, gl40, gl41, gl42, gl43, gl44, gl45, and gl46.

  • Android
	am start -n org.khronos.gl_cts/org.khronos.cts.<Submission Activity Type> -e logdir "/sdcard/logs"

Available <Submission Activity Type> are ES32Activity, ES31Activity, ES3Activity, and ES2Activity.

NOTE: A valid GLES 3.2 submission is sufficient to prove GLES 3.1 , GLES 3.0 and GLES 2.0 conformance; it is not necessary to make separate submissions for the older APIs. Similarly a valid GLES 3.1 submission is proof of GLES3.0 and GLES 2.0 conformance.

Creating a package

Once the tests are all passing or all observed failures have appropriate waivers, run the tests with the cts-runner executable or the Android conformance activity. The command must not include the parameters for verbose output. Refer to Section Running Tests its Subsection Android to learn more.

The actual submission package consists of a set of files, which should be bundled into a gzipped tar file named ESMn_<adopter>_<info>.tgz (For OpenGL ES) or GLMn_<adopter>_<info>.tgz (for OpenGL). Here <adopter> is the name of the Adopting member company, or some recognizable abbreviation (e.g. the company's extension prefix). The <info> field is optional. It may be used to uniquely identify a submission by OS, platform, date or other criteria when you are making multiple submissions within a short period of time. For example, company XYZ might make a group of submissions titled ES32_XYZ_Android4.1.1, ES32_XYZ_SupercoreV8, et cetera. The <info> field is not required, and it is permissible to make multiple distinct submissions which have the same package file name. <Mn> refers to the OpenGL or OpenGL ES major and minor version for which results are being submitted. For example, a submission for an OpenGL 3.3 implementation might be named GL33_XYZ_Windows7.tgz, a submission for an OpenGL ES 3.1 implementation might be named ES31_XYZ_Windows7.tgz .

One way to create a suitable gzipped tar file is to execute the command:

tar -cvzf <tgzfilename> -C srcDirectory .

where <srcDirectory> is the name of the directory containing the package files. A submission package should contain the files listed in the following five sections and only them.

1) Statement of Conformance

Statement of Conformance: Include a file called STATEMENT-<adopter> that describes the Implementation for which you are claiming conformance based on this submission. This should be a simple text file containing the following keywords and their values, with at most one keyword/value pair per line.

CONFORM_VERSION:    <git tag of CTS release>
PRODUCT:            <string-value>
CPU:                <string-value>
OS:                 <string-value>

CONFORM_VERSION is the CTS git release tag that you are using for your submission.

PRODUCT is the name of the product that was used to generate the test results. The product name should be the name that the product is commonly known by, and should be specific enough to identify it unambiguously. For example, hardware implementations might describe their product as "Nokia N95 mobile phone handset", or "TI OMAP2420 H4 development board". Software implementations should use the name under which the software is distributed, such as "AMD RenderMonkey 1.8 SDK". Submissions covering simulated hardware (e.g. silicon IP) should use the trade name of the IP package, e.g. "GC500 version . RTL package".

CPU and OS describe the platform on which the OpenGL ES driver client interface is exposed. Note that only one CPU and OS can be listed for a given submission. It is permissible to identify a CPU by its instruction set (e.g. ARMv8), or an OS by its major and minor release numbers (e.g. iOS 6.1). If an implementation makes use of optional CPU features, they should be identified in the CPU name (e.g. ARMv7-NEON).

2) Conformant Products List

If the submission package covers multiple products, you can list them by appending additional PRODUCT: lines to the conformance statement. For example:

   CONFORM_VERSION:         <git tag of CTS release>
   PRODUCT:                 Product A
   PRODUCT:                 Product B

Section A8 of the Conformance Procedures Document describes the similarity criteria a Product must satisfy to be included on the Conformant Products List of a submission. Additional information is given below.

Note that to be claimed as a Conformant Product under a submission, a Product's CPU must be sufficiently similar to the CPU of the Conformant Implementation that they can execute the same driver binary, or different builds of the same driver source compiled with the same compiler options. For example, products based on ARM 1136 can be claimed as part of a submission for an ARM 1176 implementation, provided the driver does not use any features that are not found in both CPUs. See section A8 of the Conformance Procedures Document.

Note also that a Conformant Product's OS must be the same as the Conformant Implementation described in the Statement of Conformance, or must differ by at most a minor revision that does not materially affect the driver binary. For example, a Product using Linux 2.6.5 may be claimed if it is not materially different from a Conformant Implementation using Linux 2.6.3.

3) Result Logs

The submission package must contain all .qpa files produced by the conformance test run, without the --verbose flag. The verbose logs are large and the verbosity complicates the review process. The submission package must also contain the summary log file, cts-run-summary.xml.

4) Changes

The CTS build must always be done from clean git repository that doesn't have any uncommitted changes. Thus it is necessary to run and capture output of git status and git log in the source directories.

4.1) Releases with Khronos Confidential CTS

For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0 this includes the root source directory and in Khronos Confidential CTS source directory:

git status > <submission pkg dir>/git-status.txt
git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt

cd external/kc-cts/src
git status > <submission pkg dir>/kc-cts-git-status.txt
git log --first-parent <release tag>^..HEAD > <submission pkg dir>/kc-cts-git-log.txt

Any changes made to CTS must be committed to the local repository, and provided as part of the submission package. This can be done by running:

git format-patch -o <submission pkg dir> <release tag>..HEAD

cd external/kc-cts/src
git format-patch -o <submission pkg dir> <release tag>..HEAD
4.2) Releases without Khronos Confidential CTS

For opengl-es-cts-3.2.4.0 and later OpenGL ES CTS releases this includes only the root source directory:

git status > <submission pkg dir>/git-status.txt
git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt

Any changes made to CTS must be committed to the local repository, and provided as part of the submission package. This can be done by running:

git format-patch -o <submission pkg dir> <release tag>..HEAD

NOTE: When cherry-picking patches on top of release tag, please use git cherry-pick -x to include original commit hash in the commit message.

Absence of any of these files (except the optional PRODUCTS-<adopter> file) from the package will invalidate the submission. Presence of extraneous files will invalidate a submission.

Clone this wiki locally