Conformance Tests for Rust (CTRS)

This is an implementation-independent test suite for The Rust Programming Language. It is intended as a regression test for the reference implementation, as well as a tool for testing conformance of alternate implementations.

It differs from the RI test suite in several ways:

• It only covers stable features.
• It has a simple implementation-independent test runner.
• It draws test cases from a wide range of sources.
• It categorizes test cases by language revision, so that multiple levels of conformance may be established.

CTRS is not a curated test of the specification - this probably isn't code you want to read for fun. Rather, it is simply a massive corpus of valid Rust code that is known to work against the reference compiler. The test suite will evolve over time as bugs are shaken out of the RI.

Execute with ./run.py to test your Rust compiler. Set the RUSTC environment variable if you must.

Criteria for inclusion

Criteria will expand over time. Currently:

• Must pass 'rustc -F deprecated -F unstable_features'
• Must pass 'rustc -F deprecated -F unstable_features --test'
• No 'extern crate' (i.e. only std)
• Single-file test cases only

Run ./stab.py on a Rust file to see if it is a candidate.

Organization

All tests are under the test directory. Immediately under test are directories for Rust language/compiler versions. Under each version are directories for test 'groups'. Test groups are drawn from many sources, and are licensed individually.

Note that even tests categorized under a later language revision may work under previous revisions. When tests are added for a new language version they are not first checked against previous versions.

Current coverage

Current focus is pulling tests out of the reference implementation that meet the criteria, as well as sourcing single-file examples from other places.

• 1.4.0
  • doc - Extracts from the RI 'doc' directory
  • doc-collections - Extracts from the RI collections API docs
  • doc-core - Extracts from the RI core library API docs
  • doc-std - Extracts from the RI std library API docs
  • run-pass - Tests from the RI test suite's 'run-pass' directory
• 1.3.0
  • doc - Extracts from the RI 'doc' directory
  • doc-collections - Extracts from the RI collections API docs
  • doc-core - Extracts from the RI core library API docs
  • doc-std - Extracts from the RI std library API docs
  • too-many-lists - Extracts from Learn Rust by writing Entirely Too Many Linked Lists
  • run-pass - Tests from the RI test suite's 'run-pass' directory
  • rust-by-example - Extracts from www.rustbyexample.com
• 1.2.0
  • doc - Extracts from the RI 'doc' directory
  • doc-collections - Extracts from the RI collections API docs
  • doc-core - Extracts from the RI core library API docs
  • doc-std - Extracts from the RI std library API docs
  • pretty - Tests from the RI test suite's 'pretty' directory
  • run-pass - Tests from the RI test suite's 'run-pass' directory
  • rust-by-example - Extracts from www.rustbyexample.com
• 1.1.0
  • doc - Extracts from the RI 'doc' directory
  • doc-collections - Extracts from the RI collections API docs
  • doc-core - Extracts from the RI core library API docs
  • doc-std - Extracts from the RI std library API docs
  • pretty - Tests from the RI test suite's 'pretty' directory
  • run-pass - Tests from the RI test suite's 'run-pass' directory
  • rust-by-example - Extracts from www.rustbyexample.com

Compiler interface requirements

Running the test suite requires a Rust compiler that generally behaves like the reference compiler. Currently the test suite expects the compiler to support the -o flag. In the future it will expect more flags to work.

Licensing

For test cases, any OSI-approved license accepted. Non-free licenses also welcome, but in another repo.

The way this is tracked currently is in a LICENSE file in each directory of code that didn't originate from The Rust Project itself, along with an ORIGIN file indicating where it came from.

Other code in CTRS is MIT/Apache-2.0.

Scripts

This test suite is a bunch of python scripts for wrangling Rust code from various places and into a form that is runnable.

• dupes.py - Lists files in the repo with identical hashes.
• delete_dupes.py - Deletes duplicates from git.
• run.py - The test suite runner. Just needs a rustc command available or the RUSTC environment variable to be set.
• slurp.py - Copies all Rust files that pass stab.py from one dir to another.
• slurp_docs.py - Extracts stable docs from one dir to another.
• undoc_dir.py - Recursively extracts doc tests from all .rs/.md files to a dir.
• stab.py - Checks whether a crate meets the criteria for inclusion.
• undoc.py - Extracts doc tests from a single file.

Future directions

• Improve accuracy of staby.py
• Add support for reference compile-fail tests
• Support aux-build tests
• Support should-panic doc tests
• Support run-fail tests
• Add support for Makefile tests?
• FFI tests, like rust_test_helpers
• Sources
  • compile-fail, run-fail, run-make tests
  • In-tree std crate docs
  • Popular blog posts
  • rustforrubyists.com
  • euler-rust https://github.com/gifnksm/ProjectEulerRust
  • http://www.programming-idioms.org/about#about-block-language-coverage
  • https://github.com/geraldstanje/rust_snippets
  • guidelines
  • http://blog.burntsushi.net/rust-error-handling/
  • Dependency-free libs e.g. math, quickcheck
  • coretest
  • automatic extraction of #[test] cases
  • tool to extract build plans from cargo graphs
  • https://www.ralfj.de/projects/rust-101/main.html
  • error index examples!