Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test harness #57

Merged
merged 28 commits into from
Sep 1, 2023
Merged

Add test harness #57

merged 28 commits into from
Sep 1, 2023

Conversation

9999years
Copy link
Member

@9999years 9999years commented Aug 28, 2023

The GhcidNg type copies a given directory to a tempdir and then launches ghcid-ng there. It also spawns an async task to read JSON log events from the log file.

GhcidNg uses thread-local storage to determine which version of ghc to use when launching tests, and will error appropriately if the thread-local storage is not set. This takes advantage of the fact that async tests run in the tokio "current-thread" runtime by default, which schedules all tasks in the test on the same thread.

The second part is a proc macro attribute exported as #[test_harness::test], which rewrites tests so that

  1. The tests are async functions which run under the default #[tokio::test] current-thread runtime.
  2. Tracing is set up in the tests so that log messages from the test-harness library are visible in the test output.
  3. The appropriate thread-local variables are set for each test (this includes the current GHC version).
  4. One test is generated for each GHC version.
  5. If tests fail, the relevant ghcid-ng logs are saved to a directory under target/ and the path is printed at the end of the tests.

Here's a sample test from #44 using this test harness:

/// Test that `ghcid-ng` can start up and then reload on changes.
#[test]
async fn can_reload() {
    let mut session = GhcidNg::new("tests/data/simple")
        .await
        .expect("ghcid-ng starts");
    session
        .wait_until_ready()
        .await
        .expect("ghcid-ng loads ghci");
    fs::append(session.path("src/MyLib.hs"), "\n\nhello = 1\n")
        .await
        .unwrap();
    session
        .wait_until_reload()
        .await
        .expect("ghcid-ng reloads on changes");
    session
        .get_log(
            Matcher::span_close()
                .in_module("ghcid_ng::ghci")
                .in_spans(["on_action", "reload"]),
        )
        .await
        .expect("ghcid-ng finishes reloading");
}

@linear
Copy link

linear bot commented Aug 28, 2023

DUX-1327 Add test harness

Add the proc-macro-based test harness itself.

This was referenced Aug 28, 2023
@9999years
Copy link
Member Author

9999years commented Aug 28, 2023

This was referenced Aug 28, 2023
@github-actions github-actions bot added the patch Bug fixes or non-functional changes label Aug 28, 2023
@9999years 9999years force-pushed the rebeccat/dux-1326-add-a-haskell-project branch from 23440e3 to 5c23795 Compare August 28, 2023 19:07
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 68ad5e0 to fc8f2c8 Compare August 28, 2023 19:07
@9999years 9999years changed the base branch from rebeccat/dux-1326-add-a-haskell-project to _rebeccat/dux-1332-add-tracing-event-matcher August 28, 2023 19:45
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from fc8f2c8 to 045003c Compare August 28, 2023 19:45
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from 42da07a to cedfa15 Compare August 28, 2023 19:56
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 045003c to 37378f1 Compare August 28, 2023 19:56
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from cedfa15 to dcd7d3d Compare August 28, 2023 20:09
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 37378f1 to 7efd4e0 Compare August 28, 2023 20:10
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from dcd7d3d to 0225fbc Compare August 28, 2023 21:24
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 7efd4e0 to 0921d3a Compare August 28, 2023 21:24
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from 0225fbc to b4d0fe9 Compare August 28, 2023 22:04
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 8fdad0a to 6cbff61 Compare August 28, 2023 22:04
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from b4d0fe9 to 3bc1d3b Compare August 28, 2023 22:24
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 6cbff61 to 96d3725 Compare August 28, 2023 22:24
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from 3bc1d3b to 6fa5f00 Compare August 28, 2023 22:43
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 96d3725 to 47bda68 Compare August 28, 2023 22:43
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from 6fa5f00 to 50ff0cb Compare August 28, 2023 22:50
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from 47bda68 to faf6711 Compare August 28, 2023 22:50
@9999years 9999years force-pushed the _rebeccat/dux-1332-add-tracing-event-matcher branch from 50ff0cb to f8fd033 Compare August 28, 2023 22:54
@9999years 9999years force-pushed the rebeccat/dux-1327-add-test-harness branch from faf6711 to cfcc024 Compare August 28, 2023 22:54
@9999years 9999years marked this pull request as ready for review August 29, 2023 23:45
flake.nix Outdated Show resolved Hide resolved
test-harness-macro/src/lib.rs Outdated Show resolved Hide resolved
test-harness-macro/src/lib.rs Outdated Show resolved Hide resolved
test-harness/src/fs.rs Outdated Show resolved Hide resolved
flake.nix Outdated Show resolved Hide resolved
flake.nix Outdated Show resolved Hide resolved
Gabriella439
Gabriella439 previously approved these changes Sep 1, 2023
}
Some(path) => {
if let Err(err) = tokio::fs::remove_dir_all(&path).await {
// Run `find` on the directory so we can see what's in it?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you had to make use of this fallback code path for removing the directory? In other words, has rm -rf succeeded where remove_dir_all failed?

test-harness/src/internal.rs Outdated Show resolved Hide resolved
test-harness/src/internal.rs Outdated Show resolved Hide resolved
@9999years 9999years merged commit 5c1f1b5 into main Sep 1, 2023
15 checks passed
@9999years 9999years deleted the rebeccat/dux-1327-add-test-harness branch September 1, 2023 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants