Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add some tests, scripted and non-scripted #26

Open
armanbilge opened this issue Dec 30, 2021 · 1 comment
Open

Add some tests, scripted and non-scripted #26

armanbilge opened this issue Dec 30, 2021 · 1 comment

Comments

@armanbilge
Copy link
Member

I think there are some good ones in sbt-spiewak we can adapt. Haven't checked the davenverse.

@armanbilge
Copy link
Member Author

armanbilge commented Feb 22, 2022

So I've been lazy about this, but I haven't forgotten about it. I've had an idea brewing:

  1. There's only so much we can test "internally".
    For example, sbt-github-actions has tests that check the contents of the generated workflow. However, the problem is the workflow can change (and frequently does). So then these tests have to be updated which is annoying and it's not really a functional test anyway i.e. actually runs the workflow.

  2. For better or worse, the real test is when this actually rolls out.
    We get along okay by virtue of the fact that sbt-typelevel is self-hosted, but obviously this doesn't test everything. Ideally, we would PR repos with the update before we roll out the official release, to check that everything is working.

  3. Which leads me to think, what we actually want are integration tests. Something like this:

    • We make our own forks of some representative Typelevel repos, that depend on a snapshot of sbt-typelevel.
    • Every time sbt-typelevel releases a snapshot, we run a custom scala-steward-action to PR the snapshot update to these forks.
    • We monitor these PRs to make sure nothing is broken, particularly before releasing a new version of sbt-typelevel.

    These forks can obviously be extremely slimmed down: we don't care so much about the project contents, but rather their real-world build configurations and git histories (e.g., to verify the mima previous version detection is working). Edit: well, since we ship scalac options maybe we should care about their contents.

    One last bit of trickyness is testing the release process itself, since the best way to do that is actually with a release. It's probably okay to leave that to sbt-typelevel itself, since it would be difficult to release a version with broken releases 😆 . However, it would still be good to exercise the artifact upload/download steps on a dummy publication job, since those are tricky on custom build matrices.

    This will take a little bit of infrastructure, but actually doesn't seem too out-of-reach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant