FR: history explorer for looking at old runs

[aspiers]

Forgive me if I'm missing something here, but it seems that stestr goes to a fair bit of trouble to store old test runs in the file- or sqlite-backed database, but doesn't actually facilitate accessing that history in any way except stestr last for looking at the most recent run. Is that right? If so it would be really nice to have a new subcommand (e.g. stestr history) for exploring old runs.

Example user stories

• As a developer who is trying to track down a heisenbug, I would like to be able to type something like stestr history --failing=my.test.which.triggers.awkward.heisenbug in order to obtain a list of all historical test runs where the heisenbug was triggered.

• As a developer who has made a mess and broken his local git branch, I would like to be able to find the exact time of the last run where all the tests passed, via something like stestr history --passing --limit 1, in order to use the git reflog to restore the code to that point.

• As a developer who is trying to pin down a strange issue stemming from the interaction of tests A and B, I would like to be able to simultaneously view in different terminals the test outputs from:

  • a passing run of just test A
  • a passing run of just test B
  • a single --serial run where test A passes and then test B fails

[mtreinish]

Yeah, the history features in stestr at this point is just what testr gave us, which was never much. My intent when I forked from testr was to expand it and leverage all the features that subunit2sql provided for doing this (which is why the experimental subunit2sql backend is there). I just got side tracked and never firmed up the subunit2sql support so I never made any further progress on this. You probably could leverage this a bit today by using subunit2sql-graph (or even openstack-health) and the experimental sql repository type. But, that's far from where I'd like to see this.

That being said I'm definitely in favor of doing something like this, and I think the 3 use cases you've outlined are a good starting point to start discussing what this should look like and how we go about writing it. I also think we should decouple it's implementation from the subunit2sql backend (because there is still work to do on the subunit2sql side around generating an output subunit stream from the DB with attachments) which might double the future work, but at least will enable us to move forward on this.

[mtreinish]

As an aside one thing you bring up is in the second example is that right now we don't track sha1 or anything from git at all. It would be cool as a simple starting point to add initial support for tracking git sha1s at stestr run. The only complexity comes from what about uncommitted changes? In my typical dev workflow I run tests before making a commit, if we do track the current git ref it won't really take that kind of workflow into account.

[aspiers]

Cool, thanks. Yes it should definitely be independent of the backend if possible.

@mtreinish commented on February 26, 2019 7:29 PM:

As an aside one thing you bring up is in the second example is that right now we don't track sha1 or anything from git at all. It would be cool as a simple starting point to add initial support for tracking git sha1s at stestr run. The only complexity comes from what about uncommitted changes? In my typical dev workflow I run tests before making a commit, if we do track the current git ref it won't really take that kind of workflow into account.

Yes, this is a problem. One simple solution (or at least way to dodge the issue) would just be to append -dirty to the sha1.

[mtreinish]

@aspiers to facilitate the second user story I pushed up: #225 which will let us store git hashes (or any other vcs support we decide to implement) in the future and find runs based on that.

[mtreinish]

Just a heads up I got a start to this in #306. It doesn't do the VCS integration component yet (so I didn't mark this as fixing this issue), but it adds a history subcommand that can be used to list all past runs, view a past run, and/or remove a run from the repository. Once the command is in place we can look at expanding it's functionality to include things like the VCS integration, search by results and metadata, and other things.

[aspiers]

Nice! Thanks for the update and hope you're well :-)