[mini.test] Recommended setup for headless execution might not exit

[adigitoleo]

Started playing aroung with mini.test because rolling my own testing ended up way more annoying than I thought. I found a small issue which is that in these situations, headless execution of lua MiniTest.run() and friends doesn't exit to the terminal, which could hang my CI:

• "No tests to execute." I guess this message is printed to the internal v:errmsg, because setting collect = { silent = true } in setup suppresses the message, but the nvim process still does not exit (which is even worse)
• syntax/Vim error in the minimal_init.lua (I think this gets picked up by the internal error buffer of the nvim process, bypassing mini.test completely, could be tricky to change)
• syntax/Vim error in a test file ("Sourcing <foo.lua> resulted in the following error: ")

I tried setting collect = { stop_on_error = true } but that didn't help, as I suspect that option means "stop testing if we encounter ANY test failure", right?

It seems like some of this could be solved by adding some checks for #vim.api.nvim_list_uis() == 0 which should trigger vim.cmd.quit or something like that.

[adigitoleo]

Of course, the issue with just changing the makefile command to run nvim -c "lua MiniTest.run()|quit" is that I lose the exit code information which is necessary to inform CI about overall test success/failure.

[echasnovski]

Glad 'mini.test' gets more traction.

However, I don't really understand the issue here. It is more or less expected that actual setup is done properly so that it can actually run. Having hints for more robust scripts that utilize 'mini.test' are welcome, but somewhat out of scope for 'mini.test' itself. If you have any particular suggestions, we can discuss them.

Also, I don't remember having any of CI problems due to listed reasons. Probably because testing locally and interactively prior to testing headless always seems like a good idea.

[adigitoleo]

It seems that the problem is that in my plugin I use vim.t.foo = "bar" kind of assignment, which sets the variables on the parent nvim process always. I checked, and when using child.lua[[vim.t.foo = "bar"]] in a mock test, then the tab-local variable is indeed only defined for the child process.

[adigitoleo]

After some proper teardown of vim.t stuff and careful use of vim.schedule I was finally able to get a test suite working, in both interactive and headless mode! It immediately told me that I had a very broken .setup() which I now fixed. There is sill a lot more work to debug various problems with the secondary features of the plugin (floating help and man buffers) but thanks to this excellent testing framework I now think it might be possible.

https://builds.sr.ht/~adigitoleo/job/1282800

[echasnovski]

Oof, testing floating terminal will be not trivial. I did not do a lot of it, but can suggest using vim.loop.sleep() (in parent instance, not child) to ensure that terminal is properly opened (as it takes time).

[adigitoleo]

I seem to be getting somewhere with this. Incidentally, using mini.nvim lead to this report: neovim/neovim#30051

I'm really enjoying the parametrised test capability in particular. I can even go crazy and pass function references as the parameters! And for the floating windows, using uv.sleep(500) seems to cover the worst cases (waiting for termopen).

Actually vim.schedule is not even required, I was just being silly before and didn't use child methods properly.

My test/init.lua (used by the setup hooks) now looks like this:

local command = vim.api.nvim_create_user_command
vim.opt.rtp:append(vim.fn.getcwd())

function TestInit()
    vim.opt.rtp:append('dep/mini.nvim')
    require('mini.test').setup({
        collect = {
            find_files = function()
                return vim.fn.globpath('test', '**/test_*.lua', true, true)
            end,
            -- silent = true,
        },
    })
    haunt = require('my_plugin')
    vim.cmd.helptags('ALL')
    my_plugin._err_blocking = true  -- Makes it use error() instead of vim.api.nvim_err_writeln so that expect.error and such work properly
end

command("TestInit", TestInit, { desc = "Initialise testing context" })
command("TestRun", [[=MiniTest.run()]], { desc = "Run test suite" })

-- Load test dependency automatically if headless.
if #vim.api.nvim_list_uis() == 0 then
    TestInit()
end

[echasnovski]

Nice! Glad it starts coming together 👍

[adigitoleo]

I agree that it would be bad practice to commit tests without running them locally, so it's not much of an issue. It just seemed like an incomplete UX to have headless mode not always automatically exit to terminal, but maybe that's just me. I could imagine a situation where CI is using a different neovim version than the local one (again, arguably PEBKAC) that introduced regressions and caused an error in a test file, preventing CI termination.

There are a few specific changes I made to the Makefile and minimal_init.lua immediately, which are not exactly related but since you asked I will list them:

• Declare FILE = $(error "specify the test file using a FILE=<name> assignment") which will cause make to throw that error when attempting to read from an unset FILE variable for the test_file target (alternatively, using an ?= assignment would allow for setting FILE as an env var instead of only inline in the make command)
• Use deps/mini.nvim/.git as the prerequisite for test and test_file targets, which will retain desired behaviour if mini.test is included as a git submodule instead of as a git clone
• In minimal_init.lua, factor the lines inside the #vim.api.nvim_list_uis() == 0 check into a function so that setup for running tests from an interactive nvim session is as easy as calling that function

Can I also just confirm that collect = { stop_on_error = true } in the config does what I think it does, i.e.

stop testing if we encounter ANY test failure

[echasnovski]

Nice tips, thanks!

Can I also just confirm that collect = { stop_on_error = true } in the config does what I think it does, i.e.

stop testing if we encounter ANY test failure

It should do what is documented: if it is true, then after the first error (as a result of test case execution) the whole process (of executing test cases) stops. This is mostly to have allow less time if all you need to know is whether the whole suite passes or not.

[adigitoleo]

Yep, it is indeed nice to have this! I saw the docs (thanks for writing nice help files and guides) but thanks for clarifying what "error" means there.