EBADF from async nio.fn.tempname() access - help debugging

[levouh]

I have a custom adapter for pytest that is somewhat ripped from neotest-python, but runs the tests in a container and redirects the output to the host filesystem where it can be parsed (--junitxml provides the results, while output from docker exec is just dumped directly to STD{OUT,ERR} as expected).

Occasionally, I run into the following issue:

Error executing luv callback:                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:95: Async task failed without callback: The coroutine failed with this message:                                                                                                                                                                                                                                                                                                                                                    
...eotest/lua/neotest/client/strategies/integrated/init.lua:32: EBADF: bad file descriptor                                                                                                                                                                                                                                                                                                                                                                                                     
stack traceback:                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        [C]: in function 'assert'                                                                                                                                                                                                                                                                                                                                                                                                                                                              
        ...eotest/lua/neotest/client/strategies/integrated/init.lua:32: in function 'func'                                                                                                                                                                                                                                                                                                                                                                                                     
        ...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:168: in function <...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:167>                                                                                                                                                                                                                                                                                                                                         
stack traceback:                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        [C]: in function 'error'                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        ...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:95: in function 'close_task'                                                                                                                                                                                                                                                                                                                                                                                               
        ...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:117: in function 'cb'                                                                                                                                                                                                                                                                                                                                                                                                      
        ...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:181: in function <...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:180>

but I'm not entirely sure what it means. Looking at the code:

  local attach_win, attach_buf, attach_chan, attach_unsubscribe
  local output_path = nio.fn.tempname()
  local open_err, output_fd = nio.uv.fs_open(output_path, "w", 438)
  assert(not open_err, open_err)

  output_accum:subscribe(nio.create(function(data)
    local write_err, _ = nio.uv.fs_write(output_fd, data)
    assert(not write_err, write_err)
  end, 1))

We are creating a temporary file, opening it to clear its content and then writing data from the subprocess that is spawned to this file. Is the general goal here to just capture all output from the process so it can be displayed?

I understand that this is largely impossible to debug from a third party because you don't have a lot of insight into my weird setup, but I am mainly trying to understand what the high level goal of this code is so I can understand where things start to go wrong.

Doing a bit of "debugging", the file that gets created is as expected output_path=/tmp/nvim.amasquelier/hjowwP/3. It seems that at some point that file handle goes bad between when it is created and when it is accessed (asynchronously?) later on? If I do something like cat /tmp/nvim.amasquelier/hjowwP/3 to look at the specific file, or ls /tmp/nvim.amasquelier/hjowwP to look at all temporary files, they are all just fine from these commands in the shell. This implies the file is just fine, but something has gone wrong with the file descriptor that has causes this issue.

Any ideas how to troubleshoot what exactly is going wrong with Neotest and its file descriptor @rcarriga? I don't see that anyone else has reported a similar issue.

It is not consistent whether or not is happens either - here is 4 runs, and it only happened on the first (it is not consistently the first run either):

[user] -- DBGSTR [init.lua:27-4] output_path=/tmp/nvim.amasquelier/2OxlGs/0
Error executing luv callback:
...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:95: Async task failed without callback: The coroutine failed with this message: 
...eotest/lua/neotest/client/strategies/integrated/init.lua:33: EBADF: bad file descriptor
stack traceback:
	[C]: in function 'assert'
	...eotest/lua/neotest/client/strategies/integrated/init.lua:33: in function 'func'
	...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:168: in function <...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:167>
stack traceback:
	[C]: in function 'error'
	...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:95: in function 'close_task'
	...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:117: in function 'cb'
	...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:181: in function <...quelier/.local/share/nvim/lazy/neotest/lua/nio/tasks.lua:180>
[user] -- DBGSTR [init.lua:27-4] output_path=/tmp/nvim.amasquelier/2OxlGs/1
[user] -- DBGSTR [init.lua:27-4] output_path=/tmp/nvim.amasquelier/2OxlGs/2
[user] -- DBGSTR [init.lua:27-4] output_path=/tmp/nvim.amasquelier/2OxlGs/3

[levouh]

I should also note that regardless of whether or not this error happens, the output window still seems to have all the output from the docker exec call (the content also matches what can be found by cat-ing the temporary file that gets created).

[rcarriga]

Is the general goal here to just capture all output from the process so it can be displayed?

That's exactly the goal 😄

As you said, this is a tough one to debug but my best guess would be the file is being closed before the final write happens. This is happening because the writes are done asynchronously. I've pushed a change that should address this so please let me know if you still see this behaviour with the latest neotest commit

[levouh]

I will update to the latest and let you know - thanks.