Broken Benchmarks in CI

[WillAyd]

Having seen this on a couple PRs now it looks like the following bechmarks are causing build failures as they fail to run

frame_methods.Repr.time_html_repr_trunc_mi

                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 1039, in main_run_server
                   main_run(run_args)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 913, in main_run
                   result = benchmark.do_run()
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 412, in do_run
                   return self.run(*self._current_params)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 506, in run
                   min_run_count=self.min_run_count)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 569, in benchmark_timing
                   timing = timer.timeit(number)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/timeit.py", line 176, in timeit
                   timing = self.inner(it, self.timer)
                 File "<timeit-src>", line 6, in inner
                 File "/home/vsts/work/1/s/asv_bench/benchmarks/frame_methods.py", line 226, in time_html_repr_trunc_mi
                   self.df3._repr_html_()
                 File "/home/vsts/work/1/s/pandas/core/frame.py", line 651, in _repr_html_
                   import IPython
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/IPython/__init__.py", line 55, in <module>
                   from .terminal.embed import embed
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/IPython/terminal/embed.py", line 16, in <module>
                   from IPython.terminal.interactiveshell import TerminalInteractiveShell
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 81, in <module>
                   if not _stream or not hasattr(_stream, 'isatty') or not _stream.isatty():
               ValueError: I/O operation on closed file

frame_methods.Repr.time_html_repr_trunc_si

                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 1039, in main_run_server
                   main_run(run_args)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 913, in main_run
                   result = benchmark.do_run()
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 412, in do_run
                   return self.run(*self._current_params)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 506, in run
                   min_run_count=self.min_run_count)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/asv/benchmark.py", line 569, in benchmark_timing
                   timing = timer.timeit(number)
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/timeit.py", line 176, in timeit
                   timing = self.inner(it, self.timer)
                 File "<timeit-src>", line 6, in inner
                 File "/home/vsts/work/1/s/asv_bench/benchmarks/frame_methods.py", line 229, in time_html_repr_trunc_si
                   self.df4._repr_html_()
                 File "/home/vsts/work/1/s/pandas/core/frame.py", line 651, in _repr_html_
                   import IPython
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/IPython/__init__.py", line 55, in <module>
                   from .terminal.embed import embed
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/IPython/terminal/embed.py", line 16, in <module>
                   from IPython.terminal.interactiveshell import TerminalInteractiveShell
                 File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 81, in <module>
                   if not _stream or not hasattr(_stream, 'isatty') or not _stream.isatty():
               ValueError: I/O operation on closed file

[WillAyd]

cc @qwhelan

[qwhelan]

@WillAyd Have a fix but trying to identify the actual cause first.

[qwhelan]

@WillAyd Can you link an example? It looks like asv calls sys.stdin.close() in certain cases prior to the benchmark being run, which causes IPython to break on import. However, it doesn't do this in all cases so I'd need an example to see if we can step around it.

For reference, the suspicious line is here: https://github.com/airspeed-velocity/asv/blob/f8e102283c0ac284b0bd8af5637617c10614141b/asv/benchmark.py#L1164

[TomAugspurger]

This is a duplicate of an issue I opened (and I have a PR fixing it).

[TomAugspurger]

#25150 If anyone wants to merge / update. I won’t be able to for a bit.