Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jupyter mixes stdout and stderr output in non-deterministic order #173

Closed
Guzzii opened this issue Oct 7, 2018 · 11 comments
Closed

Jupyter mixes stdout and stderr output in non-deterministic order #173

Guzzii opened this issue Oct 7, 2018 · 11 comments

Comments

@Guzzii
Copy link

Guzzii commented Oct 7, 2018

Description

A dummy progress bar would be created after finishing each outer iteration, looking like below. The code to reproduce the issue is attached. It works fine without print though. Is there any way to avoid this behavior?

100% (100 of 100) |######################| Elapsed Time: 0:00:01 Time:  0:00:01
 16% (16 of 100) |###                    | Elapsed Time: 0:00:00 ETA:   0:00:01
0
100% (100 of 100) |######################| Elapsed Time: 0:00:01 Time:  0:00:01
 16% (16 of 100) |###                    | Elapsed Time: 0:00:00 ETA:   0:00:00
1
100% (100 of 100) |######################| Elapsed Time: 0:00:01 Time:  0:00:01
2

Code

import time
import progressbar

for i in range(3):
    for j in progressbar.progressbar(range(0, 100)):
        time.sleep(0.01)
    print(i)

Versions

  • Python version: 3.6
  • Python distribution/environment: Anaconda
  • Operating System: macOS 10.13
  • Package version: 3.37.1
@wolph
Copy link
Owner

wolph commented Oct 9, 2018

I cannot reproduce it in either the normal Python console or Anaconda. Can you try in a regular Python console just to make sure it's not related to Anaconda?

@Guzzii
Copy link
Author

Guzzii commented Oct 9, 2018

Hi @wolph. Sorry for the confusion. You are right. It works in python/ipython console. It only seems to have issue in Jupiter notebook.

@aaronjohnson92
Copy link

I reproduced this on python 3.7 in windows cmd, as well as on arch linux.

@wolph
Copy link
Owner

wolph commented Oct 17, 2018

It seems like a bug in Jupyter outside of my control.

When running this code:

import time
import progressbar


for i in range(3):
    bar = progressbar.ProgressBar()
    
    for j in bar(range(10)):
        if j == 0:
            bar.widgets = ['bar i: %d :: ' % i] + bar.widgets
    
        time.sleep(0.1)
    print('print i: %d' % i)

The result is:

bar i: 0 :: 100% (10 of 10) |############| Elapsed Time: 0:00:01 Time:  0:00:01
bar i: 1 ::  10% (1 of 10) |#            | Elapsed Time: 0:00:00 ETA:   0:00:00
print i: 0
bar i: 1 :: 100% (10 of 10) |############| Elapsed Time: 0:00:01 Time:  0:00:01
bar i: 2 ::  10% (1 of 10) |#            | Elapsed Time: 0:00:00 ETA:   0:00:00
print i: 1
bar i: 2 :: 100% (10 of 10) |############| Elapsed Time: 0:00:01 Time:  0:00:01
print i: 2

That means that the stdout output is being buffered and printed at the wrong time. This is very similar to a bug in the Jetbrains editors (Pycharm specifically, but they all have it): #115

The good news is, as opposed to the bug in Pycharm, this bug can easily be circumvented. Simply add a sys.stdout.flush() between the print() and the start of the next bar and you're done :)

import sys
import time
import progressbar


for i in range(3):
    sys.stdout.flush()
    for j in progressbar.progressbar(range(10)):
        time.sleep(0.1)
    print('print i: %d' % i)

@Guzzii
Copy link
Author

Guzzii commented Oct 17, 2018

Thanks for looking into the issue @wolph . The trick works for me. It is a lot cleaner now

@wolph wolph closed this as completed Nov 1, 2018
@wolph wolph changed the title Dummy progress bar in nested for loop with print statement Jupyter mixes stdout and stderr output in non-deterministic order Nov 1, 2018
@HDembinski
Copy link

@wolph I just stumbled over the same issue. Why can't progressbar.progressbar call stdout.flush before it is starting to print, to avoid this issue?

@wolph
Copy link
Owner

wolph commented Apr 14, 2020

@HDembinski when stdout redirection is enabled it would do at. If stdout redirection is not enabled the progressbar assumes that it's under your control and won't touch it.

Instead of manually flushing that might be a better solution :)

@HDembinski
Copy link

HDembinski commented Apr 14, 2020

Instead of manually flushing that might be a better solution :)

Sorry, I don't follow what you mean.

I am generally very happy with progressbar, it is well designed and I can see that you care about the details and that things "just work". Perhaps as a workaround for the Jupyter notebook bug progressbar could check sys.stdout, which is an instance of ipykernel.iostream.OutStream in a notebook, and in that case flush automatically?

@wolph
Copy link
Owner

wolph commented Apr 14, 2020

I was replying from my phone so I couldn't post an example. I mean the output redirection: https://github.com/WoLpH/python-progressbar#combining-progressbars-with-print-output

import time
import progressbar

for i in progressbar.progressbar(range(100), redirect_stdout=True):
    print('Some text', i)
    time.sleep(0.1)

Although I suppose detecting for that specific stream would be an option too, I'm not entirely sure which implementation would be best here. Automatically enabling the stdout redirection as mentioned above would be the easiest and probably works well in all cases but I'm a bit worried it can break things for people that also do iostream detection.

At the very least I'll need to make sure people can still disable the behaviour just in case they're somehow dependent on it. Wouldn't be the first time ;)

XKCD springs to mind: https://xkcd.com/1172/
image

@wolph wolph reopened this Apr 14, 2020
@HDembinski
Copy link

Right, as a fellow maintainer I know. Thanks for reopening.

@stale
Copy link

stale bot commented Jun 17, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the no-activity label Jun 17, 2020
@stale stale bot closed this as completed Jun 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants