Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make DataFrame.to_string output full content by default #28052

Merged
merged 28 commits into from
Sep 16, 2019
Merged

Make DataFrame.to_string output full content by default #28052

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
af444f0
Added a parameter to pass all the way down to specify max_colwidth. N…
Aug 21, 2019
00a43cc
I have threaded the max_colwidth parameter all over the place, but I'm
Aug 21, 2019
1ebf091
Ok, I removed all the deep changes and parameter-passing. Instead, we…
Aug 21, 2019
21bbf64
For some reason, the truncation switches justification if the max_col…
Aug 21, 2019
cfde48e
Added a test to show that this option exists for to_string
Aug 21, 2019
1abc2fa
Shortened one line (split across two). It's hard to actually shorten …
Aug 21, 2019
a1d3832
Shortened all the lines even in the test to comply with PEP8
Aug 21, 2019
3cf9a6a
Adding a newline per suggestion from isort
Aug 22, 2019
762d677
Solved the justify problem, and also added some None value for the ma…
Aug 22, 2019
d64fcb8
Swap out format to be None, ignore justification issues.
Aug 22, 2019
6e792f8
Reformat blac.
Aug 22, 2019
6a5cd97
Merge branch 'master' into issue9784-to-string-truncate-long-strings
Aug 22, 2019
889284f
Merge branch 'master' into issue9784-to-string-truncate-long-strings
Aug 27, 2019
3116ea1
Use the is_nonnegative_int validator for the max_colwidth param. I di…
Aug 27, 2019
840c1a6
Added entry to whatsnew.
Aug 27, 2019
2c68bbc
Fixed formatting with black.
Aug 27, 2019
4e7fe82
Split whatsnew entry, add versionadded, reorder params
Aug 28, 2019
90f0ee0
Remove double line break
Aug 28, 2019
c2b8421
Oops, didn't mean to update this script
Aug 28, 2019
5a8a525
Merge branch 'master' into issue9784-to-string-truncate-long-strings
Aug 28, 2019
da031f2
Correct word in whatsnew.
Aug 30, 2019
20d10b5
Merge branch 'master' into issue9784-to-string-truncate-long-strings
Aug 30, 2019
0f8119e
Update doc/source/whatsnew/v1.0.0.rst
Aug 31, 2019
732c2f4
Resolve conflict in whatsnew
Sep 5, 2019
affef02
Merge suggested edits
Sep 5, 2019
3efa1da
Merge branch 'master' into issue9784-to-string-truncate-long-strings
Sep 6, 2019
6142150
Merge branch 'master' into issue9784-to-string-truncate-long-strings
Sep 14, 2019
b23da91
Resolve conflict correctly.
Sep 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 27 additions & 21 deletions pandas/core/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
import numpy.ma as ma

from pandas._config import get_option

from pandas._libs import algos as libalgos, lib
from pandas.compat import PY36, raise_with_traceback
from pandas.compat.numpy import function as nv
Expand Down Expand Up @@ -637,6 +636,7 @@ def __repr__(self):
max_rows = get_option("display.max_rows")
min_rows = get_option("display.min_rows")
max_cols = get_option("display.max_columns")
max_colwidth = get_option("display.max_colwidth")
show_dimensions = get_option("display.show_dimensions")
if get_option("display.expand_frame_repr"):
width, _ = console.get_console_size()
Expand All @@ -648,6 +648,7 @@ def __repr__(self):
min_rows=min_rows,
max_cols=max_cols,
line_width=width,
max_colwidth=max_colwidth,
show_dimensions=show_dimensions,
)

Expand Down Expand Up @@ -707,11 +708,14 @@ def to_string(
max_cols=None,
show_dimensions=False,
decimal=".",
max_colwidth=9999999,
Copy link
Author

@lshepard lshepard Aug 21, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanted to note - one of the commenters on the issue asks: "With Pandas 0.25.0, setting display.max_colwidth to a large number stops the truncation but when trying to left justify columns with df.to_string(justify='left'), that same display setting somehow pads columns on the left so they are not left aligned. Is there any present way to prevent truncation and get left justified string columns when output to a terminal? I know a pull request is in process but I would like to do this now. Thanks."

I can accomplish this in my testing by setting max_colwidth=0, which switches the padding to left. It is weird, though, that passing justify="left" does not justify it correctly. Seems like maybe a separate bug or one that I could look further into.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand this cause, but I don't think we'll want a workaround like this.

Can you post the output with max_colwidth=0, and how it compares to this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shows how the max_colwidth=0 strangely forces a left justification:

>>> print(df.to_string(max_colwidth=0))
        A       B
0 NaN      NaN   
1 -1.0000  foo   
2 -2.1234  foooo 
3  3.0000  fooooo
4  4.0000  bar

Whereas the behavior in this PR preserves the right justification that is the current default:

>>> print(df.to_string(max_colwidth=99999))
        A       B
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar

And interestingly, passing justify='left' doesn't have an effect:

>>> print(df.to_string(justify='left'))
   A      B      
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar

This is because of these lines in _make_fixed_width:

def _make_fixed_width(
    ...
    max_len = max(adj.len(x) for x in strings)
    ...
    def just(x):
        if conf_max is not None:
            if (conf_max > 3) & (adj.len(x) > max_len):
                x = x[: max_len - 3] + "..."
        return x

    strings = [just(x) for x in strings]
    result = adj.justify(strings, max_len, mode=justify)
    return result

It checks if conf_max > 3 to apply the dot truncation ... so if it's <= 3 then that isn't called. So it's not just 0 but any of 0, 1, 2, or 3 that causes the justification to line up.

I can spend some more time to better understand why this is happening. I agree that we should not rely on some incidentals of the underlying implementation to determine whether to justify the text.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I figured out the issue. There are two lines where format_array is called and the justify parameter is not passed all the way through -- so in some places, the justification is being overridden.

Note that the bug where justification doesn't happen if conf_max < 3 already appears - so I think it can probably be pulled out as a separate PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scratch all that. I re-read the docs and I see that I misinterpreted the justify param, as it only refers to the column headers, not the content. In that regard it is behaving correctly. So I think I'll leave the justification question out of this PR.

line_width=None,
):
"""
Render a DataFrame to a console-friendly tabular output.
%(shared_params)s
max_colwidth : int, optional
Max width to truncate each column in characters. By default, no limit.
TomAugspurger marked this conversation as resolved.
Show resolved Hide resolved
line_width : int, optional
Width to wrap a line in characters.
%(returns)s
Expand All @@ -730,26 +734,28 @@ def to_string(
2 3 6
"""

formatter = fmt.DataFrameFormatter(
self,
columns=columns,
col_space=col_space,
na_rep=na_rep,
formatters=formatters,
float_format=float_format,
sparsify=sparsify,
justify=justify,
index_names=index_names,
header=header,
index=index,
min_rows=min_rows,
max_rows=max_rows,
max_cols=max_cols,
show_dimensions=show_dimensions,
decimal=decimal,
line_width=line_width,
)
return formatter.to_string(buf=buf)
from pandas import option_context
with option_context('display.max_colwidth', max_colwidth):
formatter = fmt.DataFrameFormatter(
self,
columns=columns,
col_space=col_space,
na_rep=na_rep,
formatters=formatters,
float_format=float_format,
sparsify=sparsify,
justify=justify,
index_names=index_names,
header=header,
index=index,
min_rows=min_rows,
max_rows=max_rows,
max_cols=max_cols,
show_dimensions=show_dimensions,
decimal=decimal,
line_width=line_width,
)
return formatter.to_string(buf=buf)

# ----------------------------------------------------------------------

Expand Down
39 changes: 39 additions & 0 deletions pandas/tests/io/formats/test_format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,45 @@ def test_str_max_colwidth(self):
"1 foo bar stuff 1"
)

def test_to_string_truncate(self):
# GH 9784 - dont truncate when calling DataFrame.to_string
df = pd.DataFrame(
[
{
"a": "foo",
"b": "bar",
"c": "let's make this a very VERY long line that is longer "
"than the default 50 character limit",
"d": 1,
},
{"a": "foo", "b": "bar", "c": "stuff", "d": 1},
]
)
df.set_index(["a", "b", "c"])
assert df.to_string() == (
" a b "
" c d\n"
"0 foo bar let's make this a very VERY long line t"
"hat is longer than the default 50 character limit 1\n"
"1 foo bar "
" stuff 1"
)
with option_context("max_colwidth", 20):
# the display option has no effect on the to_string method
assert df.to_string() == (
" a b "
" c d\n"
"0 foo bar let's make this a very VERY long line t"
"hat is longer than the default 50 character limit 1\n"
"1 foo bar "
" stuff 1"
)
assert df.to_string(max_colwidth=20) == (
" a b c d\n"
"0 foo bar let's make this ... 1\n"
"1 foo bar stuff 1"
)

def test_auto_detect(self):
term_width, term_height = get_terminal_size()
fac = 1.05 # Arbitrary large factor to exceed term width
Expand Down