Support CSV output where JSON is allowed #281

davidag · 2019-05-09T19:18:42Z

In response to #256, which seems like an interesting use case to me, I've started to implement CSV format support in the same places where JSON is currently allowed (log/report/aggregate).

Some things are still missing, but I've created the PR to get early feedback (as suggested in the guidelines :).

TODO

MutuallyExclusiveOption prints an invalid error message when specifying an option internal name, which is the case of -j/--json (i.e. format_json).
- Handled in commit 0c7a803
Decide whether --format should be added (as discussed on Support JSON output in report command #102).
- After solving the previous point, it seems OK to have two different options for each of the two formats (i.e. json and csv), given that the plain format is the default and is hidden from the user.
Add tests
Document new command option(s) in cli.py
Add docstrings to new functions in utils.py

Cli options

Output formats for commands log, report and aggregate:

$ watson cmd --csv   # new: outputs a csv text as described below
$ watson cmd --json  # old: outputs a json in the same format as before
$ watson cmd --plain # new: this option is hidden in help/docs
                     # because it's the default output format

Handling mutually exclusive options:

$ watson cmd --csv --json
Error: The following options are mutually exclusive: `--json`, `--csv`

$ watson report --luna --week
Error: The following options are mutually exclusive: `--all`, `--luna`, `--year`, `--week`, `--month`, `--day`

CSV format

Design decisions

The generated output should contain, explicitly or implicitly, the same data available in the plain text format.
Python csv classes are instantiated with the default dialect, i.e. Excel.
Dates and times are formatted in a way that Excel can handle them (yyyy-MM-dd HH:mm:ss according to this link)

Examples

$ watson log --csv

id,start,stop,project,tags
b19d6b,2019-05-02 11:00:00,2019-05-02 13:02:34,project-2,
0ba222,2019-05-02 13:23:15,2019-05-02 14:31:25,project-2,
95c58c,2019-05-02 15:27:26,2019-05-02 20:47:36,project-2,
1f6adf,2019-05-06 15:37:38,2019-05-06 16:07:08,project-1,"tag-1, tag-2"
733381,2019-05-06 23:01:08,2019-05-06 23:26:37,project-3,"tag-3, tag-5"
e33ed7,2019-05-07 11:47:40,2019-05-07 14:06:50,project-3,"tag-3, tag-5"
1b2b89,2019-05-07 18:39:54,2019-05-07 18:57:59,project-4,"tag-5, tag-6, tag-7"
f9beb5,2019-05-07 19:14:14,2019-05-07 19:47:08,project-5,"tag-3, tag-5"
ed494a,2019-05-08 12:45:35,2019-05-08 13:00:37,project-4,"tag-5, tag-6, tag-7"
02da6f,2019-05-09 10:21:27,2019-05-09 13:19:22,project-3,"tag-3, tag-5"
45351e,2019-05-09 16:08:15,2019-05-09 19:23:15,project-3,"tag-3, tag-5"
4616a7,2019-05-09 19:13:15,2019-05-09 19:23:23,project-1,tag-1
8f4121,2019-05-09 19:23:44,2019-05-09 19:23:49,project-1,tag-2
980336,2019-05-09 19:30:16,2019-05-09 19:33:19,project-3,

$ watson report --csv

from,to,project,tag,time
2019-05-02 00:00:00,2019-05-09 23:59:59,project-1,,2383.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-1,tag-1,2378.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-1,tag-2,1775.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-2,,30654.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-3,,32437.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-3,tag-3,32254.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-3,tag-5,32254.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-4,,1987.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-4,tag-5,1987.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-4,tag-6,1987.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-4,tag-7,1987.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-5,,1974.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-5,tag-3,1974.0
2019-05-02 00:00:00,2019-05-09 23:59:59,project-5,tag-5,1974.0

$ watson aggregate --csv

from,to,project,tag,time
2019-05-02 00:00:00,2019-05-02 23:59:59,project-2,,30654.0
2019-05-06 00:00:00,2019-05-06 23:59:59,project-1,,1770.0
2019-05-06 00:00:00,2019-05-06 23:59:59,project-1,tag-1,1770.0
2019-05-06 00:00:00,2019-05-06 23:59:59,project-1,tag-2,1770.0
2019-05-06 00:00:00,2019-05-06 23:59:59,project-3,,1529.0
2019-05-06 00:00:00,2019-05-06 23:59:59,project-3,tag-3,1529.0
2019-05-06 00:00:00,2019-05-06 23:59:59,project-3,tag-5,1529.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-3,,8350.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-3,tag-3,8350.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-3,tag-5,8350.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-4,,1085.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-4,tag-5,1085.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-4,tag-6,1085.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-4,tag-7,1085.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-5,,1974.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-5,tag-3,1974.0
2019-05-07 00:00:00,2019-05-07 23:59:59,project-5,tag-5,1974.0
2019-05-08 00:00:00,2019-05-08 23:59:59,project-4,,902.0
2019-05-08 00:00:00,2019-05-08 23:59:59,project-4,tag-5,902.0
2019-05-08 00:00:00,2019-05-08 23:59:59,project-4,tag-6,902.0
2019-05-08 00:00:00,2019-05-08 23:59:59,project-4,tag-7,902.0
2019-05-09 00:00:00,2019-05-09 23:59:59,project-1,,613.0
2019-05-09 00:00:00,2019-05-09 23:59:59,project-1,tag-1,608.0
2019-05-09 00:00:00,2019-05-09 23:59:59,project-1,tag-2,5.0
2019-05-09 00:00:00,2019-05-09 23:59:59,project-3,,22558.0
2019-05-09 00:00:00,2019-05-09 23:59:59,project-3,tag-3,22375.0
2019-05-09 00:00:00,2019-05-09 23:59:59,project-3,tag-5,22375.0

jmaupetit · 2019-05-17T17:20:45Z

Keep it up @davidag! 🙏

davidag · 2019-05-25T11:59:57Z

@jmaupetit if you give me the okay, I can squash the commits to prepare the merge... or make changes if needed.

tests/test_utils.py

k4nar

Small nitpick, but LGTM otherwise 👍 .

davidag · 2019-06-04T15:51:34Z

@jmaupetit @k4nar I think there's something not working with Python 3.4, it looks related to Arrow compatibility:

ERROR: arrow requires Python '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*' but the running Python is 3.4.6

Maybe Python 3.4 support should be dropped?

k4nar · 2019-06-04T16:19:38Z

@davidag : We should probably born arrow to <0.13.2 for users of python < 3.5 instead. Adding arrow<0.13.2;python_version<"3.5" to the install_requires part of the setup.py should do the trick.

davidag · 2019-06-04T21:14:49Z

@k4nar : I see how that can solve the issue, but I'm not sure I understand why Watson must still support Python 3.4, taking into account this message in travis:

DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).

You can see other Python projects dropping support for Python 3.4 some months ago. For example, Pelican.

If you still want to support it, I suppose a different PR should be created.

jmaupetit · 2019-06-05T09:42:10Z

Good point @davidag! I think we should drop support for python 3.4 in a new PR.

davidag · 2019-06-13T18:18:49Z

@jmaupetit: Rebased, squashed and added a changelog entry. Ready to merge? 🎉

- New option --csv/-s in report, log and aggregate commands, mutually exclusive with --json/-j. - New hidden option --plain to handle the default output format in report, aggregate and log commands (i.e. plain text). This option is not intended to be used by the final user. - Add support in MutuallyExclusiveOption for options with an internal name different from the ones permitted in the command line.

jmaupetit added the feature label May 17, 2019

davidag force-pushed the csv-export branch from 1e71581 to 5fb7df8 Compare May 23, 2019 19:04

davidag changed the title ~~WIP: Support CSV output where JSON is allowed~~ Support CSV output where JSON is allowed May 24, 2019

k4nar reviewed Jun 3, 2019

View reviewed changes

tests/test_utils.py Outdated Show resolved Hide resolved

k4nar reviewed Jun 3, 2019

View reviewed changes

jmaupetit assigned davidag Jun 4, 2019

davidag force-pushed the csv-export branch 2 times, most recently from 847f97a to 9521d7a Compare June 13, 2019 18:14

davidag force-pushed the csv-export branch from 9521d7a to 899f9b8 Compare June 13, 2019 18:52

jmaupetit approved these changes Jun 14, 2019

View reviewed changes

jmaupetit merged commit dc607b6 into jazzband:master Jun 14, 2019

davidag mentioned this pull request Aug 11, 2019

Export to csv? #256

Closed

jmaupetit mentioned this pull request Aug 26, 2019

Bump release to 1.8.0 #303

Merged

davidag deleted the csv-export branch October 16, 2019 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support CSV output where JSON is allowed #281

Support CSV output where JSON is allowed #281

davidag commented May 9, 2019 •

edited

Loading

jmaupetit commented May 17, 2019 •

edited

Loading

davidag commented May 25, 2019

k4nar left a comment

davidag commented Jun 4, 2019

k4nar commented Jun 4, 2019

davidag commented Jun 4, 2019

jmaupetit commented Jun 5, 2019

davidag commented Jun 13, 2019

Support CSV output where JSON is allowed #281

Support CSV output where JSON is allowed #281

Conversation

davidag commented May 9, 2019 • edited Loading

TODO

Cli options

CSV format

Design decisions

Examples

jmaupetit commented May 17, 2019 • edited Loading

davidag commented May 25, 2019

k4nar left a comment

Choose a reason for hiding this comment

davidag commented Jun 4, 2019

k4nar commented Jun 4, 2019

davidag commented Jun 4, 2019

jmaupetit commented Jun 5, 2019

davidag commented Jun 13, 2019

davidag commented May 9, 2019 •

edited

Loading

jmaupetit commented May 17, 2019 •

edited

Loading