Argument Clinic: support diverting output to buffer, external file, etc

[larryhastings]

BPO	20287
Nosy	@birkenfeld, @pitrou, @larryhastings, @zware, @serhiy-storchaka
Files	larry.clinic.buffer.patch.1.txt larry.clinic.buffer.patch.2.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/larryhastings'
closed_at = <Date 2014-01-18.01:48:31.216>
created_at = <Date 2014-01-17.07:35:58.155>
labels = ['type-feature']
title = 'Argument Clinic: support diverting output to buffer, external file, etc'
updated_at = <Date 2014-01-18.01:48:31.214>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2014-01-18.01:48:31.214>
actor = 'larry'
assignee = 'larry'
closed = True
closed_date = <Date 2014-01-18.01:48:31.216>
closer = 'larry'
components = []
creation = <Date 2014-01-17.07:35:58.155>
creator = 'larry'
dependencies = []
files = ['33503', '33518']
hgrepos = []
issue_num = 20287
keywords = []
message_count = 8.0
messages = ['208325', '208334', '208355', '208361', '208372', '208374', '208378', '208379']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'pitrou', 'larry', 'python-dev', 'zach.ware', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20287'
versions = ['Python 3.4']

[larryhastings]

The long-awaited, highly desirable "buffer" patch is here! With features I figured would never see the light of day.

Changes in this patch:

• New directive "output":
  output <field> <destination>
  redirects a "field" (subsection of Clinic's generated code)
  to an alternate "destination" (someplace besides the output block)
  predefined destinations: block buffer file two-pass suppress

• Alternate use of "output":
  output preset <name>
  redirects all destinations en mass based on a precreated scenario
  predefined presets: original file buffer partial-buffer two-pass

• New directive "destination":
  destination <name> <type> [<filename>]
  creates new destinations for use with "output"
  valid types: buffer file two-pass suppress

• New directive "dump":
  dump <destination>
  dumps the contents of a destination into the current file

• New directive "preserve":
  preserve
  preserves the current output in an output block (used by "file"
  destinations)

• New directive "set":
  set line_prefix <prefix>
  set line_suffix <suffix>

• New output template, uses PyArg_UnpackTuple

• New compact output template, for simple METH_O, only one line

• "args" and "kwargs" variables in C renamed to "_args" and "_kwargs"

Patch is not ready for checkin, as it still needs documentation. But it's ready for code review!

There's a lot of churn in the C files. That's due to the following
changes, in order of frequency:

• "args" -> "_args"
• "kwargs" -> "_kwargs"
• new concise METH_O template
• PyArg_UnpackTuple generator

[serhiy-storchaka]

Can Argument Clinic be simplified when drop support for all alternative outputs besides a side file?

• "args" and "kwargs" variables in C renamed to "_args" and "_kwargs"

Why this is needed? If buildin function has "args" or "kwargs" keyword arguments, C variables can be renamed to args_value and kwargs_value, as for any reserved keywords (e.g. "default", "int"). Other changes look very good, but these are just add code churn.

[larryhastings]

If I removed support for all destinations except files, I might
be able to remove fifty lines or so. However, I'm not planning
on doing that. Does cryptmodule.c, which has exactly one function,
really need a 20-line separate file?

Also, the "file" approach often requires rearranging the file
before it will work, like moving struct declarations (or at
least making forward declarations) before the #include of
the generated file.

The change of "args" to "_args" is me planning ahead a little;
I plan to add the ability to get "args" and "kwargs" passed in
to the impl function, both visibly to the signature, or silently
(for bizarre functions like module_connect() in
Modules/sqlite/connect.c). It would be possible to do this
without renaming them, but then a) it would slightly complicate
the approach I plan to take, and b) it would mean adding those
ugly "_value" suffixes where they aren't strictly necessary.
So I elected to bite the bullet and have the code churn now,
in the middle of this patch that already has some code churn.

However, I've been thinking about it all morning, and maybe
I should just bite the bullet and make the code generator
understand values whose name is different in the parsing and
impl functions.

(By the way, my next major patch, enabling signatures to work
for all the other builtin types besides simple functions,
will have lots of unavoidable code churn. You are forewarned!)

[larryhastings]

I have backed out the "args" -> "_args" change. Not because I'm giving up on it, or that I necessarily think it's a bad idea, but because there
are some changes happening in bpo-20189 that will make it easier / that would conflict if I attacked it now.

The resulting patch is smaller than the first iteration. And since you said it was fine except for that, I assume this patch is fine and can go in?

[serhiy-storchaka]

The patch is too large and I don't very conversant with this code, so I just left several minor comments on Rietveld. At first glance all looks good.

All works with my patches, results are good. But I see that side fail names have extension .clinic.c. The .c.clinic extension would be better.

For now clinic_test is failed:

Error:
Destination does not exist: 'file'

[larryhastings]

I have a fix already in place for the clinic_test.py problem.

I disagree with your proposed "foo.c.clinic" nomenclature, and there are advantages to "foo.clinic.c": double-clicking on them will launch the correct editor, and editors will automatically color them correctly.

We've had this discussion before, and you weren't able to change my mind then, so please give the bikeshedding a rest.

[python-dev]

New changeset 7f6712954d5d by Larry Hastings in branch 'default':
Issue bpo-20287: Argument Clinic's output is now configurable, allowing
http://hg.python.org/cpython/rev/7f6712954d5d

[larryhastings]

Done!