Function composition and negation, take 2 #17184

andyferris · 2016-06-29T04:13:12Z

Something strange happened to my previous PR #17119, and github closed it.

This is the basic version with just an anonymous function version of ∘ and a definition of function negation. This is quite related to #17155.

@StefanKarpinski feel free to just take docstrings and tests (in general - is it better to make a PR to your PR?)

tkelman · 2016-06-29T04:16:56Z

base/operators.jl

 |>(x, f) = f(x)

+"""
+    ∘(f, g)
+    f ∘ g


would need to put the signatures into the RST docs so make docs populates the docstrings there, see details in contributing.md

I added .. function:: ∘(f, g) to base.rst and, although it worked for .. function:: !(f::Function), I get the following:

WARNING: missing docs for signature: ∘(f, g)

Is this something to do with unicode operators?

@tkelman (not sure if you'd get auto-pinged or not, here)

Yes, no need to explicitly ping me if I've already commented on this issue. It may be a unicode issue where you might have to explicitly declare unicode operators somewhere for RST/Sphinx (I'm not sure where that is though), or it may have to do with the separate signatures being on multiple lines.

OK, yes it is the latter

That's a little unfortunate though. Infix operators should allow an infix form. Even typing ?<: in the REPL gives help for <:(T1, T2) - have you ever seen a single line of Julia code written like that??

At least it shows you that operators are just standard functions (OK, not true for <:) and that you can add a method using the standard syntax. But we should definitely improve these docstrings by adding the infix forms on the second line at the minimum.

@andyferris did you include the entire signature, i.e.

.. function:: ∘(f, g) f ∘ g

or just the first line .. function:: ∘(f, g)? Signature matching requires the entire first code block, not just the first line of it.

TotalVerb · 2016-06-29T04:42:16Z

How is performance? Does the splatting penalty apply here?

Edit: Yikes, it's not good at all. This definition helps a lot:

function ∘(f, g)
    h(x) = f(g(x))
    h(xs...) = f(g(xs...))
    h
end

however it probably slows down ∘ itself.

andyferris · 2016-06-29T04:53:55Z

@TotalVerb Good question! Simple check says ~~"no"~~"yes" - splatting penalty is applying. @StefanKarpinski do you know where we are up to here? I have seen some issues that discuss this and I thought there was a simple fix in place for f(x...) = g(x...) patterns (although this is epsilon more complicated than that).

andyferris · 2016-06-29T05:05:19Z

Indeed, if we can't work around it, then maybe we have to follow @TotalVerb's suggestion until the splatting is fixed. I would suggest generating versions up to MAX_TUPLE_SPLAT (currently 16), for consistency.

andyferris · 2016-06-29T05:08:36Z

however it probably slows down ∘ itself.

A functor approach might help with that... define 17 methods globally on calling a ComposedFunction{F,G} and not generate them every time someone calls ∘.

TotalVerb · 2016-06-29T05:14:30Z

I don't like calling these "functors", especially since (from the user's perspective) they're not meaningfully different from plain old closures.

I guess you mean something like this? I did not get a chance to look at your last PR, but I assume it was a similar approach?

immutable ComposedFunction{F,G} <: Function
    f::F
    g::G
end

This is basically the same as the autogenerated closure type, but more flexible I suppose.

TotalVerb · 2016-06-29T05:25:27Z

Actually, since h is a closure, the methods should already be added only once—not once on each call—so the functor should be unnecessary. Indeed, checking code_native one observes that ∘ is compiled into a no-op.

andyferris · 2016-06-29T05:51:38Z

OK, yes, functor is a strange name. I meant exactly the thing your wrote, ComposedFunction.

I agree the h as a closure will work fast and "only be constructed once per composed function". I was suggesting that we only define the 17 splatting methods "once per session of Julia".

I actually don't care if there is a once-off overhead to ∘. Just one more reason to fix the splatting penalty!

TotalVerb · 2016-06-29T05:56:51Z

@andyferris What I meant was that my assessment was incorrect, and there is no such cost when creating a composed function, because methods are on the (abstract) closure type. So the performance of a "functor" is equivalent to the performance of a closure.

Decided to make up to 16 inputs fast, to match MAX_TUPLE_SPLAT. Conveniently, closures also allow for introspection of f and g.

andyferris · 2016-06-29T07:22:12Z

Hmm... its a bit uglier, but we now have no splatting penalty!

I just discovered closures let us have access to the two things we are composing, allowing for introspection. Cool!

(Closures also seem to be a type of "generated type", in the sense the field names and number depend on how you define them. Super cool!)

nalimilan · 2016-06-29T07:34:47Z

base/operators.jl

 """
-!(f::Function) = (x...)->!f(x...)
+function !(f::Function)


Can't you just define this as ! ∘ f to avoid repeating the same machinery as above?

I would like to, and I tried that first, but I got some kind of recursion when I tried to compile Julia. I didn't have time to investigate the cause of that yet, so I just used copy-paste.

I'm pretty sure that's because ! ∘ f doesn't parse correctly. We probably need (!) ∘ f. This is a little ugly, which makes me like !f syntax a little more.

That worked... thanks!

andyferris · 2016-06-30T03:33:41Z

I'm pretty sure that's because ! ∘ f doesn't parse correctly. We probably need (!) ∘ f. This is a little ugly, which makes me like !f syntax a little more.

We will probably see this with other operators. This is an argument for even more overloads of operators on functions, as in #17155... it would even suggest doing it for every operator. Not sure if I like that or not though!

EDIT: or at least every unary operator, since composition is more complex for binary.

mweastwood · 2016-06-30T05:57:34Z

test/operators.jl

@@ -55,3 +55,6 @@ let xs = [[i:i+4;] for i in 1:10]
        @test max(xs[1:n]...) == [n:n+4;]
    end
 end
+
+@test ((x -> x+1) ∘ (x _> 2x))(5) == 11


typo @test ((x -> x+1) ∘ (x -> 2x))(5) == 11

andyferris · 2016-07-07T12:56:02Z

@StefanKarpinski @JeffBezanson Where are we on this, and what is the probability of 0.5 inclusion?

Between here and #17155, the ∘ seems pretty well received. There are some contentions with unary operators like ! and much, much stronger contentions with binary operators like &, on the basis of not knowing where to draw the line on auto-composition.

We could maybe try ! as an experiment in 0.5? Or drop it completely?

tkelman · 2016-07-07T20:54:23Z

base/operators.jl

+"""
+    ∘(f, g)
+
+Creates a composition of two functions (or functor objects), such that


this pr doesn't include functor objects any more, does it?

No, it uses closures. But ∘ accepts functors (any callable) as arguments.

I'll edit it to make it clear.

oxinabox · 2016-08-29T11:00:55Z

I noted today a place I wanted to used this: Eager-ifying lazy operations

function lazyload_semcor(tagdir_path::AbstractString)
    Task(...)
end

load_semcor(tagdir_path::AbstractString) = collect(lazyload_semcor(tagdir_path))

This would become:

load_semcor = collect∘lazyload_semcor

Which is nice.

andyferris · 2016-08-29T11:24:40Z

Yes, that's right. I quite enjoy using these already in our CoordinateTransformations for similar reasons. You gain a lot of control on how things are done with little user effort.

The reason my original PR had a ComposedFunction object was to allow for even more in-depth introspection where composing certain things together might do something non-trivial - an example of the top of my head is map ∘ reduce -> mapreduce (because they take multiple arguments that doesn't work as I wrote it, obviously, but you can see we can skip an allocation and go straight to iteration - in other words going the opposite way you said, from eager to lazy, but only where appropriate).

tkelman · 2016-12-31T03:30:46Z

closing in favor of #17155

function negation and composition: !f, f∘g

dc8e1b5

This was referenced Jun 29, 2016

PR: Function composition and negation (a.k.a. "It's like a category, man!") #17119

Closed

function composition (f∘g) and negation (!f) #17155

Merged

andyferris force-pushed the function-compose branch from 287c228 to 992facd Compare June 29, 2016 04:16

tkelman reviewed Jun 29, 2016
View reviewed changes

Added docstrings and tests for compositions

bda9052

andyferris force-pushed the function-compose branch from 992facd to bda9052 Compare June 29, 2016 04:32

rst for docs

786c579

Changed to closure approach: no splatting penalty

21fcc91

Decided to make up to 16 inputs fast, to match MAX_TUPLE_SPLAT. Conveniently, closures also allow for introspection of f and g.

nalimilan reviewed Jun 29, 2016
View reviewed changes

Simplified definition of function negation

e6a69b7

mweastwood reviewed Jun 30, 2016
View reviewed changes

Fixed typo in test

e3d9579

tkelman reviewed Jul 7, 2016
View reviewed changes

Clarify docstring

b95404a

andyferris mentioned this pull request Aug 28, 2016

Taking vector transposes seriously #4774

Closed

oxinabox mentioned this pull request Oct 1, 2016

Function chaining #5571

Closed

tkelman mentioned this pull request Oct 4, 2016

Function composition operator? #18783

Closed

andyferris mentioned this pull request Nov 4, 2016

Notation for lazy map #19198

Open

tkelman closed this Dec 31, 2016

JeffFessler mentioned this pull request Oct 15, 2019

Enable multiple function composition in prefix form #33568

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Function composition and negation, take 2 #17184

Function composition and negation, take 2 #17184

andyferris commented Jun 29, 2016

tkelman Jun 29, 2016

andyferris Jun 29, 2016

andyferris Jun 29, 2016

tkelman Jun 29, 2016

andyferris Jun 29, 2016

andyferris Jun 29, 2016

nalimilan Jun 29, 2016

MichaelHatherly Jun 29, 2016

TotalVerb commented Jun 29, 2016 •

edited

Loading

andyferris commented Jun 29, 2016 •

edited

Loading

andyferris commented Jun 29, 2016

andyferris commented Jun 29, 2016

TotalVerb commented Jun 29, 2016 •

edited

Loading

TotalVerb commented Jun 29, 2016 •

edited

Loading

andyferris commented Jun 29, 2016

TotalVerb commented Jun 29, 2016

andyferris commented Jun 29, 2016

nalimilan Jun 29, 2016

andyferris Jun 29, 2016

TotalVerb Jun 30, 2016 •

edited

Loading

andyferris Jun 30, 2016 •

edited

Loading

andyferris commented Jun 30, 2016 •

edited

Loading

mweastwood Jun 30, 2016

andyferris Jun 30, 2016

andyferris commented Jul 7, 2016

tkelman Jul 7, 2016

andyferris Jul 7, 2016

andyferris Jul 7, 2016

oxinabox commented Aug 29, 2016 •

edited

Loading

andyferris commented Aug 29, 2016

tkelman commented Dec 31, 2016

Function composition and negation, take 2 #17184

Function composition and negation, take 2 #17184

Conversation

andyferris commented Jun 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TotalVerb commented Jun 29, 2016 • edited Loading

andyferris commented Jun 29, 2016 • edited Loading

andyferris commented Jun 29, 2016

andyferris commented Jun 29, 2016

TotalVerb commented Jun 29, 2016 • edited Loading

TotalVerb commented Jun 29, 2016 • edited Loading

andyferris commented Jun 29, 2016

TotalVerb commented Jun 29, 2016

andyferris commented Jun 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TotalVerb Jun 30, 2016 • edited Loading

Choose a reason for hiding this comment

andyferris Jun 30, 2016 • edited Loading

Choose a reason for hiding this comment

andyferris commented Jun 30, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyferris commented Jul 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oxinabox commented Aug 29, 2016 • edited Loading

andyferris commented Aug 29, 2016

tkelman commented Dec 31, 2016

TotalVerb commented Jun 29, 2016 •

edited

Loading

andyferris commented Jun 29, 2016 •

edited

Loading

TotalVerb commented Jun 29, 2016 •

edited

Loading

TotalVerb commented Jun 29, 2016 •

edited

Loading

TotalVerb Jun 30, 2016 •

edited

Loading

andyferris Jun 30, 2016 •

edited

Loading

andyferris commented Jun 30, 2016 •

edited

Loading

oxinabox commented Aug 29, 2016 •

edited

Loading