do blocks, anonymous functions, and control flow

[StefanKarpinski]

Now that we have the do-block syntax, one is tempted to write things like this:

function f(dir,file)
  cd(dir) do
    return readall(file)
  end
  error("didn't expect to get here")
end

Of course, since the do-block syntax is just shorthand for passing an anonymous function to cd, this throws an error, rather than returning the contents of "$dir/$file". I would like to keep do-blocks as just a syntactic sugar over anonymous functions. Is there some way we can make these things work more conveniently, or is that just going to cause complications? It's also convenient to use do-blocks for loop-like constructs, at which point one often wants features like break and continue, which raises similar issues.

[StefanKarpinski]

Another more subtle issue that I just encountered is this:

reqs = parse_requires("REQUIRES")
Git.each_submodule(false) do pkg, path, sha1
  reqs = [reqs,parse_requires("$path/REQUIRES")]
end

There's more context, of course, and it occurs in a function body, but basically, this doesn't work because the inner assignment to reqs causes it to be a local variable which then isn't defined on the first pass through.

[JeffBezanson]

As for the first problem: ha! have fun implementing that.

The second problem I don't believe, because variable bindings are inherited by default. I get the following:

bar(f) = f(0)

function qux()
        x = 1
       bar() do y
        x = [x,y]
       end
       x
     end

qux()
2-element Int64 Array:
 1
 0

[JeffBezanson]

I honestly don't understand why we suddenly need these do blocks all over the place. Now you have the option of handling control flow in either the caller or the callee, and it's not clear which one to use. If we want to change control flow to be closure-based, we need to take a good 3 months to redesign a lot of things.

[StefanKarpinski]

Have fun implementing what? I wasn't proposing a particular behavior, just pointing out that this is a potential point of confusion that we should consider. The second one I observed, but apparently it only happens in the repl, not inside a function as I had claimed (the repl is where I observed it, but the code example was in a function body):

julia> x = 1
1

julia> bar() do y
         x = [x,y]
       end
in anonymous: x not defined
 in anonymous at no file:2
 in bar at none:1

[JeffBezanson]

This is the same behavior we've had forever, where globals aren't overwritten inside functions by default. So perhaps do blocks are not just syntactic sugar for functions. In hindsight I can see I should not have added them so hastily. This is a good example of everything starting to fall apart once you start throwing out new features too quickly.

[StefanKarpinski]

The point of opening the issue was to discuss problems raised by do-block syntax. Honestly, I'm not entirely in love with it. Maybe it's a failed experiment, but if that's the case then we need another way to handle things like opening files with a guarantee that they'll get closed again. The fact that do-block syntax raises these issues — which it does in Ruby as well — may be a design smell. If so, let's figure out a better way to do it.

There was the idea for regex matching of using a for loop, as in:

for m = match(r"^(\w+)=(.*)$", line)
  # handle match, doesn't execute if no match
end

If there were else blocks on for loops, then you can also handle non-matching smoothly:

for m = match(r"^(\w+)=(.*)$", line)
  # handle match, doesn't execute if no match
else
  # handle not matching
end

I like this approach a lot for regex matching. It doesn't rely on a closure, inlines very nicely, and the else version relies on a generally useful feature that we wouldn't mind adding anyway. This approach, however, doesn't obviously help in the case of things like open or cd where you want to have guaranteed exit behavior in the case of errors.

For what it's worth, I experimented in the package code with both using do-blocks and using coroutines (Task objects) to do complex iterations and I much prefer using task objects with a for loop. Sure, there's some overhead in switching tasks, but then again, there's overhead in using closures. The coroutines have the clear advantage of looking like control flow and actually just being control flow (so return, break, and continue are non-issues and work as expected).

[StefanKarpinski]

A couple of ideas...

Have something like ensure as in: f = open("file") ensure close(f).

Add onexit(itr) to the iteration protocol and ensure that it is called no matter how a loop terminates, which would allow this as a file open or cd idiom:

for f = open("file")
  # use open file handle f
end

for d = cd("dir")
  # do something in directory
end

This kind of thing would arguably make Julia a for-loop-oriented language (which might not be a bad thing).

[JeffBezanson]

These are good points. I do like for/else and while/else. They also play nice with henchmen unrolling, since both require copying the condition. You can simply make the first copy of the branch point to the else block instead of the end of the loop.

[StefanKarpinski]

So one obvious takeaway is that we ought to have for-else and while-else: #1289.

[nolta]

Have something like ensure as in: f = open("file") ensure close(f).

Sort of like go's defer statement:

http://blog.golang.org/2010/08/defer-panic-and-recover.html

Add onexit(itr) to the iteration protocol and ensure that it is called no matter how a loop terminates, ...

Sort of like python's with statement:

with open(...) as f:
    f.write(...)

which translates roughly as:

f = open(...)
f.__enter__()
try:
    f.write(...)
except e:
    f.__exit__(e)

[StefanKarpinski]

Yes, both are very apt comparisons. More familiar names would, of course, be welcomed (although I'm not sure Go's choice of defer can be considered standard). One argument for having an onexit function as part of the iteration protocol is things like each_line which, when applied to some objects really ought to ensure that a file handle gets closed whenever the loop exits.

[JeffBezanson]

I realize that I like the do syntax when it only needs to be sugar for passing a function to a higher-order function, and doesn't need any magic control flow behavior. The only real problem is that its syntax obscures what is going on, so that somebody might reasonably expect return to return from the outer function. Explicit parentheses and lambdas do have advantages you know :) Maybe a more revealing syntax would be something like

f(x) with function (y,z)
  ...
end

We could also consider adding named blocks so you can return from any block at any point, including nested loops etc.

[StefanKarpinski]

The trouble with that is that the whole point of the do block syntax is that it obscures what's really going on. It almost seems like we want a here doc syntax for function arguments...

[JeffBezanson]

I'm not sure what that syntax proposal means without an example.

Obscuring what's going on doesn't seem to work well, since it makes it hard to predict what happens in edge cases like return, break, and continue without running to the manual.

[StefanKarpinski]

It wasn't even a proposal, just saying that what you're describing with named blocks sounds like here documents.

[StefanKarpinski]

I'm into the idea of a with-like syntax. Of course, in Julia, there would be something like withok and witherr generic functions that you would define methods for and then writing something like

with f = open("file")
  # do stuff with file handle f
end

would get expanded into something like this:

let f = open("file")
  try
    # do stuff with file handle f
  catch err
    witherr(f,err)
  end
  withok(f)
end

The default witherr would call withok and then rethrow the error. My main objection to this whole construct is that it almost wants to be combined with something else — possibly the for loop as I suggested above.

It would also just be generally useful to have something like Go's defer mechanism or D's scope guards. For open/close, those are easy enough:

f = open("file")
defer close(f)
# do stuff with file handle f

But what about changing directories? The code for that is significantly more complicated since the robust implementation opens the current directory, keeps a file handle to it, and later uses fchdir to get back to it by file handle rather than by name (which could have changed or disappeared in the mean time). That's the kind of stuff you really want to hide away in a with-like mechanism, rather than letting the programmer try to write it themselves.

[jsarnoff]

From a "logical construction of software" perspective, having guards ... for/else while/else until/else are guarded block expressions of a kind ... makes designing bad path avoidance and direction good program flow easier. The more capable, concise, consistent and complete guard is enactable as gatekeeper and as guide. Software development is more efficient where tangle avoidance is given consistent expression.

And it is just fine to enfold the capability, as occurs some with be-iteratively/else and do-currently/later.

[jsarnoff]

FYI the clear sense from your discussion is that Julia should ditch the do block and better solve a simpler (not less powerful, less complex and more enabling) intent.

[StefanKarpinski]

Yes, the basic issues with do blocks are:

1. They look like control flow but aren't;
2. They are anonymous functions but don't look like it.

Both of these mismatches cause problems.

[o-jasper]

I use this:

macro with(setting, body)
  w_var = isa(setting,Expr) && is(setting.head,symbol("="))
 #Make a variable if none given.
  setting = w_var ? setting : :($(gensym()) = $setting) 
  assert(length(setting.args)==2, "Cannot set more than one thing at a time.
 (incorrect number of arguments in Expr of setting; $(length(setting.args)))")
  ret= gensym()
  return esc(quote 
              local $setting
              $ret = $body
              no_longer_with($(setting.args[1]), $ret)
             end)
end
#If ret not input argument, defaults to return it.
function no_longer_with{T,With}(with::With, ret::T) 
  no_longer_with(with)
  return ret
end

no_longer_with(stream::IOStream) = close(stream) #!

It is in this file but i haven't pushed it yet.. I have also changed @with_primitive for openGL to use it, you can basically make anything work with it by defining no_longer_with.

readall(file::String) = @with stream = open(file,"r") readall(stream)

Note this has an abstraction leak, it won't call no_longer_with if you return or throw(or exit somehow) in the middle of it. In common lisp i would use unwind-protect to help against that, but not sure how that is implemented.

@with could do cd too, if cd returns something no_longer_with can work with. Alternatively symbol-pairs can be kept that link a function with their @with version.(It does set a global(nonspecial) var though?)

Maybe everything that can be done with a macro, should be? The dropped @ and (implicit)use of else, elseif and implicit begins could signify that it is a 'standard' one, maybe. Looking at it that way, maybe do has a little arbitary reversal of where the macro name is.

[timholy]

If we're re-entering a chaos period, can I nominate this issue as particularly in need of a decision?

[StefanKarpinski]

Agreed.

[vtjnash]

related: #441

[JeffBezanson]

Decision: we will keep it as-is. The reason is that often you want return to return from the inner block, for example in a callback that hangs around for a long time. In such a case, the enclosing function has returned already and cannot be returned from anyway. In the future, I'd like to add the ability to return from any specified block, at which point you'd be able to get the other behavior, so we might as well keep things as they are for now.

[StefanKarpinski]

The other part of this is that we recommend against using do blocks for iteration constructs, in favor of expressing all iteration using for loops. If you don't care too much about performance, you can use a task and get complex iteration behavior; if you do care about performance, you need to introduce an object that supports our start/done/next iteration protocol.

Another thought was that it might be more suggestive of the fact that the do block body is an anonymous function if we required parens around the block arguments:

listen("localhost", 2000) do (sock,status)
  # do something with the socket
end

[vtjnash]

while playing around with some other changes, I came across this syntax as a pretty close approximation to the do-else form (using match as an example):

if match(r"egex", "egex") do m
    println(m)
    return :something
end === nothing
    println(nothing)
end
function Base.match(f, re, str)
     m = match(re, str)
     m !== nothing && return f(m)
     nothing
end