Reimplement with (#4024)

The `with` keyword has been little used in most Pony programs. `with` was 
implemented as "sugar" over the `try/else/then` construct. Over the course of 
time, this became problematic as changes were made to make `try` more friendly. 
However, these `try` changes negatively impacted `with`. Prior to this change, 
`with` had become [barely usable](#3759)
. We've reimplemented `with` to address the usability problems that built up 
over time. `with` is no longer sugar over `try` and as such, shouldn't develop 
any unrelated problems going forward. However, the reimplemented `with` is a 
breaking change.

Because `with` was sugar over `try`, the full expression was `with/else`. Any 
error that occurred within a `with` block would be handled by provided `else`. 
The existence of `with/else` rather than pure `with` was not a principled 
choice. The `else` only existed because it was needed to satisfy error-handling 
in the `try` based implementation. Our new implementation of `with` does not 
have the optional `else` clause. All error handling is in the hands of the 
programmer like it would be with any other Pony construct.

Previously, you would have had:

```pony
with obj = SomeObjectThatNeedsDisposing() do
  // use obj
else
  // only run if an error has occurred
end
```

Now, you would do:

```pony
try
  with obj = SomeObjectThatNeedsDisposing() do
    // use obj
  end
else
  // only run if an error has occurred
end
```

Or perhaps:

```pony
with obj = SomeObjectThatNeedsDisposing() do
  try
    // use obj
  else
    // only run if an error has occurred
  end
end
```

The new `with` block guarantees that `dispose` will be called on all `with` 
condition variables before jumping away whether due to an `error` or a control 
flow structure such as `return`.

This first version of the "new `with`" maintains one weakness that the previous 
implementation suffered from; you can't use an `iso` variable as a `with` 
condition. The following code will not compile:

```pony
use @pony_exitcode[None](code: I32)

class Disposable
  var _exit_code: I32

  new iso create() =>
    _exit_code = 0

  fun ref set_exit(code: I32) =>
    _exit_code = code

  fun dispose() =>
    @pony_exitcode(_exit_code)

actor Main
  new create(env: Env) =>
    with d = Disposable do
      d.set_exit(10)
    end
```

A future improvement to the `with` implementation will allow the usage of `iso` 
variables as `with` conditions.

Internally now, we have a new abstract token `TK_DISPOSING_BLOCK` that is used to implement `with`. The code for it is still very similar to `try` except, it doesn't have an `else` clause. Instead, it has a body and a dispose clause that correspond to the body and then clauses in `try`.

Fixes #3759

.release-notes/4024.md

@@ -0,0 +1,67 @@
+## Reimplement `with`
+
+The `with` keyword has been little used in most Pony programs. `with` was implemented as "sugar" over the `try/else/then` construct. Over the course of time, this became problematic as changes were made to make `try` more friendly. However, these `try` changes negatively impacted `with`. Prior to this change, `with` had become [barely usable](https://github.com/ponylang/ponyc/issues/3759). We've reimplemented `with` to address the usability problems that built up over time. `with` is no longer sugar over `try` and as such, shouldn't develop any unrelated problems going forward. However, the reimplemented `with` is a breaking change.
+
+Because `with` was sugar over `try`, the full expression was `with/else`. Any error that occurred within a `with` block would be handled by provided `else`. The existence of `with/else` rather than pure `with` was not a principled choice. The `else` only existed because it was needed to satisfy error-handling in the `try` based implementation. Our new implementation of `with` does not have the optional `else` clause. All error handling is in the hands of the programmer like it would be with any other Pony construct.
+
+Previously, you would have had:
+
+```pony
+with obj = SomeObjectThatNeedsDisposing() do
+  // use obj
+else
+  // only run if an error has occurred
+end
+```
+
+Now, you would do:
+
+```pony
+try
+  with obj = SomeObjectThatNeedsDisposing() do
+    // use obj
+  end
+else
+  // only run if an error has occurred
+end
+```
+
+Or perhaps:
+
+```pony
+with obj = SomeObjectThatNeedsDisposing() do
+  try
+    // use obj
+  else
+    // only run if an error has occurred
+  end
+end
+```
+
+The new `with` block guarantees that `dispose` will be called on all `with` condition variables before jumping away whether due to an `error` or a control flow structure such as `return`.
+
+This first version of the "new `with`" maintains one weakness that the previous implementation suffered from; you can't use an `iso` variable as a `with` condition. The following code will not compile:
+
+```pony
+use @pony_exitcode[None](code: I32)
+
+class Disposable
+  var _exit_code: I32
+
+  new iso create() =>
+    _exit_code = 0
+
+  fun ref set_exit(code: I32) =>
+    _exit_code = code
+
+  fun dispose() =>
+    @pony_exitcode(_exit_code)
+
+actor Main
+  new create(env: Env) =>
+    with d = Disposable do
+      d.set_exit(10)
+    end
+```
+
+A future improvement to the `with` implementation will allow the usage of `iso` variables as `with` conditions.

pony.g

@@ -99,7 +99,7 @@ nextterm
   | 'while' ('\\' ID (',' ID)* '\\')? rawseq 'do' rawseq ('else' annotatedrawseq)? 'end'
   | 'repeat' ('\\' ID (',' ID)* '\\')? rawseq 'until' annotatedrawseq ('else' annotatedrawseq)? 'end'
   | 'for' ('\\' ID (',' ID)* '\\')? idseq 'in' rawseq 'do' rawseq ('else' annotatedrawseq)? 'end'
-  | 'with' ('\\' ID (',' ID)* '\\')? (withelem (',' withelem)*) 'do' rawseq ('else' annotatedrawseq)? 'end'
+  | 'with' ('\\' ID (',' ID)* '\\')? (withelem (',' withelem)*) 'do' rawseq 'end'
   | 'try' ('\\' ID (',' ID)* '\\')? rawseq ('else' annotatedrawseq)? ('then' annotatedrawseq)? 'end'
   | 'recover' ('\\' ID (',' ID)* '\\')? cap? rawseq 'end'
   | 'consume' cap? term
@@ -115,7 +115,7 @@ term
   | 'while' ('\\' ID (',' ID)* '\\')? rawseq 'do' rawseq ('else' annotatedrawseq)? 'end'
   | 'repeat' ('\\' ID (',' ID)* '\\')? rawseq 'until' annotatedrawseq ('else' annotatedrawseq)? 'end'
   | 'for' ('\\' ID (',' ID)* '\\')? idseq 'in' rawseq 'do' rawseq ('else' annotatedrawseq)? 'end'
-  | 'with' ('\\' ID (',' ID)* '\\')? (withelem (',' withelem)*) 'do' rawseq ('else' annotatedrawseq)? 'end'
+  | 'with' ('\\' ID (',' ID)* '\\')? (withelem (',' withelem)*) 'do' rawseq 'end'
   | 'try' ('\\' ID (',' ID)* '\\')? rawseq ('else' annotatedrawseq)? ('then' annotatedrawseq)? 'end'
   | 'recover' ('\\' ID (',' ID)* '\\')? cap? rawseq 'end'
   | 'consume' cap? term

src/libponyc/ast/ast.c

@@ -943,7 +943,7 @@ ast_t* ast_nearest(ast_t* ast, token_id id)
   return ast;
 }
 
-ast_t* ast_try_clause(ast_t* ast, size_t* clause)
+ast_t* ast_error_handling_clause(ast_t* ast, size_t* clause)
 {
   ast_t* last = NULL;
 
@@ -953,6 +953,7 @@ ast_t* ast_try_clause(ast_t* ast, size_t* clause)
     {
       case TK_TRY:
       case TK_TRY_NO_CHECK:
+      case TK_DISPOSING_BLOCK:
       {
         *clause = ast_index(last);
         return ast;

src/libponyc/ast/ast.h

@@ -106,7 +106,7 @@ bool ast_has_annotation(ast_t* ast, const char* name);
 void ast_erase(ast_t* ast);
 
 ast_t* ast_nearest(ast_t* ast, token_id id);
-ast_t* ast_try_clause(ast_t* ast, size_t* clause);
+ast_t* ast_error_handling_clause(ast_t* ast, size_t* clause);
 
 ast_t* ast_parent(ast_t* ast);
 ast_t* ast_child(ast_t* ast);

src/libponyc/ast/frame.c

@@ -105,6 +105,10 @@ bool frame_push(typecheck_t* t, ast_t* ast)
       t->frame->try_expr = ast;
       break;
 
+    case TK_DISPOSING_BLOCK:
+      pop = push_frame(t);
+      break;
+
     case TK_RECOVER:
       pop = push_frame(t);
       t->frame->recover = ast;

src/libponyc/ast/lexer.c

@@ -308,6 +308,8 @@ static const lextoken_t abstract[] =
 
   { "annotation", TK_ANNOTATION },
 
+  { "disposingblock", TK_DISPOSING_BLOCK },
+
   { "\\n", TK_NEWLINE },
   {NULL, (token_id)0}
 };

src/libponyc/ast/parser.c

@@ -897,24 +897,14 @@ DEF(withexpr);
   WHILE(TK_COMMA, RULE("with expression", withelem));
   DONE();
 
-// WITH [annotations] withexpr DO rawseq [ELSE annotatedrawseq] END
-// =>
-// (SEQ
-//   (ASSIGN (LET $1 initialiser))*
-//   (TRY_NO_CHECK
-//     (SEQ (ASSIGN idseq $1)* body)
-//     (SEQ (ASSIGN idseq $1)* else)
-//     (SEQ $1.dispose()*)))
-// The body and else clause aren't scopes since the sugar wraps them in seqs
-// for us.
+// WITH [annotations] withexpr DO rawseq END
 DEF(with);
   PRINT_INLINE();
   TOKEN(NULL, TK_WITH);
   ANNOTATE(annotations);
   RULE("with expression", withexpr);
   SKIP(NULL, TK_DO);
   RULE("with body", rawseq);
-  IF(TK_ELSE, RULE("else clause", annotatedrawseq));
   TERMINATE("with expression", TK_END);
   DONE();
 

src/libponyc/ast/token.h

@@ -260,6 +260,8 @@ typedef enum token_id
 
   TK_ANNOTATION,
 
+  TK_DISPOSING_BLOCK,
+
   // Pseudo tokens that never actually exist
   TK_NEWLINE,  // Used by parser macros
   TK_FLATTEN,  // Used by parser macros for tree building

src/libponyc/ast/treecheckdef.h

@@ -141,11 +141,11 @@ RULE(compile_error,
 GROUP(expr,
   local, binop, isop, assignop, asop, tuple, consume, recover, prefix, dot,
   tilde, chain, qualify, call, ffi_call, match_capture,
-  if_expr, ifdef, iftypeset, whileloop, repeat, for_loop, with, match, try_expr,
-  lambda, barelambda, array_literal, object_literal, int_literal, float_literal,
-  string, bool_literal, id, rawseq, package_ref, location,
-  this_ref, ref, fun_ref, type_ref, field_ref, tuple_elem_ref, local_ref,
-  param_ref);
+  if_expr, ifdef, iftypeset, whileloop, repeat, for_loop, with,
+  disposing_block, match, try_expr, lambda, barelambda, array_literal,
+  object_literal, int_literal, float_literal, string, bool_literal, id, rawseq,
+  package_ref, location, this_ref, ref, fun_ref, type_ref, field_ref,
+  tuple_elem_ref, local_ref, param_ref);
 
 RULE(local,
   HAS_TYPE(type)
@@ -338,10 +338,15 @@ RULE(for_loop,
 RULE(with,
   HAS_TYPE(type)
   CHILD(expr)    // With variable(s)
-  CHILD(rawseq)  // Body
-  CHILD(rawseq, none), // Else
+  CHILD(rawseq),  // Body
   TK_WITH);
 
+RULE(disposing_block,
+  HAS_TYPE(type)
+  CHILD(seq) // body
+  CHILD(seq), // dispose
+  TK_DISPOSING_BLOCK);
+
 RULE(match,
   IS_SCOPE
   HAS_TYPE(type)