[Meta] Generics/Static early symbol resolution

[mratsim]

I am starting to lose track of all bug reports linked to generics and static early symbol resolution.

Context

Quoting @Araq

The pre-instantiation lookup is required to implement the proper "open vs closed symbol lookup" rules.

http://nim-lang.org/docs/manual.html#generics-symbol-lookup-in-generics

It's a tough issue, maybe we need to weaken the prepass if a 'this' parameter is used.

Impact

• .this pragma doesn't work with generics #5053: .this pragma doesn't work with generics
• generic object with generic field evaluated too early #5540: generic object with generic field evaluated too early
• List comprehensions do not work with generic parameter #5707: List comprehensions do not work with generic parameter (needs mixin)
• Generics proc + macros: identifier resolution happens before macros #6387: Generics proc + macros: identifier resolution happens before macros
• mixin + block expression: works in Generics, not in "normal" proc #7385: mixin + block expression: works in Generics, not in "normal" proc
• {.this: self.} does not work with Static[T] #7618: {.this: self.} does not work with Static[T]
• strformat doesn't work properly inside generics #7632: strformat doesn't work properly inside generics and templates
• when in generic should fail earlier #8603: when should check the code at the same depth like if
• Usability: {.experimental: “ForLoopMacros”.} is required at all macro callsites #8676: Usability: {.experimental: “ForLoopMacros”.} is required at all macro callsites
• Documentation with when branch is missing field types #9899: Documentation with when branch is missing field types
• The async pragma doesn't work properly with a nested proc inside a generic function #10818: The async pragma doesn't work properly with a nested proc inside a generic function
• untyped macro nested in template: getting sym instead of idents #11091: [template] untyped macro nested in template: getting sym instead of idents
• Invoking generic procedure that uses varargs implicit conversion acts incorrectly #11154: Invoking generic procedure that uses varargs implicit conversion acts incorrectly
• for loop invoked from generic procedure defined in another module cant find items iterator #11167: Template with for loop invoked from generic procedure defined in another module cant find items iterator
• symbol resolution issues when ambiguous call happens in generic proc #11188: symbol resolution issues when ambiguous call happens in generic proc
• Generics "sandwiched" between two modules don't mixin their scope symbols properly #11225: Generics "sandwiched" between two modules don't mixin their scope symbols properly

Solution

Most advanced discussion is in #8603
Quoting @zah

Er, it's a bug and a rather simple one. The generic pre-pass in semgnrc.nim have to examine the when condition expressions in the same way it examines all other expressions.

The fix I imagine is that you should just call semGenericStmt on the when condition expression (n.sons[0])

I'll have to look at some specific examples. Inside the compiler, we keep track of which generic symbols are in "unresolved" state and we need to add guards in the code detecting such symbols in many places. I suspect something similar is going on here as well.

[mratsim]

Relevant discussion and planning to tackle the issue: https://irclogs.nim-lang.org/09-05-2019.html#11:12:59

[mratsim]

Another instance of the bug affecting nimpy:

• nimpy cannot be used in generic procs yglukhov/nimpy#153
• https://forum.nim-lang.org/t/6321#38994

You cannot use the natural numpy syntax (or nimpy at all until we have a workaround) in generic procs due to early symbol resolution:

import ../src/arraymancer
import nimpy
import nimpy/raw_buffers
import sequtils

proc `+`[T](p: ptr T, val: int) : ptr T {.inline.}=
  cast[ptr T](cast[uint](p) + cast[uint](val * sizeof(T)))

proc pyBufToTensor[T](ndArray: PyObject): Tensor[T]=
  # Convert PyObject to RawPyBuffer
  var aBuf: RawPyBuffer
  ndArray.getBuffer(aBuf, PyBUF_WRITABLE or PyBUF_ND)
  aBuf.release()

  # Copy buffer into Tensor
  var shape: seq[int]
  for i in 0..<aBuf.ndim:
    shape.add((aBuf.shape+i)[])

  # Get memory asdress of buffer
  var bufPtr = cast[ptr UncheckedArray[T]](aBuf.buf)
  # Get underlying data buffer of Tensor
  result = newTensor[T](shape)
  var tensorDataPtr = cast[ptr UncheckedArray[T]](result.get_data_ptr)
  # Copy Buffer into Tensor
  var length = shape.foldl(a*b)
  copyMem(tensorDataPtr, bufPtr, length*sizeof(T))

let np = pyImport("numpy")

proc nimfftshift[T](t: Tensor[T]): Tensor[T]=
  ## Reshape PyObject to Arraymancer Tensor
  #var ndArray = wrappy.w_fftshift(t.toSeq, t.shape.toSeq)
  var shape = np.`array`(t.shape.toSeq)
  var ndArray = np.`array`(t.toSeq)
  ndArray = np.reshape(ndArray, shape)
  ndArray = np.fft.fftshift(ndArray)

  # Convert RawPyBuffer to Tensor
  result = pyBufToTensor[T](ndArray)

proc main() =
  let a = randomTensor([3, 3], 3).asType(float32)
  echo a
  let b = nimfftshift(a)
  echo b

main()

raises semcheck issue:

proc reshape(t: Tensor; new_shape: MetadataArray): Tensor
  first type mismatch at position: 1
  required type for t: Tensor
  but expression 'np' is of type: PyObject
proc reshape(t: Tensor; new_shape: varargs[int]): Tensor
  first type mismatch at position: 1
  required type for t: Tensor
  but expression 'np' is of type: PyObject
proc reshape[TT](a: Variable[TT]; shape: MetadataArray): Variable[TT]
  first type mismatch at position: 1
  required type for a: Variable[reshape.TT]
  but expression 'np' is of type: PyObject
proc reshape[TT](a: Variable[TT]; shape: varargs[int]): Variable[TT]
  first type mismatch at position: 1
  required type for a: Variable[reshape.TT]
  but expression 'np' is of type: PyObject

expression: reshape(np, ndArray, shape)

while this compiles

import ../src/arraymancer
import nimpy
import nimpy/raw_buffers
import sequtils

proc `+`[T](p: ptr T, val: int) : ptr T {.inline.}=
  cast[ptr T](cast[uint](p) + cast[uint](val * sizeof(T)))

proc pyBufToTensor[T](ndArray: PyObject): Tensor[T]=
  # Convert PyObject to RawPyBuffer
  var aBuf: RawPyBuffer
  ndArray.getBuffer(aBuf, PyBUF_WRITABLE or PyBUF_ND)
  aBuf.release()

  # Copy buffer into Tensor
  var shape: seq[int]
  for i in 0..<aBuf.ndim:
    shape.add((aBuf.shape+i)[])

  # Get memory asdress of buffer
  var bufPtr = cast[ptr UncheckedArray[T]](aBuf.buf)
  # Get underlying data buffer of Tensor
  result = newTensor[T](shape)
  var tensorDataPtr = cast[ptr UncheckedArray[T]](result.get_data_ptr)
  # Copy Buffer into Tensor
  var length = shape.foldl(a*b)
  copyMem(tensorDataPtr, bufPtr, length*sizeof(T))

let np = pyImport("numpy")

proc nimfftshift(t: Tensor[float32]): Tensor[float32]=
  ## Reshape PyObject to Arraymancer Tensor
  #var ndArray = wrappy.w_fftshift(t.toSeq, t.shape.toSeq)
  var shape = np.`array`(t.shape.toSeq)
  var ndArray = np.`array`(t.toSeq)
  ndArray = np.reshape(ndArray, shape)
  ndArray = np.fft.fftshift(ndArray)

  # Convert RawPyBuffer to Tensor
  result = pyBufToTensor[float32](ndArray)

proc main() =
  let a = randomTensor([3, 3], 3).asType(float32)
  echo a
  let b = nimfftshift(a)
  echo b

main()

[Araq]

Well this is not all "the same bug". There are different bugs, some of which are closed already. This is not actionable. It's not "early symbol resolution" either, the symbol resolution has 2 steps and the instantation scope is considered too for mixin symbols. And frankly, the fact that nimpy uses experimental dot operators makes it all even less actionable.

[Araq]

I might as well call this issue "Different DSLs don't compose well" or "untyped parameters must die". We need a decent problem analysis and not conflate everything under the "Generics" label. And yes, different DSLs tend to not compose well, it's a tough open research problem.

[bung87]

update
the long list remained opened items:

• Documentation with when branch is missing field types #9899
• untyped macro nested in template: getting sym instead of idents #11091
• Invoking generic procedure that uses varargs implicit conversion acts incorrectly #11154
• for loop invoked from generic procedure defined in another module cant find items iterator #11167