[Feature request] Memory management and garbage collection

[KexinFeng]

Description

The current memory management struggles in dealing with the following scenario.

try (NDManager manager = NDManager.newBaseManager()) {
  NDArray array;
  NDArray arrayDelta;
  array = manager.create(new float[]{1f, 2f, 3f, 4f}, new Shape(2, 2));
  arrayDelta = manager.create(new float[]{5f, 6f, 7f, 8f}, new Shape(2, 2));

    // inside a loop
    array = NDArrays.concat(new NDList(array, arrayDelta));
}

In one iteration of the concat, the array will be overwritten by the returned array, which will leave the original array orphaned and not recycled. As iteration goes, this will cause memory leak. This can be seen by setting breakpoints and watching manager.resources. It keeps growing while the array before concatenation should be recycled in time.

Another example is the following:

cosSimilarity = topkHiddenStates.batchMatMul(contextHiddenStates.transpose(0, 2, 1));

In this one-line NDArray operation, it adds 2 more ndarrays into the manager, which includes the intermediate array after the transpose. The intermediate array won't be released until the end.

The current solution is someting like below.

try (NDManager manager = NDManager.newBaseManager()) {
  NDArray array;
  NDArray arrayDelta;
  NDArray arrayTmp;
  array = manager.create(new float[]{1f, 2f, 3f, 4f}, new Shape(2, 2));
  arrayDelta = manager.create(new float[]{5f, 6f, 7f, 8f}, new Shape(2, 2));

    // inside a loop
    arrayTmp = NDArrays.concat(new NDList(array, arrayDelta));
    array.close();
    array = arrayTmp;
}

In this solution, manager is not helpful since inside concat, array and arrayTmp will be attached to the same manager. Selectively closing the original array is also inconvenient.

The NDScope in #2321 is not helpful, since it only recycles newly created arrays, but here the request is to recycle the old arrays after generating the concatenated array, which is kept for the next iteration.

try (NDScope scope = new NDScope()) {
     // Create NDArrays
} // NDArrays created in this scope will be closed automatically

---

Ideally, for the NDArray operations, the user should be freed from the memory management.Probably the references below is still relevant to solving this.

#2273

[KexinFeng]

One idea to solve this issue is that, at the end of a NDScope where the release of NDArrays created inside is called, do it selectively. Iterate over the resources under a manager, and release only those orphaned NDArrays. In this way, the temporary memory in NDArrays is recycled all together. But the key is to trace which Resources are orphaned and automatically mark them (reference count them), which is very similar to java GC, and indeterministic.

But this is very much needed in intensive NDArray computation where the above issue is a clear painpoint. Maybe NDScope is a good way to control such garbage collection behaviour only within the array computaiton intenstive part, decided by the user. The techniques in #2273 may be relevant.

[KexinFeng]

This issue can be solved by NDScope:

try(NDScope scope = new NDScope()){
  scope.suppressNotUsedWarning();
  ...
  NDSope.unregister(NDArrays_to_keep)
}

[KexinFeng]

An example of extensive usage of this NDScope to avoid memory leak is in this PR: https://github.com/deepjavalibrary/djl/pull/2637/files
This NDScope is implemented here: #2321