How to properly free model resources

[sc12752]

Hi,

I want to validate a number of models and pick the best one, here's how my scala code looks like:

lazy val features: NDList
val classifier: AbstractBlock
val parentDir: File

private def evaluateModel(subdirectory: File) =
  Using(Model newInstance "TimeSeriesClassifier") { model =>
    val path = Path.of(subdirectory.getAbsolutePath)
    model.setBlock(classifier)
    model.load(path)

    Using(this getPredictor model) { predictor =>
      val predictions = predictor.predict(features).head
      (doEvaluate(predictions, subdirectory), subdirectory)
    }
  }

def bestModelFile: (Double, File) =
  parentDir.listFiles.toList
    .filter(_.isDirectory)
    .map(evaluateModel)
    .map(_.get.get)
    .maxBy(_._1)

Basically evaluateModel function gets called many times and each time it loads a new model, does a prediction and then closes model and predictor (Using in scala is specifically for this purpose and is similar to how try block is used in java).

The problem is that memory usage grows very fast and program soon crashes with out of memory error, it looks like each model resources are kept in place despite it being closed on each method call. Any hints on how this can be fixed?

Thanks!

[sc12752]

I was able to pin it down further, here's a simplified code:

    val model = Model.newInstance("TimeSeriesClassifier")
    val path = Path.of(dir + "200/")
    model.setBlock(classifier)
    model.load(path)

    val predictor = model.newPredictor(new NoopTranslator, Device.cpu)
    val features = list.get(0).toType(DataType.FLOAT32, false).get("-40:").asNDList
    predictor.predict(features)
    predictor.predict(features)
    predictor.predict(features)
    predictor.predict(features)
    predictor.predict(features)
    predictor.predict(features)
    predictor.predict(features)
    predictor.close()
    model.close()

    Thread.sleep(10000)

That row of predictor.predict adds about 1GB of memory each and then memory stays filled despite closing statements.

[sc12752]

Turned out I should also close a manager which is used to create a features NDList. My solution is to create a submanager for each iteration and then close it once prediction is obtained.

[zachgk]

Right. When doing prediction with an NDList input or output, the intermediate computations for executing the model will typically get attached to the same manager as the input NDList. So, you would need to create an iteration submanager to close like you are doing. You can also switch to using a different input/output type like float[] and then the memory management is mostly automatic. You can find more details in our memory management guide