Fix double accounting of memory when spilling aggregation is used #413

sopel39 · 2019-03-08T15:26:50Z

No description provided.

findepi

(everything except last commit) lgtm % minor comments

findepi · 2019-03-11T15:04:45Z

presto-main/src/main/java/io/prestosql/operator/HashAggregationOperator.java

@@ -272,6 +273,7 @@ public OperatorFactory duplicate()
    private final HashCollisionsCounter hashCollisionsCounter;

    private HashAggregationBuilder aggregationBuilder;
+    private LocalMemoryContext aggregationBuilderMemoryContext;


memoryContext?

findepi · 2019-03-11T15:08:58Z

presto-main/src/main/java/io/prestosql/operator/HashAggregationOperator.java

@@ -368,6 +370,10 @@ public void addInput(Page page)
        if (aggregationBuilder == null) {
            // TODO: We ignore spillEnabled here if any aggregate has ORDER BY clause or DISTINCT because they are not yet implemented for spilling.
            if (step.isOutputPartial() || !spillEnabled || hasOrderBy() || hasDistinct()) {
+                aggregationBuilderMemoryContext = operatorContext.localUserMemoryContext();
+                if (useSystemMemory) {
+                    aggregationBuilderMemoryContext = operatorContext.localSystemMemoryContext();


Why not pass MemoryContext into InMemoryHashAggregationBuilder?

Or, maybe better, why not have this assignment in the constructor?
As it is now, it looks like creating a Mem Context multiple times (which i think is not really the case).

Why not pass MemoryContext into InMemoryHashAggregationBuilder?

I tried that approach, but since InMemoryHashAggregationBuilder is used within SpillableHashAggregationBuilder it cannot account memory on it's own. This is because SpillableHashAggregationBuilder changes memory pools from revocable to user and InMemoryHashAggregationBuilder doesn't (and shouldn't) know about this flip.
Ideally (TODO), SpillableHashAggregationBuilder should use underlying accumulators and GroupByHash directly and omit InMemoryHashAggregationBuilder.

Or, maybe better, why not have this assignment in the constructor?

That makes sense.

findepi · 2019-03-11T15:10:19Z

presto-main/src/main/java/io/prestosql/operator/HashAggregationOperator.java

@@ -510,6 +522,10 @@ private void closeAggregationBuilder()
            // aggregationBuilder.close() will release all memory reserved in memory accounting.
            // The reference must be set to null afterwards to avoid unaccounted memory.
            aggregationBuilder = null;
+            if (aggregationBuilderMemoryContext != null) {


it can't be null at this point

findepi · 2019-03-11T15:16:20Z

presto-main/src/main/java/io/prestosql/operator/HashAggregationOperator.java

-                        true,
-                        useSystemMemory);
+                        () -> {
+                            aggregationBuilderMemoryContext.setBytes(((InMemoryHashAggregationBuilder) aggregationBuilder).getSizeInMemory());


InMemoryHashAggregationBuilder instance gets a callback in its constructor, but the callback refers to that instance. Should InMemoryHashAggregationBuilder use this callback in the ctor to update initial memory occupancy, this would fail.
To fix this, UpdateMemory should take long bytes parameter.

To fix this, UpdateMemory should take long bytes parameter.

GroupByHash would need to be passed LocalMemoryContext as an argument then. I also tried this approach, but it explodes the changes and still doesn't nicely solve problem of flipping memory from revocable to user (e.g: GroupByHash would need to keep getEstimatedSize method, but also accept LocalMemoryContext as arg).

Objects should either:

expose it's memory usage via getEstimatedSize method (if they don't "own" the memory)

account for memory themselves (via LocalMemoryContext). In this case AggregatedMemoryContext should be passed to constructor of such objects.

Mixing both usage patterns is confusing.

findepi · 2019-03-11T15:33:57Z

presto-main/src/main/java/io/prestosql/operator/HashAggregationOperator.java

@@ -510,6 +522,10 @@ private void closeAggregationBuilder()
            // aggregationBuilder.close() will release all memory reserved in memory accounting.
            // The reference must be set to null afterwards to avoid unaccounted memory.
            aggregationBuilder = null;
+            if (aggregationBuilderMemoryContext != null) {
+                aggregationBuilderMemoryContext.setBytes(0);


You're zeroing memory contexts at the end of this method.
Either add operatorContext.localSystemMemoryContext().setBytes(0); there (instead of this line), or remove the all other .setBytes(0) calls there. Or explain :)

True. Didn't notice that.
Just memoryContext.setBytes(0); at end of this method is enough

...n/src/main/java/io/prestosql/operator/aggregation/builder/MergingHashAggregationBuilder.java

...src/main/java/io/prestosql/operator/aggregation/builder/SpillableHashAggregationBuilder.java

findepi · 2019-03-11T15:42:20Z

...src/main/java/io/prestosql/operator/aggregation/builder/SpillableHashAggregationBuilder.java

+        // if before building result from hashAggregationBuilder we would convert it to "read only" version.
+        // Read only version of GroupByHash from hashAggregationBuilder could be compacted by dropping
+        // most of it's field, freeing up some memory that could be used for sorting.
+        return hashAggregationBuilder.getSizeInMemory() + hashAggregationBuilder.getGroupIdsSortingSize();


Why hashAggregationBuilder.getSizeInMemory() does not include hashAggregationBuilder.getGroupIdsSortingSize()?

hashAggregationBuilder.getGroupIdsSortingSize() part is only materialized in memory when buildHashSortedResult is called. This happens only when unspilling. Therefore getSizeInMemoryWhenUnspilling is used to estimate aggregation size when unspilling only.

findepi · 2019-03-11T15:45:12Z

...src/main/java/io/prestosql/operator/aggregation/builder/SpillableHashAggregationBuilder.java

@@ -219,6 +210,15 @@ private boolean shouldMergeWithMemory(long memorySize)
        }
    }

+    private long getSizeInMemoryWhenUnspilling()


I would keep the old method name getSizeInMemory.

the fact that it's only invoked after spiller is created seems not relevant to semantics of the method

the method is invoked before spilling (ie before getFutureValue(spillToDisk());), so it's not "when unspilling"

the method is invoked before spilling (ie before getFutureValue(spillToDisk());), so it's not "when unspilling"

It is used to estimate size of memory during unspilling phase and to decide if we should merge with mem or not. This method has very limited use case and the previous name was confusing

InMemoryHashAggregationBuilder was doing it's own memory accounting for the purpose of GroupByHash yielding. When SpillableHashAggregationBuilder is used it also accounts InMemoryHashAggregationBuilder as a revocable memory. Therefore memory was accounted twice. This commit removes memory accoutning from InMemoryHashAggregationBuilder.

sopel39 requested a review from findepi March 8, 2019 15:26

cla-bot bot added the cla-signed label Mar 8, 2019

sopel39 added 3 commits March 11, 2019 12:09

Add OperatorContext#aggregateSystemMemoryContext method

3043538

Update peak memory reservations when revocable memory is updated

b471178

Update revocableMemoryFuture when revocable memory is udpated

d315d73

sopel39 force-pushed the ks/fix_spill_aggregation branch from 4bd7cc7 to f10654b Compare March 11, 2019 11:10

findepi approved these changes Mar 11, 2019

View reviewed changes

sopel39 force-pushed the ks/fix_spill_aggregation branch from f10654b to 84aae4d Compare March 12, 2019 11:37

findepi approved these changes Mar 12, 2019

View reviewed changes

Rename getSizeInMemory to getSizeInMemoryWhenUnspilling

0b20a46

sopel39 force-pushed the ks/fix_spill_aggregation branch from 84aae4d to 0b20a46 Compare March 12, 2019 12:12

sopel39 merged commit fe4271c into trinodb:master Mar 12, 2019

sopel39 deleted the ks/fix_spill_aggregation branch March 12, 2019 15:10

findepi added this to the 306 milestone Mar 13, 2019

sopel39 mentioned this pull request Mar 13, 2019

Release notes for 306 #410

Closed

6 tasks

sopel39 mentioned this pull request Jan 9, 2020

Add 2019 summary post trinodb/trino.io#66

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix double accounting of memory when spilling aggregation is used #413

Fix double accounting of memory when spilling aggregation is used #413

sopel39 commented Mar 8, 2019

findepi left a comment

findepi Mar 11, 2019

findepi Mar 11, 2019

sopel39 Mar 12, 2019

findepi Mar 11, 2019

findepi Mar 11, 2019

sopel39 Mar 12, 2019

findepi Mar 11, 2019

sopel39 Mar 12, 2019

findepi Mar 11, 2019

sopel39 Mar 12, 2019

findepi Mar 11, 2019

sopel39 Mar 12, 2019

Fix double accounting of memory when spilling aggregation is used #413

Fix double accounting of memory when spilling aggregation is used #413

Conversation

sopel39 commented Mar 8, 2019

findepi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment