Skip to content

Commit

Permalink
[SPARK-48391][CORE] Using addAll instead of add function in fromAccum…
Browse files Browse the repository at this point in the history
…ulatorInfos method of TaskMetrics Class

### What changes were proposed in this pull request?

Using addAll instead of add function in fromAccumulators method of TaskMetrics.

### Why are the changes needed?

To Improve performance. In the fromAccumulators method of TaskMetrics,we should use `
tm._externalAccums.addAll` instead of `tm._externalAccums.add`, as _externalAccums is a instance of CopyOnWriteArrayList

### Does this PR introduce _any_ user-facing change?

yes.

### How was this patch tested?

No Tests.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#46705 from monkeyboy123/fromAccumulators-accelerate.

Authored-by: Dereck Li <monkeyboy.ljh@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
monkeyboy123 authored and cloud-fan committed May 31, 2024
1 parent 96365c8 commit 3cd35f8
Showing 1 changed file with 5 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -328,16 +328,19 @@ private[spark] object TaskMetrics extends Logging {
*/
def fromAccumulators(accums: Seq[AccumulatorV2[_, _]]): TaskMetrics = {
val tm = new TaskMetrics
val externalAccums = new java.util.ArrayList[AccumulatorV2[Any, Any]]()
for (acc <- accums) {
val name = acc.name
val tmpAcc = acc.asInstanceOf[AccumulatorV2[Any, Any]]
if (name.isDefined && tm.nameToAccums.contains(name.get)) {
val tmAcc = tm.nameToAccums(name.get).asInstanceOf[AccumulatorV2[Any, Any]]
tmAcc.metadata = acc.metadata
tmAcc.merge(acc.asInstanceOf[AccumulatorV2[Any, Any]])
tmAcc.merge(tmpAcc)
} else {
tm._externalAccums.add(acc)
externalAccums.add(tmpAcc)
}
}
tm._externalAccums.addAll(externalAccums)
tm
}
}

0 comments on commit 3cd35f8

Please sign in to comment.