[BUG] In Spark 3.2.0 and above dynamic partition pruning and AQE are not mutually exclusive

[revans2]

The existing tests for this failed initially because they are testing that when DPP happens AQE is disabled. That is no longer the case and we need to look at that situation and see if we are doing the right thing. We should also add in tests to verify that we are doing the right thing.

[andygrove]

I started looking at this and I think it will be simple enough to add new tests but I am going to wait until #2073 is merged since branch-0.5 does not currently compile with Spark 3.2.0

[sameerz]

Removing from the sprint since we are not supporting Spark 3.2.0 in the 0.5 release.

[revans2]

So the tests that are failing now for AdaptiveQueryExecSuite are all failing in Spark code, specifically in AQEShuffleReadExec.scala:86 where it has a hard coded set of partitioning that is allowed.

      // For coalesced shuffle read, the data distribution is not changed, only the number of
      // partitions is changed.
      child.outputPartitioning match {
        case h: HashPartitioning =>
          CurrentOrigin.withOrigin(h.origin)(h.copy(numPartitions = partitionSpecs.length))
        case r: RangePartitioning =>
          CurrentOrigin.withOrigin(r.origin)(r.copy(numPartitions = partitionSpecs.length))
        // This can only happen for `REBALANCE_PARTITIONS_BY_NONE`, which uses
        // `RoundRobinPartitioning` but we don't need to retain the number of partitions.
        case r: RoundRobinPartitioning =>
          r.copy(numPartitions = partitionSpecs.length)
        case other => throw new IllegalStateException(
          "Unexpected partitioning for coalesced shuffle read: " + other)
      }

This includes the one DPP test Join partitioned tables DPP fallback so I don't know what the next steps for this are. Do we need to get something into Spark before 3.2.0 ships or if there is a way for us to work around this.

[revans2]

As always Andy is on top of this.

https://issues.apache.org/jira/browse/SPARK-36666

That didn't make it into 3.2.0-RC2 but should be in the next one.

[revans2]

I think I can just close this. If I manually update things to test on 3.2.1-SNAPHSOT (which includes the above fix) there are no failures in the AQE tests