Skip to content

Commit

Permalink
Increase AWS Glue get partition max results to 1000
Browse files Browse the repository at this point in the history
- Increased the value from earlier default of 128 to
  reduce number of API requests made to Glue during
  query planning.
  • Loading branch information
hashhar authored and ebyhr committed Aug 25, 2020
1 parent 268fcd1 commit 45dc37d
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ public class GlueHiveMetastore
private static final String WILDCARD_EXPRESSION = "";
private static final int BATCH_GET_PARTITION_MAX_PAGE_SIZE = 1000;
private static final int BATCH_CREATE_PARTITION_MAX_PAGE_SIZE = 100;
private static final int AWS_GLUE_GET_PARTITIONS_MAX_RESULTS = 128;
private static final int AWS_GLUE_GET_PARTITIONS_MAX_RESULTS = 1000;
private static final Comparator<Partition> PARTITION_COMPARATOR =
comparing(Partition::getValues, lexicographical(String.CASE_INSENSITIVE_ORDER));

Expand Down

0 comments on commit 45dc37d

Please sign in to comment.