Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Glue Catalog Table creation fails when there are multiple partition keys #7

Closed
SanjuMLGeek opened this issue Apr 5, 2023 · 1 comment · Fixed by #8
Closed

Glue Catalog Table creation fails when there are multiple partition keys #7

SanjuMLGeek opened this issue Apr 5, 2023 · 1 comment · Fixed by #8
Labels
bug 🐛 An issue with the system

Comments

@SanjuMLGeek
Copy link

Describe the Bug

Hello,

I am trying to create Glue catalog table using this module and I have tables with multiple partitions such as below.

{name= "schemaid" , type = "string"}
{name= "batchtimestamp" , type = "string"}

Per variable definition, it's just a map of strings and I believe I can pass only 1 key value pair as below.

partition_keys = {
name= "schemaid" ,
type = "string"
}

If I try to define the list of string maps or put map of string maps, it fails

For a local defined as list:

partition_keys = [
{name= "schemaid" , type = "string"},
{name= "batchtimestamp" , type = "string"}
]

image

For a local defined as map of string maps:

partition_keys = {
{name= "schemaid" , type = "string"},
{name= "batchtimestamp" , type = "string"}
}

image

Can you please help to see if that's a bug or I need to define the partition keys differently. Also I am trying to use same multiple columns for partition Indexes too and facing same challenges.

Expected Behavior

while using aws_glue_catalog_table resource directly, I could successfully build the table using below code for multiple partition keys and multiple indexes out of those partition keys.

image

Steps to Reproduce

define local as one of the below way and try creating table with module and partition_keys = local.partition_keys

partition_keys = [
{name= "schemaid" , type = "string"},
{name= "batchtimestamp" , type = "string"}
]

partition_keys = {
{name= "schemaid" , type = "string"},
{name= "batchtimestamp" , type = "string"}
}

Screenshots

No response

Environment

No response

Additional Context

No response

@SanjuMLGeek SanjuMLGeek added the bug 🐛 An issue with the system label Apr 5, 2023
@aknysh aknysh closed this as completed in #8 Apr 8, 2023
@joe-niland
Copy link
Member

@SanjuMLGeek thanks for reporting this issue. Let us know if there is anything further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 An issue with the system
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants