Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(canner): implement new data source canner #823

Merged
merged 16 commits into from
Oct 22, 2024

Conversation

grieve54706
Copy link
Contributor

@grieve54706 grieve54706 commented Oct 14, 2024

Summary

Support new data source Canner Enterprise. It provides the PG wire protocol, so we can connect it by following Postgres.

Connection info

  • host: str
  • port: str
  • user: str
  • pat: str
  • workspace: str
  • enable_ssl: bool, If true, will connect with HTTPS

Additional information

  • Canner enterprise does not support CREATE TEMPORARY VIEW to get the schema for ibis. We need to build schema by dry-run.
  • Canner enterprise does not support dry-run, so we have to query with a limit of zero
  • The table and column comments are stored in another service of Canner Enterprise. We need to query them by GraphQL.
  • Preparing Canner Enterprise to test is difficult, so we don't test this feature in the CI.

@grieve54706 grieve54706 requested review from goldmedal and removed request for goldmedal October 14, 2024 06:11
@grieve54706 grieve54706 force-pushed the feature/canner-enterprise branch from 61f40b2 to 83629e2 Compare October 14, 2024 07:40
@grieve54706 grieve54706 requested a review from goldmedal October 16, 2024 04:16
@grieve54706 grieve54706 force-pushed the feature/canner-enterprise branch from 71159d0 to 04faf51 Compare October 16, 2024 05:16
@grieve54706 grieve54706 marked this pull request as ready for review October 16, 2024 05:17
@github-actions github-actions bot added ibis dependencies Pull requests that update a dependency file python Pull requests that update Python code labels Oct 22, 2024
Copy link
Contributor

@goldmedal goldmedal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @grieve54706 overall looks good to me. Only some minor comments.

Comment on lines 17 to 18
- The table `orders` must be with a comment `This is a table comment`
- The table `orders` must have a column `o_comment` with a comment `This is a comment`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The table `orders` must be with a comment `This is a table comment`
- The table `orders` must have a column `o_comment` with a comment `This is a comment`
- The table `orders` must be with a description `This is a table comment`
- The table `orders` must have a column `o_comment` with a description `This is a comment`

In the canner side, I think it's called description.


pytestmark = pytest.mark.canner

client = TestClient(app)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can use the with client clause here, too.

Comment on lines 49 to 52
"rls": {
"name": "SESSION_STATUS",
"operator": "EQUALS",
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused field.

Suggested change
"rls": {
"name": "SESSION_STATUS",
"operator": "EQUALS",
},

@goldmedal goldmedal merged commit 3cb776d into main Oct 22, 2024
7 checks passed
@goldmedal goldmedal deleted the feature/canner-enterprise branch October 22, 2024 06:33
grieve54706 added a commit that referenced this pull request Dec 13, 2024
* feat(canner): implement new data source `canner`

* feat(canner): get type name via query and cache the result

* test(canner): add test case and fix dry_run

* feat(canner): implement metadata

* chore(ci): not test canner

* chore(poetry): update

* test(canner): adjust test

* chore: remove unused

* chore(test): move test order

* chore(canner): move comment

* chore(canner): adjust method _oid_to_ibis_type to _to_ibis_type

* chore(canner): improve readability

* chore(comment): use description instead of comment to fit canner enterprise design

* test(canner): adjust the pattern

* test(canner): remove unused
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file ibis python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants