-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: Support ClickHouse driver #221
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
aad22e1
feat(extension-driver-clickhouse): create clickhouse driver extension
kokokuo 466b899
chore(extension-driver-clickhouse): update README
kokokuo 135dfb5
feat(docs): add ClickHouse connector to docs package
kokokuo 385b908
feat(extension-driver-clickhouse): add test cases and refactor queryi…
kokokuo 3e43ae8
chore(docs, extension-driver-clickhouse): optimizing and correcting c…
kokokuo 3ccd301
chore(labs, extension-driver-clickhouse): update Makefile to add exte…
kokokuo 9f3e4ce
fix(extension-driver-clickhouse): fix the typo and make the clickhous…
kokokuo c7b26e4
chore(extension-driver-clickhouse): normalize to convert to Bool clic…
kokokuo 59d4c39
chore(docs): update clickhouse document
kokokuo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # ClickHouse | ||
|
|
||
| ## Installation | ||
|
|
||
| 1. Install the package: | ||
|
|
||
| **If you are developing with binary, the package is already bundled in the binary. You can skip this step.** | ||
|
|
||
| ```bash | ||
| npm i @vulcan-sql/extension-driver-clickhouse | ||
| ``` | ||
|
|
||
| 2. Update your `vulcan.yaml` file to enable the extension: | ||
|
|
||
| ```yaml | ||
| extensions: | ||
| ... | ||
| // highlight-next-line | ||
| ch: '@vulcan-sql/extension-driver-clickhouse' # Add this line | ||
| ``` | ||
|
|
||
| 3. Create a new profile in your `profiles.yaml` file or in the designated profile paths. For example: | ||
|
|
||
| ```yaml | ||
| - name: ch # profile name | ||
| type: clickhouse | ||
| connection: | ||
| host: www.example.com:8123 | ||
| request_timeout: 60000 | ||
| compression: | ||
| request: true | ||
| max_open_connections: 10 | ||
| username: user | ||
| password: pass | ||
| database: hello-clickhouse | ||
| allow: '*' | ||
| ``` | ||
|
|
||
| ## Configuration | ||
|
|
||
| For more information, please refer to the [ClickHouse Client documentation](https://clickhouse.com/docs/en/integrations/language-clients/nodejs) to learn about the available arguments for the ClickHouse Client. | ||
|
|
||
| | Name | Required | Default | Description | | ||
| | -------------------- | -------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | host | N | http://localhost:8123 | ClickHouse instance URL. | | ||
| | request_timeout | N | 30000 | Request timeout in milliseconds. | | ||
| | max_open_connections | N | Infinity | Maximum number of sockets to allow per host. | | ||
| | compression | N | | Compression settings for data transfer. Currently, only GZIP compression using zlib is supported. See [Compression docs](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#compression) for details. | | ||
| | username | N | default | The name of the user on whose behalf requests are made. | | ||
| | password | N | | The user's password. | | ||
| | application | N | VulcanSQL | The name of the application using the Node.js client. | | ||
| | database | N | default | Database name to use. | | ||
| | clickhouse_settings | N | | ClickHouse settings to apply to all requests. For all available settings, see [Advance Settings](https://clickhouse.com/docs/en/operations/settings), and For the definition, see [Definition](https://github.com/ClickHouse/clickhouse-js/blob/0.1.1/src/settings.ts) | | ||
| | tls | N | | Configure TLS certificates. See [TLS docs](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#tls-certificates). | | ||
| | session_id | N | | ClickHouse Session ID to send with every request. | | ||
| | keep_alive | N | | HTTP Keep-Alive related settings. See [Keep Alive docs](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#keep-alive) | | ||
|
|
||
| The `log` option is not included above because it requires defining a Logger class and assigning it. Therefore, it cannot be set through `profiles.yaml`. | ||
|
|
||
| ## Note | ||
|
|
||
| ClickHouse supports parameterized queries to prevent SQL injection using prepared statements. Named placeholders are defined using the `{name:type}` syntax. For more information, refer to the [Query with Parameters](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#queries-with-parameters) section in the ClickHouse documentation. | ||
|
|
||
| However, the VulcanSQL API supports JSON format for API query parameters and does not support the full range of types available in ClickHouse. VulcanSQL only supports the conversion of the following types: | ||
|
|
||
| - `boolean` to ClickHouse type `Bool` | ||
| - `number` to ClickHouse types `Int` or `Float` | ||
| - `string` to ClickHouse type `String` | ||
|
|
||
| Therefore, if you need to query data with special types in ClickHouse, such as `Array(Unit8)`, `Record<K, V>`, `Date`, `DateTime`, and so on, you can use ClickHouse [Regular Functions](https://clickhouse.com/docs/en/sql-reference/functions) or [Type Conversion Functions](https://clickhouse.com/docs/en/sql-reference/functions/type-conversion-functions) to handle them. | ||
|
|
||
| Example: | ||
|
|
||
| ```sql | ||
| -- If the `val` from the API query parameter is '1990-11-01' and the `born_date` column is of type `Date32`, | ||
| -- you can use the toDate function to convert the value. See https://clickhouse.com/docs/en/sql-reference/functions/type-conversion-functions#todate | ||
| SELECT * FROM users WHERE born_date = toDate({val:String}); | ||
| ``` | ||
|
|
||
| ## ⚠️ Caution | ||
|
|
||
| The ClickHouse driver currently does not support caching datasets. If you use the ClickHouse driver with caching dataset features, it will result in failure. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # ClickHouse | ||
|
|
||
| Connect to your ClickHouse servers using the official [Node.js Driver](https://clickhouse.com/docs/en/integrations/language-clients/nodejs). | ||
|
|
||
| ## Installation | ||
|
|
||
| 1. Install package | ||
|
|
||
| ```bash | ||
| npm i @vulcan-sql/extension-driver-clickhouse | ||
| ``` | ||
|
|
||
| :::info | ||
| If you run VulcanSQL with Docker, you should use the command `vulcan-install @vulcan-sql/extension-driver-clickhouse` instead. | ||
|
|
||
| ::: | ||
|
|
||
| 2. Update `vulcan.yaml`, and enable the extension. | ||
|
|
||
| ```yaml | ||
| extensions: | ||
| ... | ||
| // highlight-next-line | ||
| ch: '@vulcan-sql/extension-driver-clickhouse' # Add this line | ||
| ``` | ||
|
|
||
| 3. Create a new profile in `profiles.yaml` or in your profile files. For example: | ||
|
|
||
| ```yaml | ||
| - name: ch # profile name | ||
| type: clickhouse | ||
| connection: | ||
| host: www.example.com:8123 | ||
| request_timeout: 60000 | ||
| compression: | ||
| request: true | ||
| max_open_connections: 10 | ||
| username: user | ||
| password: pass | ||
| database: hello-clickhouse | ||
| allow: '*' | ||
| ``` | ||
|
|
||
| ## Configuration | ||
|
|
||
| For more information, please refer to the [ClickHouse Client documentation](https://clickhouse.com/docs/en/integrations/language-clients/nodejs) to learn about the available arguments for the ClickHouse Client. | ||
|
|
||
| | Name | Required | Default | Description | | ||
| | -------------------- | -------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | host | N | http://localhost:8123 | ClickHouse instance URL. | | ||
| | request_timeout | N | 30000 | Request timeout in milliseconds. | | ||
| | max_open_connections | N | Infinity | Maximum number of sockets to allow per host. | | ||
| | compression | N | | Compression settings for data transfer. Currently, only GZIP compression using zlib is supported. See [Compression docs](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#compression) for details. | | ||
| | username | N | default | The name of the user on whose behalf requests are made. | | ||
| | password | N | | The user's password. | | ||
| | application | N | VulcanSQL | The name of the application using the Node.js client. | | ||
| | database | N | default | Database name to use. | | ||
| | clickhouse_settings | N | | ClickHouse settings to apply to all requests. For all available settings, see [Advance Settings](https://clickhouse.com/docs/en/operations/settings), and For the definition, see [Definition](https://github.com/ClickHouse/clickhouse-js/blob/0.1.1/src/settings.ts) | | ||
| | tls | N | | Configure TLS certificates. See [TLS docs](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#tls-certificates). | | ||
| | session_id | N | | ClickHouse Session ID to send with every request. | | ||
| | keep_alive | N | | HTTP Keep-Alive related settings. See [Keep Alive docs](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#keep-alive) | | ||
|
|
||
| The `log` option is not included above because it requires defining a Logger class and assigning it. Therefore, it cannot be set through `profiles.yaml`. | ||
|
|
||
| ## Note | ||
|
|
||
| ClickHouse supports parameterized queries to prevent SQL injection using prepared statements. Named placeholders are defined using the `{name:type}` syntax. For more information, refer to the [Query with Parameters](https://clickhouse.com/docs/en/integrations/language-clients/nodejs#queries-with-parameters) section in the ClickHouse documentation. | ||
|
|
||
| However, the VulcanSQL API supports JSON format for API query parameters and does not support the full range of types available in ClickHouse. VulcanSQL only supports the conversion of the following types: | ||
|
|
||
| - `boolean` to ClickHouse type `Bool` | ||
| - `number` to ClickHouse types `Int` or `Float` | ||
| - `string` to ClickHouse type `String` | ||
|
|
||
| Therefore, if you need to query data with special types in ClickHouse, such as `Array(Unit8)`, `Record<K, V>`, `Date`, `DateTime`, and so on, you can use ClickHouse [Regular Functions](https://clickhouse.com/docs/en/sql-reference/functions) or [Type Conversion Functions](https://clickhouse.com/docs/en/sql-reference/functions/type-conversion-functions) to handle them. | ||
|
|
||
| Example: | ||
|
|
||
| ```sql | ||
| -- If the `val` from the API query parameter is '1990-11-01' and the `born_date` column is of type `Date32`, | ||
| -- you can use the toDate function to convert the value. See https://clickhouse.com/docs/en/sql-reference/functions/type-conversion-functions#todate | ||
| SELECT * FROM users WHERE born_date = toDate({val:String}); | ||
| ``` | ||
|
|
||
| ## ⚠️ Caution | ||
|
|
||
| The ClickHouse driver currently does not support caching datasets. If you use the ClickHouse driver with caching dataset features, it will result in failure. | ||
|
|
||
| ## Testing | ||
|
|
||
| To run tests for the `extension-driver-clickhouse` module, use the following command: | ||
|
|
||
| ```bash | ||
| nx test extension-driver-clickhouse | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this necessary? It seems like the vulcan.yaml didn't use the clickhouse extension