Skip to content

Commit

Permalink
2
Browse files Browse the repository at this point in the history
  • Loading branch information
@acelyc111
acelyc111 committed Apr 22, 2024
1 parent db1cae2 commit 7f43e1f
Show file tree
Hide file tree
Showing 2 changed files with 284 additions and 16 deletions.
264 changes: 263 additions & 1 deletion _docs/en/administration/table-env.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,266 @@
permalink: administration/table-env
---

TRANSLATING
# Introduction

In order to control some of the behaviors of the table, Pegasus provides the table environment variables, also known as `app envs`.

The table environment variables are stored in the metadata of the table in the form of a map (in the Thrift structure `app_info`) and persisted to the Apache Zookeeper.

View the table information through the shell tool's [`ls` command](/docs/tools/shell/#ls), and the last column `envs_count` indicates the number of environment variables of the table. For example:
```
>>> ls
[general_info]
app_id status app_name app_type partition_count replica_count is_stateful create_time drop_time drop_expire envs_count
2 AVAILABLE temp pegasus 8 3 true 2024-04-15_06:58:38 - - 1
```

If you want to view specific table environment variables, you need to use the [get_app_envs](#get_app_envs) command.

Table environment variables have the following characteristics:
* Persist the metadata of the table onto Apache Zookeeper.
* It can be dynamically modified through the command line, and once the modification is successful, it will be immediately updated to Zookeeper.
* Synchronize the variables though the periodic synchronization message `config_sync` between the meta server and the replica server to take effect on each replica server.
> Due to periodic synchronization, the updated environment variables may not take immediate effect on the replica server, but rather have a delay. This delay time depends on the value of the configuration 'config_sync_interval_ms', which defaults to 30 seconds.
* The key of environmental variables is usually separated by `.` for easy classification.

The functions currently supported through table environment variables include:
* [Manual-Compact](manual-compact)
* [Usage-Scenario](usage-scenario)

# Control commands

Pegasus provides [Shell commands](/docs/tools/shell/) for manipulating table environment variables.
> Before executing these commands, `use <table_name>` needs to be executed to select the table that needs to be operated on.
## get_app_envs

Get a list of environment variables, usage: `get_app_envs [-j|--json]`

示例：
```
>>> use temp
OK
>>> get_app_envs
[app_envs]
rocksdb.usage_scenario : normal
```

## set_app_envs

设置环境变量，用法：`set_app_envs <key> <value> [key value...]`

示例：
```
>>> use temp
OK
>>> set_app_envs rocksdb.usage_scenario bulk_load
set app envs succeed
```

## del_app_envs

删除环境变量，用法：`del_app_envs <key> [key...]`

示例：
```
>>> use temp
OK
>>> set_app_envs rocksdb.usage_scenario bulk_load
set app envs succeed
>>> del_app_envs rocksdb.usage_scenario
del app envs succeed
=============================
deleted keys:
rocksdb.usage_scenario
=============================
```

## clear_app_envs

清空环境变量，或者叫批量删除环境变量，用法：`clear_app_envs <-a|--all> <-p|--prefix str>`

支持两种方式：
* 全部清理：使用 `-a` 选项。
* 通过前缀清理：使用 `-p` 选项指定前缀，匹配时会先自动在前缀后面加上 `.`，然后按照字符串前缀匹配。

譬如：
```
>>> use temp
OK
>>> set_app_envs manual_compact.once.trigger_time 1713700000
set app envs succeed
>>> set_app_envs manual_compact.once.target_level -1
set app envs succeed
>>> set_app_envs manual_compact.once.bottommost_level_compaction force
set app envs succeed
>>> set_app_envs rocksdb.usage_scenario bulk_load
set app envs succeed
>>> clear_app_envs -p manual_compact
clear app envs succeed
=============================
deleted keys:
manual_compact.once.bottommost_level_compaction
manual_compact.once.target_level
manual_compact.once.trigger_time
=============================
>>> get_app_envs
[app_envs]
rocksdb.usage_scenario : bulk_load
```

# Supported environment variables

Starting from Pegasus 2.6, all supported table environment variables can be obtained through Meta Server [HTTP API](/api/http) `/envs/list`.
For example:

```
$ curl 127.0.0.1:34601/envs/list
{
"business.info": {
"limitation": "",
"sample": "",
"type": "string"
},
"default_ttl": {
"limitation": ">= 0",
"sample": "86400",
"type": "unsigned int32"
},
"manual_compact.disabled": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"manual_compact.max_concurrent_running_count": {
"limitation": ">= 0",
"sample": "8",
"type": "unsigned int32"
},
"manual_compact.once.bottommost_level_compaction": {
"limitation": "force | skip",
"sample": "skip",
"type": "string"
},
"manual_compact.once.target_level": {
"limitation": "-1 or >= 1",
"sample": "6",
"type": "unsigned int64"
},
"manual_compact.once.trigger_time": {
"limitation": "> 0, timestamp (in seconds) to trigger the once manual compaction",
"sample": "1700000000",
"type": "unsigned int64"
},
"manual_compact.periodic.bottommost_level_compaction": {
"limitation": "force | skip",
"sample": "skip",
"type": "string"
},
"manual_compact.periodic.target_level": {
"limitation": "-1 or >= 1",
"sample": "6",
"type": "unsigned int64"
},
"manual_compact.periodic.trigger_time": {
"limitation": "",
"sample": "",
"type": "string"
},
"replica.backup_request_throttling": {
"limitation": "<size[K|M]>*<delay|reject>*<milliseconds>",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica.deny_client_request": {
"limitation": "timeout*all | timeout*write | timeout*read | reconfig*all | reconfig*write | reconfig*read",
"sample": "timeout*all",
"type": "string"
},
"replica.read_throttling": {
"limitation": "<size[K|M]>*<delay|reject>*<milliseconds>",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica.read_throttling_by_size": {
"limitation": "",
"sample": "20000*delay*100,20000*reject*100",
"type": "string"
},
"replica.rocksdb_block_cache_enabled": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"replica.rocksdb_iteration_threshold_time_ms": {
"limitation": ">= 0",
"sample": "1000",
"type": "unsigned int64"
},
"replica.slow_query_threshold": {
"limitation": ">= 20",
"sample": "1000",
"type": "unsigned int64"
},
"replica.split.validate_partition_hash": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"replica.write_throttling": {
"limitation": "<size[K|M]>*<delay|reject>*<milliseconds>",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica.write_throttling_by_size": {
"limitation": "<size[K|M]>*<delay|reject>*<milliseconds>",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica_access_controller.allowed_users": {
"limitation": "",
"sample": "",
"type": "string"
},
"replica_access_controller.ranger_policies": {
"limitation": "",
"sample": "",
"type": "string"
},
"rocksdb.allow_ingest_behind": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"rocksdb.checkpoint.reserve_min_count": {
"limitation": "> 0",
"sample": "2",
"type": "unsigned int32"
},
"rocksdb.checkpoint.reserve_time_seconds": {
"limitation": ">= 0",
"sample": "3600",
"type": "unsigned int32"
},
"rocksdb.num_levels": {
"limitation": "In range [1, 10]",
"sample": "6",
"type": "unsigned int64"
},
"rocksdb.usage_scenario": {
"limitation": "bulk_load | normal | prefer_write",
"sample": "normal",
"type": "string"
},
"rocksdb.write_buffer_size": {
"limitation": "In range [16777216, 536870912]",
"sample": "16777216",
"type": "unsigned int64"
},
"user_specified_compaction": {
"limitation": "",
"sample": "",
"type": "string"
}
}
```
36 changes: 21 additions & 15 deletions _docs/zh/administration/table-env.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
permalink: administration/table-env
---

# 功能目标
# 介绍

为了对 Table 的一些行为进行控制，Pegasus 提供了 Table 环境变量，又称之为 `app envs`。
为了对 table 的一些行为进行控制，Pegasus 提供了 table 环境变量，又称之为 `app envs`。

Table 环境变量以 map 的形式存储在 Table 的元数据中（Thrift 结构 `app_info` 中），并持久化到 Zookeeper 上。
Table 环境变量以 map 的形式存储在该表的元数据中（Thrift 结构 `app_info` 中），并持久化到 Apache Zookeeper 上。

通过 shell 工具的 [`ls` 命令](/tools/shell/#ls) 查看表信息，最后一列 `envs_count` 记录了 table 环境变量的个数。例如：
通过 shell 工具的 [`ls` 命令](/docs/tools/shell/#ls) 查看表信息，最后一列 `envs_count` 指明了该表的环境变量个数。例如：
```
>>> ls
[general_info]
Expand All @@ -19,20 +19,22 @@ app_id status app_name app_type partition_count replica_count is_statef
如果要查看具体的 table 环境变量，则需要使用 [get_app_envs](#get_app_envs) 命令。

Table 环境变量具有如下特性：
* 作为 Table 的元数据持久化到 Zookeeper 上。
* 可以通过命令行动态修改，修改成功后会立即更新到 Zookeeper 上。
* 通过 meta server 和 replica server 的定期同步消息 `config_sync` 同步给各个 replica server 生效。
> 由于是定期同步，所以环境变量更新后可能不会在 replica server 上立即生效，而是有一个延迟。这个延迟时间依赖于配置 `config_sync_interval_ms` 的值，默认是 30 秒。
* 作为 table 的元数据持久化到 Apache Zookeeper 上。
* 可以通过命令行动态修改，修改成功后会立即更新到 Apache Zookeeper 上。
* 通过 meta server 和 replica server 的周期性同步消息 `config_sync` 同步给各个 replica server 生效。
> 由于是周期性同步，所以环境变量更新后可能不会在 replica server 上立即生效，而是有一个延迟。这个延迟时间依赖于配置 `config_sync_interval_ms` 的值，默认是 30 秒。
* 环境变量的 key 通常使用 `.` 分隔，方便分类。

目前通过 Table 环境变量支持的功能如：
目前通过 table 环境变量支持的功能如：
* [Manual-Compact 功能](manual-compact)
* [Usage-Scenario 功能](usage-scenario)

# 操作命令
Pegasus 的 [Shell 工具](/tools/shell/) 中提供了操作 table 环境变量的命令。这些命令执行前都需要先执行 `use <table_name>` 选择需要操作的表。
# 控制命令

Pegasus 的 [Shell 工具](/docs/tools/shell/) 中提供了操作 table 环境变量的命令。这些命令执行前都需要先执行 `use <table_name>` 选择需要操作的表。

## get_app_envs

获取环境变量列表，用法：`get_app_envs [-j|--json]`

示例：
Expand All @@ -43,7 +45,9 @@ OK
[app_envs]
rocksdb.usage_scenario : normal
```

## set_app_envs

设置环境变量，用法：`set_app_envs <key> <value> [key value...]`

示例：
Expand All @@ -55,6 +59,7 @@ set app envs succeed
```

## del_app_envs

删除环境变量，用法：`del_app_envs <key> [key...]`

示例：
Expand All @@ -72,7 +77,8 @@ deleted keys:
```

## clear_app_envs
清理环境变量，或者叫批量删除环境变量，用法：`clear_app_envs <-a|--all> <-p|--prefix str>`

清空环境变量，或者叫批量删除环境变量，用法：`clear_app_envs <-a|--all> <-p|--prefix str>`

支持两种方式：
* 全部清理：使用 `-a` 选项。
Expand Down Expand Up @@ -103,10 +109,10 @@ deleted keys:
rocksdb.usage_scenario : bulk_load
```

# 支持列表
# 支持的环境变量

可通过 meta server 的 [HTTP 接口](/api/http) `/envs/list` 获取所有支持的 table 环境变量。
例如，2.6 版本支持的环境变量如下：
从 Pegasus 2.6 开始，可通过 meta server 的 [HTTP 接口](/api/http) `/envs/list` 获取所有支持的 table 环境变量。
例如：

```
$ curl 127.0.0.1:34601/envs/list
Expand Down

0 comments on commit 7f43e1f

Please sign in to comment.