From 2045ed968e1c5609a08e76b10b92368e8907afde Mon Sep 17 00:00:00 2001 From: Yingchun Lai Date: Thu, 23 Nov 2023 22:33:57 +0800 Subject: [PATCH] Update some links --- _data/docs_menu.yml | 2 +- _docs/zh/administration/rebalance.md | 2 +- _docs/zh/clients/clients.md | 11 +- _docs/zh/clients/java-client.md | 4 +- _docs/zh/clients/node-client.md | 2 +- _docs/zh/clients/python2-client.md | 465 --------------------------- _docs/zh/clients/python3-client.md | 4 +- _docs/zh/clients/scala-client.md | 8 +- _docs/zh/tools/shell.md | 2 +- _overview/en/overview.md | 2 +- _overview/zh/overview.md | 2 +- 11 files changed, 19 insertions(+), 485 deletions(-) delete mode 100755 _docs/zh/clients/python2-client.md diff --git a/_data/docs_menu.yml b/_data/docs_menu.yml index 405f3d01..adffcdde 100644 --- a/_data/docs_menu.yml +++ b/_data/docs_menu.yml @@ -15,7 +15,7 @@ - name: title_cpp-client link: /clients/cpp-client - name: title_go-client - link: https://github.com/XiaoMi/pegasus-go-client + link: https://github.com/apache/incubator-pegasus/tree/master/go-client - name: title_python2-client link: /clients/python2-client - name: title_python3-client diff --git a/_docs/zh/administration/rebalance.md b/_docs/zh/administration/rebalance.md index e9fcc0a8..09ee9584 100644 --- a/_docs/zh/administration/rebalance.md +++ b/_docs/zh/administration/rebalance.md @@ -242,7 +242,7 @@ Succeed count: 1 Failed count: 0 ``` -[remote_command](https://github.com/XiaoMi/rdsn/blob/a8c2d46568889902df820ac65e045a1e78aa84d4/include/dsn/tool-api/command_manager.h)是pegasus的一个特性, 允许一个server注册一些命令，然后命令行可以通过rpc调用这些命令。这里我们使用**help**来访问meta server leader，获取meta server端支持的所有命令。例子中已经略掉了所有不相关的行，只留下以"meta.lb"开头的所有和负载均衡相关的命令。 +[remote_command](https://github.com/apache/incubator-pegasus/blob/master/src/utils/command_manager.h)是pegasus的一个特性, 允许一个server注册一些命令，然后命令行可以通过rpc调用这些命令。这里我们使用**help**来访问meta server leader，获取meta server端支持的所有命令。例子中已经略掉了所有不相关的行，只留下以"meta.lb"开头的所有和负载均衡相关的命令。 由于文档和代码的不一致问题，文档里不一定覆盖了当前meta所有的lb控制命令。如果想获取最新的命令列表，请用最新的代码手动执行一下help。 diff --git a/_docs/zh/clients/clients.md b/_docs/zh/clients/clients.md index 7a281948..729082fa 100644 --- a/_docs/zh/clients/clients.md +++ b/_docs/zh/clients/clients.md @@ -5,9 +5,8 @@ permalink: clients/index.html Pegasus目前提供以下多种客户端支持： | 编程语言 | 项目链接 | -| Java | | -| Scala | | -| Go | | -| Python2 | | -| Python3 | | -| NodeJs | | +| Java | | +| Scala | | +| Go | | +| Python3 | | +| NodeJs | | diff --git a/_docs/zh/clients/java-client.md b/_docs/zh/clients/java-client.md index 450b7a70..b02c73fc 100755 --- a/_docs/zh/clients/java-client.md +++ b/_docs/zh/clients/java-client.md @@ -4,12 +4,12 @@ permalink: clients/java-client # 获取Java客户端 -项目地址：[Pegasus Java Client](https://github.com/XiaoMi/pegasus-java-client) +项目地址：[Pegasus Java Client](https://github.com/apache/incubator-pegasus/tree/master/java-client) 下载： ```bash -git clone https://github.com/XiaoMi/pegasus-java-client.git +git clone https://github.com/apache/incubator-pegasus.git cd pegasus-java-client ``` diff --git a/_docs/zh/clients/node-client.md b/_docs/zh/clients/node-client.md index 5bfdeb57..2d8aa583 100755 --- a/_docs/zh/clients/node-client.md +++ b/_docs/zh/clients/node-client.md @@ -3,7 +3,7 @@ permalink: clients/node-client --- # 安装NodeJs客户端 -项目地址：[Pegasus NodeJS Client](https://github.com/XiaoMi/pegasus-nodejs-client) +项目地址：[Pegasus NodeJS Client](https://github.com/apache/incubator-pegasus/tree/master/nodejs-client) 下载并将客户端依赖添加到package.json中： `npm install pegasus-nodejs-client --save` # 创建/关闭客户端 diff --git a/_docs/zh/clients/python2-client.md b/_docs/zh/clients/python2-client.md deleted file mode 100755 index a5941871..00000000 --- a/_docs/zh/clients/python2-client.md +++ /dev/null @@ -1,465 +0,0 @@ ---- -permalink: clients/python2-client ---- - -# pegasus python client - -## 项目地址 - - - -## 版本要求 - -Python 2.* - -## 安装 - -`pip install pypegasus` - -## 使用 - -pegasus python client使用了[twisted](https://github.com/twisted/twisted), 编写的代码会带有twisted的影子。 - -### 示例 - -完整的示例请参考[sample](https://github.com/XiaoMi/pegasus-python-client/blob/master/sample.py)。以下是简单的示例： - -``` -#!/usr/bin/env python -# coding:utf-8 - -from pypegasus.pgclient import Pegasus - -from twisted.internet import reactor -from twisted.internet.defer import inlineCallbacks - - -@inlineCallbacks -def basic_test(): - # init - c = Pegasus(['127.0.0.1:34601', '127.0.0.1:34602'], 'temp') - - suc = yield c.init() - if not suc: - reactor.stop() - print('ERROR: connect pegasus server failed') - return - - # set - try: - ret = yield c.set('hkey1', 'skey1', 'value', 0, 500) - print('set ret: ', ret) - except Exception as e: - print(e) - - # get - ret = yield c.get('hkey1', 'skey1') - print('get ret: ', ret) - - reactor.stop() - - -if __name__ == "__main__": - reactor.callWhenRunning(basic_test) - reactor.run() -``` - -### log配置文件 - -pegasus python client使用了[logging](https://docs.python.org/2/library/logging.html)日志包，默认配置如下： - -``` -[loggers] -keys=root -[logger_root] -level=INFO -handlers=hand01 -propagate=0 -[handlers] -keys=hand01 -[handler_hand01] -class=handlers.RotatingFileHandler -formatter=form01 -args=('pegasus.log', 'a', 100*1024*1024, 10) -[formatters] -keys=form01 -[formatter_form01] -format=%(asctime)s [%(thread)d] [%(levelname)s] %(filename)s:%(lineno)d %(message)s -datefmt=%Y-%m-%d %H:%M:%S -``` - -如果用户有定制需求，可以在自己的代码目录添加配置文件`logger.conf` - -### API说明 - -#### 初始化 - -初始化先构造Pegasus对象，在使用init函数完成初始化： - -``` -class Pegasus(object): - """ - Pegasus client class. - """ - - def __init__(self, meta_addrs=None, table_name='', - timeout=DEFAULT_TIMEOUT): - """ - :param meta_addrs: (list) pagasus meta servers list. - example: ['127.0.0.1:34601', '127.0.0.1:34602', '127.0.0.1:34603'] - :param table_name: (str) table name/app name used in pegasus. - :param timeout: (int) default timeout in milliseconds when communicate with meta sever and replica server. - """ -``` - -``` - def init(self): - """ - Initialize the client before you can use it. - - :return: (DeferredList) True when initialized succeed, others when failed. - """ -``` - -#### ttl - -判断key的剩余的ttl时间 - -``` -def ttl(self, hash_key, sort_key, timeout=0): - """ - Get ttl(time to live) of the data. - - :param hash_key: (str) which hash key used for this API. - :param sort_key: (str) which sort key used for this API. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, ttl) - code: error_types.ERR_OK.value when data exist, error_types.ERR_OBJECT_NOT_FOUND.value when data not found. - ttl: in seconds, -1 means forever. - """ -``` - -#### exist - -判断key是否存在 - -``` -def exist(self, hash_key, sort_key, timeout=0): - """ - Check value exist. - - :param hash_key: (str) which hash key used for this API. - :param sort_key: (str) which sort key used for this API. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, ign) - code: error_types.ERR_OK.value when data exist, error_types.ERR_OBJECT_NOT_FOUND.value when data not found. - ign: useless, should be ignored. - """ -``` - -#### set - -插入一条数据（若已存在则会覆盖） - -``` -def set(self, hash_key, sort_key, value, ttl=0, timeout=0): - """ - Set value to be stored in . - - :param hash_key: (str) which hash key used for this API. - :param sort_key: (str) which sort key used for this API. - :param value: (str) value to be stored under . - :param ttl: (int) ttl(time to live) in seconds of this data. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, ign) - code: error_types.ERR_OK.value when data stored succeed. - ign: useless, should be ignored. - """ -``` - -#### multi_set - -同时写一条hashkey的多条sortkey数据 - -``` -def multi_set(self, hash_key, sortkey_value_dict, ttl=0, timeout=0): - """ - Set multiple sort_keys-values under hash_key to be stored. - - :param hash_key: (str) which hash key used for this API. - :param sortkey_value_dict: (dict) pairs in dict. - :param ttl: (int) ttl(time to live) in seconds of these data. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, ign) - code: error_types.ERR_OK.value when data stored succeed. - ign: useless, should be ignored. - """ -``` - -#### get - -获取一条数据 - -``` -def get(self, hash_key, sort_key, timeout=0): - """ - Get value stored in . - - :param hash_key: (str) which hash key used for this API. - :param sort_key: (str) which sort key used for this API. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, value). - code: error_types.ERR_OK.value when data got succeed, error_types.ERR_OBJECT_NOT_FOUND.value when data not found. - value: data stored in this - """ -``` - -#### multi_get - -同时读一条hashkey的多条sortkey数据 - -``` -def multi_get(self, hash_key, - sortkey_set, - max_kv_count=100, - max_kv_size=1000000, - no_value=False, - timeout=0): - """ - Get multiple values stored in pairs. - - :param hash_key: (str) which hash key used for this API. - :param sortkey_set: (set) sort keys in set. - :param max_kv_count: (int) max count of k-v pairs to be fetched. max_fetch_count <= 0 means no limit. - :param max_kv_size: (int) max total data size of k-v pairs to be fetched. max_fetch_size <= 0 means no limit. - :param no_value: (bool) whether to fetch value of these keys. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, kvs) - code: error_types.ERR_OK.value when data got succeed. - kvs: pairs in dict. - """ -``` - -#### multi_get_opt - -同时读一条hashkey的多条sortkey数据, 读取的数据根据`multi_get_options`参数指定的模式确定。 - -``` -def multi_get_opt(self, hash_key, - start_sort_key, stop_sort_key, - multi_get_options, - max_kv_count=100, - max_kv_size=1000000, - timeout=0): - """ - Get multiple values stored in hash_key, and sort key range in [start_sort_key, stop_sort_key) as default. - - :param hash_key: (str) which hash key used for this API. - :param start_sort_key: (str) returned k-v pairs is start from start_sort_key. - :param stop_sort_key: (str) returned k-v pairs is stop at stop_sort_key. - :param multi_get_options: (MultiGetOptions) configurable multi_get options. - :param max_kv_count: (int) max count of k-v pairs to be fetched. max_fetch_count <= 0 means no limit. - :param max_kv_size: (int) max total data size of k-v pairs to be fetched. max_fetch_size <= 0 means no limit. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, kvs) - code: error_types.ERR_OK.value when data got succeed. - kvs: pairs in dict. - """ -``` - -其中，`MultiGetOptions`可以指定sortkey的范围、是否包含边界、子串匹配、是否返回value、是否逆序等，具体定义如下： - -``` -class MultiGetOptions(object): - """ - configurable options for multi_get. - """ - - def __init__(self): - self.start_inclusive = True - self.stop_inclusive = False - self.sortkey_filter_type = filter_type.FT_NO_FILTER - self.sortkey_filter_pattern = "" - self.no_value = False - self.reverse = False - -class filter_type: - FT_NO_FILTER = 0 - FT_MATCH_ANYWHERE = 1 - FT_MATCH_PREFIX = 2 - FT_MATCH_POSTFIX = 3 -``` - -#### remove - -删除一条数据 - -``` -def remove(self, hash_key, sort_key, timeout=0): - """ - Remove the entire -value in pegasus. - - :param hash_key: (str) which hash key used for this API. - :param sort_key: (str) which sort key used for this API. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, ign) - code: error_types.ERR_OK.value when data stored succeed. - ign: useless, should be ignored. - """ -``` - -#### multi_del - -批量删除一个hashkey下的多条sortkey数据 - -``` -def multi_del(self, hash_key, sortkey_set, timeout=0): - """ - Remove multiple entire -values in pegasus. - - :param hash_key: (str) which hash key used for this API. - :param sortkey_set: (set) sort keys in set. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, count). - code: error_types.ERR_OK.value when data got succeed. - count: count of deleted k-v pairs. - """ -``` - -#### sort_key_count - -获取一个hashkey下的sortkey数量 - -``` -def sort_key_count(self, hash_key, timeout=0): - """ - Get the total sort key count under the hash_key. - - :param hash_key: (str) which hash key used for this API. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, count) - code: error_types.ERR_OK.value when data got succeed, error_types.ERR_OBJECT_NOT_FOUND.value when data not found. - value: total sort key count under the hash_key. - """ -``` - -#### get_sort_keys - -获取一个hashkey下的sortkey值 - -``` -def get_sort_keys(self, hash_key, - max_kv_count=100, - max_kv_size=1000000, - timeout=0): - """ - Get multiple sort keys under hash_key. - - :param hash_key: (str) which hash key used for this API. - :param max_kv_count: (int) max count of k-v pairs to be fetched. max_fetch_count <= 0 means no limit. - :param max_kv_size: (int) max total data size of k-v pairs to be fetched. max_fetch_size <= 0 means no limit. - :param timeout: (int) how long will the operation timeout in milliseconds. - if timeout > 0, it is a timeout value for current operation, - else the timeout value specified to create the instance will be used. - :return: (tuple) (code, ks) - code: error_types.ERR_OK.value when data got succeed. - ks: pairs in dict, ign will always be empty str. - """ -``` - -#### get_scanner - -获取scanner对象，用于指定范围的数据扫描。可以通过`scan_options`参数指定扫描的模式。 - -``` -def get_scanner(self, hash_key, - start_sort_key, stop_sort_key, - scan_options): - """ - Get scanner for hash_key, start from start_sort_key, and stop at stop_sort_key. - Whether the scanner include the start_sort_key and stop_sort_key is configurable by scan_options - - :param hash_key: (str) which hash key used for this API. - :param start_sort_key: (str) returned scanner is start from start_sort_key. - :param stop_sort_key: (str) returned scanner is stop at stop_sort_key. - :param scan_options: (ScanOptions) configurable scan options. - :return: (PegasusScanner) scanner, instance of PegasusScanner. - """ -``` - -其中，`ScanOptions`可以指定是否包含边界、超时时间、一次从replica server批量获取的sortkey-value数量等，具体定义如下： - -``` -class ScanOptions(object): - """ - configurable options for scan. - """ - - def __init__(self): - self.timeout_millis = 5000 - self.batch_size = 1000 - self.start_inclusive = True - self.stop_inclusive = False - self.snapshot = None # for future use -``` - -#### get_unordered_scanners - -一次性获取多个scanner，用于整个table的数据扫描。可以通过`scan_options`参数指定扫描的模式。 - -``` -def get_unordered_scanners(self, max_split_count, scan_options): - """ - Get scanners for the whole pegasus table. - - :param max_split_count: (int) max count of scanners will be returned. - :param scan_options: (ScanOptions) configurable scan options. - :return: (list) instance of PegasusScanner list. - each scanner in this list can scan separate part of the whole pegasus table. - """ -``` - -#### scanner对象 - -用于数据扫描的对象，由`get_scanner`和`get_unordered_scanners`返回。使用它的`next`函数执行扫描过程。 - -``` -class PegasusScanner(object): - """ - Pegasus scanner class, used for scanning data in pegasus table. - """ -``` - -#### get_next - -获取扫描得到的数据，需要循环执行，直到返回`None`结束扫描。 - -``` -def get_next(self): - """ - scan the next k-v pair for the scanner. - - :return: (tuple, value> or None) - all the sort_keys returned by this API are in ascend order. - """ -``` diff --git a/_docs/zh/clients/python3-client.md b/_docs/zh/clients/python3-client.md index f9a0efa2..940fcb5a 100755 --- a/_docs/zh/clients/python3-client.md +++ b/_docs/zh/clients/python3-client.md @@ -6,7 +6,7 @@ permalink: clients/python3-client ## 项目地址 - + ## 版本要求 @@ -24,7 +24,7 @@ pegasus python3 client 从 python2-client 改动而来，对原有的接口参 ### 示例 -完整的示例请参考[sample](https://github.com/XiaoMi/pegasus-python-client/blob/python3/sample.py)。以下是简单的示例： +完整的示例请参考[sample](https://github.com/apache/incubator-pegasus/tree/master/python-client/sample.py)。以下是简单的示例： ``` #!/usr/bin/env python diff --git a/_docs/zh/clients/scala-client.md b/_docs/zh/clients/scala-client.md index e5762bd6..56b7791e 100755 --- a/_docs/zh/clients/scala-client.md +++ b/_docs/zh/clients/scala-client.md @@ -4,13 +4,13 @@ permalink: clients/scala-client # 获取客户端 -项目地址：[Pegasus scala client](https://github.com/xiaomi/pegasus-scala-client) +项目地址：[Pegasus scala client](https://github.com/apache/incubator-pegasus/tree/master/scala-client) 下载： ```bash -git clone https://github.com/XiaoMi/pegasus-scala-client.git -cd pegasus-scala-client +git clone git@github.com:apache/incubator-pegasus.git +cd incubator-pegasus/scala-client ``` -选择所使用的版本并构建，建议使用master版本。同时注意，scala客户端构建依赖[Java客户端](https://github.com/XiaoMi/pegasus-java-client)，请参考[获取Java客户端](/clients/java-client#获取java客户端)在项目中添加Java依赖。你可以打包成Jar包进行使用： +选择所使用的版本并构建，建议使用master版本。同时注意，scala客户端构建依赖[Java客户端](https://github.com/apache/incubator-pegasus/tree/master/java-client)，请参考[获取Java客户端](/clients/java-client#获取java客户端)在项目中添加Java依赖。你可以打包成Jar包进行使用： ```bash sbt package ``` diff --git a/_docs/zh/tools/shell.md b/_docs/zh/tools/shell.md index 1b9ceecc..c994f5f8 100755 --- a/_docs/zh/tools/shell.md +++ b/_docs/zh/tools/shell.md @@ -746,7 +746,7 @@ USAGE: multi_get_range * `-y|--sort_key_filter_pattern`参数：指定SortKey的过滤模式串，空串相当于无过滤。 * `-n|--max_count`参数：指定最多读取的数据条数。 * `-i|--no_value`参数：指定是否只返回HashKey和SortKey，不返回Value数据，默认为false。 -* `-r|--reverse`参数：是否逆向扫描数据库，从后往前查找数据，但是查找得到的结果在list中还是按照SortKey从小到大顺序存放。该参数从[v1.8.0版本](https://github.com/xiaomi/pegasus/releases/tag/v1.8.0)开始支持。 +* `-r|--reverse`参数：是否逆向扫描数据库，从后往前查找数据，但是查找得到的结果在list中还是按照SortKey从小到大顺序存放。该参数从[v1.8.0版本](https://github.com/apache/incubator-pegasus/releases/tag/v1.8.0)开始支持。 示例： ``` diff --git a/_overview/en/overview.md b/_overview/en/overview.md index 79d82c37..3e66b100 100755 --- a/_overview/en/overview.md +++ b/_overview/en/overview.md @@ -4,7 +4,7 @@ lang: en --- [PacificA]: https://www.microsoft.com/en-us/research/publication/pacifica-replication-in-log-based-distributed-storage-systems/ -[pegasus-rocksdb]: https://github.com/xiaomi/pegasus-rocksdb +[pegasus-rocksdb]: https://rocksdb.org/ [hbase]: https://hbase.apache.org/ Apache Pegasus is a distributed key-value storage system which is designed to be: diff --git a/_overview/zh/overview.md b/_overview/zh/overview.md index 47af2941..373f3642 100755 --- a/_overview/zh/overview.md +++ b/_overview/zh/overview.md @@ -6,7 +6,7 @@ Apache Pegasus是一个分布式Key-Value存储系统，它的设计目标是具 - **高扩展性**：通过哈希分片实现分布式横向扩展。 - **强一致性**：通过[PacificA](https://www.microsoft.com/en-us/research/publication/pacifica-replication-in-log-based-distributed-storage-systems/)一致性协议保证。 -- **高性能**：底层使用[RocksDB](https://github.com/xiaomi/pegasus-rocksdb)作为存储引擎。 +- **高性能**：底层使用[RocksDB](https://rocksdb.org/)作为存储引擎。 - **简单易用**：基于Key-Value的良好接口。 ## 背景