Skip to content

Commit

Permalink
Bump to version 1.1.0 (#322)
Browse files Browse the repository at this point in the history
  • Loading branch information
Xuanwo authored May 14, 2020
1 parent 23b12df commit 9ed5dde
Show file tree
Hide file tree
Showing 18 changed files with 343 additions and 107 deletions.
36 changes: 35 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [v1.1.0] - 2020-05-14

### Proposal

- services, design: Propose and implement 21-split-segmenter (#270)

### Added

- types/pairs: Add more comment for work dir
- pkg/httpclient: Add stream-oriented http client support (#274)
- services/qingstor: Detect bucket location automatically (#278)
- ci: Setup drone for integration test (#282)
- services/qingstor: Allow read with offset and size (#283)
- services: Add service level pair support (#311)
- services/*: Add service level storage class support (#313)
- services/qingstor: Add disable uri cleaning support (#314)

### Changed

- types/pairs: Allow parse from plain string value (#281)
- docs: Use vuepress instead (#286)
- docs: Import user experience (#287)
- docs: Get ready for i18n docs (#289)
- docs: Add crowdin support for i18n (#290)
- *: Refactor definitions generator (#303)
- definitions: Auto inject http_client_options for all services (#321)

### Fixed

- internal/cmd: Fix -ignore flag in go generate matches unexpected files (#268)
- services/azblob: Fix context deadline exceeded while reading (#275)
- pkg/httpclient: Fix connection closed while writing or reading (#305)

## [v1.0.0] - 2020-04-23

### Proposal
Expand Down Expand Up @@ -238,7 +271,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
- Add pair based option and metadata support.
- Add qingstor services.

[Unreleased]: https://github.com/Xuanwo/storage/compare/v1.0.0...HEAD
[Unreleased]: https://github.com/Xuanwo/storage/compare/v1.1.0...HEAD
[v1.1.0]: https://github.com/Xuanwo/storage/compare/v1.0.0...v1.1.0
[v1.0.0]: https://github.com/Xuanwo/storage/compare/v0.9.0...v1.0.0
[v0.9.0]: https://github.com/Xuanwo/storage/compare/v0.8.0...v0.9.0
[v0.8.0]: https://github.com/Xuanwo/storage/compare/v0.7.2...v0.8.0
Expand Down
2 changes: 2 additions & 0 deletions crowdin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@ commit_message: '[ci skip] i18n: Translate %original_file_name% to %language%'
append_commit_message: false
files:
- source: /docs/**/*.md
ignore:
- /docs/%locale%/
translation: /docs/%locale%/**/%original_file_name%
94 changes: 47 additions & 47 deletions docs/zh-CN/README.md
Original file line number Diff line number Diff line change
@@ -1,52 +1,52 @@
# storage

[![Build Status](https://travis-ci.com/Xuanwo/storage.svg?branch=master)](https://travis-ci.com/Xuanwo/storage) [![GoDoc](https://godoc.org/github.com/Xuanwo/storage?status.svg)](https://godoc.org/github.com/Xuanwo/storage) [![Go Report Card](https://goreportcard.com/badge/github.com/Xuanwo/storage)](https://goreportcard.com/report/github.com/Xuanwo/storage) [![codecov](https://codecov.io/gh/Xuanwo/storage/branch/master/graph/badge.svg)](https://codecov.io/gh/Xuanwo/storage) [![License](https://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/Xuanwo/storage/blob/master/LICENSE) [![Codacy Badge](https://api.codacy.com/project/badge/Grade/15867a455afc4f24a763a5ed1011e05a)](https://app.codacy.com/manual/Xuanwo/storage?utm_source=github.com&utm_medium=referral&utm_content=Xuanwo/storage&utm_campaign=Badge_Grade_Settings) [![Join the chat](https://img.shields.io/badge/chat-online-blue?style=flat&logo=telegram)](https://t.me/storage_dev)
[![构建状态](https://travis-ci.com/Xuanwo/storage.svg?branch=master)](https://travis-ci.com/Xuanwo/storage) [![GoDoc](https://godoc.org/github.com/Xuanwo/storage?status.svg)](https://godoc.org/github.com/Xuanwo/storage) [![Go Report Card](https://goreportcard.com/badge/github.com/Xuanwo/storage)](https://goreportcard.com/report/github.com/Xuanwo/storage) [![codecov](https://codecov.io/gh/Xuanwo/storage/branch/master/graph/badge.svg)](https://codecov.io/gh/Xuanwo/storage) [![License](https://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/Xuanwo/storage/blob/master/LICENSE) [![Codacy Badge](https://api.codacy.com/project/badge/Grade/15867a455afc4f24a763a5ed1011e05a)](https://app.codacy.com/manual/Xuanwo/storage?utm_source=github.com&utm_medium=referral&utm_content=Xuanwo/storage&utm_campaign=Badge_Grade_Settings) [![加入聊天](https://img.shields.io/badge/chat-online-blue?style=flat&logo=telegram)](https://t.me/storage_dev)

An application-oriented unified storage layer for Golang.
面向应用程序的 Golang 统一存储层。

## Goal
## 目标

- Production ready
- High performance
- Vendor lock free
- 可用于生产环境
- 高性能
- 无供应商锁定

## Features
## 功能

### Servicer Level
### 服务级别

- Basic operations across implemented storage services with the same API
- List: list all Storager in service
- Get: get a Storager via name
- Create: create a Storager
- Delete: delete a Storager
- 使用相同的 API 对存储服务进行基本操作
- List: 列出所有 Storager
- Get: 通过名称获取 Storager
- Create: 创建一个 Storager
- Delete: 删除一个 Storager

### Storager Level
### 存储级别

- Basic operations across all storage services with the same API
- Read: read file content
- Write: write content into file
- List: list files under a dir or prefix
- Stat: get file's metadata
- Delete: delete a file
- Metadata: get storage service's metadata
- Advanced operations across implemented storage services with the same API
- Copy: copy a file
- Move: move a file
- Reach: generate a public accessible url
- Statistical: get storage service's statistics
- Segment: Full support for Segment, aka, Multipart
- 使用相同的 API 对存储服务进行基本操作
- Read: 读取文件内容
- Write: 将内容写入文件
- List: 列取一个目录或者前缀下的文件
- Stat: 获取文件的元数据
- Delete: 删除一个文件
- Metadata: 获取存储服务的元数据
- 使用相同的 API 对存储服务进行高级操作
- Copy: 复制一个文件
- Move: 移动一个文件
- Reach: 生成一个可公开访问的 URL
- Statistical: 获取存储服务的统计数据
- Segment: 对分块/分段的完整支持

### File Level
### 文件级别

- Metadata
- Content Length / Size: Full support via [RFC 2616](https://tools.ietf.org/html/rfc2616)
- Content MD5 / ETag: Full support via [proposal](docs/design/14-normalize-content-hash-check.md)
- Content Type: Full support via [RFC 2616](https://tools.ietf.org/html/rfc2616)
- Storage Class: Full support via [proposal](docs/design/8-normalize-metadata-storage-class.md)
- 元数据
- Content Length / Size: 通过 [RFC 2616](https://tools.ietf.org/html/rfc2616) 实现完整支持
- Content MD5 / ETag: 通过 [proposal](docs/design/14-normalize-content-hash-check.md) 实现完整支持
- Content Type: 通过 [RFC 2616](https://tools.ietf.org/html/rfc2616) 实现完整支持
- Storage Class: 通过 [proposal](docs/design/8-normalize-metadata-storage-class.md) 实现完整支持

## Installation
## 安装

Install will `go get`
通过 `go get` 安装

```bash
go get github.com/Xuanwo/storage
Expand All @@ -58,31 +58,31 @@ Import
import "github.com/Xuanwo/storage"
```

## Quick Start
## 快速开始

```go
// Init a service.
// 初始化服务。
store, err := coreutils.OpenStorager("fs", pairs.WithWorkDir("/tmp"))
if err != nil {
log.Fatalf("service init failed: %v", err)
}

// Use Storager API to maintain data.
// 使用 Storager API 来维护数据。
r, err := store.Read("path/to/file")
if err != nil {
log.Printf("storager read: %v", err)
}
```

## Services
## 服务

- [azblob](./services/azblob/): [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/)
- [cos](./services/cos/): [Tencent Cloud Object Storage](https://cloud.tencent.com/product/cos)
- [dropbox](./services/dropbox/): [Dropbox](https://www.dropbox.com)
- [fs](./services/fs/): Local file system
- [gcs](./services/gcs/): [Google Cloud Storage](https://cloud.google.com/storage/)
- [zblob](./services/azblob/): [Azure Blob 存储](https://docs.microsoft.com/en-us/azure/storage/blobs/)
- [cos](./services/cos/): [腾讯云对象存储](https://cloud.tencent.com/product/cos)
- [Dropbox](./services/dropbox/): [Dropbox](https://www.dropbox.com)
- [fs](./services/fs/): 本地文件系统
- [gcs](./services/gcs/): [Google 云存储](https://cloud.google.com/storage/)
- [kodo](./services/kodo/): [qiniu kodo](https://www.qiniu.com/products/kodo)
- [oss](./services/oss/): [Aliyun Object Storage](https://www.aliyun.com/product/oss)
- [qingstor](./services/qingstor/): [QingStor Object Storage](https://www.qingcloud.com/products/qingstor/)
- [oss](./services/oss/): [Aliyun 对象存储](https://www.aliyun.com/product/oss)
- [qingstor](./services/qingstor/): [QingStor 对象存储](https://www.qingcloud.com/products/qingstor/)
- [s3](./services/s3/): [Amazon S3](https://aws.amazon.com/s3/)
- [uss](./services/uss/): [UPYUN Storage Service](https://www.upyun.com/products/file-storage)
- [uss](./services/uss/): [UPYUN 存储服务](https://www.upyun.com/products/file-storage)
11 changes: 7 additions & 4 deletions docs/zh-CN/design/11-error-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ adds:

## Background

[storage](https://github.com/Xuanwo/storage) intends to be a production ready storage layer, and error handling is one of the most important parts.
[storage][] intends to be a production ready storage layer, and error handling is one of the most important parts.

While facing an error, user should be capable to do following things:

Expand All @@ -24,10 +24,10 @@ In order to provide those capabilities, we should return error with contextual i

So I propose following changes:

- Add spec [spec: error handling](../spec/1-error-handling.md) to normalize error handling across the whole lib
- Add spec [spec: error handling][] to normalize error handling across the whole lib
- All error handling related code should be refactored

Take package `segment` as example, as described in [spec: error handling](../spec/1-error-handling.md), we will have an `Error` struct to carry contextual information:
Take package `segment` as example, as described in [spec: error handling][], we will have an `Error` struct to carry contextual information:

```go
type SegError struct {
Expand Down Expand Up @@ -101,8 +101,11 @@ if err != nil && errors.As(err, &e) {

## Compatibility

Error returned by [storage](https://github.com/Xuanwo/storage) could be changed.
Error returned by [storage][] could be changed.

## Implementation

Most of the work would be done by the author of this proposal.

[storage]: https://github.com/Xuanwo/storage
[spec: error handling]: ../spec/1-error-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ updates:

## Background

We removed recursive support in [2-use-callback-in-list-operations](./2-use-callback-in-list-operations.md), and in the same proposal, we outlined that:
We removed recursive support in [2-use-callback-in-list-operations][], and in the same proposal, we outlined that:

> Directory based storage will only list one directory, and prefix based storage will only list one prefix without a delimiter.
Expand Down Expand Up @@ -47,4 +47,6 @@ No breaking changes.

## Implementation

Most of the work would be done by the author of this proposal.
Most of the work would be done by the author of this proposal.

[2-use-callback-in-list-operations]: ./2-use-callback-in-list-operations.md
15 changes: 10 additions & 5 deletions docs/zh-CN/design/13-remove-config-string.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ deprecates:

## Background

[storage](https://github.com/Xuanwo/storage) added config string support in proposal [3-support-service-init-via-config-string](https://github.com/Xuanwo/storage/blob/master/docs/design/3-support-service-init-via-config-string.md), and updated in proposal [9-remove-storager-init](https://github.com/Xuanwo/storage/blob/master/docs/design/9-remove-storager-init.md). These proposals allow users init services like following:
[storage][] added config string support in proposal [3-support-service-init-via-config-string][], and updated in proposal [9-remove-storager-init][]. These proposals allow users init services like following:

```go
srv, store, err := coreutils.Open("fs:///?work_dir=/path/to/dir")
Expand All @@ -19,9 +19,9 @@ if err != nil {
}
```

With time goes on, and deeper understanding of [storage](https://github.com/Xuanwo/storage)'s configuration, I found `config string` is not a good solution. `config string` does have some benefits: simple string, easy to construct, easy to understand(?).
With time goes on, and deeper understanding of [storage][]'s configuration, I found `config string` is not a good solution. `config string` does have some benefits: simple string, easy to construct, easy to understand(?).

However, after some experience on demo project [bard](https://github.com/Xuanwo/bard), which is a paste bin service built upon [storage](https://github.com/Xuanwo/storage), I found the `config string` deeply influences end user side configuration. [bard](https://github.com/Xuanwo/bard)'s config looks like following:
However, after some experience on demo project [bard][], which is a paste bin service built upon [storage][], I found the `config string` deeply influences end user side configuration. [bard][]'s config looks like following:

```yaml
public_url: http://127.0.0.1:8080
Expand All @@ -37,7 +37,7 @@ database:
storage: "fs:///?work_dir=/tmp/bard/data"
```
Every application built upon [storage](https://github.com/Xuanwo/storage) either exposes config string to end user directly or writes a format config function to convert their own config to [storage](https://github.com/Xuanwo/storage) config string. This is unexpected.
Every application built upon [storage][] either exposes config string to end user directly or writes a format config function to convert their own config to [storage][] config string. This is unexpected.
Not only that, config string also makes it hard to construct type safe pairs. We need to parse them from string and can't have effective use of existing configuration formats.
Expand Down Expand Up @@ -72,4 +72,9 @@ Following packages will be affected:

## Implementation

Most of the work would be done by the author of this proposal.
Most of the work would be done by the author of this proposal.

[storage]: https://github.com/Xuanwo/storage
[3-support-service-init-via-config-string]: https://github.com/Xuanwo/storage/blob/master/docs/design/3-support-service-init-via-config-string.md
[9-remove-storager-init]: https://github.com/Xuanwo/storage/blob/master/docs/design/9-remove-storager-init.md
[bard]: https://github.com/Xuanwo/bard
9 changes: 6 additions & 3 deletions docs/zh-CN/design/14-normalize-content-hash-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ So I propose following changes:

HTTP Related Standards

- [Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content](https://www.rfc-editor.org/rfc/rfc7231)
- [Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests](https://www.rfc-editor.org/rfc/rfc7232)
- [Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content][]
- [Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests][]
- [Permanent Message Header Field Names](https://www.iana.org/assignments/message-headers/message-headers.xml#perm-headers)

Storage Service Reference Document
Expand All @@ -46,4 +46,7 @@ No break changes

## Implementation

Most of the work would be done by the author of this proposal.
Most of the work would be done by the author of this proposal.

[Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content]: https://www.rfc-editor.org/rfc/rfc7231
[Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests]: https://www.rfc-editor.org/rfc/rfc7232
29 changes: 17 additions & 12 deletions docs/zh-CN/design/15-release-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,32 +8,32 @@ updated_at: 2020-03-10

## Background

Golang team makes sure the go 1 compatibility in [Go 1 and the Future of Go Programs](https://golang.org/doc/go1compat):
Golang team makes sure the go 1 compatibility in [Go 1 and the Future of Go Programs][]:

> It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification.
[storage](https://github.com/Xuanwo/storage) should have similar statement about its release policy.
[storage][] should have similar statement about its release policy.

## Proposal

So I propose that [storage](https://github.com/Xuanwo/storage) should obey the following policies:
So I propose that [storage][] should obey the following policies:

### Semantic Versioning

**[storage](https://github.com/Xuanwo/storage) SHOULD follow [Semantic Versioning](https://semver.org/)**
**[storage][] SHOULD follow [Semantic Versioning](https://semver.org/)**

- After `1.0.0` released, [storage](https://github.com/Xuanwo/storage) will follow semantic versioning strictly
- All exported items in [storage](https://github.com/Xuanwo/storage) except `internal` and `tests` will be included in semantic versioning
- After `1.0.0` released, [storage][] will follow semantic versioning strictly
- All exported items in [storage][] except `internal` and `tests` will be included in semantic versioning

[storage](https://github.com/Xuanwo/storage) uses [dependbot](https://dependabot.com/) to upgrade its dependences automatically, the upgrade process will be like follows:
[storage][] uses [dependbot][] to upgrade its dependences automatically, the upgrade process will be like follows:

- [dependbot](https://dependabot.com/) will create new PRs to branch [dependence](https://github.com/Xuanwo/storage/tree/dependence) and merge them after build succeeded
- [storage](https://github.com/Xuanwo/storage) will merge branch [dependence](https://github.com/Xuanwo/storage/tree/dependence) in every release PR
- After release PR get merged, branch [dependence](https://github.com/Xuanwo/storage/tree/dependence) should be reset to branch master
- [dependbot][] will create new PRs to branch [dependence][] and merge them after build succeeded
- [storage][] will merge branch [dependence][] in every release PR
- After release PR get merged, branch [dependence][] should be reset to branch master

### Target Golang Versions

**[storage](https://github.com/Xuanwo/storage) SHOULD be compatible with the last two golang major versions**
**[storage][] SHOULD be compatible with the last two golang major versions**

Assuming the current version is go 1.14

Expand All @@ -52,4 +52,9 @@ None

## Implementation

No code related changes.
No code related changes.

[Go 1 and the Future of Go Programs]: https://golang.org/doc/go1compat
[storage]: https://github.com/Xuanwo/storage
[dependbot]: https://dependabot.com/
[dependence]: https://github.com/Xuanwo/storage/tree/dependence
6 changes: 4 additions & 2 deletions docs/zh-CN/design/16-loose-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ updated_at: 2020-03-12

## Background

Current [storage](https://github.com/Xuanwo/storage)'s pair handle behavior is inconsistent.
Current [storage][]'s pair handle behavior is inconsistent.

In all `parseStoragePairXXX` functions, we will ignore not supported pairs via only pick supported one:

Expand Down Expand Up @@ -95,4 +95,6 @@ None.

## Implementation

Most of the work would be done by the author of this proposal.
Most of the work would be done by the author of this proposal.

[storage]: https://github.com/Xuanwo/storage
Loading

1 comment on commit 9ed5dde

@vercel
Copy link

@vercel vercel bot commented on 9ed5dde May 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.