feat(services/ipfs): Move services ipfs back

.github/dependabot.yml

@@ -27,20 +27,24 @@ updates:
   - package-ecosystem: gomod
     directory: "/services/dropbox"
     schedule:
-      interval: daily  
+      interval: daily
   - package-ecosystem: gomod
     directory: "/services/ftp"
     schedule:
       interval: daily
   - package-ecosystem: gomod
     directory: "/services/gcs"
     schedule:
-      interval: daily    
+      interval: daily
   - package-ecosystem: gomod
     directory: "/services/gdrive"
     schedule:
-      interval: daily    
+      interval: daily
   - package-ecosystem: gomod
     directory: "/services/hdfs"
     schedule:
-      interval: daily 
+      interval: daily
+  - package-ecosystem: gomod
+    directory: "/services/ipfs"
+    schedule:
+      interval: daily

.github/workflows/services-test-ipfs.yml

@@ -0,0 +1,43 @@
+name: "Services Test Ipfs"
+
+on: 
+   push:
+     paths:
+       - 'services/ipfs/**'
+   pull_request:
+     paths:
+       - 'services/ipfs/**'
+
+jobs:
+  integration_test:
+    name: "Integration Test"
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        go: [ "1.15", "1.16" ]
+        ipfs: ["0.8", "0.9"]
+
+    steps:
+      - name: Set up Go 1.x
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ matrix.go }}
+
+      - name: Checkout repository
+        uses: actions/checkout@v2
+
+      - name: Set up IPFS ${{ matrix.ipfs }}
+        uses: ibnesayeed/setup-ipfs@8f99e662ba25be9a1158fffdeea57e9024468bfd
+        id: ipfs_setup
+        with:
+          ipfs_version: ${{ matrix.ipfs }}
+          run_daemon: true
+
+      - name: Test
+        env:
+          STORAGE_IPFS_INTEGRATION_TEST: "on"
+          STORAGE_IPFS_ENDPOINT: "http:127.0.0.1:5001"
+          STORAGE_IPFS_GATEWAY: "http:127.0.0.1:8080"
+        working-directory: services/ipfs
+        run: make integration_test

services/ipfs/.gitignore

@@ -0,0 +1,7 @@
+coverage.*
+bin/
+Makefile.env
+
+# IDE
+.idea
+.vscode

services/ipfs/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Change Log
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/)
+and this project adheres to [Semantic Versioning](https://semver.org/).
+
+## [v0.2.0] - 2021-09-13
+
+### Added
+
+- docs: RFC-13 Add Gateway Pair (#13)
+
+### Changed
+
+- ci: Enable auto merge for Dependabot
+- docs: Update README (#22)
+
+### Fixed
+
+- fix: Code not generated
+
+### Upgraded
+
+- ci: Upgrade fetch-metadata
+- build(deps): Bump github.com/beyondstorage/go-endpoint to 1.1.0 (#19)
+
+## v0.1.0 - 2021-07-15
+
+### Added
+
+- Implement IPFS Storager, Copier, Mover, Direr.
+
+[v0.2.0]: https://github.com/beyondstorage/go-service-ipfs/compare/v0.1.0...v0.2.0

services/ipfs/Makefile

@@ -0,0 +1,43 @@
+SHELL := /bin/bash
+
+-include Makefile.env
+
+.PHONY: all check format lint build test generate tidy
+
+help:
+	@echo "Please use \`make <target>\` where <target> is one of"
+	@echo "  check               to do static check"
+	@echo "  build               to create bin directory and build"
+	@echo "  generate            to generate code"
+	@echo "  test                to run test"
+
+check: vet
+
+format:
+	go fmt ./...
+
+vet:
+	go vet ./...
+
+generate:
+	@echo "generate code"
+	go generate ./...
+	go fmt ./...
+
+build: tidy generate check
+	go build ./...
+
+test:
+	go test -race -coverprofile=coverage.txt -covermode=atomic -v .
+	go tool cover -html="coverage.txt" -o "coverage.html"
+
+integration_test:
+	go test -race -count=1 -covermode=atomic -v ./tests
+
+tidy:
+	go mod tidy
+	go mod verify
+
+clean:
+	@echo "clean generated files"
+	find . -type f -name 'generated.go' -delete

services/ipfs/Makefile.env.example

@@ -0,0 +1,3 @@
+export STORAGE_IPFS_INTEGRATION_TEST=on
+export STORAGE_IPFS_ENDPOINT=http:127.0.0.1:5001
+export STORAGE_IPFS_GATEWAY=http:127.0.0.1:8080

services/ipfs/README.md

@@ -0,0 +1,35 @@
+[![Services Test Ipfs](https://github.com/beyondstorage/go-storage/actions/workflows/services-test-ipfs.yml/badge.svg)](https://github.com/beyondstorage/go-storage/actions/workflows/services-test-ipfs.yml)
+
+# ipfs
+
+[InterPlanetary File System(IPFS)](https://ipfs.io/) support for [go-storage](https://github.com/beyondstorage/go-storage).
+
+## Install
+
+```go
+go get go.beyondstorage.io/services/ipfs
+```
+
+## Usage
+
+```go
+import (
+	"log"
+
+	_ "go.beyondstorage.io/services/ipfs"
+	"go.beyondstorage.io/v5/services"
+)
+
+func main() {
+	store, err := services.NewStoragerFromString("ipfs:///path/to/workdir?endpoint=<ipfs_http_api_endpoint>&gateway=<ipfs_http_gateway>")
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	// Write data from io.Reader into hello.txt
+	n, err := store.Write("hello.txt", r, length)
+}
+```
+
+- See more examples in [go-storage-example](https://github.com/beyondstorage/go-storage-example).
+- Read [more docs](https://beyondstorage.io/docs/go-storage/services/ipfs) about go-service-ipfs.

services/ipfs/doc.go

@@ -0,0 +1,6 @@
+/*
+Package ipfs provided support for the InterPlanetary File System (IPFS).
+*/
+package ipfs
+
+//go:generate go run -tags tools go.beyondstorage.io/v5/cmd/definitions service.toml

services/ipfs/docs/rfcs/0-example.md

@@ -0,0 +1,41 @@
+- Author: (fill me in with `name <mail>`, e.g., zu1k <i@lgf.im>)
+- Start Date: (fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [beyondstorage/go-service-ipfs#0](https://github.com/beyondstorage/go-service-ipfs/issues/0)
+- Tracking Issue: [beyondstorage/go-service-ipfs#0](https://github.com/beyondstorage/go-service-ipfs/issues/0)
+
+# RFC-0: <proposal name>
+
+- Updates: (delete this part if not applicable)
+    - [GSP-20](./20-abc): Deletes something
+- Updated By: (delete this part if not applicable)
+    - [GSP-10](./10-do-be-do-be-do): Adds something
+    - [GSP-1000](./1000-lalala): Deprecates this RFC
+
+## Background
+
+Explain why we are doing this.
+
+Related issues and early discussions can be linked, but the RFC should try to be self-contained if possible.
+
+## Proposal
+
+<proposal's content>
+
+## Rationale
+
+<proposal's rationale content, other implementations>
+
+Possible content:
+
+- Design Principles
+- Drawbacks
+- Alternative implementations and comparison
+- Possible Q&As
+
+## Compatibility
+
+<proposal's compatibility statement>
+
+## Implementation
+
+Explain what steps should be done to implement this proposal.

services/ipfs/docs/rfcs/13-add-gateway-pair.md

@@ -0,0 +1,69 @@
+- Author: zu1k <i@lgf.im>
+- Start Date: 2021-07-15
+- RFC PR: [beyondstorage/go-service-ipfs#13](https://github.com/beyondstorage/go-service-ipfs/pull/13)
+- Tracking Issue: [beyondstorage/go-service-ipfs#14](https://github.com/beyondstorage/go-service-ipfs/issues/14)
+
+# RFC-13: Add Gateway Pair
+
+Releated issue: [beyondstorage/go-service-ipfs#5](https://github.com/beyondstorage/go-service-ipfs/issues/5)
+
+## Background
+
+With IPFS, we can get the `CID` of an object, then we can use it with a public or internal IPFS Gateway to splice out the access link.
+
+When using internal gateway like `127.0.0.1:8080`, the access link are inaccessible from external network.
+
+When using a public gateway, e.g. `ipfs.io`, `cf-ipfs.com`, if the service is an intranet IPFS cluster, it is possible that the files are not accessible from these public gateway as well.
+
+### How to get the `CID` of an object?
+
+We can [stat](https://docs.ipfs.io/reference/http/api/#api-v0-files-stat) the object, the `hash` would be its `CID`.
+
+```
+$ ipfs files stat /part1
+QmSvxBhBHn2gu9kAdjCTYfUgDNnyBcNoXXVojRyKdBjZUA  # CID
+Size: 262144
+CumulativeSize: 262158
+ChildBlocks: 0
+Type: file
+```
+
+### How to splice out the access link?
+
+- https://{gateway URL}/ipfs/{CID}/{optional path to resource}
+- https://{CID}.ipfs.{gatewayURL}/{optional path to resource}
+
+## Proposal
+
+I propose to add a pair to let the user specify the `gateway`.
+
+- The `type` of `gateway` should be `String`
+- The `format` of `gateway` should follow [go-endpoint](https://github.com/beyondstorage/go-endpoint/blob/master/README.md)
+- The `value` of `gateway` should be parsed into `HTTP` or `HTTPS`
+- Now we use `gateway` only in `Reach` operation
+
+## Rationale
+
+### Why not infer the `gateway` from the `endpoint`?
+
+If the `endpoint` used is not the loopback address, it is possible to infer the `gateway` address.
+
+For example, if a user tells IPFS to listen to `0.0.0.0` and use the public IP to access, the `endpoint` may be `<IP>:5001` or `<domain>:5001`, we can infer the `gateway` as `<IP>:8080` or `<domain>:8080`.
+
+The problem with this approach is that it is very unstable and the IPFS api does not come with a forensic mechanism, so it is almost impossible for users to access directly through the public IP, so the inferred gateway is most likely invalid.
+
+### Why not use the `gateway` in the configuration?
+
+We can get the `gateway` of IPFS through the [config-show api](https://docs.ipfs.io/reference/http/api/#api-v0-config-show).
+
+This method works if the user has modified the default gateway. 
+
+However, we cannot guarantee that the user has modified this configuration item, and it may be more common for users to use reverse proxy to make IPFS public.
+
+## Compatibility
+
+No compatibility issues at this time.
+
+## Implementation
+
+First we define a pair of `String` type with the name `gateway` in `service.toml`. Then we `generate` code and implement the `Reacher` interface.