Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: migrate to viper configure manage #1946

Merged
merged 3 commits into from
Sep 23, 2021

Conversation

bzp2010
Copy link
Contributor

@bzp2010 bzp2010 commented Jun 21, 2021

Please answer these questions before submitting a pull request, or your PR will get closed.

Why submit this pull request?

  • Bugfix
  • New feature provided
  • Improve performance
  • Backport patches

What changes will this PR take into?

The configuration management is implemented by viper library instead of reading yaml file directly.
It will provide support for YAML, JSON, TOML and other configuration file formats, and even remote loading through etcd, which will be controlled by -c or --config parameters.

And no incompatible updates introduced.

Related issues

part of #1931

Checklist:

  • Did you explain what problem does this PR solve? Or what new features have been added?
  • Have you added corresponding test cases?
  • Have you modified the corresponding document?
  • Is this PR backward compatible? If it is not backward compatible, please discuss on the mailing list first

@bzp2010 bzp2010 added enhancement New feature or request backend labels Jun 21, 2021
@bzp2010 bzp2010 self-assigned this Jun 21, 2021
@netlify
Copy link

netlify bot commented Jun 21, 2021

✔️ Deploy Preview for apisix-dashboard ready!

🔨 Explore the source changes: 92eeef1

🔍 Inspect the deploy log: https://app.netlify.com/sites/apisix-dashboard/deploys/60e67a5136b608000732921d

😎 Browse the preview: https://deploy-preview-1946--apisix-dashboard.netlify.app

@codecov-commenter
Copy link

codecov-commenter commented Jun 21, 2021

Codecov Report

Merging #1946 (a54fe02) into master (3cfc327) will increase coverage by 2.33%.
The diff coverage is 49.09%.

❗ Current head a54fe02 differs from pull request most recent head 92eeef1. Consider uploading reports for the commit 92eeef1 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1946      +/-   ##
==========================================
+ Coverage   66.79%   69.13%   +2.33%     
==========================================
  Files         182       60     -122     
  Lines        6575     3428    -3147     
  Branches      753        0     -753     
==========================================
- Hits         4392     2370    -2022     
+ Misses       1907      785    -1122     
+ Partials      276      273       -3     
Flag Coverage Δ
backend-e2e-test 44.22% <49.09%> (-0.24%) ⬇️
backend-e2e-test-ginkgo 46.70% <49.09%> (-0.63%) ⬇️
backend-unit-test 52.26% <ø> (ø)
frontend-e2e-test ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
api/internal/conf/conf.go 53.26% <48.14%> (-2.04%) ⬇️
api/cmd/root.go 78.04% <100.00%> (ø)
api/internal/core/server/server.go 35.84% <0.00%> (-13.21%) ⬇️
api/internal/utils/closer.go 33.33% <0.00%> (ø)
api/internal/core/store/store.go 88.30% <0.00%> (ø)
web/src/pages/Upstream/List.tsx
web/src/pages/Dashboard/service.ts
web/src/components/Plugin/UI/proxy-mirror.tsx
...nents/Upstream/components/active-check/Timeout.tsx
web/src/global.tsx
... and 117 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cfc327...92eeef1. Read the comment docs.

@bzp2010 bzp2010 force-pushed the feat-migrate-viper branch 9 times, most recently from 4403a17 to 31526f1 Compare June 21, 2021 09:22
@bzp2010 bzp2010 changed the title feat: migrate to viper configure manage refactor: migrate to viper configure manage Jun 21, 2021
@bzp2010
Copy link
Contributor Author

bzp2010 commented Jun 22, 2021

2021-06-22 Update1

There are some key problems in refactoring.

  1. some functions of configuration file path and work-dir parameter overlap
  2. ci test script execution path problem causes the program unable to automatically find the configuration file location

I'm trying to refactor the server module and the service daemon module.

@bzp2010 bzp2010 mentioned this pull request Jun 22, 2021
8 tasks
@bzp2010 bzp2010 force-pushed the feat-migrate-viper branch 2 times, most recently from 120a4cb to 92eeef1 Compare July 8, 2021 04:08
@bzp2010 bzp2010 marked this pull request as ready for review July 8, 2021 04:23
github.com/prometheus/client_golang v1.8.0 // indirect
github.com/satori/go.uuid v1.2.0
github.com/shiningrush/droplet v0.2.6-0.20210127040147-53817015cd1b
github.com/shiningrush/droplet/wrapper/gin v0.2.1
github.com/sirupsen/logrus v1.7.0 // indirect
github.com/sony/sonyflake v1.0.0
github.com/spf13/cobra v0.0.3
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.8.1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember that the license of a dependency of viper was not compatible with Apache 2.0, please confirm.

Copy link
Contributor Author

@bzp2010 bzp2010 Jul 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @nic-chen. I'm sorry for the slow response.

I used google/go-licenses to generate a list of the spf13/viper's dependencies and licenses.

library URL license
github.com/mitchellh/mapstructure https://github.com/mitchellh/mapstructure/blob/master/LICENSE MIT
github.com/magiconair/properties https://github.com/magiconair/properties/blob/master/LICENSE.md BSD-2-Clause
github.com/spf13/afero https://github.com/spf13/afero/blob/master/LICENSE.txt Apache-2.0
github.com/spf13/pflag https://github.com/spf13/pflag/blob/master/LICENSE BSD-3-Clause
github.com/hashicorp/hcl https://github.com/hashicorp/hcl/blob/master/LICENSE MPL-2.0
github.com/pelletier/go-toml https://github.com/pelletier/go-toml/blob/master/LICENSE Apache-2.0
github.com/spf13/viper https://github.com/spf13/viper/blob/master/LICENSE MIT
github.com/spf13/cast https://github.com/spf13/cast/blob/master/LICENSE MIT
github.com/spf13/jwalterweatherman https://github.com/spf13/jwalterweatherman/blob/master/LICENSE MIT
github.com/subosito/gotenv https://github.com/subosito/gotenv/blob/master/LICENSE MIT
github.com/fsnotify/fsnotify https://github.com/fsnotify/fsnotify/blob/master/LICENSE BSD-3-Clause
gopkg.in/yaml.v2 https://github.com/go-yaml/yaml Apahce-2.0
golang.org/x/text https://golang.org/x/text BSD-3-Clause
gopkg.in/ini.v1 https://gopkg.in/ini.v1 Apache-2.0

Include Apache-2.0, MIT, MPL-2.0, BSD-2-Clause and BSD-3-Clause.

And these licenses are also used in current Manager API dependencies.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about MPL-2.0, @moonming please have a look, thanks.

Copy link
Contributor Author

@bzp2010 bzp2010 Jul 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Description

Viper library uses the API of hashicorp/hcl open source repository using MPL-2.0 license, but does not use its code directly.

Solution

If importing a library using MPL-2.0 license is not feasible, I will consider implementing a shim code and replacing it with go mod replace. It emulates the API of that library and provides empty results.

It's like the following image.
image

So is such a solution feasible?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I searched the go project under apache, and many projects also use viper
for example:

https://github.com/apache/skywalking-banyandb

https://github.com/apache/camel-k

I have no problem here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MPL-2.0 maybe can NOT in apache projects: https://www.apache.org/legal/resolved.html

Copy link
Contributor Author

@bzp2010 bzp2010 Jul 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @moonming. Sorry to bother you again.

🤔 What do you think of this? Thx.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @moonming Any changes here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a dependency with a Category B license is fine

@juzhiyuan juzhiyuan requested a review from nic-chen July 25, 2021 09:26
Copy link
Member

@juzhiyuan juzhiyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@juzhiyuan juzhiyuan merged commit 0972f16 into apache:master Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants