Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add multiconfig for launch workers #1056

Merged
merged 70 commits into from
Dec 20, 2022
Merged

Conversation

BillyWooo
Copy link
Collaborator

as the title.
Launch workers with [local parachain / staging parachain / production parachain]

resolves #1000

Kailai-Wang and others added 30 commits July 5, 2022 20:20
* some more renaming

* adjustment to IMP mock

* change to sgx-runtime pallets
jingleizhang and others added 10 commits December 12, 2022 12:48
* add id_graph to ocall extrinsics

* adjust callback extrinsic

* add cross-env&&dotenv

* set env

* random shard for testing

* Change the versions of @PolkaDot

* change the version of @polkadot/types

* rm yarn.lock

* slight renaming

* change maxVerificationDelay to 30min

* change yarn.lock

* generateChallengeCode

* modify yarn run command

* change defaultSinger

* add getSinger

* add Sign functions

* add getSinger && generateChallengeCode

* add ID_HUB_URL for sign message

* modify defaultSigner

* modify shard

* add ethers provider&&wallet

* add eth endpoint

* add @ethersproject/providers&&ethers

* substrate&&ethereum tests

* web3 types

* nonce

* change methods name

* change methods name

* change events

* change methods name

* change event

* modify method name

* ts-test support `id_graph`

* fix bug

* remove log

* resolve conflicts

Co-authored-by: Verin1005 <daqingchong0809@gmail.com>
Co-authored-by: zzz <zhiming.zhong@litentry.com>
* merge tee-dev branch into dev branch (#902)

* adjust docker tag for tee-dev

* use integritee/pallets master branch

* add teeracle to tee-dev

* switch to sdk-v0.9.0-polkadot-v0.9.26 as release branch

* add back the dummy method for testing purpose

* fix make clippy

* small cleanup

* add back teeracle

* use specific branch of tee-primitives (todo:revert it)

* correctify branch for tee-primitives

* temporarily use TEE pallets master branch

* add EnsureOrigin impl and tests

* its better to place it in runtime_common

* remove the dummy extrinsic (used for testing previous)

Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>
Co-authored-by: BillyWooo <yang@litentry.com>

* 0.9.29 update (#891)

* feat: rough update to 0.9.29

* debug:temp fmt

* debug: 0.9.29 without trystate

* debug: Integration with latest try-runtime

* debug: update weight template

* debug: fmt

* debug: patch toml try-runtime

* debug: fmt

* debug: revert template non-weight part

* debug: add teeracle try-runtime

* debug: cargo lock

* fix Cargo.lock (#904)

* lift the bridge transfer restriciton and bump version (#907)

* use hash to compute the raw payload (#910)

* remove Litmus sudo (#876)

* sudo

* change Root into Root +democracy

* debug: recover chainspec rootkey

* fix clippy

* try to fix #900

* add debug log #900

* make CI tests pass after remove litmus `Sudo`

* fmt

* debug

* debug

* rococo/litentry align with litmus

* remove useless test code

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: BillyWooo <yang@litentry.com>
Co-authored-by: zhiming-zhong <zhiming.zhong@litentry.com>

* Use rococo as dev network (#918)

* add rococo staging

* cargo update

* update README

* fix bad english

* Bump aes-gcm from `be0d6c2` to `5eb6d1a` (#925)

Bumps [aes-gcm](https://github.com/RustCrypto/AEADs) from `be0d6c2` to `5eb6d1a`.
- [Release notes](https://github.com/RustCrypto/AEADs/releases)
- [Commits](RustCrypto/AEADs@be0d6c2...5eb6d1a)

---
updated-dependencies:
- dependency-name: aes-gcm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump scale-info from 2.2.0 to 2.3.0 (#926)

Bumps [scale-info](https://github.com/paritytech/scale-info) from 2.2.0 to 2.3.0.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/scale-info/commits/v2.3.0)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Minor update (#928)

* add dwellir bootnode

* bump litmus runtime version

* cargo update

* feat: relocate remove_sudo migration code (#936)

* feat: relocate remove_sudo migration code

* debug: fmt

* debug: change staking parameter (#931)

* debug: change staking parameter

* debug: bump rococo.litentry spec version

* Use sccache in Dockerfile (#937)

* try to switch to github-runner

* use sccache in Dockerfile for self-hosted runner

* fix syntax

* download sccache

* fix path and use pre-built taplo

* intentionally change lib.rs

* intentionally update a pallet only

* try to add/stop sccache server

* minor update

* update README

* try to use rust-cache

* use /root

* use GITHUB_REF_NAME in hash-key

* I dislike github syntax

* another try

* deprecate set-output

* revert back lib.rs

* dintinguish between release and prod

* fix RUN order

* use head_ref or ref_name

Co-authored-by: BillyWooo <yang@litentry.com>

* Update v0.9.30 (#942)

* some deps update to v0.9.30

* deps update to v0.9.30

* format code

* change deps

* fix docker file

* fix docker file

* change docker file

* change docker file

* fix docker yml file

* fix docker yml file

* fix docker yml file

* add install dependencis shell

* fix install dependencis shell

* fix install dependencis shell

* fix docker file

* fix yml file

* add sudo permisson

* add some dep

Co-authored-by: Chengcheng Shi <Chengcheng@litentry>

* optimize cli commands (#947)

* optimize cli commands

* Change some command's names

* format docker file

Co-authored-by: SCC <root@LAPTOP-73QQ3GCK.localdomain>

* Add onfinality script (#952)

* minor update

* bump version and cargo update

* [benchmarking bot] Auto commit generated weights files (#951)

Co-authored-by: Kailai-Wang <Kailai-Wang@users.noreply.github.com>

* try to use new weights

* fix typo

* add missing mods

* fix compile

* add onFinality image version update

* fix indention

* add continue-on-error

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kailai-Wang <Kailai-Wang@users.noreply.github.com>

* Enable parachain staking extrinsic (#955)

* debug: change extrinsic filter

* debug: fmt and clippy

* debug: revert cargo.lock

* Add release bot GHA (#957)

* add release bot yml

* small update

* Issue 958 (#959)

* fix parachain staking benchmark code

* fix syntax

* omit CollatorSelection benchmark code

* [benchmarking bot] Auto commit generated weights files (#961)

Co-authored-by: Jinsipang <Jinsipang@users.noreply.github.com>

Co-authored-by: Chengcheng Shi
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Bump serde_json from 1.0.87 to 1.0.88 (#967)

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.87 to 1.0.88.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](serde-rs/json@v1.0.87...v1.0.88)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Optimize parachain-staking benchmark code to shorten running time (#974)

* remove reward for delegator warn logs

* fix benchmark shell

* fix some syntax

* fix shell

* fix shells

* optimized benchmark code and shell

* fix some params

* optimize benchmark code

* [benchmarking bot] Auto commit generated weights files (#973)

Co-authored-by: Jinsipang

* fix benchmark code

* add staking benchmark code at runtime

* fix benchmark code

* fix benchmark code

* fix benchmark code

* format code

* change benchmark code args

Co-authored-by: Chengcheng Shi <chengcheng@litentry.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Bump aes-gcm from `86250ac` to `6ac2d1c` (#970)

Bumps [aes-gcm](https://github.com/RustCrypto/AEADs) from `86250ac` to `6ac2d1c`.
- [Release notes](https://github.com/RustCrypto/AEADs/releases)
- [Commits](RustCrypto/AEADs@86250ac...6ac2d1c)

---
updated-dependencies:
- dependency-name: aes-gcm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump serde from 1.0.147 to 1.0.148 (#991)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.147 to 1.0.148.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.147...v1.0.148)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump serde_json from 1.0.88 to 1.0.89 (#990)

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.88 to 1.0.89.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](serde-rs/json@v1.0.88...v1.0.89)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add PolkadotXCM filter (#993)

* update Polkadot v0.9.32 (#1043)

* update deps to polkaodt-v0.9.31

* update node cli

* fix some test code

* update to polkadot-v0.9.32

* fix buuld benchmark failed and run test failed issues

* fix node code

* [benchmarking bot] Auto commit generated weights files (#1046)

Co-authored-by: Chengcheng Shi <chengcheng@litentry.com>

* Support standalone node for parachain (#1059)

* use unified crate version and cargo update

* initial add standalone mode

* fix chain spec id

* fix clippy

* add script to start standalone node

* add Makefile

* use 9944 as parachain port consistently

* adjust bridge endpoint

* minor update

* recover docker image

* minor update

* bump version

* update README

* Bump decode-uri-component from 0.2.0 to 0.2.2 in /ts-tests (#1060)

Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](SamVerschueren/decode-uri-component@v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Backport from release-v0.9.13 banch (#1067)

* minor fix of text search

* fix small typo

* Bump express from 4.17.2 to 4.18.2 in /ts-tests (#1071)

Bumps [express](https://github.com/expressjs/express) from 4.17.2 to 4.18.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](expressjs/express@4.17.2...4.18.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Runtime upgrade simulation (#1081)

* update script

* use relative path

* update script

* improve arg handling

* fix trap

* simulmation runtime upgrade demo impl

* runtime-upgrade ts impl

* fix some syntax

* fix makefile

* optimized scripts && cli

* use master branch of fork-off-substrate

* optimize ts code

* fix description

* fix action sytanx

* fix cli syntax

* fix cli syntax

* fix cli syntax

* fix cli syntax

* fix cli syntax

* fix cli syntax

* fix cli syntax

* fix cli syntax

* optimize ts-code & cli

* adjustment of script and GHA

* try to fix CI

Co-authored-by: Scc <root@LAPTOP-73QQ3GCK.localdomain>
Co-authored-by: Scc <chengcheng@litentry.com>

* update dependencies from fork pallets to origin integritee pallets v0.9.32 (#1083)

* switch to temp branch

* update to integritee-pallets v0.9.32

* update pallets repo source

* add up missing lock file

* [benchmarking bot] Auto commit generated weights files

* [benchmarking bot] Auto commit generated weights files (#1082)

Co-authored-by: BillyWooo <BillyWooo@users.noreply.github.com>

* Revert "[benchmarking bot] Auto commit generated weights files (#1082)"

This reverts commit 6b11ff2.

* Revert "Merge remote-tracking branch 'origin/benchmarking-bot-3656108355' into pr-integritee-pallets-v0.9.32"

This reverts commit 3cb2961, reversing
changes made to d93c244.

Co-authored-by: BillyWooo <BillyWooo@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zhizming-zhong <zhiming.zhong@litentry.com>

* Bump serde from 1.0.149 to 1.0.150 (#1085)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.149 to 1.0.150.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.149...v1.0.150)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump scale-info from 2.3.0 to 2.3.1 (#1086)

Bumps [scale-info](https://github.com/paritytech/scale-info) from 2.3.0 to 2.3.1.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](paritytech/scale-info@v2.3.0...v2.3.1)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add test local pallets benchmarking shell (#1087)

* add local benchmark testing shell

* fix benmarking scripts

* fix some syntax

* fix some syntax

* fix some syntax

Co-authored-by: SCC <chengcheng@litentry.com>

* Revert "Add test local pallets benchmarking shell (#1087)" (#1091)

This reverts commit 1c1ee63.

* update tee-worker to upstream/master (3cc02342)

* bump up version to 0.9.32

* cargo build / fmt / clippy / test pass

* parachain build

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>
Co-authored-by: BillyWooo <yang@litentry.com>
Co-authored-by: WMQ <46511820+wangminqi@users.noreply.github.com>
Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: zhiming-zhong <zhiming.zhong@litentry.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: SCC <87012234+Jinsipang@users.noreply.github.com>
Co-authored-by: Chengcheng Shi <Chengcheng@litentry>
Co-authored-by: SCC <root@LAPTOP-73QQ3GCK.localdomain>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kailai-Wang <Kailai-Wang@users.noreply.github.com>
Co-authored-by: Chengcheng Shi <chengcheng@litentry.com>
Co-authored-by: BillyWooo <BillyWooo@users.noreply.github.com>
Co-authored-by: ericzhang <jingleizhang@users.noreply.github.com>
@BillyWooo BillyWooo marked this pull request as draft December 18, 2022 21:37
@BillyWooo BillyWooo marked this pull request as ready for review December 18, 2022 21:37
@BillyWooo BillyWooo changed the base branch from tee-dev to dev December 18, 2022 21:37
@BillyWooo BillyWooo self-assigned this Dec 18, 2022
@codecov-commenter
Copy link

codecov-commenter commented Dec 18, 2022

Codecov Report

Merging #1056 (f7b0ea6) into dev (8b7f31b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #1056   +/-   ##
=======================================
  Coverage   20.70%   20.70%           
=======================================
  Files         217      217           
  Lines        9013     9013           
=======================================
  Hits         1866     1866           
  Misses       7147     7147           
Impacted Files Coverage Δ
...tentry/core/data-providers/src/twitter_official.rs 0.00% <ø> (ø)
tee-worker/service/src/main.rs 0.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Collaborator

@Kailai-Wang Kailai-Wang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks better, thanks.

The biggest concern is still what should be defined in dev/staging/prod configs -- I'll forward this to @jingleizhang

help: Cleans and purges any previous state and key files and generates them anew before starting.
- running-mode:
long: running-mode
help: Litentry TEE service running mode <dev/stage/prod>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think staging is the conventional word 😂, same for worker_config_*

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤣

@@ -0,0 +1,75 @@
#!/usr/bin/env bash
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when shall we run launch.sh and when launch.py?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are fine. launch.sh would be more preferred for local launch & test. launch.py + config.json would have more flexibility on ports parameter inputs (launch.sh will just increase the port number between different worker. config.json can give any port number).

"443"
"443",
"running-mode",
"stage"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think -u and -m can be confusing/conflicting: can I use -u wss://parachain-eu-0.staging.litentry.io and -m dev at the same time? Again it depends what are dev/staging/prod supposed to do

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is no shortcut for 'running-mode'. Always need to type the full name. I also feel confusing about these parameter shortcuts. That's the reason NO shortcut provided here.

@@ -0,0 +1,8 @@
{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a difference between dev/staging/prod, this needs a clear definition: what exactly should differ in these different configs? -- needs to be evaluated/defined by who opens this issue (maybe @jingleizhang)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. For the moment. These three files are almost dummy placeholders. Need to define/clarify later.

Copy link
Contributor

@jingleizhang jingleizhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least we have those prod/staging/dev config files so that we can work with the same application code in different environments.

All parameters should be moved into the config file, and only one parameter that represents the env needs to be kept, actually. But for now, we're just using the config to save some no-more hard-coded information. At least it helps us keep some api keys safe.

Please correct me if I'm wrong: prod stands for Litentry, staging means parachain-eu-0.staging.litentry.io, and dev means rococo.

I think we could change those names to something more clear if we think it's necessary. 😂

@BillyWooo BillyWooo merged commit e83b7f5 into dev Dec 20, 2022
@BillyWooo BillyWooo deleted the 1000_muti_config_to_launch_workers branch December 20, 2022 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

multiple configs for different envioments
5 participants