Merge branch 'qa' and 'master' into develop (ethereum#663)

* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)

* Added changes per discussion in PR, more changes may be necessary

* Adding prerelease true

* Disabling goreleaser

* Removing README swap file

* change bor_dir and add bor user for v0.3.0 release

* rollback bor user and use root

* metrics: handle equal to separated config flag (ethereum#596)

* metrics: handle  based config path

* internal/cli/server: add more context to logs

* use space separated flag and value in bor.service

* fixed static-nodes related buf (os independent) (ethereum#598)

* fixed static-nodes related buf (os independent)

* taking static-nodes as input if default not present

* Update default flags (ethereum#600)

* internal/cli/server: use geth's default for txpool.pricelimit and add comments

* builder/files: update config.toml for mainnet

* packaging/templates: update defaults for mainnet and mumbai

* internal/cli/server: skip overriding cache

* packaging/templates: update cache value for mainnet

* packaging/templates: update gcmode for archive mumbai node

* metrics: handle nil telemetry config (ethereum#601)

* resolve merge conflicts

* update go version in release.yml

* update goversion in makefile

* update Docker login for goreleaser-cross v1.19

* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist

* added a toml configuration file with comments describing each flag (ethereum#607)

* added a toml configuration file with comments describing each flag

* internal/cli/server: update flag description

* docs/cli: update example config and description of flags

* docs: update new-cli docs

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update

* added ancient datadir flag and toml field, need to decide on default value and update the conversion script

* updated toml files with ancient field

* Add support for new flags in new config.toml, which were present in old config.toml (ethereum#612)

* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml

* added RAW fields for these time.Duration flags

* updated the conversion script to support these extra 4 flags

* removed hcl and json config tests as we are only supporting toml config files

* updated toml files with cache.timeout field

* updated toml files with jsonrpc.timeouts field

* tests/bor: expect a call for latest checkpoint

* tests/bor: expect a call for latest checkpoint

* packaging/templates: update cache values for archive nodes

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* remove unwanted code

* Fix docker publish authentication issue

In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.

* Revert "update Docker login for goreleaser-cross v1.19"

This reverts commit 4d19cf5.

* Bump version to stable

* Revert "Merge pull request ethereum#435 from maticnetwork/POS-553"

This reverts commit 657d262, reversing
changes made to 88dbfa1.

* revert change for release for go1.19

* Add default values to CLI helper and docs

This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.

* Add a summary of new CLI in docs

* Updating packager as binutils changed version so that apt-get installs current versions

* Add state pruning to new CLI

* Minor wording fix in prune state description

* Bumping control file versions

* Mainnet Delhi fork

* Set version to stable

* change delhi hardfork block number

* handle future chain import and skip peer drop (ethereum#650)

* handle future chain import and skip peer drop

* add block import metric

* params: bump version to v0.3.3-stable

* Bump bor version in control files for v0.3.3 mainnet release

Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>

.github/workflows/release.yml

@@ -21,14 +21,20 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@master
         with:
-          go-version: 1.17.x
+          go-version: 1.19.x
 
       - name: Prepare
         id: prepare
         run: |
             TAG=${GITHUB_REF#refs/tags/}
             echo ::set-output name=tag_name::${TAG}
 
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB }}
+          password: ${{ secrets.DOCKERHUB_KEY }}
+
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v1
 
@@ -39,5 +45,3 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           VERSION: ${{ steps.prepare.outputs.tag_name }}
           SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
-          DOCKER_USERNAME: ${{ secrets.DOCKERHUB }}
-          DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_KEY }}

.goreleaser.yml

@@ -1,7 +1,7 @@
 project_name: bor
 
 release:
-  disable: false
+  disable: true
   draft: true
   prerelease: auto
 

Dockerfile

@@ -1,17 +1,17 @@
 FROM golang:latest
 
-ARG BOR_DIR=/bor
+ARG BOR_DIR=/var/lib/bor
 ENV BOR_DIR=$BOR_DIR
 
 RUN apt-get update -y && apt-get upgrade -y \
     && apt install build-essential git -y \
-    && mkdir -p /bor
+    && mkdir -p ${BOR_DIR}
 
 WORKDIR ${BOR_DIR}
 COPY . .
 RUN make bor
 
-RUN cp build/bin/bor /usr/local/bin/
+RUN cp build/bin/bor /usr/bin/
 
 ENV SHELL /bin/bash
 EXPOSE 8545 8546 8547 30303 30303/udp

Dockerfile.alltools

@@ -13,6 +13,6 @@ RUN set -x \
     && apk add --update --no-cache \
        ca-certificates \
     && rm -rf /var/cache/apk/*
-COPY --from=builder /bor/build/bin/* /usr/local/bin/
+COPY --from=builder /bor/build/bin/* /usr/bin/
 
 EXPOSE 8545 8546 30303 30303/udp

Dockerfile.release

@@ -1,10 +1,15 @@
 FROM alpine:3.14
 
+ARG BOR_DIR=/var/lib/bor
+ENV BOR_DIR=$BOR_DIR
+
 RUN apk add --no-cache ca-certificates && \
-    mkdir -p /etc/bor
-COPY bor /usr/local/bin/
-COPY builder/files/genesis-mainnet-v1.json /etc/bor/
-COPY builder/files/genesis-testnet-v4.json /etc/bor/
+    mkdir -p ${BOR_DIR}
+
+WORKDIR ${BOR_DIR}
+COPY bor /usr/bin/
+COPY builder/files/genesis-mainnet-v1.json ${BOR_DIR}
+COPY builder/files/genesis-testnet-v4.json ${BOR_DIR}
 
 EXPOSE 8545 8546 8547 30303 30303/udp
 ENTRYPOINT ["bor"]

Makefile

@@ -37,7 +37,7 @@ protoc:
 generate-mocks:
 	go generate mockgen -destination=./tests/bor/mocks/IHeimdallClient.go -package=mocks ./consensus/bor IHeimdallClient
 	go generate mockgen -destination=./eth/filters/IBackend.go -package=filters ./eth/filters Backend
-	
+
 geth:
 	$(GORUN) build/ci.go install ./cmd/geth
 	@echo "Done building."
@@ -199,7 +199,7 @@ geth-windows-amd64:
 	@ls -ld $(GOBIN)/geth-windows-* | grep amd64
 
 PACKAGE_NAME          := github.com/maticnetwork/bor
-GOLANG_CROSS_VERSION  ?= v1.18.1
+GOLANG_CROSS_VERSION  ?= v1.19.1
 
 .PHONY: release-dry-run
 release-dry-run:
@@ -227,6 +227,7 @@ release:
 		-e DOCKER_PASSWORD \
 		-e SLACK_WEBHOOK \
 		-v /var/run/docker.sock:/var/run/docker.sock \
+		-v $(HOME)/.docker/config.json:/root/.docker/config.json \
 		-v `pwd`:/go/src/$(PACKAGE_NAME) \
 		-w /go/src/$(PACKAGE_NAME) \
 		goreleaser/goreleaser-cross:${GOLANG_CROSS_VERSION} \

builder/files/bor.service

@@ -6,7 +6,7 @@
 [Service]
   Restart=on-failure
   RestartSec=5s
-  ExecStart=/usr/local/bin/bor server -config="/var/lib/bor/config.toml"
+  ExecStart=/usr/local/bin/bor server -config "/var/lib/bor/config.toml"
   Type=simple
   User=bor
   KillSignal=SIGINT

builder/files/config.toml

@@ -6,6 +6,7 @@ chain = "mainnet"
 # identity = "Pratiks-MacBook-Pro.local"
 # log-level = "INFO"
 datadir = "/var/lib/bor/data"
+# ancient = ""
 # keystore = "/var/lib/bor/keystore"
 syncmode = "full"
 # gcmode = "full"
@@ -52,46 +53,50 @@ syncmode = "full"
   # pricebump = 10
 
 [miner]
-  gaslimit = 20000000
+  gaslimit = 30000000
   gasprice = "30000000000"
   # mine = true
   # etherbase = "VALIDATOR ADDRESS"
   # extradata = ""
 
 
 # [jsonrpc]
-  # ipcdisable = false
-  # ipcpath = ""
-  # gascap = 50000000
-  # txfeecap = 5.0
-  # [jsonrpc.http]
-    # enabled = false
-    # port = 8545
-    # prefix = ""
-    # host = "localhost"
-    # api = ["eth", "net", "web3", "txpool", "bor"]
-    # vhosts = ["*"]
-    # corsdomain = ["*"]
-  # [jsonrpc.ws]
-    # enabled = false
-    # port = 8546
-    # prefix = ""
-    # host = "localhost"
-    # api = ["web3", "net"]
-    # origins = ["*"]
-  # [jsonrpc.graphql]
-    # enabled = false
-    # port = 0
-    # prefix = ""
-    # host = ""
-    # vhosts = ["*"]
-    # corsdomain = ["*"]
+#   ipcdisable = false
+#   ipcpath = ""
+#   gascap = 50000000
+#   txfeecap = 5.0
+#   [jsonrpc.http]
+#     enabled = false
+#     port = 8545
+#     prefix = ""
+#     host = "localhost"
+#     api = ["eth", "net", "web3", "txpool", "bor"]
+#     vhosts = ["*"]
+#     corsdomain = ["*"]
+#   [jsonrpc.ws]
+#     enabled = false
+#     port = 8546
+#     prefix = ""
+#     host = "localhost"
+#     api = ["web3", "net"]
+#     origins = ["*"]
+#   [jsonrpc.graphql]
+#     enabled = false
+#     port = 0
+#     prefix = ""
+#     host = ""
+#     vhosts = ["*"]
+#     corsdomain = ["*"]
+#   [jsonrpc.timeouts]
+#     read = "30s"
+#     write = "30s"
+#     idle = "2m0s"
 
-# [gpo]
+[gpo]
   # blocks = 20
   # percentile = 60
   # maxprice = "5000000000000"
-  # ignoreprice = "2"
+  ignoreprice = "30000000000"
 
 [telemetry]
   metrics = true
@@ -122,6 +127,7 @@ syncmode = "full"
   # preimages = false
   # txlookuplimit = 2350000
   # triesinmemory = 128
+  # timeout = "1h0m0s"
 
 [accounts]
   # allow-insecure-unlock = true

builder/files/genesis-mainnet-v1.json

@@ -15,14 +15,17 @@
     "londonBlock": 23850000,
     "bor": {
       "jaipurBlock": 23850000,
+      "delhiBlock": 38189056,
       "period": {
         "0": 2
       },
       "producerDelay": {
-        "0": 6
+        "0": 6,
+        "38189056": 4
       },
       "sprint": {
-        "0": 64
+        "0": 64,
+        "38189056": 16
       },
       "backupMultiplier": {
         "0": 2

core/blockchain.go

@@ -68,6 +68,7 @@ var (
 	snapshotStorageReadTimer = metrics.NewRegisteredTimer("chain/snapshot/storage/reads", nil)
 	snapshotCommitTimer      = metrics.NewRegisteredTimer("chain/snapshot/commits", nil)
 
+	blockImportTimer     = metrics.NewRegisteredMeter("chain/imports", nil)
 	blockInsertTimer     = metrics.NewRegisteredTimer("chain/inserts", nil)
 	blockValidationTimer = metrics.NewRegisteredTimer("chain/validation", nil)
 	blockExecutionTimer  = metrics.NewRegisteredTimer("chain/execution", nil)
@@ -1518,6 +1519,11 @@ func (bc *BlockChain) insertChain(chain types.Blocks, verifySeals, setHead bool)
 	it := newInsertIterator(chain, results, bc.validator)
 	block, err := it.next()
 
+	// Update the block import meter; it will just record chains we've received
+	// from other peers. (Note that the actual chain which gets imported would be
+	// quite low).
+	blockImportTimer.Mark(int64(len(headers)))
+
 	// Check the validity of incoming chain
 	isValid, err1 := bc.forker.ValidateReorg(bc.CurrentBlock().Header(), headers)
 	if err1 != nil {

core/forkchoice.go

@@ -114,9 +114,7 @@ func (f *ForkChoice) ReorgNeeded(current *types.Header, header *types.Header) (b
 func (f *ForkChoice) ValidateReorg(current *types.Header, chain []*types.Header) (bool, error) {
 	// Call the bor chain validator service
 	if f.validator != nil {
-		if isValid := f.validator.IsValidChain(current, chain); !isValid {
-			return false, nil
-		}
+		return f.validator.IsValidChain(current, chain)
 	}
 
 	return true, nil

core/forkchoice_test.go

@@ -13,15 +13,15 @@ import (
 
 // chainValidatorFake is a mock for the chain validator service
 type chainValidatorFake struct {
-	validate func(currentHeader *types.Header, chain []*types.Header) bool
+	validate func(currentHeader *types.Header, chain []*types.Header) (bool, error)
 }
 
 // chainReaderFake is a mock for the chain reader service
 type chainReaderFake struct {
 	getTd func(hash common.Hash, number uint64) *big.Int
 }
 
-func newChainValidatorFake(validate func(currentHeader *types.Header, chain []*types.Header) bool) *chainValidatorFake {
+func newChainValidatorFake(validate func(currentHeader *types.Header, chain []*types.Header) (bool, error)) *chainValidatorFake {
 	return &chainValidatorFake{validate: validate}
 }
 
@@ -46,18 +46,18 @@ func TestPastChainInsert(t *testing.T) {
 	getTd := func(hash common.Hash, number uint64) *big.Int {
 		return big.NewInt(int64(number))
 	}
-	validate := func(currentHeader *types.Header, chain []*types.Header) bool {
+	validate := func(currentHeader *types.Header, chain []*types.Header) (bool, error) {
 		// Put all explicit conditions here
 		// If canonical chain is empty and we're importing a chain of 64 blocks
 		if currentHeader.Number.Uint64() == uint64(0) && len(chain) == 64 {
-			return true
+			return true, nil
 		}
 		// If canonical chain is of len 64 and we're importing a past chain from 54-64, then accept it
 		if currentHeader.Number.Uint64() == uint64(64) && chain[0].Number.Uint64() == 55 && len(chain) == 10 {
-			return true
+			return true, nil
 		}
 
-		return false
+		return false, nil
 	}
 	mockChainReader := newChainReaderFake(getTd)
 	mockChainValidator := newChainValidatorFake(validate)
@@ -116,18 +116,18 @@ func TestFutureChainInsert(t *testing.T) {
 	getTd := func(hash common.Hash, number uint64) *big.Int {
 		return big.NewInt(int64(number))
 	}
-	validate := func(currentHeader *types.Header, chain []*types.Header) bool {
+	validate := func(currentHeader *types.Header, chain []*types.Header) (bool, error) {
 		// Put all explicit conditions here
 		// If canonical chain is empty and we're importing a chain of 64 blocks
 		if currentHeader.Number.Uint64() == uint64(0) && len(chain) == 64 {
-			return true
+			return true, nil
 		}
 		// If length of future chains > some value, they should not be accepted
 		if currentHeader.Number.Uint64() == uint64(64) && len(chain) <= 10 {
-			return true
+			return true, nil
 		}
 
-		return false
+		return false, nil
 	}
 	mockChainReader := newChainReaderFake(getTd)
 	mockChainValidator := newChainValidatorFake(validate)
@@ -174,18 +174,18 @@ func TestOverlappingChainInsert(t *testing.T) {
 	getTd := func(hash common.Hash, number uint64) *big.Int {
 		return big.NewInt(int64(number))
 	}
-	validate := func(currentHeader *types.Header, chain []*types.Header) bool {
+	validate := func(currentHeader *types.Header, chain []*types.Header) (bool, error) {
 		// Put all explicit conditions here
 		// If canonical chain is empty and we're importing a chain of 64 blocks
 		if currentHeader.Number.Uint64() == uint64(0) && len(chain) == 64 {
-			return true
+			return true, nil
 		}
 		// If length of chain is > some fixed value then don't accept it
 		if currentHeader.Number.Uint64() == uint64(64) && len(chain) <= 20 {
-			return true
+			return true, nil
 		}
 
-		return false
+		return false, nil
 	}
 	mockChainReader := newChainReaderFake(getTd)
 	mockChainValidator := newChainValidatorFake(validate)
@@ -227,7 +227,7 @@ func (c *chainReaderFake) GetTd(hash common.Hash, number uint64) *big.Int {
 func (w *chainValidatorFake) IsValidPeer(remoteHeader *types.Header, fetchHeadersByNumber func(number uint64, amount int, skip int, reverse bool) ([]*types.Header, []common.Hash, error)) (bool, error) {
 	return true, nil
 }
-func (w *chainValidatorFake) IsValidChain(current *types.Header, headers []*types.Header) bool {
+func (w *chainValidatorFake) IsValidChain(current *types.Header, headers []*types.Header) (bool, error) {
 	return w.validate(current, headers)
 }
 func (w *chainValidatorFake) ProcessCheckpoint(endBlockNum uint64, endBlockHash common.Hash) {}

docs/README.md

@@ -1,9 +1,7 @@
 
 # Documentation
 
-- [Command-line-interface](./cli)
-
-- [Configuration file](./config.md)
+[The new command line interface (CLI)](./cli) in this version of Bor aims to give users more control over the codebase when interacting with and starting a node. We have made every effort to keep most of the flags similar to the old CLI, except for a few notable changes. One major change is the use of the --config flag, which previously represented fields without available flags. It now represents all flags available to the user, and will overwrite any other flags if provided. As a node operator, you still have the flexibility to modify flags as needed. Please note that this change does not affect the internal functionality of the node, and it remains compatible with Geth and the Ethereum Virtual Machine (EVM).
 
 ## Additional notes
 
@@ -13,8 +11,16 @@
   $ bor server <flags>
   ```
 
-- Toml files used earlier just to configure static/trusted nodes are being deprecated. Instead, a toml file now can be used instead of flags and can contain all configuration for the node to run. The link to a sample config file is given above. To simply run bor with a configuration file, the following command can be used. 
+  See [here](./cli/server.md) for more flag details.
+
+- The `bor dumpconfig` sub-command prints the default configurations, in the TOML format, on the terminal. One can `pipe (>)` this to a file (say `config.toml`) and use it to start bor. 
+
+- A toml file now can be used instead of flags and can contain all configuration for the node to run. To simply run bor with a configuration file, the following command can be used. 
 
   ```
   $ bor server --config <path_to_config.toml>
   ```
+
+- You can find an example config file [here](./cli/example_config.toml) to know more about what each flag is used for, what are the defaults and recommended values for different networks. 
+
+- Toml files used earlier (with `--config` flag) to configure additional fields (like static and trusted nodes) are being deprecated and have been converted to flags.