Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test local pallets benchmarking shell #1087

Merged
merged 7 commits into from
Dec 13, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions scripts/local-pallets-benchmark.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
#!/usr/bin/env bash

# This script can be used for running litentry's benchmarks.
#
# The litentry binary is required to be compiled with make build-node-benchmarks
# in release mode.

set -e

BINARY="./target/release/litentry-collator"


if [[ ! -f "${BINARY}" ]]; then
echo "binary '${BINARY}' does not exist."
echo "ensure that the litentry binary is compiled with 'make build-node-benchmarks' and in release mode."
exit 1
fi

function help {
echo "USAGE:"
echo " ${0} litentry|litmuts|rococo pallet_name benchmark_method [--check]"
echo ""
echo "EXAMPLES:"
echo " ${0} " "list all benchmarks and provide a selection to choose from"
echo " ${0} --check " "list all benchmarks and provide a selection to choose from, runs in 'check' mode (reduced steps and repetitions)"
echo " ${0} foo bar " "run a benchmark for pallet 'foo' and benchmark 'bar'"
echo " ${0} foo bar all " "run the pallet all benchmark method"
}

WEIGHTS_PATH="$(pwd)/weights"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is guaranteed to be wrong: pwd can be anything, you need to use absolute path or cd to a known folder before this

if [ ! -d $WEIGHTS_PATH ];then
mkdir -p weights/

fi

CHAIN_TYPE="--chain=${1}-dev"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It contradicts with what's written in help
./scripts/local-pallets-benchmark.sh --check give you the wrong chain-type

PALLET=${2//-/_}

EXTRINSIC=

if [[ ${3} == 'all' ]];then
EXTRINSIC=*
else
EXTRINSIC=${3//-/_}
fi

function choose_and_bench {
readarray -t options < <(${BINARY} benchmark pallet --list $CHAIN_TYPE | sed 1d)
options+=('EXIT')

select opt in "${options[@]}"; do
IFS=', ' read -ra parts <<< "${opt}"
[[ "${opt}" == 'EXIT' ]] && exit 0

bench "${parts[0]}" "${parts[1]}" "${2}"
break
done
}

function bench {
echo "pallet ${1}-weights.rs"
OUTPUT="--output=$(pwd)/weights/${1}_weights.rs"

echo "benchmarking '${1}::${2}' --check=${3}, writing results to '${OUTPUT}'"

if [[ $PALLET == *"parachain_staking"* ]]; then
echo "will run ${1} benchmark code"
STEPS=25
REPEAT=20
else
echo "will run other pallet (${1}) benchmark code"
STEPS=20
REPEAT=50
fi

WASMTIME_BACKTRACE_DETAILS=1 ${BINARY} benchmark pallet \
$CHAIN_TYPE \
--execution=wasm \
--db-cache=20 \
--wasm-execution=compiled \
--pallet="$PALLET" \
--extrinsic="$EXTRINSIC" \
--heap-pages=4096 \
--steps="$STEPS" \
--repeat="$REPEAT" \
--header=./LICENSE_HEADER \
$TEMPLATE \
$OUTPUT

}

if [[ "${@}" =~ "--help" ]]; then
help
else
CHECK=0
if [[ "${@}" =~ "--check" ]]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the handling of --check is wrongly implemented.

tried:

./scripts/local-pallets-benchmark.sh rococo --check
# select 66
# got error 
Error: Input("No benchmarks found which match your input.")

tried ./scripts/local-pallets-benchmark.sh rococo pallet-proxy --check, same error:

pallet  pallet-proxy-weights.rs
benchmarking 'pallet-proxy::' --check=1, writing results to '--output=/home/kai/litentry-parachain/weights/pallet-proxy_weights.rs'
will run other pallet (pallet-proxy) benchmark code
2022-12-13 12:57:03 💥 keeping old session because of empty collator set!
2022-12-13 12:57:03 💥 keeping old session because of empty collator set!
Error: Input("No benchmarks found which match your input.")

tried ./scripts/local-pallets-benchmark.sh rococo pallet-proxy add_proxy --check, select 76) pallet_membership, swap_member, it seems add_proxy is benchmarked, but the generated weight file is under Created file: "/home/kai/litentry-parachain/weights/pallet_membership_weights.rs"

It's a mess

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only thing that worked is benchmarking the whole pallet, e.g.

./scripts/local-pallets-benchmark.sh rococo pallet-extrinsic-filter

CHECK=1
set -o noglob && set -- ${@/'--check'} && set +o noglob
fi

ALL=0
if [[ "${@}" =~ "--all" ]]; then
ALL=1
fi

if [[ "${ALL}" -eq 1 ]]; then
bench '*' '*' "${CHECK}" "weights/"
elif [[ $# -ne 2 ]]; then
choose_and_bench "${CHECK}"
else
bench "${2}" "${3}" "${CHECK}"
fi
fi