Fishtest Server Setup

Server setup

1. Use a clean install of Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy), on a virtual machine, cloud instance etc.
   1. If you use VirtualBox on Windows, you will need to disable the Windows Hypervisor.
      See: https://forums.virtualbox.org/viewtopic.php?f=25&t=99390 and unplug your PC afterwards!
      More from Microsoft about this, here.
2. Copy the script setup_fishtest.sh:
   1. Write your password in the variable usr_pwd
   2. (Optional to use https) Write your fully qualified domain name in the variable server_name
3. Run the setup script using bash as:

   sudo bash setup_fishtest.sh 2>&1 | tee setup_fishtest.sh.log

Nets workflow setup

1. Download the nets required by the tests (e.g. the default one) from the official fishtest server NN Stats page
2. Open a web browser using the ip_address of the fishtest server (http://ip_address/login)
3. Login as user01 (with password user01)
4. Select the "NN Upload page" (http://ip_address/upload)
5. Upload the net. The net is written in /home/fishtest/net-server/nn
6. (Optional) Use the script /home/fishtest/net-server/set_net_server.sh to set the server (the official server or the local development server) from which to download the nets during the tests

Create tests

Use these users

• To approve test:
  • User: user00
  • Password: user00
• To create test:
  • User: user01
  • Password: user01

Connect a worker

Use the ip_address of the fishtest server
To have multiple workers make some copies of the worker folder.

python3 worker.py <username> <password> --protocol <http/https> --host <ip_address> --port <80/443/custom> --concurrency <n_cores> 

Start/stop fishtest services

• Start the services

  sudo systemctl start fishtest@{6543..6544}.service

• Stop the services

  sudo systemctl stop fishtest@{6543..6544}.service

• Restart the services

  sudo systemctl restart fishtest@{6543..6544}.service

Debug the server

Using the Pyramid Debug Toolbar

1. Login on Ubuntu
2. Use the following commands to start/stop the fishtest_dbg.service
   • Start the debug session

     sudo systemctl start fishtest_dbg.service

   • Stop the debug session

     sudo systemctl stop fishtest_dbg.service

3. Open a browser using the port 6542 (http://ip_address:6542).

---

Scripts

Server setup

For Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy)

Click to view

#!/bin/bash
# 240519
# to setup a fishtest server on Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy), simply run:
# sudo bash setup_fishtest.sh 2>&1 | tee setup_fishtest.sh.log
#
# to use fishtest connect a browser to:
# http://<ip_address> or http://<fully_qualified_domain_name>

user_name='fishtest'
user_pwd='<your_password>'
# try to find the ip address
server_name=$(hostname --all-ip-addresses)
server_name=$(echo $server_name)
# use a fully qualified domain names (http/https)
# server_name='<fully_qualified_domain_name>'

git_user_name='your_name'
git_user_email='you@example.com'

# create user for fishtest
useradd -m -s /bin/bash ${user_name}
echo ${user_name}:${user_pwd} | chpasswd
usermod -aG sudo ${user_name}
sudo -i -u ${user_name} << EOF
mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
EOF

# get the user $HOME
user_home=$(sudo -i -u ${user_name} << 'EOF'
echo ${HOME}
EOF
)

# add some bash variables
sudo -i -u ${user_name} << 'EOF'
cat << 'EOF0' >> .profile

export FISHTEST_HOST=127.0.0.1
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export VENV="$HOME/fishtest/server/env"
export NSVENV="$HOME/net-server/env"
EOF0
EOF

# set secrets
sudo -i -u ${user_name} << EOF
echo '' > fishtest.secret
echo '' > fishtest.captcha.secret
echo 'http://127.0.0.1/upload_net/' > fishtest.upload

cat << EOF0 > .netrc
# GitHub authentication to raise API rate limit
# create a <personal-access-token> https://github.com/settings/tokens
#machine api.github.com
#login <personal-access-token>
#password x-oauth-basic
EOF0
chmod 600 .netrc
EOF

# install required packages
apt update && apt full-upgrade -y && apt autoremove -y && apt clean
apt purge -y apache2 apache2-data apache2-doc apache2-utils apache2-bin
apt install -y bash-completion cpulimit curl exim4 git gnupg mutt nginx pigz procps ufw

# configure ufw
ufw allow ssh
ufw allow http
ufw allow https
ufw allow 6542
ufw --force enable
ufw status verbose

# configure nginx
# check connections: netstat -anp | grep python3 | grep ESTAB | wc -l
cat << EOF > /etc/nginx/sites-available/fishtest.conf
upstream backend_6543 {
    server 127.0.0.1:6543;
    keepalive 64;
}

upstream backend_6544 {
    server 127.0.0.1:6544;
    keepalive 64;
}

upstream backend_6545 {
    server 127.0.0.1:6545;
    keepalive 64;
}

upstream backend_8000 {
    server 127.0.0.1:8000;
}

map \$uri \$backends {
    /upload_net/                              backend_8000;
    /tests                                    backend_6544;
    ~^/api/(actions|active_runs|calc_elo)     backend_6545;
    ~^/api/(nn/|pgn/|run_pgns/|upload_pgn)    backend_6545;
    ~^/tests/(finished|machines|user)         backend_6545;
    ~^/(actions/|contributors)                backend_6545;
    ~^/(api|tests)/                           backend_6543;
    default                                   backend_6544;
}

server {
    listen 80;
    listen [::]:80;

    server_name ${server_name};

    location ~ ^/(css/|html/|img/|js/|favicon.ico\$|robots.txt\$) {
        root        ${user_home}/fishtest/server/fishtest/static;
        expires     1y;
        add_header  Cache-Control public;
        access_log  off;
    }

    location /nn/ {
        root         ${user_home}/net-server;
        gzip_static  always;
        gunzip       on;
    }

    location / {
        rewrite ^/$       /tests permanent;
        rewrite ^/tests/$ /tests permanent;
        proxy_set_header Connection "";
        proxy_set_header X-Forwarded-Proto  \$scheme;
        proxy_set_header X-Forwarded-For    \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host   \$host:\$server_port;
        proxy_set_header X-Forwarded-Port   \$server_port;

        client_max_body_size        120m;
        client_body_buffer_size     128k;
        proxy_connect_timeout       5s;
        proxy_send_timeout          30s;
        proxy_read_timeout          30s;
        proxy_max_temp_file_size    0;
        proxy_redirect              off;
        proxy_http_version          1.1;

        gunzip on;

        proxy_pass http://\$backends;
    }
}
EOF

unlink /etc/nginx/sites-enabled/default
ln -sf /etc/nginx/sites-available/fishtest.conf /etc/nginx/sites-enabled/fishtest.conf
usermod -aG ${user_name} www-data
systemctl enable nginx.service
systemctl restart nginx.service

# setup pyenv and install the latest python version
# https://github.com/pyenv/pyenv
apt update
apt install -y build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

sudo -i -u ${user_name} << 'EOF'
git clone https://github.com/pyenv/pyenv.git "${HOME}/.pyenv"

cat << 'EOF0' >> ${HOME}/.profile

# pyenv: keep at the end of the file
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
EOF0

cat << 'EOF0' >> ${HOME}/.bashrc

# pyenv: keep at the end of the file
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
EOF0
EOF

# optimized python build: LTO takes some time to run the tests and make the second optimized build
# consider to install a newer GCC (https://stackoverflow.com/questions/67298443/when-gcc-11-will-appear-in-ubuntu-repositories)
# CONFIGURE_OPTS="--enable-optimizations --with-lto" MAKE_OPTS="--jobs 2" PYTHON_CFLAGS="-march=native -mtune=native" pyenv install ${python_ver}
sudo -i -u ${user_name} << 'EOF'
python_ver="3.12.3"
pyenv install ${python_ver}
pyenv global ${python_ver}
EOF

# install mongodb community edition for Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy)
curl -fsSL https://pgp.mongodb.com/server-6.0.asc | gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg --dearmor
ubuntu_release=$(lsb_release -c | awk '{print $2}')
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu ${ubuntu_release}/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
apt update
apt install -y mongodb-org

# set the cache size in /etc/mongod.conf
#  wiredTiger:
#    engineConfig:
#      cacheSizeGB: 2.25
cp /etc/mongod.conf mongod.conf.bkp
sed -i 's/^#  wiredTiger:/  wiredTiger:\n    engineConfig:\n      cacheSizeGB: 2.25/' /etc/mongod.conf
# set the memory decommit
sed -i '/^## Enterprise-Only Options:/i\setParameter:\n  tcmallocAggressiveMemoryDecommit: 1\n' /etc/mongod.conf
# setup logrotate for mongodb
sed -i '/^  logAppend: true/a\  logRotate: reopen' /etc/mongod.conf

cat << 'EOF' > /etc/logrotate.d/mongod
/var/log/mongodb/mongod.log
{
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0600 mongodb mongodb
    sharedscripts
    postrotate
        /bin/kill -SIGUSR1 $(pgrep mongod 2>/dev/null) 2>/dev/null || true
    endscript
}
EOF

# download fishtest
sudo -i -u ${user_name} << EOF
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email "${git_user_email}"
git config user.name "${git_user_name}"
EOF

# setup fishtest
sudo -i -u ${user_name} << 'EOF'
python3 -m venv ${VENV}
${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
cd ${HOME}/fishtest/server
${VENV}/bin/python3 -m pip install -e .
EOF

# install fishtest as systemd service
cat << EOF > /etc/systemd/system/fishtest@.service
[Unit]
Description=Fishtest Server port %i
After=network.target mongod.service

[Service]
Type=simple
ExecStart=${user_home}/fishtest/server/env/bin/pserve production.ini http_port=%i
Restart=on-failure
RestartSec=3
User=${user_name}
WorkingDirectory=${user_home}/fishtest/server

[Install]
WantedBy=multi-user.target
EOF

# install also fishtest debug as systemd service
cat << EOF > /etc/systemd/system/fishtest_dbg.service
[Unit]
Description=Fishtest Server Debug port 6542
After=network.target mongod.service

[Service]
Type=simple
ExecStart=${user_home}/fishtest/server/env/bin/pserve development.ini --reload
User=${user_name}
WorkingDirectory=${user_home}/fishtest/server

[Install]
WantedBy=multi-user.target
EOF

# enable the autostart for mongod.service and fishtest@.service
# check the log with: sudo journalctl -u fishtest@6543.service --since "2 days ago"
systemctl daemon-reload
systemctl enable mongod.service
systemctl enable fishtest@{6543..6545}.service

# start fishtest server
systemctl start mongod.service
systemctl start fishtest@{6543..6545}.service

# add mongodb indexes
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 ${HOME}/fishtest/server/utils/create_indexes.py actions flag_cache pgns runs users
EOF

# add some default users:
# "user00" (with password "user00"), as approver
# "user01" (with password "user01"), as normal user
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 << EOF0
from fishtest.rundb import RunDb
rdb = RunDb()
for i in range(10):
    user_name = f"user{i:02d}"
    user_mail = f"{user_name}@example.org"
    rdb.userdb.create_user(user_name, user_name, user_mail)
    if i == 0:
        rdb.userdb.add_user_group(user_name, "group:approvers")
    user = rdb.userdb.get_user(user_name)
    user["blocked"] = False
    user["pending"] = False
    user["machine_limit"] = 100
    rdb.userdb.save_user(user)
EOF0
EOF

sudo -i -u ${user_name} << 'EOF'
(crontab -l; cat << EOF0
VENV=${HOME}/fishtest/server/env
UPATH=${HOME}/fishtest/server/utils

# Backup mongodb database and upload to s3
# keep disabled on dev server
# 3 */6 * * * /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- sh \${UPATH}/backup.sh

# Update the users table
1,16,31,46 * * * * /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- \${VENV}/bin/python3 \${UPATH}/delta_update_users.py

# Purge old pgn files
33 3 * * *  /usr/bin/nice -n 10 /usr/bin/cpulimit -l 20 -f -m -- \${VENV}/bin/python3 \${UPATH}/purge_pgns.py

# Clean up old mail (more than 9 days old)
33 5 * * * /usr/bin/nice -n 10 screen -D -m mutt -e 'push D~d>9d<enter>qy<enter>'

EOF0
) | crontab -
EOF

# setup net-server
sudo -i -u ${user_name} << 'EOF'
mkdir -p ${HOME}/net-server/nn
python3 -m venv ${NSVENV}
${NSVENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
${NSVENV}/bin/python3 -m pip install --upgrade fastapi gunicorn orjson python-multipart uvicorn[standard]

cat << EOF0 > ${HOME}/net-server/net_server.py
import gzip
import hashlib
from pathlib import Path

from fastapi import FastAPI, HTTPException, UploadFile

app = FastAPI()


@app.post("/upload_net/", status_code=201)
async def create_upload_net(upload: UploadFile) -> None:
    net_file = upload.filename
    net_file_gz = Path("${HOME}/net-server/nn/") / (net_file + ".gz")
    try:
        with gzip.open(net_file_gz, "xb") as f:
            f.write(await upload.read())
    except FileExistsError as e:
        detail = f"File {net_file} already uploaded"
        print(detail, e, flush=True)
        raise HTTPException(
            status_code=409,
            detail=detail,
        )
    except Exception as e:
        net_file_gz.unlink(missing_ok=True)
        detail = f"Failed to write file {net_file}"
        print(detail, e, flush=True)
        raise HTTPException(
            status_code=500,
            detail=detail,
        )

    try:
        net_data = gzip.decompress(net_file_gz.read_bytes())
    except Exception as e:
        detail = f"Failed to read uploaded file {net_file}"
        print(detail, e, flush=True)
        raise HTTPException(
            status_code=500,
            detail=detail,
        )

    net_hash = hashlib.sha256(net_data).hexdigest()[:12]

    if net_hash != net_file[3:15]:
        net_file_gz.unlink()
        detail = f"Invalid hash for uploaded file {net_file}"
        print(detail, flush=True)
        raise HTTPException(
            status_code=500,
            detail=detail,
        )
EOF0
EOF

# script to set the server from which to download the nets
sudo -i -u ${user_name} << EOF
cat << EOF0 > \${HOME}/net-server/set_net_server.sh
#!/bin/bash

_usage () {
cat << EOF1
usage: bash \\\${0} <o|l>
  set the server to download the nets from:
  o: the "official" server used in fishtest
  l: this development "local" server
EOF1
exit
}

if [[ \\\${#} == '0' ]] ; then
  _usage
fi

if [[ \\\${1} == 'l' ]]; then
  sed -i 's/"https:\/\/data.stockfishchess.org\/nn\/"/"http:\/\/${server_name}\/nn\/"/' \\\${HOME}/fishtest/server/fishtest/api.py
elif [[ \\\${1} == 'o' ]]; then
  sed -i 's/"http:\/\/${server_name}\/nn\/"/"https:\/\/data.stockfishchess.org\/nn\/"/' \\\${HOME}/fishtest/server/fishtest/api.py
else
  _usage
fi

echo 'fishtest restart to apply the new setting:'
sudo systemctl restart fishtest@{6543..6545}
EOF0
EOF

### install net-server as systemd service
cat << EOF > /etc/systemd/system/net-server.service
[Unit]
Description=fastapi server for chess engine networks
After=network.target

[Service]
Type=simple
ExecStart=${user_home}/net-server/env/bin/gunicorn net_server:app --timeout 120 --workers 4 --worker-class uvicorn.workers.UvicornWorker
Restart=on-failure
RestartSec=3
User=${user_name}
WorkingDirectory=${user_home}/net-server

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable net-server.service
systemctl start net-server.service

cat << EOF
connect a browser to:
http://${server_name}
EOF

Let's Encrypt setup

For Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy)

Click to view

# sudo bash setup-certbot.sh 2>&1 | tee setup-certbot.sh.log
# install certbot to setup let's encrypt
# https://certbot.eff.org/
# requires a DNS and a fully qualified domain name as servername
snap install core
snap refresh core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
cat << EOF
to configure let's encrypt run:
sudo certbot --nginx
EOF

Test a PR/branch

Server

#!/bin/bash
# to update a fishtest server simply run:
# sudo bash update_fishtest.sh 2>&1 | tee update_fishtest.sh.log
#
# to use fishtest connect a browser to:
# http://<ip_address>
user_name='fishtest'
# backup
sudo -i -u ${user_name} << EOF
tar -czvf fishtest.$(date +%Y%m%d%H%M%S --utc).tgz fishtest
EOF
systemctl stop cron
systemctl stop fishtest@{6543..6545}
sleep 5
# download and prepare fishtest
sudo -i -u ${user_name} << EOF
rm -rf fishtest
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email 'you@example.com'
git config user.name 'your_name'
# add here the upstream branch to be tested
#git remote add <your_upstream> https://github.com/<your_username>/fishtest
#git pull --no-edit --rebase <your_upstream> <your_branch>
# add here the PR/PRs to be tested
#git pull --no-edit --rebase origin pull/<PR_number>/head
#git pull --no-edit --rebase origin pull/<PR_number>/head
# setup fishtest
sudo -i -u ${user_name} << 'EOF'
python3 -m venv ${VENV}
${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
cd ${HOME}/fishtest/server
${VENV}/bin/python3 -m pip install -e .
EOF
# start fishtest
systemctl start cron
systemctl start fishtest@{6543..6545}
cat << EOF
connect a browser to:
http://$(hostname --all-ip-addresses)/tests
EOF

Worker

#!/bin/bash
# requirements:
# sudo apt update && sudo apt install -y python3 python3-pip python3-venv git build-essential libnuma-dev

test_folder=/<full_path>/__test_folder
virtual_env=${test_folder}/env

rm -rf ${test_folder}
mkdir -p ${test_folder}
cd ${test_folder}
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email "you@example.com"
git config user.name "your_name"

# add here the upstream branches to be tested
#git remote add <upstream_0> https://github.com/<username_0>/fishtest
#git pull --no-edit --rebase <upstream_0> <branch_0>
#git remote add <upstream_1> https://github.com/<username_1>/fishtest
#git pull --no-edit --rebase <upstream_1> <branch_1>

# add here the PRs to be tested
#git pull --no-edit --rebase origin pull/<PR_number_0>/head
#git pull --no-edit --rebase origin pull/<PR_number_1>/head

cd worker
python3 -m venv ${virtual_env}
${virtual_env}/bin/python3 -m pip install --upgrade pip setuptools wheel

${virtual_env}/bin/python3 worker.py user00 user00 --protocol http --host <ip-address> --port 80 --concurrency 3 

Mongodb: backup and restore

Click to view

Use the mongodb tools in a temporary folder, e.g.

• Backup

  mongodump --gzip && tar -cvf dump.tar dump && rm -rf dump
  

• Restore

  tar -xvf dump.tar && mongorestore --gzip --drop && rm -rf dump
  

Stop fishtest and cron services before a mongodb restore:

sudo systemctl stop fishtest@{6543..6545}
sudo systemctl stop cron

---

FAQ

Bad worker

Sometime a badly configured worker client may post lots of losses on time during tests, or cause some tests to stop
The best policy to follow in these cases would be:

1. Login with your own approver or user username/password
2. Click on a worker name in either the Events Log or the Workers table
3. All the workers names are now a hyperlink
4. You can block/unblock the worker. If you are an approver continue with the other steps
5. Write in the Message box the reason for the block and some short instruction to solve the issue. If blocked, a worker is not able to get a new task and it shows the Message
6. Raise your concerns about the worker anomalous behavior in the appropriate channel inside the Discord server
7. If necessary, write an email to the user asking to control the worker

Block a user

Approvers (and only approvers) can now block malicious users on fishtest:

1. Login with your own approver username/password
2. Click on a run to see the test page
3. All the workers names are now a hyperlink for the approvers
4. Click on a worker name (field "username")
5. You view some info about the user (e.g. the email)
6. You can block/unblock the user. If blocked, the user cannot login in fishtest (e.g. submitting a test) and the workers cannot join the framework