Pyinstabot-downloader

Table of contents

• About this project
• Project architecture
• Requirements
• Environment variables
• Prepare and configure environment
  • WebDav as the target storage of the content
  • Bot configuration source and supported parameters
  • Bot persistent data storage
• How to run project locally

About this project

This project is a Telegram bot that allows you to upload posts from your Instagram profile to WebDav compatible storage.

Main functions

• a backup copy of a specific post by link
• a backup copy of list of posts by links

Preview of the bot in action

Project architecture

Users flow

Code structure

Requirements

• Vault Server
  • store the project configuration in kv2
  • generate access credentials in the database
  • prepare the vault server
• WebDav Cloud
  • remote directory url
  • username and password
• Telegram Bot
  • api token
• Instagram Account
  • username and password
• Postgresql Database
  • empty database

Environment variables

Variable	Description	Default value
LOGGER_LEVEL	The logging level of the logging module	INFO
TELEGRAM_BOT_NAME	The name of the bot, used to determine the unique mount point in the vault	pyinstabot-downloader
MESSAGES_CONFIG	The path to the message template file	src/configs/messages.json
VAULT_*	All supported vault environment variables can be found here	-

Prepare and configure environment

WebDav as the target storage of the content

• Create an account in any cloud that supports WebDav
• Get the WebDav directory url from your cloud provider
• Get the WebDav username and password from your cloud provider

Bot configuration source and supported parameters

All bot configuration is stored in the Vault Secrets (except for the part of the configuration that configures the connection to Vault)

• pyinstabot-downloader-database - vault database engine mount point, returns the temporary username and password for the database. More information about the database engine can be found here and here
  

• configuration/database: database connection parameters

  {
    "dbname": "pyinstabot-downloader",
    "host": "postgresql.example.com",
    "port": "5432",
    "connections": "10"
  }

  
  

• configuration/telegram: telegram bot configuration

  {
    "token": "123456:qwerty"
  }

  
  

• configuration/downloader-api: downloader module configuration (for downloading content from instagram)

  {
    "enabled": "True",
    "2fa-enabled": "False",
    "2fa-seed": "my_2fa_secret",
    "country-code": "1",
    "delay-requests": "1",
    "locale": "en_US",
    "username": "my_username",
    "password": "my_password",
    "session-file": "session.json",
    "timezone-offset": "10800",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
    "request-timeout": "10"
  }

  Description of parameters

  • delay-requests: the delay between requests to the instagram api in seconds
  • locale: the locale of the instagram account
  • session-file: the path to the file where the session data will be stored
  • timezone-offset: the offset of the timezone in seconds
  • user-agent: the user-agent of the instagram account
  • 2fa-enabled: two-factor authentication status
  • 2fa-seed: two-factor authentication secret
  • country-code: the country code of the instagram account
  • enabled: the status of the downloader module
  • username: the username of the instagram account
  • password: the password of the instagram account
  • request-timeout: the timeout for requests to the instagram api
  
  

• configuration/uploader-api: uploader module configuration (for upload content to the target storage)

  {
    "enabled": "True",
    "destination-directory": "cloud-directory/",
    "password": "qwerty123",
    "source-directory": "data/",
    "username": "username1",
    "url": "https://webdav.example.com/directory"
  }

  Description of parameters

  • destination-directory: the directory in the target storage where the content will be uploaded
  • source-directory: the directory where the content will be stored before uploading (temporary directory)
  • url: the url of the target webdav directory
  • enabled: the status of the uploader module
  • username: the username of the webdav account
  • password: the password of the webdav account
  
  

• configuration/users/<telegram_user_id>: users permissions and attributes

  {
    "requests": "{\"requests_per_day\": 10, \"requests_per_hour\": 1, \"random_shift_minutes\": 60}",
    "roles": "[\"post\", \"posts_list\"]",
    "status": "allowed"
  }

  Description of parameters

  • requests: the number of requests that the user can make per day and per hour, as well as the random shift in minutes (scheduling of message processing from the queue works on the basis of this parameter)
  • roles: list of roles that allow to use the corresponding functionality (available roles).
  • status: allowed or denied user access to the bot

You can use an existing vault-server or launch a new one using docker-compose

Scripts for configuring the vault-server are located in the vault-init.sh

• instructions for starting and configuring a new vault-server

# Clone the repository
git clone https://github.com/obervinov/pyinstabot-downloader.git
cd pyinstabot-downloader

# Run vault-server
docker-compose -f docker-compose.yml up vault-server -d

# Initialize and unseal new vault-server
export VAULT_ADDR=http://0.0.0.0:8200
vault operator init
vault operator unseal

# Run the script for configuring the vault-server for this bot project
export VAULT_TOKEN=hvs.123456qwerty
./scripts/vault-init.sh

• instructions for configuring an existing vault server

# Clone the repository
git clone https://github.com/obervinov/pyinstabot-downloader.git
cd pyinstabot-downloader

# Run the script for configuring the vault-server for this bot project
export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=hvs.123456qwerty
./scripts/vault-init.sh

Bot persistent data storage

Persistent data storage is implemented using Postgresql

• data structure, tables and assignment of tables here
• migrations here

The database structure is created automatically when the bot starts:

1. bot checks the database structure and creates missing tables if necessary
2. after checking the database structure, the bot executes the migrations in the order of their numbering

To quickly prepare an instance, you can execute the psql-init.sh script

git clone https://github.com/obervinov/pyinstabot-downloader.git
cd pyinstabot-downloader

export PGHOST=<host>
export PGPORT=<port>
export PGUSER=<user>
export PGPASSWORD=<password>
export PGDATABASE=postgres
./scripts/psql-init.sh

What data is stored in tables:

• user request queue
• user metadata
• user request logs
• processed messages
• migrations
• bot messages

How to run project locally

export VAULT_APPROLE_ID={change_me}
export VAULT_APPROLE_SECRET_ID={change_me}
export VAULT_ADDR={change_me}
docker compose -f docker-compose.yml up -d

GitHub Actions

Name	Version
GitHub Actions Templates	v2.0.2