Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

General updates #21

Merged
merged 7 commits into from
Nov 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ jobs:
name: Publish to Cloudflare Pages

if: ${{
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false) ||
(github.event_name == 'pull_request_target' &&
Expand Down
25 changes: 25 additions & 0 deletions components/DiagramImage.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import Image from 'next/image'
import { useTheme } from 'next-themes'
import { useMounted } from 'nextra/hooks'

export function DiagramImage() {
const { resolvedTheme } = useTheme()
const mounted = useMounted()

if (!mounted) return null
return resolvedTheme === 'light' ? (
<Image
src="/images/Unownhash-Services-v3-light.jpg"
alt="Unown# service diagram"
width={900}
height={900}
/>
) : (
<Image
src="/images/Unownhash-Services-v3-dark.jpg"
alt="Unown# service diagram"
width={900}
height={900}
/>
)
}
2 changes: 1 addition & 1 deletion pages/docs/dragonite/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Dragonite is **closed source** software. We are releasing compiled binaries to t
1. Koji integration for hands free route creation
1. Built in level mode that is fast
1. Highly scalable, can run 1000+ workers in a single installation
1. grpc raw send to golbat, can send raws to multiple destinations (including RDM)
1. gRPC raw send to golbat, can send raws to multiple destinations (including RDM)
1. Built in prometheus stats, and raw stats table for detailed performance measurements
1. API support for modifying settings; and a beautiful admin tool

Expand Down
2 changes: 1 addition & 1 deletion pages/docs/golbat/index.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Golbat

[**Golbat**](https://github.com/UnownHash/Golbat) is our data processor. Its job is to accept HTTP or GRPC requests, format them into something useful, save them to your database, and optionally send webhooks to external services. We have optimized it over some with large instances processing many thousands of requests per second.
[**Golbat**](https://github.com/UnownHash/Golbat) is our data processor. Its job is to accept HTTP or gRPC requests, format them into something useful, save them to your database, and optionally send webhooks to external services. Golbat has been thoroughly tested with large instances processing many thousands of requests per second.

One of Golbat's best features is that pokemon encounters are stored in memory. This allows maps, like ReactMap, to connect to Golbat over an API allowing for a vastly improved experience over SQL queries. This also reduces server operations as your database will not constantly be filled full of short lived events that are deleted within a hour.

Expand Down
2 changes: 1 addition & 1 deletion pages/docs/golbat/pvp.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { SourceLink } from '../../../components/SourceLink'
import { SourceLink } from '@/components/SourceLink'

# PvP Calculation

Expand Down
2 changes: 1 addition & 1 deletion pages/docs/golbat/stats.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import Image from 'next/image'

Golbat comes with basic database statistics built in. These tables are named with the `_stats` suffix. With these tables you can use raw SQL queries or hook into a third party visualization tool to visualize the data that Golbat is parsing.

[Blissey](/docs/blissey) and our [Prometheus](https://github.com/UnownHash/Golbat/pull/100) integration can be used to get started on visualizing the data.
[Blissey](/docs/other/blissey) and our [Prometheus](https://github.com/UnownHash/Golbat/pull/100) integration can be used to get started on visualizing the data.

<br/>

Expand Down
4 changes: 2 additions & 2 deletions pages/docs/index.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import Image from 'next/image'
import { DiagramImage } from '@/components/DiagramImage'

# Intro

Expand All @@ -11,4 +11,4 @@ Make sure to have a functional [Database Setup](/docs/other/database) prior to s

## High Level Diagram

<Image src="/images/Unownhash-Services-v2.jpg" alt="Unown# service diagram" width={900} height={900} />
<DiagramImage/>
6 changes: 2 additions & 4 deletions pages/docs/other/blissey.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import Image from 'next/image'

# Blissey

[**Blissey**](https://github.com/UnownHash/Blissey) processes reporting data from Golbat into separate database tables for visualization in [Grafana](https://grafana.com/).
[**Blissey**](https://github.com/UnownHash/Blissey) processes reporting data from Golbat and Dragonite into separate database tables for visualization in [Grafana](https://grafana.com/).

## Showcase

Expand All @@ -24,6 +24,4 @@ The below images are a real instance of Grafana displaying a few of the released

## Setup

<Callout type="info" emoji="🚧">
This section is under construction
</Callout>
The setup process for Blissey is documented in the project README on [GitHub](https://github.com/UnownHash/Blissey).
52 changes: 37 additions & 15 deletions pages/docs/other/database.mdx
Original file line number Diff line number Diff line change
@@ -1,38 +1,52 @@
# Database

To use the Unown# services, you will need a database server with two different databases configured. One for **Golbat** and one for **Dragonite**.
To use Unown# services, you will need a database server with two different databases configured. One for **Golbat** and one for **Dragonite**. If you have an existing database feel free to continue using your existing service.

## Supported

- MariaDB 10.6 or higher (recommended)
- MySQL 8 or higher

## Create a database

Feel free to choose one of the following install methods. No matter your choice of Standard or Docker for hosting your database it is highly recommended that you create backups of your database.

### Standard database setup

Standard installation is thoroughly documented on external sites like [https://mariadb.org/](https://mariadb.org/) and [https://www.mysql.com/](https://www.mysql.com/). Please use one of those community guides for your specific platform.

### Docker database setup

1. Create a folder for storing your database files. This should ideally live on an fast and redundant SSD volume.

```sh
mkdir database
```

1. Sample service with docker:

```yml filename="docker-compose.yml"
version: '3.7'
services:
db:
db:
image: mariadb:10.11
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --default-authentication-plugin=mysql_native_password --binlog-expire-logs-seconds=86400
container_name: db
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: YourStrongRootPassw0rd!
MYSQL_DATABASE: _temp
MYSQL_USER: unownhash
MYSQL_PASSWORD: YourStrongPassw0rd!
MYSQL_ROOT_PASSWORD: YourStrongRootPassw0rd!
MYSQL_DATABASE: _temp
MYSQL_USER: unownhash
MYSQL_PASSWORD: YourStrongPassw0rd!
ports:
- 3306:3306
- 3306:3306
volumes:
- ./database:/var/lib/mysql
#- ./my.cnf:/etc/mysql/conf.d/my.cnf # Example below
#- /etc/localtime:/etc/localtime:ro
- ./database:/var/lib/mysql
#- ./my.cnf:/etc/mysql/conf.d/my.cnf # Example below
#- /etc/localtime:/etc/localtime:ro

networks:
default:
default:
name: scanner
```

Expand All @@ -42,7 +56,7 @@ To use the Unown# services, you will need a database server with two different d
docker compose up -d
```

## Setup
## Create your database tables

Once the database service is running, you will need to connect to the database service with [TablePlus](https://tableplus.com/), [Workbench](https://www.mysql.com/products/workbench/), Command Line, etc. The default Encoding and Collation values should be fine for most configurations.

Expand All @@ -57,15 +71,15 @@ These options can help you quite significantly with performance.
# This should be 50% of RAM, leaving space for golbat
innodb_buffer_pool_size = 64G

# Log file size should certainly be >= 1GB, but on a big system, this is more appropriate
# Log file size, should certainly be >= 1GB, but on a big system this is more appropriate
innodb_log_file_size = 16G

# This should be the number of cores
# This should be number of cores
innodb_read_io_threads = 10
innodb_write_io_threads = 10
innodb_purge_threads = 10

# Some people recommend at least 1 per gb, so could be increased above
# Some people receommend at least 1 per gb, so could be increased above
innodb_buffer_pool_instances = 8

# allow big sorts, in memory temp tables
Expand All @@ -86,3 +100,11 @@ innodb_io_capacity_max=2000
# Trust disk system at the expense of recovery
innodb_doublewrite = 0
```

These last two options prevent your ibdata1 file from growing continuously and space not being re-used. If you have this problem you can add these but be aware this requires recreation of ibdata.

```yml filename="my.cnf"
# Efficiently use ibdata
innodb_undo_log_truncate = 1
innodb_undo_tablespaces = 4
```
2 changes: 1 addition & 1 deletion pages/docs/other/third-party.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { SourceLink } from '../../../components/SourceLink'
import { SourceLink } from '@/components/SourceLink'
import Image from 'next/image'

# Third Party
Expand Down
2 changes: 1 addition & 1 deletion pages/docs/setup/docker.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ import { Callout } from 'nextra-theme-docs'

> **Note**: this doesn't cover 100% of the configuration just the basics to get you started. We **highly recommend** setting a `raw_bearer` secret if these services are accessible from the public internet.

Using Golbat's GRPC endpoint (default port 50001) is recommended over the raw endpoint (default 9001).
Using Golbat's gRPC endpoint (default port 50001) is recommended over the raw endpoint (default 9001).

```toml filename="dragonite_config.toml" {2,5,7,9,11-16}
[rotom]
Expand Down
2 changes: 1 addition & 1 deletion pages/docs/setup/standard.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { Callout } from 'nextra-theme-docs'
- [Go 1.21](https://go.dev/doc/install)
- [Node 18](https://nodejs.org/en/download)
- [Git](https://git-scm.com/downloads) _(`git` might be pre-installed)_
- [PM2](https://pm2.io/docs/runtime/guide/installation/) `yarn global add pm2`
- [PM2](https://pm2.io/docs/runtime/guide/installation/) `npm install -g pm2` or `yarn global add pm2`
- [jq](https://jqlang.github.io/jq/download/) `sudo apt-get install jq`
- Compatible Database - See [Database Setup](/docs/other/database)

Expand Down
Binary file removed public/images/Unownhash-Services-v2.jpg
Binary file not shown.
Binary file added public/images/Unownhash-Services-v3-dark.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/images/Unownhash-Services-v3-light.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 4 additions & 1 deletion tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,10 @@
"moduleResolution": "node",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve"
"jsx": "preserve",
"paths": {
"@/components/*": ["./components/*"]
}
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
"exclude": ["node_modules"]
Expand Down