Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature/psa-metrics #969

Merged
merged 7 commits into from
Feb 23, 2022
Merged

feature/psa-metrics #969

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5f5967e
chore: add configs
Feb 10, 2022
9ccc73a
feat: add upload type and pin requests metrics
Feb 10, 2022
ff9425e
fix: edit copy
Feb 14, 2022
0b5a205
chore: use client for query
Feb 14, 2022
7bea36c
fix: pass upload type directly to getUploadTypeMetrics
Feb 21, 2022
0d1e497
fix: typo
Feb 21, 2022
7b7ee64
fix: decouple db client from metric values
Feb 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/api/src/constants.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,5 @@ export const LOCAL_ADD_THRESHOLD = 1024 * 1024 * 2.5
export const DAG_SIZE_CALC_LIMIT = 1024 * 1024 * 9
// Maximum permitted block size in bytes.
export const MAX_BLOCK_SIZE = 1 << 20 // 1MiB
export const UPLOAD_TYPES = ['Car', 'Blob', 'Multipart', 'Upload']
export const PIN_STATUSES = ['PinQueued', 'Pinning', 'Pinned', 'PinError']
52 changes: 28 additions & 24 deletions packages/api/src/metrics.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/* global Response caches */

import { METRICS_CACHE_MAX_AGE } from './constants.js'
import { METRICS_CACHE_MAX_AGE, PIN_STATUSES, UPLOAD_TYPES } from './constants.js'

/**
* Retrieve metrics in prometheus exposition format.
Expand All @@ -21,21 +21,29 @@ export async function metricsGet (request, env, ctx) {
const [
usersTotal,
uploadsTotal,
uploadMetrics,
contentTotalBytes,
pinsTotal,
pinsQueuedTotal,
pinsPinningTotal,
pinsPinnedTotal,
pinsFailedTotal
pinsMetrics,
pinsRequestsTotal
] = await Promise.all([
env.db.getMetricsValue('users_total'),
env.db.getMetricsValue('uploads_total'),
Promise.all(
UPLOAD_TYPES.map(async (t) => ({
type: t,
total: await env.db.getMetricsValue(`uploads_${t.toLowerCase()}_total`)
}))
),
env.db.getMetricsValue('content_bytes_total'),
env.db.getMetricsValue('pins_total'),
env.db.getMetricsValue('pins_status_queued_total'),
env.db.getMetricsValue('pins_status_pinning_total'),
env.db.getMetricsValue('pins_status_pinned_total'),
env.db.getMetricsValue('pins_status_failed_total')
Promise.all(
PIN_STATUSES.map(async (s) => ({
status: s,
total: await env.db.getMetricsValue(`pins_status_${s.toLowerCase()}_total`)
}))
),
env.db.getMetricsValue('pin_requests_total')
])

const metrics = [
Expand All @@ -46,6 +54,10 @@ export async function metricsGet (request, env, ctx) {
'# HELP web3storage_uploads_total Total number of user uploads.',
'# TYPE web3storage_uploads_total counter',
`web3storage_uploads_total ${uploadsTotal}`,
...uploadMetrics.map(
({ type, total }) =>
`web3storage_uploads_total{type="${type}"} ${total || 0}`
),

'# HELP web3storage_content_bytes_total Total bytes of all unique DAGs stored.',
'# TYPE web3storage_content_bytes_total counter',
Expand All @@ -54,22 +66,14 @@ export async function metricsGet (request, env, ctx) {
'# HELP web3storage_pins_total Total number of pins on the IPFS Cluster',
'# TYPE web3storage_pins_total counter',
`web3storage_pins_total ${pinsTotal}`,
...pinsMetrics.map(
({ status, total }) =>
`web3storage_pins_total{status="${status}"} ${total || 0}`
),

'# HELP web3storage_pins_status_queued_total Total number of pins that are queued.',
'# TYPE web3storage_pins_status_queued_total counter',
`web3storage_pins_status_queued_total ${pinsQueuedTotal}`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to change this. But we need to take into account older values. Should we rename things here to Prometheus best practises and do a sum in Grafana query with the older name?

cc @hugomrdias @alanshaw

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes!


'# HELP web3storage_pins_status_pinning_total Total number of pins that are pinning.',
'# TYPE web3storage_pins_status_pinning_total counter',
`web3storage_pins_status_pinning_total ${pinsPinningTotal}`,

'# HELP web3storage_pins_status_pinned_total Total number of pins that are pinned.',
'# TYPE web3storage_pins_status_pinned_total counter',
`web3storage_pins_status_pinned_total ${pinsPinnedTotal}`,

'# HELP web3storage_pins_status_failed_total Total number of pins that are failed.',
'# TYPE web3storage_pins_status_failed_total counter',
`web3storage_pins_status_failed_total ${pinsFailedTotal}`
'# HELP web3storage_pin_requests_total Total number of pin requests via Pinning Service API.',
'# TYPE web3storage_pin_requests_total counter',
`web3storage_pin_requests_total ${pinsRequestsTotal}`
].join('\n')

res = new Response(metrics, {
Expand Down
13 changes: 9 additions & 4 deletions packages/api/test/metrics.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,16 @@ describe('GET /metrics', () => {
const text = await res.text()
assert(text.includes('web3storage_users_total'))
assert(text.includes('web3storage_uploads_total'))
assert(text.includes('web3storage_uploads_total{type="Car"}'))
assert(text.includes('web3storage_uploads_total{type="Blob"}'))
assert(text.includes('web3storage_uploads_total{type="Multipart"}'))
assert(text.includes('web3storage_uploads_total{type="Upload"}'))
assert(text.includes('web3storage_content_bytes_total'))
assert(text.includes('web3storage_pins_total'))
assert(text.includes('web3storage_pins_status_queued_total'))
assert(text.includes('web3storage_pins_status_pinning_total'))
assert(text.includes('web3storage_pins_status_pinned_total'))
assert(text.includes('web3storage_pins_status_failed_total'))
assert(text.includes('web3storage_pins_total{status="PinQueued"}'))
assert(text.includes('web3storage_pins_total{status="Pinning"}'))
assert(text.includes('web3storage_pins_total{status="Pinned"}'))
assert(text.includes('web3storage_pins_total{status="PinError"}'))
assert(text.includes('web3storage_pin_requests_total'))
})
})
8 changes: 8 additions & 0 deletions packages/api/wrangler.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,14 @@ vars = { CLUSTER_API_URL = "https://leslieoa-cluster-api-web3-storage.loca.lt",
[env.leslieoa.durable_objects]
bindings = [{ name = "NAME_ROOM", class_name = "NameRoom0" }]

[env.alexandra]
workers_dev = true
account_id = "a454b0d1615f83ee5d00a63ae152f36f"
vars = { CLUSTER_API_URL = "https://alexandra-cluster-api-web3-storage.loca.lt", PG_REST_URL = "https://alexandra-postgres-api-web3-storage.loca.lt", ENV = "dev" }

[env.alexandra.durable_objects]
bindings = [{ name = "NAME_ROOM", class_name = "NameRoom0" }]

# Add your env here. Override the the values you need to change.

# Create your env name as the value of `whoami` on your system, so you can run `npm start` to run in dev mode.
Expand Down
1 change: 1 addition & 0 deletions packages/db/index.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ export { gql }
export class DBClient {
constructor(config: { endpoint?: string; token: string, postgres?: boolean })
client: PostgrestClient
getMetricsValue (key: string): Promise<{ total: number }>
upsertUser (user: UpsertUserInput): Promise<UpsertUserOutput>
getUser (issuer: string): Promise<UserOutput>
getUsedStorage (userId: number): Promise<number>
Expand Down
19 changes: 15 additions & 4 deletions packages/db/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ import {
getPinMetrics,
getPinStatusMetrics,
getContentMetrics,
getPinBytesMetrics
getPinBytesMetrics,
getPinRequestsMetrics,
getUploadTypeMetrics
} from './metrics.js'

const uploadQuery = `
Expand Down Expand Up @@ -796,6 +798,12 @@ export class DBClient {
case 'uploads_total':
res = await getUploadMetrics(this._client)
return res.total
case 'uploads_car_total':
case 'uploads_blob_total':
case 'uploads_multipart_total':
case 'uploads_upload_total':
res = await getUploadTypeMetrics(this._client, key)
return res.total
case 'content_bytes_total':
res = await getContentMetrics(this._client)
return res.totalBytes
Expand All @@ -805,14 +813,17 @@ export class DBClient {
case 'pins_bytes_total':
res = await getPinBytesMetrics(this._client)
return res.totalBytes
case 'pins_status_queued_total':
case 'pins_status_pinqueued_total':
case 'pins_status_pinning_total':
case 'pins_status_pinned_total':
case 'pins_status_failed_total':
case 'pins_status_pinerror_total':
res = await getPinStatusMetrics(this._client, key)
return res.total
case 'pin_requests_total':
res = await getPinRequestsMetrics(this._client)
return res.total
default:
throw new Error('unknown metric requested')
throw new Error(`unknown metric requested: ${key}`)
}
}

Expand Down
39 changes: 37 additions & 2 deletions packages/db/metrics.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,26 @@ export async function getUploadMetrics (client) {
}
}

const uploadTypeMapping = {
uploads_car_total: 'Car',
uploads_blob_total: 'Blob',
uploads_multipart_total: 'Multipart',
uploads_upload_total: 'Upload'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DB client shouldn't be concerned with the keys we're using in the metrics. An actual upload "type" should be passed to getUploadTypeMetrics.

}

export async function getUploadTypeMetrics (client, key) {
const uploadType = uploadTypeMapping[key]
const { data, error } = await client.rpc('uploads_by_type', { query_type: uploadType })

if (error) {
throw new DBError(error)
}

return {
total: data
}
}

export async function getContentMetrics (client) {
const { data, error } = await client.rpc('content_dag_size_total')
if (error) {
Expand Down Expand Up @@ -68,10 +88,10 @@ export async function getPinMetrics (client) {
}

const pinStatusMapping = {
pins_status_queued_total: 'PinQueued',
pins_status_pinqueued_total: 'PinQueued',
pins_status_pinning_total: 'Pinning',
pins_status_pinned_total: 'Pinned',
pins_status_failed_total: 'PinError'
pins_status_pinerror_total: 'PinError'
}

export async function getPinStatusMetrics (client, key) {
Expand All @@ -86,3 +106,18 @@ export async function getPinStatusMetrics (client, key) {
total: data
}
}

export async function getPinRequestsMetrics (client, key) {
const { count, error } = await client
.from('psa_pin_request')
.select('*', { head: true, count: 'exact' })
.range(0, 1)

if (error) {
throw new DBError(error)
}

return {
total: count
}
}
13 changes: 13 additions & 0 deletions packages/db/postgres/functions.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,19 @@ BEGIN
END
$$;

CREATE OR REPLACE FUNCTION uploads_by_type(query_type TEXT) RETURNS TEXT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering why do we need a procedure here? Shouldn't we just use client count API?
I see that's how it is done for other metrics, but I'm not sure if/why it's needed?

I wonder if was done like this because of the Supabase client issue loading the all table even when just "counting"?
But that can be workarounded by adding range(0, 1)

Probably one for @vasco-santos

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sum we can't do with the client, and that was the main motivation to not use the client. The getPinStatusMetrics using pin_from_status_total I don't recall if there was a reason. Either the cast to pin status, or was just added here with the sums and we didn't consider using the client.

We use the client for other metrics counts, so if we can use it here, I think we should

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexandrastoica did you find a reason to not have it using supabase client?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will change to client! :)

LANGUAGE plpgsql
AS
$$
BEGIN
return(
select count(*)
from upload
where type = (query_type)::upload_type
)::TEXT;
END
$$;

CREATE OR REPLACE FUNCTION find_deals_by_content_cids(cids text[])
RETURNS TABLE
(
Expand Down