Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local validation for forked PRs #11

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/process _fork.yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this, it's unrelated with validate workflow – and not needed anyway IMO.

Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Process newly added JSON

on:
pull_request:
types: [closed]
branches:
- main

permissions:
contents: write

concurrency:
group: process-json-${{ github.ref }}
cancel-in-progress: true

jobs:
process-json:
# Only run if the PR was merged and it is opened from a forked repo
if: github.event.pull_request.merged == true && github.event.pull_request.head.repo.full_name == github.repository

runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
# Needed to push changes back to the repo
fetch-depth: 0

# Must be done before setup-node.
- name: Enable Corepack
run: corepack enable

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22"
cache: "yarn"
cache-dependency-path: actions/yarn.lock

- name: Install Dependencies
run: yarn install --frozen-lockfile
working-directory: ./actions

- name: Find newly added JSON files
id: find-json
run: |
# Get the list of added JSON files in the records/new/ directory
ADDED_FILES=$(git diff ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} --diff-filter=A --name-only | grep '^records/new/.*\.json$' || true)
echo "NEW_JSON_FILES=$ADDED_FILES" >> $GITHUB_ENV
if [ -z "$ADDED_FILES" ]; then
echo "No new JSON files found."
fi

- name: Process and move files
if: env.NEW_JSON_FILES
env:
BLUESKY_IDENTIFIER_NODEJS_ORG: nodejs.org
BLUESKY_APP_PASSWORD_NODEJS_ORG: ${{ secrets.BLUESKY_APP_PASSWORD_NODEJS_ORG }}
run: |
for file in $NEW_JSON_FILES; do
echo "Processing $file..."
node actions/process.js "$file" validate_cred_fork.js
Copy link
Member

@joyeecheung joyeecheung Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow doesn't look right? It should run validate_cred_fork.js on the files and it should not run the commit and push stuff?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It pushes changes only after logging into the account; otherwise, it will result in an error.

done

- name: Commit and push changes
if: env.NEW_JSON_FILES
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git add records/*
git commit -m "Process new JSON files from #${{ github.event.pull_request.number }}" || exit 0
git push
6 changes: 3 additions & 3 deletions .github/workflows/process.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ concurrency:

jobs:
process-json:
# Only run if the PR was merged
if: github.event.pull_request.merged == true
# Only run if the PR was merged and it is opened from same repo
if: github.event.pull_request.merged == true && github.event.pull_request.head.repo.full_name == github.repository

runs-on: ubuntu-latest

Expand Down Expand Up @@ -60,7 +60,7 @@ jobs:
run: |
for file in $NEW_JSON_FILES; do
echo "Processing $file..."
node actions/process.js "$file"
node actions/process.js "$file" validate_cred.js
done

- name: Commit and push changes
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,8 @@ jobs:

- name: Validate files
if: env.NEW_JSON_FILES
env:
BLUESKY_IDENTIFIER_NODEJS_ORG: nodejs.org
BLUESKY_APP_PASSWORD_NODEJS_ORG: ${{ secrets.BLUESKY_APP_PASSWORD_NODEJS_ORG }}
run: |
for file in $NEW_JSON_FILES; do
echo "Processing $file..."
node actions/login-and-validate.js "$file"
node actions/validate_request.js "$file"
done
36 changes: 13 additions & 23 deletions actions/lib/posts.js
Original file line number Diff line number Diff line change
@@ -1,23 +1,15 @@
import AtpAgent, { RichText } from "@atproto/api";
import assert from 'node:assert';

// URL format:
// 1. https://bsky.app/profile/${handle}/post/${postId}
// 2. https://bsky.app/profile/${did}/post/${postId}
// TODO(joyeecheung): consider supporting base other than bsky.app.
const kURLPattern = /https:\/\/bsky\.app\/profile\/(.+)\/post\/(.+)/;

/**
* @param {string} url
*/
export function validatePostURL(url) {
const match = url.match(kURLPattern);
assert(match, `Post URL ${url} does not match the expected pattern`);

export function PostURL(url) {
const kURLPattern = /https:\/\/bsky\.app\/profile\/(.+)\/post\/(.+)/;
const match_url = url.match(kURLPattern);
return {
handle: match[1],
postId: match[2],
isDid: match[1].startsWith('did:')
handle: match_url[1],
postId: match_url[2],
isDid: match_url[1].startsWith('did:')
};
}

Expand All @@ -26,7 +18,7 @@ export function validatePostURL(url) {
* @param {string} postUrl
*/
export async function getPostInfoFromUrl(agent, postUrl) {
const { handle, postId, isDid } = validatePostURL(postUrl);
const { handle, postId, isDid } = PostURL(postUrl);
let did;
if (isDid) {
did = handle;
Expand All @@ -42,15 +34,13 @@ export async function getPostInfoFromUrl(agent, postUrl) {
return { uri, cid };
}

// URI format: at://${did}/app.bsky.feed.post/${postId}
const kURIPattern = /at:\/\/(.*)+\/app\.bsky\.feed\.post\/(.*)+/
export function validatePostURI(uri) {
const match = uri.match(kURIPattern);
assert(match, `Post URI ${uri} does not match the expected pattern`);

export function PostURI(uri) {
const kURIPattern = /at:\/\/(.*)+\/app\.bsky\.feed\.post\/(.*)+/
const match_uri = uri.match(kURIPattern);
return {
did: match[1],
postId: match[2]
did: match_uri[1],
postId: match_uri[2]
};
}

Expand All @@ -59,7 +49,7 @@ export function validatePostURI(uri) {
* @param {string} uri
*/
export async function getPostURLFromURI(agent, uri) {
const { did, postId } = validatePostURI(uri);
const { did, postId } = PostURI(uri);
const profile = await agent.getProfile({ actor: did });
const handle = profile.data.handle;

Expand Down
10 changes: 5 additions & 5 deletions actions/lib/validator.js
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ export function validateRequest(request) {
* @param {object} request
* @param {string} fieldName
*/
async function validatePostURLInRequest(agent, request, fieldName) {
async function PostURLInRequest(agent, request, fieldName) {
let result;
try {
result = await getPostInfoFromUrl(agent, request[fieldName]);
Expand All @@ -60,21 +60,21 @@ async function validatePostURLInRequest(agent, request, fieldName) {
}

/**
* Validate the post URLs in the request and extend them into { uri, cid } pairs
* Extend them into { uri, cid } pairs
* if necessary.
* @param {import('@atproto/api').AtpAgent} agent
* @param {object} request
*/
export async function validateAndExtendRequestReferences(agent, request) {
export async function ExtendRequestReferences(agent, request) {
switch(request.action) {
case 'repost':
case 'quote-post': {
const info = await validatePostURLInRequest(agent, request, 'repostURL');
const info = await PostURLInRequest(agent, request, 'repostURL');
request.repostInfo = info;
break;
}
case 'reply': {
const info = await validatePostURLInRequest(agent, request, 'replyURL');
const info = await PostURLInRequest(agent, request, 'replyURL');
request.replyInfo = info;
break;
}
Expand Down
8 changes: 5 additions & 3 deletions actions/process.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,19 @@ import process from 'node:process';
import path from 'node:path';
import { login } from './lib/login.js';
import { post } from './lib/posts.js';
import { validateAccount, validateRequest, validateAndExtendRequestReferences } from './lib/validator.js';
import { validateAccount, validateRequest, ExtendRequestReferences } from './lib/validator.js';

// This script takes a path to a JSON with the pattern $base_path/new/$any_name.json,
// where $any_name can be anything, and then performs the action specified in it.
// If the action succeeds, it moves the JSON file to
// $base_path/processed/$YYYY-$MM-$DD-$ID.json, where $ID is an incremental number
// starting from 0 based on the number of existing JSONs processed on the same date
// and already in the processed directory.
// Validation checks performed based on where pr is opened

assert(process.argv[2], `Usage: node process.js $base_path/new/$any_name.json`);
const { agent, request, requestFilePath } = await import('./login-and-validate.js');
assert(process.argv[3], `Usage: node process.js $base_path/new/$any_name.json required_auth.js`);
const { agent, request, requestFilePath } = await import(`./${process.argv[3]}`);

let result;
switch(request.action) {
Expand All @@ -25,7 +27,7 @@ switch(request.action) {
};
case 'repost': {
console.log('Reposting...', request.repostURL);
assert(request.repostInfo); // Extended by validateAndExtendRequestReferences.
assert(request.repostInfo); // Extended by ExtendRequestReferences.
result = await agent.repost(request.repostInfo.uri, request.repostInfo.cid);
break;
}
Expand Down
7 changes: 3 additions & 4 deletions actions/login-and-validate.js → actions/validate_cred.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import fs from 'node:fs';
import process from 'node:process';
import path from 'node:path';
import { login } from './lib/login.js';
import { validateAccount, validateRequest, validateAndExtendRequestReferences } from './lib/validator.js';
import { validateAccount, ExtendRequestReferences } from './lib/validator.js';

// The JSON file must contains the following fields:
// - "account": a string field indicating the account to use to perform the action.
Expand All @@ -15,12 +15,11 @@ const request = JSON.parse(fs.readFileSync(requestFilePath, 'utf8'));

// Validate the account field.
const account = validateAccount(request, process.env);
validateRequest(request);

// Authenticate.
const agent = await login(account);

// Validate and extend the post URLs in the request into { cid, uri } records.
await validateAndExtendRequestReferences(agent, request);
// Extend the post URLs in the request into { cid, uri } records.
await ExtendRequestReferences(agent, request);

export { agent, request, requestFilePath };
25 changes: 25 additions & 0 deletions actions/validate_cred_fork.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import fs from 'node:fs';
import process from 'node:process';
import path from 'node:path';
import { login } from './lib/login.js';
import { validateAccount, ExtendRequestReferences } from './lib/validator.js';

// The JSON file must contains the following fields:
// - "account": a string field indicating the account to use to perform the action.
// For it to work, this script expects BLUESKY_IDENTIFIER_$account and
// BLUESKY_APP_PASSWORD_$account to be set in the environment variables.
// - "action": currently "post", "repost", "quote-post", "reply" are supported.

const requestFilePath = path.resolve(process.argv[2]);
const request = JSON.parse(fs.readFileSync(requestFilePath, 'utf8'));

// Validate the account field.
const account = validateAccount(request, process.env);

// Authenticate.
const agent = await login(account);

// Extend the post URLs in the request into { cid, uri } records.
await ExtendRequestReferences(agent, request);

export { agent, request, requestFilePath };
48 changes: 48 additions & 0 deletions actions/validate_request.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import fs from 'node:fs';
import process from 'node:process';
import path from 'node:path';
import { validateRequest } from './lib/validator';
import assert from 'node:assert';

const requestFilePath = path.resolve(process.argv[2]);
const request = JSON.parse(fs.readFileSync(requestFilePath, 'utf8'));

//Check the format of the request
validateRequest(request);


// URL format:
// 1. https://bsky.app/profile/${handle}/post/${postId}
// 2. https://bsky.app/profile/${did}/post/${postId}
// TODO(joyeecheung): consider supporting base other than bsky.app.
const kURLPattern = /https:\/\/bsky\.app\/profile\/(.+)\/post\/(.+)/;
let url;
if (request.action!='post'){
switch(request.action) {
case 'repost':
case 'quote-post': {
url=request.repostURL;
break;
}
case 'reply': {
url=request.replyURL;
break;
}
default:
break;
}
match_url = url.match(kURLPattern);
assert(match_url, `Post URL ${url} does not match the expected pattern`);
}







// // URI format: at://${did}/app.bsky.feed.post/${postId}
// const kURIPattern = /at:\/\/(.*)+\/app\.bsky\.feed\.post\/(.*)+/

// const match_uri = uri.match(kURIPattern);
// assert(match_uri, `Post URI ${uri} does not match the expected pattern`);