Skip to content

Commit

Permalink
Document local audius-compose DDEX steps (#7865)
Browse files Browse the repository at this point in the history
  • Loading branch information
theoilie authored Mar 19, 2024
1 parent f6bc750 commit 02b67d9
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 39 deletions.
1 change: 1 addition & 0 deletions dev-tools/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@
"DDEX_KEY": "49d5e13d355709b615b7cce7369174fb240b6b39",
"DDEX_SECRET": "2b2c2b90d9a489234ae629a5284de84fb0633306257f17667aaebf2345d92152",
"SESSION_SECRET": "something random",
"DDEX_ADMIN_ALLOWLIST": "127559427",
"NODE_ENV": "stage"
}
}
94 changes: 55 additions & 39 deletions packages/ddex/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,40 @@ Processes and uploads DDEX releases to Audius.
## Production setup
Use audius-docker-compose to run a production DDEX instance. After you've installed audius-docker-compose, set the following required environment variables in override.env (in the audius-docker-compose repository, not here).

### Creating a bucket in S3
1. Create a new bucket in the S3 console with the name `ddex-[dev|staging]-<label/distributor>-raw`. Use all the defaults, including "ACLs disabled"
2. Do the same for a bucket named `ddex-[dev|staging]-<label/distributor>-crawled`. Use all the defaults, including "ACLs disabled"
3. Create an IAM Policy (here](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-west-2#/policies/create) (or search IAM and click Policies > Create Policy). Select S3.
* Under `Read` choose `GetObject` and `GetObjectAttributes`.
* Under `Write` choose `DeleteObject` and `PutObject`.
* Under `List` choose `ListBucket`.
* Click `Add Arn` for object actions, enter the bucket name ending with `raw`, and check the box for `Any object name`.
* Click `Add Arn` for object actions again, enter the bucket name ending with `crawled`, and check the box for `Any object name`.
* Click `Add Arn` for bucket actions and enter the bucket name ending with `raw`.
* Click `Add Arn` for bucket actions again and enter the bucket name ending with `crawled`.
* Click Next, and then name the policy `ddex-[dev|staging]-<label/distributor>-policy`.
4. Create an IAM User [here](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-west-2#/users/create) (or search IAM and click Users > Create User).
* Name the user `ddex-[dev|staging]-<label/distributor>-user` and press Next.
* Select "Attach policies directly," and search for the policy you created (`ddex-[dev|staging]-<label/distributor>-policy`). Check the box next to it and press Next and then Create User.
5. Search for your new user and press "Create access key" and then "Third-party service." Copy the access key and secret access key into your `.env` file (assuming you've already done `cp .env.[dev|stage] .env`).
6. Go back to the bucket ending with `raw`, and add CORS at the bottom of the Permissions tab. Here's an example for dev, but for a prod environment you'll wnat to replace "*" in "AllowedOrigins" with the DNS that the frontend will be served from:
```json
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
```

### AWS environment variables:
Set up your buckets by following the "Creating a bucket in S3" section below. Then, set these environment variables:
- `AWS_ACCESS_KEY_ID`: the access key for the IAM user you created
Expand Down Expand Up @@ -38,50 +72,32 @@ Fill in all missing values. See the `Creating a bucket in S3` section below for
For docker compose to work: `cat packages/ddex/.env >> dev-tools/compose/.env`

### One-time setup
`audius-compose connect` to update your `/etc/hosts`
1. `audius-compose connect` to update your `/etc/hosts`
2. Install the AWS cli and configure it for local dev:
```sh
pip install awscli && \
aws configure set aws_access_key_id test && \
aws configure set aws_secret_access_key test && \
aws configure set region us-west-2
```
3. To use the DDEX webapp as an admin, add your decoded staging user ID to `extra-env.DDEX_ADMIN_ALLOWLIST` in `../../dev-tools/config.json`
- Find your user ID by going to `https://discoveryprovider.staging.audius.co/v1/full/users/handle/<your staging handle>`, searching for `id`, and then decoding it by pasting it into the "Encoded" textbox [here](https://healthz.audius.co/#/utils/id) and copying the "Integer" value
- Note that this requires a restart if the app is already running (`audius-compose down && audius-compose up -ddex-only`)


### Bring up the ddex stack subsequently
`audius-compose up -ddex-only`
### Bring up the ddex stack locally
Run `audius-compose up -ddex-only` and navigate to `http://localhost:9000` to view the DDEX webapp

To access the ddex db via the mongo shell: `docker exec -it ddex-mongo mongosh -u mongo -p mongo --authenticationDatabase admin` then `use ddex`
To upload a delivery to be processed:
1. Create buckets: `aws --endpoint=http://localhost:4566 s3 mb s3://audius-test-raw && aws --endpoint=http://localhost:4566 s3 mb s3://audius-test-crawled`
2. Upload your file: `aws --endpoint=http://localhost:4566 s3 cp <file from your computer> s3://audius-test-raw`. Example: `aws --endpoint=http://localhost:4566 s3 cp ./ingester/e2e_test/fixtures/release_by_release/ern381/sony1.zip s3://audius-test-raw`
3. Wait 3 minutes and it'll be processed and display in the UI (localhost:9000)
To access the ddex db via the mongo shell: `docker exec -it ddex-mongo mongosh -u mongo -p mongo --authenticationDatabase admin`, and then `use ddex`.
### Develop with hot reloading
Each service can be run independently as long as `ddex-mongo` is up (from `audius-compose up --ddex-only` and then optionally stopping individual services). See the respective subdirectories' READMEs.

### Creating a bucket in S3
1. Create a new bucket in the S3 console with the name `ddex-[dev|staging]-<label/distributor>-raw`. Use all the defaults, including "ACLs disabled"
2. Do the same for a bucket named `ddex-[dev|staging]-<label/distributor>-crawled`. Use all the defaults, including "ACLs disabled"
3. Create an IAM Policy (here](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-west-2#/policies/create) (or search IAM and click Policies > Create Policy). Select S3.
* Under `Read` choose `GetObject` and `GetObjectAttributes`.
* Under `Write` choose `DeleteObject` and `PutObject`.
* Under `List` choose `ListBucket`.
* Click `Add Arn` for object actions, enter the bucket name ending with `raw`, and check the box for `Any object name`.
* Click `Add Arn` for object actions again, enter the bucket name ending with `crawled`, and check the box for `Any object name`.
* Click `Add Arn` for bucket actions and enter the bucket name ending with `raw`.
* Click `Add Arn` for bucket actions again and enter the bucket name ending with `crawled`.
* Click Next, and then name the policy `ddex-[dev|staging]-<label/distributor>-policy`.
4. Create an IAM User [here](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-west-2#/users/create) (or search IAM and click Users > Create User).
* Name the user `ddex-[dev|staging]-<label/distributor>-user` and press Next.
* Select "Attach policies directly," and search for the policy you created (`ddex-[dev|staging]-<label/distributor>-policy`). Check the box next to it and press Next and then Create User.
5. Search for your new user and press "Create access key" and then "Third-party service." Copy the access key and secret access key into your `.env` file (assuming you've already done `cp .env.[dev|stage] .env`).
6. Go back to the bucket ending with `raw`, and add CORS at the bottom of the Permissions tab. Here's an example for dev, but for a prod environment you'll wnat to replace "*" in "AllowedOrigins" with the DNS that the frontend will be served from:
```json
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
```

### Running / debugging the e2e test
* Run `audius-compose test down && audius-compose test run ddex-e2e-release-by-release` to start the ddex stack and run the e2e test for the Release-By-Release choreography. You can replace `ddex-e2e-release-by-release` with `ddex-e2e-batched` to run the e2e test for the Batched choreography.
* To debug S3:
Expand All @@ -96,7 +112,7 @@ Each service can be run independently as long as `ddex-mongo` is up (from `audiu
3. The Parser app watches for new releases and processes each one into a format that the Publisher app can use to upload to Audius.
4. When the release date is reached for a release, the Publisher app uploads the release to Audius.

### Glossary
## Glossary
- **DDEX**: Digital Data Exchange, a standard for music metadata exchange
- **ERN**: Electronic Release Notification, a DDEX message that contains metadata about a release
- **Choreography**: A DDEX term for the way that ERNs are sent and received. There are two main choreographies: Release-By-Release and Batched
Expand Down

0 comments on commit 02b67d9

Please sign in to comment.