Skip to content

Latest commit

 

History

History
212 lines (160 loc) · 8.05 KB

DEVELOPMENT.md

File metadata and controls

212 lines (160 loc) · 8.05 KB
Raw

Development

Prerequisites

Developing Hive locally requires you to have the following software installed locally:

  • Node.js 21 (or nvm or fnm)
  • pnpm v9
  • Docker version 26.1.1 or later(previous versions will not work correctly on arm64)
  • make sure these ports are free: 5432, 6379, 9000, 9001, 8123, 9092, 8081, 8082, 9644, 3567, 7043

Setup Instructions

  • Clone the repository locally
  • Make sure to install the recommended VSCode extensions (defined in .vscode/extensions.json)
  • In the root of the repo, run nvm use to use the same version of node as mentioned
  • Create .env file in the root, and use the following:
ENVIRONMENT=local
  • Run pnpm i at the root to install all the dependencies and run the hooks
  • Run pnpm local:setup to run Docker compose dependencies, create databases and migrate database

Solving permission problems on this step:

export UID=$(id -u)
export GID=$(id -g)

Add "user" field to docker-compose.dev.yml

  clickhouse:
    user: '${UID}:${GID}'
  db:
    user: '${UID}:${GID}'
  • Run pnpm generate to generate the typings from the graphql files (use pnpm graphql:generate if you only need to run GraphQL Codegen)
  • Run pnpm build to build all services
  • Click on Start Hive in the bottom bar of VSCode
  • Open the UI (http://localhost:3000 by default) and Sign in with any of the identity provider
  • Once this is done, you should be able to log in and use the project
  • Once you generate the token against your organization/personal account in hive, the same can be added locally to hive.json within packages/libraries/cli which can be used to interact via the hive cli with the registry (Use http://localhost:3001/graphql as the registry.endpoint value in hive.json)
  • Now you can use Hive locally. All other steps in this document are optional and only necessary if you work on specific features.

Development Seed

We have a script to feed your local instance of Hive with initial seed data. This step is optional.

  1. Use Start Hive to run your local Hive instance
  2. Make sure usage and usage-ingestor are running as well (with pnpm dev)
  3. Open Hive app, create a project and a target, then create a token
  4. Run the seed script: TOKEN="MY_TOKEN_HERE" pnpm seed
  5. This should report a dummy schema and some dummy usage data to your local instance of Hive, allowing you to test features e2e

Note: You can set STAGING=1 in order to target staging env and seed a target there. Same for development env, you can use DEV=1

Note: You can set FEDERATION=1 in order to publish multiple subgraphs.

To send more operations and test heavy load on Hive instance, you can also set OPERATIONS (amount of operations in each interval round, default is 1) and INTERVAL (frequency of sending operations, default: 1000ms). For example, using INTERVAL=1000 OPERATIONS=1000 will send 1000 requests per second.

Troubleshooting

We recommend the following flow if you are having issues with running Hive locally:

  1. Stop all Docker containers: docker kill $(docker ps -q)
  2. Clear all local Docker environment: docker system prune --all --force --volumes
  3. Delete all generated local .env files: find . -name '.env' | xargs rm
  4. Delete local docker/.hive and docker/.hive-dev dir used by Docker volumes.
  5. Reinstall dependencies using pnpm install
  6. Force-generate new .env files: pnpm env:sync --force

Publish your first schema (manually)

  1. Start Hive locally

  2. Create a project and a target

  3. Create a token from that target

  4. Go to packages/libraries/cli and run pnpm build

  5. Inside packages/libraries/cli, run:

    pnpm start schema:publish --registry.accessToken "YOUR_TOKEN_HERE" --registry.endpoint "http://localhost:3001/graphql" examples/single.graphql

    The registry endpoint is the GraphQL endpoint of your local server service. You can also edit the hive.json file in the cli package to avoid passing the accessToken and endpoint every time.

Setting up Slack App for developing

  1. Download Loophole CLI (same as ngrok but supports non-random urls)
  2. Log in to Loophole $ loophole account login
  3. Start the proxy by running $ loophole http 3000 --hostname hive-<your-name> (@kamilkisiela I use hive-kamil). It creates https://hive-<your-name>.loophole.site endpoint.
  4. Message @kamilkisiela and send him the url (He will update the list of accepted redirect urls in Slack App).
  5. Update APP_BASE_URL in packages/web/app/.env to the proxy URL (e.g. https://hive-<your-name>.loophole.site)
  6. Run packages/web/app and open https://hive-<your-name>.loophole.site.

We have a special Slack channel called #hive-tests to not spam people :)

Setting up GitHub App for developing

Starting a proxy for GitHub App

  1. Follow first two steps from Setting up Slack App for developing (download loophole and log in).
  2. Start web app proxy: $ loophole http 3000 --hostname hive-<your-name>
  3. Start server proxy: $ loophole http 3001 --hostname hive-<your-name>

Creating a GitHub App

  1. Go to Settings -> Developer settings -> GitHub Apps, and click on the New GitHub App button.
  2. Provide a name for your app, and set the Homepage URL to https://hive-<your-name>.loophole.site. Then set the callback URL to https://hive-<your-name>.loophole.site/api/github/callback, and post installation's callback URL to https://hive-<your-name>.loophole.site/api/github/setup-callback.
  3. Click on Create GitHub App.

Setting up env variables

  1. Server: Set the following env variables in packages/services/server/.env:

    INTEGRATION_GITHUB=1
INTEGRATION_GITHUB_GITHUB_APP_ID=<your-github-app-id>

    You'll find the GitHub App ID and private key in the General tab of your GitHub App.

    Store the Github private key next to the .env file with the name github-app.pem (packages/services/server/github-app.pem)

  2. Web App: Set the following in packages/web/app/.env:

    INTEGRATION_GITHUB_APP_NAME=<your-github-app-name>

Installing the GitHub App

Open Hive UI and go to your organization's settings page. Find Integrations section and click on Connect GitHub. You should be redirected to GitHub where you can grant repository access. After installing the app, you should be redirected back to Hive.

Testing

  1. Create a project and a target.
  2. Create a token from that target.
  3. Setup a GitHub repo with CI/CD actions like this one: https://github.com/n1ru4l/hive-federation-subgraph/.
  4. Add the token to the repo's secrets as HIVE_TOKEN.
  5. Add Hive endpoint to the repo's secrets as HIVE_ENDPOINT (https://hive-<your-name>.loophole.site/graphql).
  6. Make sure your GitHub app is installed on that repo.
  7. Push a commit to the repo and check if the CI/CD action is triggered.

Local OIDC Testing

The docker-compose.dev.yml files includes a mock OIDC server that can be used for testing the OIDC login/logout flow locally. The server tuns on port 7043.

Please make sure to set the AUTH_ORGANIZATION_OIDC environment variables for the server and app to "1".

You can use the following values for connecting an integration to an OIDC provider.

# Token Endpoint
http://localhost:7043/connect/token
# User Info Endpoint
http://localhost:7043/connect/userinfo
# Authorization Endpoint
http://localhost:7043/connect/authorize
# Client ID
implicit-mock-client
# Client Secret
client-credentials-mock-client-secret

For login use the following credentials.

# Username
test-user
# Password
password

Run Hive

  1. Click on Start Hive in the bottom bar of VSCode
  2. Open the UI (http://localhost:3000 by default) and register any email and password
  3. Sending e-mails is mocked out during local development, so in order to verify the account find the verification link by visiting the email server's /_history endpoint - http://localhost:6260/_history by default.
    • Searching for token should help you find the link.