Github-Scrapper Demo

This Project include helm chart for deploying github-scrapper in k8s.
Charts in the projecs:

• github-scrapper
• MongoDB
• fluent-bit
• elasticsearch
• kibana
• aws-ebs-csi (for k8s persistent volumes in amazon)

Prerequisites

• Terraform
• Terraform RKE provider
• Terraform RKE module
• HELM-v3
• Helmfile
• Kubectl

Limitations

• Terraform RKE module is a project that still in development early stages:

  -- no terraform outputs.
  -- do not change the ami var (must run within RancherOS).

Project Guidelines

• Create new tf file like rke.tf exmaple.
• For provisioning k8s cluster, run:
  terraform init && terraform apply
  You can increase the amount of instances in place by edit instance_count tfvar map object.
• kube_config_cluster.yml will be created in the project directory after running terraform and you will need to export KUBECONFIG and point this file.
• Create secret file github-auth for github authentication like the exmaple below by running:
  kubectl -n github-scrapper create secret generic github-auth --from-literal=GITHUB_USER='my-user' --from-literal=GITHUB_PASSWORD='my_password'
• Enter users to search seprated by space in helmfile.yaml github-scrapper release env GITHUB_SEARCH_USERS
• For installing the chart within helmfile, run:
  helmfile -i -f helmfile.yml apply

Kibana

• Checking logs can be access within kubectl port-forward:
  kubectl -n kibana port-forward svc/kibana-kibana 5601:5601