README.md

OpenSearch experiments

Experiments configuring OpenSearch to work with our common application patterns.

OpenSearch is an open search fork of the elk stack which consists of Elastic Search, Logstash and Kibana. OpenSeach Dashboards is the open source fork of Kibana.

NOTE

This other pages on this site are intended to be rendered in a docker container. Some markdown links will only work when viewed from a container. The instructions will describe how to launch that container.

References

• Learning the Elastic Stack
• Elastic Search in Depth
• OpenSearch Docs
• Logstash Documentation
  • Input Plugins
  • Filter Plugins
  • Output Plugins

General Instructions

Docker is required for this tutorial. See our docker tutorial for background.

These examples should work on Linux or Mac. Contributions are welcome to make these examples work on Windows.

These instructions presume that ports 8080-8099 are open to internet.

• Ports 8086, 8094 and 8080 are used in these examples.

Overview

  graph TD;
      Tutorial[[Tutorial Files]]
      Markdown(Markdown Renderer for Tutorial)
      OpenSearch(OpenSearch Service)
      Logstash(Logstash - upload log files to OpenSearch)
      OpenSearchDashboards(OpenSearch Dashboards)
      Init(Initialization Container)
      Tutorial-->Markdown
      Tutorial-->Logstash
      Tutorial-->Init
      Init-->OpenSearchDashboards
      Logstash-->OpenSearch
      OpenSearch<-->OpenSearchDashboards
      Markdown-.->OpenSearchDashboards
      subgraph OptionalServices
        Tomcat(Tomcat)
        Rails(Rails)
        Minio(Minio - CloudStorage)
        Logfiles[[Application Log Files]]
      end
      Tomcat-->Logfiles
      Rails-->Logfiles
      Logfiles-->Logstash
      Minio-->Logstash

Loading

If running locally...

export MYHOSTNAME=$(hostname)

If running on a server...

export MYHOSTNAME=$(hostname).$(domainname)

Start the Tutorial

docker-compose up -d --build
echo "Open http://${MYHOSTNAME}:8086/" in your browser to view these instructions.

Run each example of interest

• Start the docker stack
• Explore the configuration files
• Explore the open search dashboard to see the results
  • For each example in this tutorial, a sample search was constructed from the opensearch dashboard screen and then shared as a permalink
• Stop the docker stack
• Continue to the next example

Demos

• Explore Kibana / OpenSearch Dashboards
• Sample Files - No Json Filter
• Sample Files - Json Filter
• Sample Files - Date Normalization
• ALB Logs
• ALB Logs from S3
• Catalina Logs from running tomcat
• Access Logs from running tomcat
• Json Application Logs from running tomcat
• Handle 3 types of tomcat logs
• Rails logs
• Sample Files - Update Json via document id

UC3 specific Extensions

These examples are used internally by our development team. The sample data is not available for this tutorial. These examples load "saved objects" (searches, visualizations, dashboards) into opensearch dashboards.

• docker-compose -f docker-compose.yml -f logstash-viz.yml up -d
  • https://github.com/CDLUC3/mrt-cron/blob/main/viz/README.md
• docker-compose -f docker-compose.yml -f logstash-mimefile.yml up -d
  • https://github.com/CDLUC3/mrt-cron/blob/main/coll-health/README.md
• docker-compose -f docker-compose.yml -f objhealth.yml up -d
  • https://github.com/CDLUC3/mrt-cron/blob/main/coll-health-obj-analysis/README.md

Other useful notes

• Create OpenSearch Permalinks for Tutorial Pages