All Logs Data

A complete logs tracking service which works independendly of other services in the system. Supports features such as:

• Log dump for specific days.
• Realtime grepping of statements

The overall flow of the system is as follows.

Real-Time analysis

This would enable specific log statements to be grepped and counted in real time (just like axiom). Before triggering a bot, the real-time analysis needs to be enabled or switched on, which would start monitoring incoming logs and start creating a dump of analyzed data in a file. This data would include the following information:

• Overview data
• Grepped statement count of all services

The configuration of greppable statements would be in the following format:

services = [
    {
        'service_name': 'UCI API',
        'service_container': 'uci-api-service',
        'match_statements': {
            'BotServiceError': 'ERROR [BotService]',
            'DeviceAdded': 'Added device to registry',
            'ResolveUser': 'ServiceService::resolve: Resolving users'
        }
    },
    {
        'service_name': 'Broadcast Transformer',
        'service_container': 'docker-deploy-prod_broadcast-transformer_1',
        'match_statements': {
            'Kafka Push to outbound': 'Kafka::Send: Pushed to topic process-outbound',
            'Broadcast Processed': 'Notification processed by broadcast-transformer:',
        }
    },
]

Log Format Spec

This is the format that would be followed while creation of logs. This spec allows UCI-API to read data without communicating directly with the LogBase container.

Endpoints

These endpoints work in correspondence with uci-apis service. It is not a dependency for LogBase but rather a REST API that retrieves the data generated by LogBase. Any other REST API implementation may be written as well on top of LogBase.

• Fetch Service log GET /admin/monitoring/logs/[SERVICE_NAME]?lines=[NUMBER OF LINES]&date=dd_mm_yyyy
• Trigger Real-Time data analysis POST /admin/monitoring/realTime/start
• Fetch Real-Time data GET /admin/monitoring/realtime?date=ss_mi_hh_dd_mm_yyyy
• Stop Real-Time data analysis POST /admin/monitoring/realtime/stop
• Get available realtime data files GET /admin/monitoring/realtime/available
• Fetch Overview data GET /admin/monitoring/overview?date=ss_mi_hh_dd_mm_yyyy
• Get download link for log GET /admin/monitoring/logs/[SERVICE_NAME]/download/log?date=dd_mm_yyyy
• Get download link for error logs GET /admin/monitoring/logs/[SERVICE_NAME]/download/error?date=dd_mm_yyyy