Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logstash persistent queue improvement #2744

Merged
merged 2 commits into from
Feb 15, 2021
Merged

Logstash persistent queue improvement #2744

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d9d995
Logstash persistent queue improvement
PMazarovich Jan 28, 2021
86b1a4e
Merge branch 'develop' into logstash_pq
Feb 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Logstash is improved for using with configurable elasticsearch outputs (<https://github.com/openvinotoolkit/cvat/pull/2531>)
- Bumped nuclio version to 1.5.16
- All methods for interative segmentation accept negative points as well
- Persistent queue added to logstash (<https://github.com/openvinotoolkit/cvat/pull/2744>)

### Deprecated

Expand Down
2 changes: 1 addition & 1 deletion components/analytics/docker-compose.analytics.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ services:
cvat:
environment:
DJANGO_LOG_SERVER_HOST: logstash
DJANGO_LOG_SERVER_PORT: 5000
DJANGO_LOG_SERVER_PORT: 8080
DJANGO_LOG_VIEWER_HOST: kibana
DJANGO_LOG_VIEWER_PORT: 5601
CVAT_ANALYTICS: 1
Expand Down
5 changes: 3 additions & 2 deletions components/analytics/logstash/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,6 @@ FROM docker.elastic.co/logstash/logstash-oss:${ELK_VERSION}
RUN logstash-plugin install logstash-input-http logstash-filter-aggregate \
logstash-filter-prune logstash-output-email

COPY logstash.conf /usr/share/logstash/pipeline/
EXPOSE 5000
COPY logstash.yml /usr/share/logstash/config/
COPY logstash.conf /usr/share/logstash/pipeline/
EXPOSE 8080
21 changes: 19 additions & 2 deletions components/analytics/logstash/logstash.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,22 @@
input {
tcp {
port => 5000
http {
port => 8080
codec => json
}
}

filter {
mutate {
add_field => {"logger_name" => ""}
add_field => {"path" =>""}
}
mutate {
copy => {"[extra][logger_name]" => "logger_name" }
copy => {"[extra][path]"=>"path"}
}
prune {
blacklist_names => ["type","logsource","extra","program","pid","headers"]
}
if [logger_name] =~ /cvat.client/ {
# 1. Decode the event from json in 'message' field
# 2. Remove unnecessary field from it
Expand All @@ -14,6 +25,9 @@ filter {
mutate {
rename => { "message" => "source_message" }
}
mutate {
add_field => {"[@metadata][target_index_client]" => "cvat.client.%{+YYYY}.%{+MM}"}
}

json {
source => "source_message"
Expand Down Expand Up @@ -77,6 +91,9 @@ filter {
# 2. Remove unnecessary field from it
# 3. Type it as server
if [logger_name] =~ /cvat\.server\.task_[0-9]+/ {
mutate {
add_field => {"[@metadata][target_index_server]" => "cvat.server.%{+YYYY}.%{+MM}"}
}
mutate {
rename => { "logger_name" => "task_id" }
gsub => [ "task_id", "cvat.server.task_", "" ]
Expand Down
3 changes: 3 additions & 0 deletions components/analytics/logstash/logstash.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
queue.type: persisted
queue.max_bytes: 1gb
queue.checkpoint.writes: 20
2 changes: 1 addition & 1 deletion cvat/requirements/base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ rq-scheduler==0.10.0
sqlparse==0.3.1
django-sendfile==0.3.11
dj-pagination==2.5.0
python-logstash==0.4.6
python-logstash-async==2.2.0
django-revproxy==0.10.0
rules==2.2
GitPython==3.1.8
Expand Down
17 changes: 15 additions & 2 deletions cvat/settings/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,10 @@ def add_ssh_keys():
os.makedirs(STATIC_ROOT, exist_ok=True)

DATA_ROOT = os.path.join(BASE_DIR, 'data')
LOGSTASH_DB = os.path.join(DATA_ROOT,'logstash.db')
os.makedirs(DATA_ROOT, exist_ok=True)
if not os.path.exists(LOGSTASH_DB):
os.mknod(LOGSTASH_DB)

MEDIA_DATA_ROOT = os.path.join(DATA_ROOT, 'data')
os.makedirs(MEDIA_DATA_ROOT, exist_ok=True)
Expand Down Expand Up @@ -366,6 +369,11 @@ def add_ssh_keys():
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'logstash': {
'()': 'logstash_async.formatter.DjangoLogstashFormatter',
'message_type': 'python-logstash',
'fqdn': False, # Fully qualified domain name. Default value: false.
},
'standard': {
'format': '[%(asctime)s] %(levelname)s %(name)s: %(message)s'
}
Expand All @@ -386,11 +394,16 @@ def add_ssh_keys():
},
'logstash': {
'level': 'INFO',
'class': 'logstash.TCPLogstashHandler',
'class': 'logstash_async.handler.AsynchronousLogstashHandler',
'formatter': 'logstash',
'transport': 'logstash_async.transport.HttpTransport',
'ssl_enable': False,
'ssl_verify': False,
'host': os.getenv('DJANGO_LOG_SERVER_HOST', 'localhost'),
'port': os.getenv('DJANGO_LOG_SERVER_PORT', 5000),
'port': os.getenv('DJANGO_LOG_SERVER_PORT', 8080),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason to change the port from 5000 to 8080?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For http protocol, port 8080 is standard one, and logstash now works via http because of persistent queue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azhavoro , we use 8080 for cvat_proxy. Can it be a problem?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not open port, I don't see any problem here. We use the same port number for cvat_server https://github.com/openvinotoolkit/cvat/blob/develop/Dockerfile#L157

'version': 1,
'message_type': 'django',
'database_path': LOGSTASH_DB,
}
},
'loggers': {
Expand Down