Filebeat lose metadata of docker container when container stops

[dengqingpei1990]

Environment:
filebeat version: 6.0.0-beta1(containerized)
Operating System: CentOS 7.3, Docker-ce 17.06

As it is described at https://discuss.elastic.co/t/filebeat-miss-metadata-of-docker-container-when-container-stops/98037, filebeat failed to resolve the container's metadata information when the container stops. Ping @exekias
Here are the steps to reproduce:

1. Create a swarm service to the swarm:

docker service create --name smallredspot \ 
  --replicas 5 \
  --network dev \
  --detach=false \
  myregistry.sample.com:5000/smallredspot:latest

The smallredspot is a simple java springboot container which will print multiline logs to stdout.
2. Create a filebeat service to collect the logs:

docker service create --name filebeat \
  --mode global \
  --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
  --mount type=bind,src=/var/lib/docker/containers/,dst=/var/lib/docker/containers/ \
  --user root \
  --config src=filebeat,target=/usr/share/filebeat/filebeat.yml
docker.elastic.co/beats/filebeat:6.0.0-beta1

And the filebeat.yml:

filebeat.prospectors:
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  multiline.pattern: '^[[:space:]]+|^Caused by:'
  multiline.negate: false
  multiline.match: after
  tail_files: true
  processors:
  - add_docker_metadata: ~
output.logstash:
  hosts: ["192.168.1.120:5044"]

Also logstash.conf

input {
  beats {
    port => 5044
    type => "docker_logs"
  }
}
output {
  if [type] == "docker_logs" {
    elasticsearch {
      hosts => "http://192.168.1.120:9200"
      index => "%{[docker][container][labels][com][docker][swarm][service][name]}-%{+YYYY.MM}"
    }
}

Label com.docker.swarm.service.name was generated automatically by swarm when Creating the service. In this case, it will be the name of the service: smallredspot. So all the 5 containers' logs will goes into the smallredspot-* index following the logic. It works as expected till now.
3. Update the smallredspot service

docker service update --force smallredspot

This instruction will first stop and kill the container, then remove it and run a new one with new metadata but the service name will not change. Problem comes : a strange index is created on elasticsearch:

And the message is

{
    "_index": "%{[docker][container][labels][com][docker][swarm][service][name]}-2017.08",
    "_type": "docker_logs",
    "_id": "AV4N5o1hw7w152Pm1x8z",
    "_version": 1,
    "_score": 1,
    "_source": {
        "@timestamp": "2017-08-23T07:03:56.459Z",
        "docker": {
            "container": {
                "id": "43c3759ce800b61f5e1b1b7ff3b4431e7dc1d2cd3c70ff62a5b1e94c07420629"
            }
        },
        "source": "/var/lib/docker/containers/43c3759ce800b61f5e1b1b7ff3b4431e7dc1d2cd3c70ff62a5b1e94c07420629/43c3759ce800b61f5e1b1b7ff3b4431e7dc1d2cd3c70ff62a5b1e94c07420629-json.log",
        "offset": 8570,
        "stream": "stdout",
        "time": "2017-08-23T07:03:49.054752452Z",
        "log": "2017-08-23 15:03:49.054 INFO 1 --- [ Thread-2] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown",
        "beat": {
            "name": "60e1969b6f44",
            "hostname": "60e1969b6f44",
            "version": "6.0.0-beta1"
        },
        "prospector": {
            "type": "log"
        },
        "@version": "1",
        "host": "60e1969b6f44",
        "type": "docker_logs",
        "tags": [
            "beats_input_raw_event"
        ]
    }
}

It seems the last log line while the container is shutting down. But the container's metadata is missing. Due to this the log goes into wrong place and will not be shown in smallredspot-* index.