Indexing rate decays dramatically over time

[calispqr]

Hi,

When initiating the _reindex API I've noticed initially the DPS is around 2.5K docs/second. The problem I'm seeing is that over time DPS starts to degrade and eventually getting down to 600 DPS.

Cluster information:
single node cluster 3 shards no replicas,
64GB Ram (32GB allocated to ES)
index1 containing 65 Million documents. The index takes up 12GB of space so I'm getting an average doc size of .203KB Pretty small.
Disk space 1.7TB

The new index is on the same machine and you can see the output of the indexing rate over an extended period of time.

In addition I've followed most of the indexing performance enhancement tips suggest by ES. i.e. refresh_interval : -1

I can provide additional metrics if necessary. Also I was thinking that it might be an issue with the search so I turned the debug and info logging up but no search logs are coming through. Is a scroll / scan not considered a search? Doesn't make sense.

Thanks,
Michael Hunn

[marevol]

Hmm, could you change more shards(10 or the above) for the new index?