Deleting points for release RAM usage

[xtyDoge]

I user qdrant v1.8.3, and I encountered a problem about points deleting.
I have a huge collection, every day it has incremental record inserted. So I decided to evict some out-dated record and only maintain recent 15 days. I will drop records out of ttl at begining of the day. I use scalar quantizitaion in RAM, so I expected RAM release when deleting points. Delete records is approximately 1/15 = 6.7% of whole collection.

I use /collections/{collectionName}/points/delete API for points deleting, but the RAM usage does'nt decrease as I expected.

I try to do some troubleshooting. First I find the config of collection optimizers_config.deleted_threshold and optimizers_config.vacuum_min_vector_number, the default value is 0.2 and 1000, and I set it to 0.05 and 500. But nothing happened.

Then I get the source code at tag v1.8.3, lib/segment/src/segment.rs:1158, delete_point method. It says

// Before, we propagated point deletions to also delete its vectors. This turns
// out to be problematic because this sometimes makes us loose vector data
// because we cannot control the order of segment flushes.
// Disabled until we properly fix it or find a better way to clean up old
// vectors.
//
// // Propagate point deletion to all its vectors
// for vector_data in segment.vector_data.values() {
//     let mut vector_storage = vector_data.vector_storage.borrow_mut();
//     vector_storage.delete_vector(internal_id)?;
// }

And telemetry logs of one segment is

        "info": {
            "segment_type": "indexed",
            "num_vectors": 62077,
            "num_points": 56716,
            "num_indexed_vectors": 62077,
            "num_deleted_vectors": 5361,
            "ram_usage_bytes": 0,
            "disk_usage_bytes": 0,
            "is_appendable": false,
            "index_schema": {
                "time_id": {
                    "data_type": "integer",
                    "points": 56716
                },
                "from_user_id": {
                    "data_type": "keyword",
                    "points": 56716
                },
                "to_user_id": {
                    "data_type": "keyword",
                    "points": 56716
                }
            },
            "vector_data": {
                "": {
                    "num_vectors": 62077,
                    "num_indexed_vectors": 62077,
                    "num_deleted_vectors": 0
                }
            }
        }

So It means I should use collections/{collectionName}/points/delete API and vector : [""] params to delete vectors separately?
Or upgrade qrdant version will resolve this problem?

Thx for answers!

That's the config of the collection, after I modified the params of my collection.

{
  "result": {
    "status": "green",
    "optimizer_status": "ok",
    "vectors_count": 4899961,
    "indexed_vectors_count": 4890303,
    "points_count": 4783500,
    "segments_count": 72,
    "config": {
      "params": {
        "vectors": {
          "size": 768,
          "distance": "Dot",
          "on_disk": true
        },
        "shard_number": 9,
        "replication_factor": 2,
        "write_consistency_factor": 1,
        "on_disk_payload": true
      },
      "hnsw_config": {
        "m": 16,
        "ef_construct": 128,
        "full_scan_threshold": 10,
        "max_indexing_threads": 3,
        "on_disk": false
      },
      "optimizer_config": {
        "deleted_threshold": 0.05,
        "vacuum_min_vector_number": 500,
        "default_segment_number": 0,
        "max_segment_size": null,
        "memmap_threshold": 20000,
        "indexing_threshold": 10000,
        "flush_interval_sec": 10,
        "max_optimization_threads": 2
      },
      "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
      },
      "quantization_config": {
        "scalar": {
          "type": "int8",
          "quantile": 0.99,
          "always_ram": true
        }
      }
    },
    "payload_schema": {
      "from_user_id": {
        "data_type": "keyword",
        "points": 4783500
      },
      "time_id": {
        "data_type": "integer",
        "points": 4783500
      },
      "to_user_id": {
        "data_type": "keyword",
        "points": 4783500
      }
    }
  },
  "status": "ok",
  "time": 0.000991873
}

[timvisee]

I use /collections/{collectionName}/points/delete API for points deleting, but the RAM usage does'nt decrease as I expected.

Could you elaborate on what the change is, and what you expect?

About the above code comment - you don't have to worry about this. Not propagating the deletes to vectors does not mean they are not deleted, and they are still picked up by the vacuum optimizer.

Do you happen to know if the vacuum optimizer never runs at all? Or does it just run less often than you expect?

So It means I should use collections/{collectionName}/points/delete API and vector : [""] params to delete vectors separately?

No. Deleting the point itself is the right approach.

Or upgrade qrdant version will resolve this problem?

I don't think so. At least, I don't remember us having solved problems around this. Though, it's always worth to stay up to date for all the other improvements we make.

[xtyDoge]

I delete 30k records in this collection, 6％ of all. The expection is, Ram usage of quantization, vector index and payload index will be release soon. But it does not.
And how can I monitor the RAM usage of each collection？(or shard, segments) Does Telemery results have these metrics？

[xtyDoge]

And I see changelog in v1.12.0

https://github.com/qdrant/qdrant/pull/5188 - Use mmap vector storage by default. This change have several important implications:
We avoid using expensive rocksdb to make upload significantly faster, especially for large datasets.
Vectors are stored in memory-mapped files, which means that the OS will manage the memory for us. Qdrant process will consume less RSS Anon memory, but will use disk cache instead. That means that in case of high memory usage qdrant will slow down instead of crashing with OOM.

Is that useful for my conditions?

[timvisee]

No, I do not expect that to have any significant effect on this.

[xtyDoge]

I want to catch some logs at instance OOM, but I does'nt find any in stdout. Is there any way to do this?

[timvisee]

Logs are outputted to stdout by default. But there aren't any special logs at the time of OOM. At OOM, the kernel just kills our process without asking so there isn't much we can do.

[xtyDoge]

Maybe there should be some monitoring on RAM usage of vectors, hnsw index, quantizations and payloads, for every collection. I think it's helpful for oom problems troubleshooting. Can these features in use now？or in timetable

[xtyDoge]

So how can I drop the out-date points corrently? create shard by days and delete the shard?

[generall]

either this, or delete by payload filter

[xtyDoge]

Delete by payload cannot release all RAM usage of deleted points, how does it by deleting by shard?

[generall]

if eventually release memory after reindexing

[xtyDoge]

How to trigger reindexing? By insert or search point, or other api callings? Can I trigger it manually