Take care the potential performance regression caused by virtual method for filters

[liliu-z]

          Saying that it is also a concern.

1. IVF can have super high filtering rate but still uses virtual method.
2. To my experience the, HNSW validation is not negligible. We noticed 10% performance regression in cardinal if we try to do one more deference for that.

Originally posted by @liliu-z in #746 (comment)

[liliu-z]

/assign
/assign @alexanderguzhva

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.