Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDK]: Client support partition key #24402

Open
1 task done
xige-16 opened this issue May 25, 2023 · 1 comment
Open
1 task done

[SDK]: Client support partition key #24402

xige-16 opened this issue May 25, 2023 · 1 comment
Labels
area/api kind/enhancement Issues or changes related to enhancement kind/feature Issues related to feature request from users

Comments

@xige-16
Copy link
Contributor

xige-16 commented May 25, 2023

Is there an existing issue for this?

  • I have searched the existing issues

Is your feature request related to a problem? Please describe.

I would like to introduce another level of abstract, partition key.
So when user create a collection

from pymilvus import CollectionSchema, FieldSchema, DataType
book_id = FieldSchema(
name="book_id",
dtype=DataType.INT64,
is_primary=True,
)
book_name = FieldSchema(
name="book_name",
dtype=DataType.VARCHAR,
max_length=200,
is_partition_key=true,
)
book_intro = FieldSchema(
name="book_intro",
dtype=DataType.FLOAT_VECTOR,
dim=2
)
default_schema = CollectionSchema(
fields=[book_id, book_name, book_group, book_intro],
description="Test book search",
)

collection = Collection(name="hello_milvus", 
schema=default_schema, 
shards_num=1, 
num_partitions=1024)

So you will see book group is the parititon key, and the collection will create 1024 partitions by default.

When insert happens:

hash(book_name)%1024 to decide which segment should be written into
others are exactly same as current insert path.

When search:

User must specify a group of partition key when search (Performance may degrade if not specified)
Milvus first pick the correct segment to work with
Then do scalar filtering on the segment based on partition keys and other filter conditions.

Describe the solution you'd like.

@XuanYang-cn @longjiquan @congqixia @yhmo @yelusion2 @shanghaikid @yah01 @nameczz

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

@yhmo
Copy link
Contributor

yhmo commented May 25, 2023

milvus-io/milvus-sdk-java#492
Already done./

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api kind/enhancement Issues or changes related to enhancement kind/feature Issues related to feature request from users
Projects
None yet
Development

No branches or pull requests

3 participants