[Bug]: [JSON] Query failed when specifying JSON field as output_fields reporting "Unsupported scalar type"

[binbinlv]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20230515-30415e1b
- Deployment mode(standalone or cluster): both
- MQ type(rocksmq, pulsar or kafka):   all 
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.4.0.dev27
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Query failed specifying JSON field as output_fields reporting "Unsupported scalar type"

>>> res= collection.query(expr="json_field[\"number\"] >= 0", output_fields=["json_field"])
RPC error: [query], <MilvusException: (code=1, message=Unsupported scalar type)>, <Time:{'RPC start': '2023-05-16 14:45:43.623094', 'RPC error': '2023-05-16 14:46:56.839104'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 750, in query
    res = conn.query(self._name, expr, output_fields, partition_names,
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 959, in query
    num_entities = len_of(next(it))
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/utils.py", line 134, in len_of
    raise MilvusException(message="Unsupported scalar type")
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Unsupported scalar type)>

Expected Behavior

Query successfully specifying JSON field as output_fields

Steps To Reproduce

from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility

connections.connect()
dim = 128
int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
float_field = FieldSchema(name="float", dtype=DataType.FLOAT)
bool_field = FieldSchema(name="bool", dtype=DataType.BOOL)
string_field = FieldSchema(name="string", dtype=DataType.VARCHAR, max_length=65535)
json_field = FieldSchema(name="json_field", dtype=DataType.JSON)
float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
schema = CollectionSchema(fields=[int64_field, float_field, bool_field, json_field, float_vector])
collection = Collection("test_search_collection_binbin_tmp", schema=schema)
import random
nb=3000
vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
import numpy as np
res = collection.insert([[i for i in range(nb)], [np.float32(i) for i in range(nb)], [np.bool_(i) for i in range(nb)], [{"number": i, "string": "milvus", "bool": True, "list": [j for j in range(0, i)]} for i in range(0, nb)], vectors])
index_param = {"index_type": "IVF_SQ8", "metric_type": "L2", "params": {"nlist": 8192}}
collection.flush()
collection.create_index("float_vector", index_param, index_name="index_name_1")
collection.load()
res= collection.query(expr="json_field[\"number\"] >= 0", output_fields=["json_field"])

Milvus Log

No response

Anything else?

No response

[yah01]

/assign

[yah01]

/assign @binbinlv
fixed with latest pymilvus 2.4.0.dev30

[binbinlv]

Verified and fixed:
pymilvus: 2.4.0.dev30
milvus: master-20230516-ccd68501

>>> res= collection.query(expr="json_field[\"number\"] <= 0", output_fields=["json_field"])
>>> res[0]
{'int64': 0, 'json_field': [b'{"number":0,"string":"milvus","bool":true,"list":[]}']}

[binbinlv]

The behavior on 2.2.* branch is:
query with output_fields specifying json_field, it reports unexpected error.

Reopen this issue for tracking 2.2.* code fix.

>>> res= collection.query(expr="int64 >= 0", output_fields=["json_field"])
RPC error: [query], <MilvusException: (code=1, message=fail to query on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Query, QueryNode ID = 15, reason=stream operation failed: [UnexpectedError] std::bad_alloc
attempt #2:context canceled
)>, <Time:{'RPC start': '2023-05-18 15:28:05.539853', 'RPC error': '2023-05-18 15:28:05.670499'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 704, in query
    res = conn.query(self._name, expr, output_fields, partition_names,
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 950, in query
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to query on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Query, QueryNode ID = 15, reason=stream operation failed: [UnexpectedError] std::bad_alloc
attempt #2:context canceled

[binbinlv]

Verified and fixed in 2.2.* branch too:
milvus: 2.2.0-20230520-1eb804fc
pymilvus: 2.2.9.dev12

>>> res= collection.query(expr="json_field[\"number\"] >= 0", output_fields=["json_field"])
>>>
>>> res[0]
{'json_field': b'{"number":0,"string":"milvus","bool":true,"list":[]}', 'int64': 0}