milvus java sdk 使用 QueryParam 普通标量字段查询时返回结果集时有时无的问题

milvus 版本:

[appadmin@localhost ~]$ sudo docker  ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED       STATUS                   PORTS                                                                                      NAMES
518d96fcd2f5   zilliz/attu:v2.1.0                         "docker-entrypoint.s…"   2 weeks ago   Up 2 weeks               0.0.0.0:8000->3000/tcp, :::8000->3000/tcp                                                  my-attu
4578cdbafdf3   milvusdb/milvus:v2.1.4                     "/tini -- milvus run…"   2 weeks ago   Up 2 weeks (unhealthy)   0.0.0.0:9091->9091/tcp, :::9091->9091/tcp, 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp   milvus-standalone
c24c8adf2eae   minio/minio:RELEASE.2022-03-17T06-34-49Z   "/usr/bin/docker-ent…"   2 weeks ago   Up 2 weeks (healthy)     0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp       milvus-minio
5f6355043877   quay.io/coreos/etcd:v3.5.0                 "etcd -advertise-cli…"   2 weeks ago   Up 2 weeks (healthy)     2379-2380/tcp  

milvus java sdk 版本:

   <dependency>
            <groupId>io.milvus</groupId>
            <artifactId>milvus-sdk-java</artifactId>
            <version>2.1.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

有问题的 query 查询代码:

 public ImageVector2025VO queryByMilvusId(String milvusId,int retryTime,int maxRetryTime) {
        if( retryTime > maxRetryTime ){
            return null;
        }
        List<String> outFields = new ArrayList<>();
        outFields.add( "id" );
        outFields.add( "image_vector" );
        outFields.add( "mat_name" );
        outFields.add( "img_name" );
        outFields.add( "data_source_code" );
        outFields.add( "fan_model_combination_id" );
        QueryParam queryParam = QueryParam.newBuilder()
                                            .withCollectionName( this.collectionName )
                                            .withOutFields( outFields )
                                            .withExpr("id == \"" + milvusId + "\"")
                                            .build();
        R<QueryResults> response = this.milVusService.getMilvusServiceClient().query(queryParam);
        Integer status = response.getStatus();
        if( status.intValue() != 0 ){
            log.error( "查询失败,cause " + response.getException().getMessage() );
            return null;
        }
        QueryResults queryResults = response.getData();
        try {
            String id                 = queryResults.getFieldsData(0).getScalars().getStringData().getData( 0 );
            List<Float> imageVector = queryResults.getFieldsData(1).getVectors().getFloatVector().getDataList();
            String matName          = queryResults.getFieldsData(2).getScalars().getStringData().getData( 0 );
            String imgName          = queryResults.getFieldsData(3).getScalars().getStringData().getData( 0 );
            String data_source_code          = queryResults.getFieldsData(4).getScalars().getStringData().getData( 0 );
            Long fan_model_combination_id          = queryResults.getFieldsData(5).getScalars().getLongData().getData( 0 );
            ImageVector2025VO imageVectorVO = new ImageVector2025VO();
            imageVectorVO.setId( id );
            imageVectorVO.setImageVector( imageVector );
            imageVectorVO.setImgName( imgName );
            imageVectorVO.setMatName( matName );
            imageVectorVO.setDataSourceCode( data_source_code );
            imageVectorVO.setFanModelCombinationId( fan_model_combination_id );
            imageVectorVO.setRetryTime( retryTime );
            log.info( "查询成功!" );
            return imageVectorVO;
        }catch ( Exception e ){
            // e.printStackTrace();
            // log.error( "查询失败",e );
            return this.queryByMilvusId( milvusId,retryTime+1,maxRetryTime );
        }
    }

所以这里要加 retry 机制,经测试重试不超过10次,一定有一次成功,否则表示数据确实不存在,其实这里的原因是返回的结果集 fieldsDataList 中字段的顺序和输入参数 outFields 中字段的顺序并不是一致的,经过测试时随机的,也可能是我的 milvus 版本或者 milvus java sdk 版本太低导致的吧!

修复后的 query 代码如下所示:

 public ImageVector2025VO queryByMilvusId(String milvusId) {
        if( milvusId == null ){
            return null;
        }
        // ps:响应结果中 fieldsDataList 集合中字段的顺序并不一定和 outFields 中字段的顺序一致,每次都是随机的,所以需要根据字段名称做映射
        List<String> outFields = Arrays.asList("id", "image_vector", "img_name","mat_name","data_source_code","fan_model_combination_id");
        String expr = "id == \"" + milvusId + "\"";

        QueryParam queryParam = QueryParam.newBuilder()
                                            .withCollectionName( this.collectionName )
                                            .withOutFields(outFields)
                                            .withExpr(expr)
                                            .build();

        R<QueryResults> response = this.milVusService.getMilvusServiceClient().query(queryParam);
        if (response.getStatus() != R.Status.Success.getCode()) {
            log.error("查询失败: {}", response.getMessage());
            return null;
        }
        QueryResults results = response.getData();
        List<FieldData> fieldsDataList = results.getFieldsDataList();
        if( fieldsDataList == null || fieldsDataList.size() == 0 ){
            return null;
        }
        int dataCount = this.milVusService.calcDataCount( fieldsDataList );
        if( dataCount == 0 ){
            return null;
        }
        ImageVector2025VO vo = new ImageVector2025VO();
        for( FieldData fieldsData:fieldsDataList ){
            String fieldName = fieldsData.getFieldName();
            if( "id".equals( fieldName ) ||
                    "img_name".equals( fieldName ) ||
                    "mat_name".equals( fieldName ) ||
                    "data_source_code".equals( fieldName ) ){
                String value = fieldsData.getScalars().getStringData().getData( 0 );
                if( "id".equals( fieldName ) ){
                    vo.setId( value );
                }else if( "img_name".equals( fieldName ) ){
                    vo.setImgName( value );
                }else if( "mat_name".equals( fieldName ) ){
                    vo.setMatName( value );
                }else if( "data_source_code".equals( fieldName ) ){
                    vo.setDataSourceCode( value );
                }
            }else if( "fan_model_combination_id".equals( fieldName ) ){
                Long value = fieldsData.getScalars().getLongData().getData( 0 );
                vo.setFanModelCombinationId( value );
            }else if( "image_vector".equals( fieldName ) ){
                List<Float> floatList = fieldsData.getVectors().getFloatVector().getDataList();
                vo.setImageVector( floatList );
            }
        }
        return vo;
    }

ImageVector2025VO.java:

@Getter
@Setter
public class ImageVector2025VO {

    private String id;
    private String imgName;
    private String matName;
    private List<Float> imageVector;
    private String dataSourceCode;
    private Long fanModelCombinationId;
    private int retryTime;

    @Override
    public String toString() {
        return "{" +
                "\n\tid='" + id + '\'' +
                ", \n\timgName='" + imgName + '\'' +
                ", \n\tmatName='" + matName + '\'' +
                ", \n\timageVector=" + "[ " + imageVector.get( 0 ) + "," + imageVector.get( 1 ) + ",...," + imageVector.get( 254 ) + ","+ imageVector.get( 255 ) +" ]" +
                ", \n\tdataSourceCode='" + dataSourceCode + '\'' +
                ", \n\tfanModelCombinationId=" + fanModelCombinationId +
                "\n}";
    }
}

calcDataCount 方法:

    /**
     *  ps:是根据 id 字段去判断的,所以必须包含 id 字段
     * @param fieldsDataList
     * @return
     */
    @Override
    public int calcDataCount(List<FieldData> fieldsDataList) {
        if( fieldsDataList == null || fieldsDataList.size() == 0 ){
            return 0;
        }
        for( FieldData fieldsData:fieldsDataList ){
            String fieldName = fieldsData.getFieldName();
            if( "id".equals( fieldName ) ){
                return  fieldsData.getScalars().getStringData().getDataCount();
            }
        }
        return 0;
    }

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值