Java 类名:com.alibaba.alink.operator.batch.similarity.TextApproxNearestNeighborPredictBatchOp
Python 类名:TextApproxNearestNeighborPredictBatchOp

功能介绍

文本相似度是在字符串相似度的基础上,基于词,计算两两文章或者句子之间的相似度,文章或者句子需要以空格分割的文本,计算方式和字符串相似度类似:支持SimHashHamming,MinHash和Jaccard三种近似相似度计算方式,通过选择metric参数可计算不同的相似度。
该功能由训练和预测组成,支持计算1. 求最近邻topN 2. 求radius范围内的邻居。该功能由预测时候的topN和radius参数控制, 如果填写了topN,则输出最近邻,如果填写了radius,则输出radius范围内的邻居。
SimhashHamming(SimHash_Hamming_Distance)相似度=1-距离/64.0,应选择metric的参数为SIMHASH_HAMMING_SIM。
MinHash应选择metric的参数为MINHASH_SIM。
Jaccard应选择metric的参数为JACCARD_SIM。

参数说明

| 名称 | 中文名称 | 描述 | 类型 | 是否必须? | 取值范围 | 默认值 | | —- | —- | —- | —- | —- | —- | —- |

| selectedCol | 选中的列名 | 计算列对应的列名 | String | ✓ | | |

| modelFilePath | 模型的文件路径 | 模型的文件路径 | String | | | null |

| outputCol | 输出结果列 | 输出结果列列名,可选,默认null | String | | | null |

| radius | radius值 | radius值 | Double | | | null |

| reservedCols | 算法保留列名 | 算法保留列 | String[] | | | null |

| topN | TopN的值 | TopN的值 | Integer | | [1, +inf) | null |

| numThreads | 组件多线程线程个数 | 组件多线程线程个数 | Integer | | | 1 |

代码示例

Python 代码

  1. from pyalink.alink import *
  2. import pandas as pd
  3. useLocalEnv(1)
  4. df = pd.DataFrame([
  5. [0, "a b c d e", "a a b c e"],
  6. [1, "a a c e d w", "a a b b e d"],
  7. [2, "c d e f a", "b b c e f a"],
  8. [3, "b d e f h", "d d e a c"],
  9. [4, "a c e d m", "a e e f b c"]
  10. ])
  11. inOp = BatchOperator.fromDataframe(df, schemaStr='id long, text1 string, text2 string')
  12. train = TextApproxNearestNeighborTrainBatchOp().setIdCol("id").setSelectedCol("text1").setMetric("SIMHASH_HAMMING_SIM").linkFrom(inOp)
  13. predict = TextApproxNearestNeighborPredictBatchOp().setSelectedCol("text2").setTopN(3).linkFrom(train, inOp)
  14. predict.print()

Java 代码

  1. import org.apache.flink.types.Row;
  2. import com.alibaba.alink.operator.batch.BatchOperator;
  3. import com.alibaba.alink.operator.batch.similarity.TextApproxNearestNeighborPredictBatchOp;
  4. import com.alibaba.alink.operator.batch.similarity.TextApproxNearestNeighborTrainBatchOp;
  5. import com.alibaba.alink.operator.batch.source.MemSourceBatchOp;
  6. import org.junit.Test;
  7. import java.util.Arrays;
  8. import java.util.List;
  9. public class TextApproxNearestNeighborPredictBatchOpTest {
  10. @Test
  11. public void testTextApproxNearestNeighborPredictBatchOp() throws Exception {
  12. List <Row> df = Arrays.asList(
  13. Row.of(0, "a b c d e", "a a b c e"),
  14. Row.of(1, "a a c e d w", "a a b b e d"),
  15. Row.of(2, "c d e f a", "b b c e f a"),
  16. Row.of(3, "b d e f h", "d d e a c"),
  17. Row.of(4, "a c e d m", "a e e f b c")
  18. );
  19. BatchOperator <?> inOp = new MemSourceBatchOp(df, "id int, text1 string, text2 string");
  20. BatchOperator <?> train = new TextApproxNearestNeighborTrainBatchOp().setIdCol("id").setSelectedCol("text1")
  21. .setMetric("SIMHASH_HAMMING_SIM").linkFrom(inOp);
  22. BatchOperator <?> predict = new TextApproxNearestNeighborPredictBatchOp().setSelectedCol("text2").setTopN(3)
  23. .linkFrom(train, inOp);
  24. predict.print();
  25. }
  26. }

运行结果

| id | text1 | text2 | | —- | —- | —- |

| 0 | a b c d e | {“ID”:”[0,1,2]”,”METRIC”:”[0.953125,0.921875,0.90625]”} |

| 1 | a a c e d w | {“ID”:”[0,1,4]”,”METRIC”:”[0.9375,0.90625,0.859375]”} |

| 2 | c d e f a | {“ID”:”[0,1,4]”,”METRIC”:”[0.890625,0.859375,0.8125]”} |

| 3 | b d e f h | {“ID”:”[4,2,1]”,”METRIC”:”[0.9375,0.90625,0.890625]”} |

| 4 | a c e d m | {“ID”:”[1,0,4]”,”METRIC”:”[0.921875,0.921875,0.90625]”} |