Java 类名：com.alibaba.alink.operator.stream.nlp.DocCountVectorizerPredictStreamOp
Python 类名：DocCountVectorizerPredictStreamOp

功能介绍

根据文本中词语的特征信息，将每条文本转化为稀疏向量。

使用方式

该组件是预测组件，需要配合训练组件 DocCountVectorizerTrainBatchOp 使用。

代码示例

Python 代码

  1. df = pd.DataFrame([
  2.     [0, u'二手旧书:医学电磁成像'],
  3.     [1, u'二手美国文学选读（ 下册 ）李宜燮南开大学出版社 9787310003969'],
  4.     [2, u'二手正版图解象棋入门/谢恩思主编/华龄出版社'],
  5.     [3, u'二手中国糖尿病文献索引'],
  6.     [4, u'二手郁达夫文集（ 国内版 ）全十二册馆藏书']
  7. ])
  9. inOp1 = BatchOperator.fromDataframe(df, schemaStr='id int, text string')
  10. segment = SegmentBatchOp().setSelectedCol("text").linkFrom(inOp1)
  11. train = DocCountVectorizerTrainBatchOp().setSelectedCol("text").linkFrom(segment)
  12. inOp2 = StreamOperator.fromDataframe(df, schemaStr='id int, text string')
  13. segment2 = SegmentStreamOp().setSelectedCol("text").linkFrom(inOp2)
  14. predictStream = DocCountVectorizerPredictStreamOp(train).setSelectedCol("text").linkFrom(segment2)
  15. predictStream.print()
  16. StreamOperator.execute()

Java 代码

  1. import org.apache.flink.types.Row;
  3. import com.alibaba.alink.operator.batch.BatchOperator;
  4. import com.alibaba.alink.operator.batch.nlp.DocCountVectorizerPredictBatchOp;
  5. import com.alibaba.alink.operator.batch.nlp.DocCountVectorizerTrainBatchOp;
  6. import com.alibaba.alink.operator.batch.nlp.SegmentBatchOp;
  7. import com.alibaba.alink.operator.batch.source.MemSourceBatchOp;
  8. import org.junit.Test;
  10. import java.util.Arrays;
  11. import java.util.List;
  13. public class DocCountVectorizerPredictStreamOpTest {
  14.     @Test
  15.     public void testDocCountVectorizerPredictStreamOp() throws Exception {
  16.         List <Row> df = Arrays.asList(
  17.             Row.of(0, "二手旧书:医学电磁成像"),
  18.             Row.of(1, "二手美国文学选读（ 下册 ）李宜燮南开大学出版社 9787310003969"),
  19.             Row.of(2, "二手正版图解象棋入门/谢恩思主编/华龄出版社"),
  20.             Row.of(3, "二手中国糖尿病文献索引"),
  21.             Row.of(4, "二手郁达夫文集（ 国内版 ）全十二册馆藏书")
  22.         );
  23.         BatchOperator <?> inOp1 = new MemSourceBatchOp(df, "id int, text string");
  24.         BatchOperator <?> segment = new SegmentBatchOp().setSelectedCol("text").linkFrom(inOp1);
  25.         BatchOperator <?> train = new DocCountVectorizerTrainBatchOp().setSelectedCol("text").linkFrom(segment);
  26.         StreamOperator <?> inOp2 = new MemSourceStreamOp(df, "id int, text string");
  27.         StreamOperator <?> segment2 = new SegmentStreamOp().setSelectedCol("text").linkFrom(inOp2);
  28.         StreamOperator <?> predictStream = new DocCountVectorizerPredictStreamOp(train).setSelectedCol("text").linkFrom(
  29.             segment2);
  30.         predictStream.print();
  31.         StreamOperator.execute();
  32.     }
  33. }

运行结果

| id | text | | —- | —- |

| 0 | $37$10:1.0 14:1.0 18:1.0 25:1.0 29:1.0 34:1.0 |

| 4 | $37$0:1.0 1:1.0 2:1.0 3:1.0 11:1.0 15:1.0 21:1.0 24:1.0 27:1.0 29:1.0 30:1.0 |

| 3 | $37$8:1.0 9:1.0 16:1.0 29:1.0 32:1.0 |

| 2 | $37$5:1.0 6:1.0 12:1.0 19:1.0 20:1.0 23:1.0 26:1.0 28:1.0 29:1.0 31:1.0 36:2.0 |

| 1 | $37$0:1.0 1:1.0 4:1.0 7:1.0 13:1.0 17:1.0 22:1.0 26:1.0 29:1.0 33:1.0 35:1.0 |