Java 类名:com.alibaba.alink.operator.batch.sql.UnionBatchOp
Python 类名:UnionBatchOp

功能介绍

对批式数据进行sql的UNION操作(去重)。

参数说明

| 名称 | 中文名称 | 描述 | 类型 | 是否必须? | 取值范围 | 默认值 | | —- | —- | —- | —- | —- | —- | —- |

代码示例

Python 代码

  1. from pyalink.alink import *
  2. import pandas as pd
  3. useLocalEnv(1)
  4. df1 = pd.DataFrame([
  5. ['Ohio', 2000, 1.5],
  6. ['Ohio', 2000, 1.5],
  7. ['Ohio', 2002, 3.6],
  8. ['Nevada', 2001, 2.4],
  9. ['Nevada', 2002, 2.9],
  10. ['Nevada', 2003, 3.2]
  11. ])
  12. df2 = pd.DataFrame([
  13. ['Nevada', 2001, 2.4],
  14. ['Nevada', 2003, 3.2]
  15. ])
  16. batch_data1 = BatchOperator.fromDataframe(df1, schemaStr='f1 string, f2 bigint, f3 double')
  17. batch_data2 = BatchOperator.fromDataframe(df2, schemaStr='f1 string, f2 bigint, f3 double')
  18. UnionBatchOp().linkFrom(batch_data1, batch_data2).print()

Java 代码

  1. import com.alibaba.alink.operator.batch.BatchOperator;
  2. import com.alibaba.alink.operator.batch.source.CsvSourceBatchOp;
  3. import com.alibaba.alink.operator.batch.sql.UnionBatchOp;
  4. import org.junit.Test;
  5. public class UnionBatchOpTest {
  6. @Test
  7. public void testUnionBatchOp() throws Exception {
  8. List <Row> df1 = Arrays.asList(
  9. Row.of("Ohio", 2000, 1.5),
  10. Row.of("Ohio", 2000, 1.5),
  11. Row.of("Ohio", 2002, 3.6),
  12. Row.of("Nevada", 2001, 2.4),
  13. Row.of("Nevada", 2002, 2.9),
  14. Row.of("Nevada", 2003, 3.2)
  15. );
  16. List <Row> df2 = Arrays.asList(
  17. Row.of("Nevada", 2001, 2.4),
  18. Row.of("Nevada", 2003, 3.2)
  19. );
  20. BatchOperator <?> data1 = new MemSourceBatchOp(df1, "f1 string, f2 int, f3 double");
  21. BatchOperator <?> data2 = new MemSourceBatchOp(df2, "f1 string, f2 int, f3 double");
  22. BatchOperator <?> union = new UnionBatchOp();
  23. union.linkFrom(data1, data2).print();
  24. }
  25. }

运行结果

| f1 | f2 | f3 | | —- | —- | —- |

| Nevada | 2001 | 2.4000 |

| Nevada | 2002 | 2.9000 |

| Nevada | 2003 | 3.2000 |

| Ohio | 2000 | 1.5000 |

| Ohio | 2002 | 3.6000 |