1.1 基于docker的spark配置文件

docker-compose-spark.yml 

  1. version: '2'
  3. services:
  4.   spark:
  5.     image: docker.io/bitnami/spark:3
  6.     environment:
  7.       - SPARK_MODE=master
  8.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  9.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  10.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  11.       - SPARK_SSL_ENABLED=no
  12.     ports:
  13.       - '8080:8080'
  14.   spark-worker-1:
  15.     image: docker.io/bitnami/spark:3
  16.     environment:
  17.       - SPARK_MODE=worker
  18.       - SPARK_MASTER_URL=spark://spark:7077
  19.       - SPARK_WORKER_MEMORY=1G
  20.       - SPARK_WORKER_CORES=1
  21.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  22.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  23.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  24.       - SPARK_SSL_ENABLED=no
  25.   spark-worker-2:
  26.     image: docker.io/bitnami/spark:3
  27.     environment:
  28.       - SPARK_MODE=worker
  29.       - SPARK_MASTER_URL=spark://spark:7077
  30.       - SPARK_WORKER_MEMORY=1G
  31.       - SPARK_WORKER_CORES=1
  32.       - SPARK_RPC_AUTHENTICATION_ENABLED=no
  33.       - SPARK_RPC_ENCRYPTION_ENABLED=no
  34.       - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
  35.       - SPARK_SSL_ENABLED=no

该目录下执行

  1. docker-compose up

在浏览器中输入 localhost:8080 访问master 的web UI

pyspark 本地demo

  1. from pyspark import SparkContext
  2. sc = SparkContext("local", "count app")
  3. words = sc.parallelize(
  4.     ["scala",
  5.      "java",
  6.      "hadoop",
  7.      "spark",
  8.      "akka",
  9.      "spark vs hadoop",
  10.      "pyspark",
  11.      "pyspark and spark"
  12.      ])
  13. counts = words.count()
  14. print("Number of elements in RDD -> %i" % counts)

参考：

  1. https://www.cnblogs.com/jiujiubashiyi/p/15600979.html