1. 注意事项

  1. 使用 vscode 远程配置开发环境时，python file.py 主要依赖环境变量中 $PYTHONPATH 这个变量，因此需要设置该环境变量，否则无法导入对应pyspark 包； ```bash vim ~/.bashrc

export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.3-src.zip:$PYTHONPATH

source ~/.bashrc

  1. <br />测试：  <br />运行测试`python wordcount.py`<br />检查运行结果可在`/root/output`文件夹下查看 
  3. <a name="b192dada"></a>
  4. ## 2. 集成开发环境测试
  6. <a name="f44a6838"></a>
  7. ### 2.1 spark local 模式测试
  9. 直接 `python wordcount.py`
  11. ```python
  12. from pyspark import SparkContext
  13. from pyspark import SparkConf
  16. def CreateSparkContext():
  17.     sparkConf=SparkConf().setAppName("WordCounts").set("spark.ui.showCOnsolePrgoress","false")
  18.     sc=SparkContext(conf=sparkConf)
  19.     print("master="+sc.master)
  20.     SetLogger(sc)
  21.     SetPath(sc)
  22.     return(sc)
  24. def SetLogger(sc):
  25.     logger=sc._jvm.org.apache.log4j
  26.     logger.LogManager.getLogger("org").setLevel(logger.Level.ERROR)
  27.     logger.LogManager.getLogger("akka").setLevel(logger.Level.ERROR)
  28.     logger.LogManager.getRootLogger().setLevel(logger.Level.ERROR)
  30. def SetPath(sc):
  31.     global Path
  32.     if sc.master[0:5]=="local":
  33.         Path = "/root/pythonwork/test/"
  34.     else:
  35.         Path="hdfs://master:9000/user/"
  37. if __name__=="__main__":
  38.     print("开始执行 WordCount")
  39.     sc=CreateSparkContext()
  41.     print("开始读取文本文件...")
  42.     textFile=sc.textFile(Path+"test.txt")
  43.     print("文件共"+str(textFile.count())+"行")
  45.     countsRDD=textFile.flatMap(lambda line:line.split(" ")).map(lambda word:(word,1)).reduceByKey(lambda x,y:x+y)
  46.     print("文字共"+str(countsRDD.count())+"项数据")
  48.     print("开始保存至文本文件...")
  49.     countsRDD.saveAsTextFile("/root/pythonwork/test/output")

2.2 创建测试目录

  1. 创建测试文件上传HDFS目录 
    1. mkdir -p ~/pythonwork/project/data
    2. cp /usr/local/spark/README.md ~/pythonwork/project/data
  1. 启动 Hadoop Multi Node Cluster 
    1. start-all.sh
  1. 创建测试目录 
    1. hadoop fs -mkdir -p /user/root/data
    2. hadoop fs -copyFromLocal ~/pythonwork/project/data/README.md /user/root/data
    3. hadoop fs -ls /user/root/data/

2.3 Spark local 模式

  1. spark-submit 常用选项 
    1. --master MASTER_RUL
    2. - Local
    3. - Local[n]
    4. - Local[*]
    5. - spark://HOST:PORT
    6. - mesos L://HOST:PORT
    7. - YARN
    8. --dirver-memory MEM
    9. --executor-memory MEM
    10. --name NAME
  1. 测试 ```python vim ~/pythonwork/project/wordcount.py

from pyspark import SparkContext from pyspark import SparkConf

def CreateSparkContext(): sparkConf=SparkConf().setAppName(“WordCounts”).set(“spark.ui.showCOnsolePrgoress”,”false”) sc=SparkContext(conf=sparkConf) print(“master=”+sc.master) SetLogger(sc) SetPath(sc) return(sc)

def SetLogger(sc): logger=sc._jvm.org.apache.log4j logger.LogManager.getLogger(“org”).setLevel(logger.Level.ERROR) logger.LogManager.getLogger(“akka”).setLevel(logger.Level.ERROR) logger.LogManager.getRootLogger().setLevel(logger.Level.ERROR)

def SetPath(sc): global Path if sc.master[0:5]==”local”: Path = “/root/pythonwork/project/data/“ else: Path=”hdfs://master:9000/user/root/data/“

if name==”main“: print(“开始执行 WordCount”) sc=CreateSparkContext()

  1. print("开始读取文本文件...")
  2. textFile=sc.textFile(Path+"README.md")
  3. print("文件共"+str(textFile.count())+"行")
  5. countsRDD=textFile.flatMap(lambda line:line.split(" ")).map(lambda word:(word,1)).reduceByKey(lambda x,y:x+y)
  6. print("文字共"+str(countsRDD.count())+"项数据")
  8. print("开始保存至文本文件...")
  9. countsRDD.saveAsTextFile("/root/pythonwork/project/data/output"
  3. <a name="8abbb696"></a>
  4. ### 2.4 Spark YARN-Client 模式
  6. ```bash
  7. cd ~/pythonwork/project/
  9. HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop spark-submit --driver-memory 512m --executor-cores 1 --master yarn --deploy-mode client wordcount.py

2.5 Spark Standalone Cluster 模式

  1. cd ~/pythonwork/project/
  3. /usr/local/spark/sbin/start-all.sh
  5. spark-submit --master spark://master:7077 --deploy-mode client --executor-memory 512m --total-executor-cores 2 wordcount.py