问题

1028 mac上pyspark执行失败,提示

Exception: Python in worker has different version 3.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

解决方案


修改全局的环境配置/etc/profile

export PYSPARK_PYTHON=/Users/leo/opt/anaconda3/envs/dgldemo/bin/python
export PYSPARK_DRIVER_PYTHON=/Users/leo/opt/anaconda3/envs/dgldemo/bin/python

最后source /etc/profile,即可解决问题。

注意:修改~/.bash_profile是不能生效的,因为这只是对当前用户生效的

另:windows中解决该问题,也是修改环境变量,但是一定要修改系统变量,修改用户变量是不生效的。