1.3.1 Python之禅

  1. import this
  2. """
  3. The Zen of Python, by Tim Peters
  5. Beautiful is better than ugly.
  6. Explicit is better than implicit.
  7. Simple is better than complex.
  8. Complex is better than complicated.
  9. Flat is better than nested.
  10. Sparse is better than dense.
  11. Readability counts.
  12. Special cases aren't special enough to break the rules.
  13. Although practicality beats purity.
  14. Errors should never pass silently.
  15. Unless explicitly silenced.
  16. In the face of ambiguity, refuse the temptation to guess.
  17. There should be one-- and preferably only one --obvious way to do it.
  18. Although that way may not be obvious at first unless you're Dutch.
  19. Now is better than never.
  20. Although never is often better than *right* now.
  21. If the implementation is hard to explain, it's a bad idea.
  22. If the implementation is easy to explain, it may be a good idea.
  23. Namespaces are one honking great idea -- let's do more of those!
  24. """
  26. """
  27. •优美胜于丑陋(Python 以编写优美的代码为目标)
  28. •明了胜于晦涩(优美的代码应当是明了的，命名规范，风格相似)
  29. •简洁胜于复杂(优美的代码应当是简洁的，不要有复杂的内部实现)
  30. •复杂胜于凌乱(如果复杂不可避免，那代码间也不能有难懂的关系，要保持接口简洁)
  31. •扁平胜于嵌套(优美的代码应当是扁平的，不能有太多的嵌套)
  32. •间隔胜于紧凑(优美的代码有适当的间隔，不要奢望一行代码解决问题)
  33. •可读性很重要(优美的代码是可读的)
  34. •即便假借特例的实用性之名，也不可违背这些规则(这些规则至高无上)
  35. •不要包容所有错误，除非你确定需要这样做(精准地捕获异常，不写 except:pass 风格的代码)
  36. •当存在多种可能，不要尝试去猜测
  37. •而是尽量找一种，最好是唯一一种明显的解决方案(如果不确定，就用穷举法)
  38. •虽然这并不容易，因为你不是 Python 之父(这里的 Dutch 是指 Guido )
  39. •做也许好过不做，但不假思索就动手还不如不做(动手之前要细思量)
  40. •如果你无法向人描述你的方案，那肯定不是一个好方案；反之亦然(方案测评标准)
  41. •命名空间是一种绝妙的理念，我们应当多加利用(倡导与号召)
  42. """

1.3.2 Python教程

来自阿里的Python训练营(2020.11)，178页，19小节，包含四大任务。12天，每天1小时即可学完。

1.3.2.1 小节摘要

  • Python介绍
  1. print("Hello Python.")
  • 变量、运算符与数据类型
  1. num = "123"
  2. print(int(num) + 123)
  • 位运算
  1. print(~1, 1 & 0, 1 | 0, 1 ^ 0, 100 >> 1, 100 << 1, 1<< 2)
  • 条件语句
  1. num = 123
  2. if num > 10:
  3.     print("1")
  4. elif num > 20:
  5.     print("2")
  6. else:
  7.     print("3")
  • 循环语句
  1. passwdList = [str(i) for i in range(232, 421)]
  2. valid = False
  3. count = 3
  4. while count > 0:
  5.     password = input('enter password:')
  6.     for item in passwdList:
  7.         if password == item:
  8.             valid = True
  9.             break
  11.     if not valid:
  12.         print('invalid input')
  13.         count -= 1
  14.         continue
  15.     else:
  16.         break
  • 异常处理
  1. try:
  2.     print(1+"1")
  3. except Exception as e:
  4.     print(e)
  5. finally:
  6.     print("END")
  • 元组
  1. [1, 2, 3], (1, 2, 3)
  • 字符串
  1. print("%.4f" % int("123,2.3"[:-1].strip(".").split(",")[0][::-1]))
  • 字典
  1. dit = {"name": "X", "age": 20}
  • 集合
  1. basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}
  • 序列
  1. t = "ABC"
  2. list(t), tuple(t), len(t), max(t), min(t)
  • 函数与lambda表达式
  1. def f(x):
  2.     return x+2
  4. f = lambda x: x+2
  • 类与对象
  1. class C(object):
  2.     def __init__(self):
  3.         self.__x = None
  5.     def getx(self):
  6.         return self.__x
  8.     def setx(self, value):
  9.         self.__x = value
  11.     def delx(self):
  12.         del self.__x
  • 魔法方法
  1. def libs(n):
  2.     a = 0
  3.     b = 1
  4.     while True:
  5.         a, b = b, a + b
  6.         if a > n:
  7.             return 
  8.         yield a
  10. for each in libs(100):
  11.     print(each, end=' ')
  • 模块
  1. import hello
  2. hello.hi()
  • datetime模块
  1. import datetime
  2. dt = datetime.datetime(year=2020, month=6, day=25, hour=11, minute=51, second=49)
  3. s = dt.strftime("%Y/%m/%d %H:%M:%S")
  4. print(s)
  • 文件与文件系统
  1. with open("c001.py", "r") as f:
  2.     for i in f:
  3.         print(i)
  • OS模块
  1. import os
  2. for item in os.listdir():
  3.     print(item)
  • 序列化与反序列化
  1. import pickle
  2. data = {"name": "X", "age": 20}
  4. with open("dataFile.pkl", "wb") as f:
  5.     pickle.dump(data, f, -1)
  7. with open("dataFile.pkl", "rb") as f:
  8.     print(pickle.load(f))

1.3.2.2 四大任务

  • Task1

Python基础知识介绍，包含变量、位运算、条件语句和异常处理；

  • Task2

Python中的6大数据结构，包括列表、元组、字符串、字典、集合、序列；

  • Task3

Python中的函数、lambda表达式、类与对象、魔法方法；

  • Task4

数据分析实战项目，从0开始动手；

1.3.3 Python书籍

https://www.runoob.com/python/python-tutorial.html https://www.runoob.com/python3/python3-tutorial.html https://www.w3cschool.cn/pythonlearn/ https://www.liaoxuefeng.com/wiki/1016959663602400 《Python学习手册》 《Python语言及其应用》 《像计算机科学家一样思考Python》

1.3.4 Python Help使用

直接调用help()语句，如下：

  1. import this
  2. help(this)
  3. """
  4. Help on module this:
  6. NAME
  7.     this
  9. MODULE REFERENCE
  10.     https://docs.python.org/3.7/library/this
  12.     The following documentation is automatically generated from the Python
  13.     source files.  It may be incomplete, incorrect or include features that
  14.     are considered implementation detail and may vary between Python
  15.     implementations.  When in doubt, consult the module reference at the
  16.     location listed above.
  18. DATA
  19.     c = 97
  20.     d = {'A': 'N', 'B': 'O', 'C': 'P', 'D': 'Q', 'E': 'R', 'F': 'S', 'G': ...
  21.     i = 25
  22.     s = "Gur Mra bs Clguba, ol Gvz Crgref\n\nOrnhgvshy vf o...bar ubaxvat ...
  24. FILE
  25.     /python3/lib/python3.7/this.py
  26. """

或者

  1. def f(para1, para2):
  2.     """Get the min one."""
  3.     return min(para1, para2)
  4. help(f)
  5. """
  6. Help on function f in module __main__:
  8. f(para1, para2)
  9.     Get the min one.
  10. """

1.3.5 Python包

1.3.5.1 Python包缺失

执行代码时报错如下：

  1. import data
  2. """
  3. ---------------------------------------------------------------------------
  4. ModuleNotFoundError                       Traceback (most recent call last)
  5. <ipython-input-1090-699327cd1cad> in <module>()
  6. ----> 1 import data
  8. ModuleNotFoundError: No module named 'data'
  9. """

即缺失该data工具包。

1.3.5.2 Python包安装

一般的，可以使用Anaconda进行Python库安装及管理，其中包括Conda、Python以及一大堆安装好的工具包，比如：numpy、pandas等。

https://mirror.tuna.tsinghua.edu.cn/help/anaconda/

对于Anaconda未预装的Python包就需要自行安装，如下：

（1）在线安装

  1. pip install data -i https://pypi.tuna.tsinghua.edu.cn/simple

对于较新版本，推荐使用：

  1. python3 -m pip install data -i https://pypi.tuna.tsinghua.edu.cn/simple

若显示Successfully installed XXX即安装成功。

  1. """
  2. Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  3. Collecting data
  4.   Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ed/e9/623be82fac4250fc614741f5b1ead83d339794f94b19ac8665b6ea12ee05/data-0.4.tar.gz (7.0 kB)
  5. ...
  6. Building wheels for collected packages: data
  7.   Building wheel for data (setup.py) ... done
  8.   Created wheel for data: filename=data-0.4-py3-none-any.whl size=7247 sha256=32a13cabc2205de95c0be8d38888b6642b602dca08e2951f4d3c200128c6ff9d
  9.   Stored in directory: /pip/wheels/35/20/c4/885e3ce9b69e4852259f20fe1ee95b5d6eb7dcaafae23f4051
  10. Successfully built data
  11. Installing collected packages: data
  12. Successfully installed data-0.4
  13. """

（2）离线安装
大部分时候，安装Python包的银行环境属于内网独立网段，我们可以预先download好Python包用于离线安装。

  1. pip download data
  2. """
  3. Collecting data
  4.   Downloading data-0.4.tar.gz (7.0 kB)
  5. Collecting decorator
  6.   Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
  7. Collecting funcsigs
  8.   Downloading funcsigs-1.0.2-py2.py3-none-any.whl (17 kB)
  9. Collecting six
  10.   Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
  11. Saved ./data-0.4.tar.gz
  12. Saved ./decorator-4.4.2-py2.py3-none-any.whl
  13. Saved ./funcsigs-1.0.2-py2.py3-none-any.whl
  14. Saved ./six-1.15.0-py2.py3-none-any.whl
  15. Successfully downloaded data decorator funcsigs six
  16. """

可以看到连带依赖包也下载好，可以打包带走，带到另一个地方进行离线安装。

  1. cd packages
  2. pip install six-1.15.0-py2.py3-none-any.whl

（3）源码安装
这里注意到data下载后是压缩包data-0.4.tar.gz，此时需要源码安装。

  1. # 解压，手动解压也一样
  2. tar -zxvf data-0.4.tar.gz 
  3. """
  4. data-0.4/
  5. data-0.4/setup.cfg
  6. data-0.4/PKG-INFO
  7. data-0.4/setup.py
  8. data-0.4/README.rst
  9. data-0.4/data.egg-info/
  10. data-0.4/data.egg-info/top_level.txt
  11. data-0.4/data.egg-info/requires.txt
  12. data-0.4/data.egg-info/dependency_links.txt
  13. data-0.4/data.egg-info/SOURCES.txt
  14. data-0.4/data.egg-info/PKG-INFO
  15. data-0.4/data/
  16. data-0.4/data/decorators.py
  17. data-0.4/data/__init__.py
  18. """
  20. # 去到该路径
  21. cd data-0.4
  23. # Build
  24. python setup.py build
  25. """
  26. running build
  27. running build_py
  28. creating build
  29. creating build/lib
  30. creating build/lib/data
  31. copying data/__init__.py -> build/lib/data
  32. copying data/decorators.py -> build/lib/data
  33. """
  35. # Install
  36. python setup.py install
  37. """
  38. running install
  39. running bdist_egg
  40. running egg_info
  41. writing data.egg-info/PKG-INFO
  42. writing dependency_links to data.egg-info/dependency_links.txt
  43. writing requirements to data.egg-info/requires.txt
  44. writing top-level names to data.egg-info/top_level.txt
  45. reading manifest file 'data.egg-info/SOURCES.txt'
  46. writing manifest file 'data.egg-info/SOURCES.txt'
  47. installing library code to build/bdist.linux-x86_64/egg
  48. running install_lib
  49. running build_py
  50. ...
  51. Using /usr/local/lib/python3.6/dist-packages
  52. Finished processing dependencies for data==0.4
  53. """

（4）卸载管理

  1. pip uninstall data   
  2. """
  3. Found existing installation: data 0.4
  4. Uninstalling data-0.4:
  5.   Would remove:
  6.     /python3/lib/python3.7/site-packages/data-0.4.dist-info/*
  7.     /python3/lib/python3.7/site-packages/data/*
  8. Proceed (y/n)? y
  9.   Successfully uninstalled data-0.4
  10. """

1.3.5.3 Python包验证

如上，我们已经安装data包，验证之。

  1. python -c "import data"

若无输出即安装无误。

  • PS：在Windows环境下，上述命令需先通过win+R，输入cmd调出命令提示符操作。
    除切换路径时改用cd /d D:\Python包，其余一致

1.3.6 Python IDE

1.3.6.1 Jupyter

  • Jupyter魔法函数

所谓魔法函数，实际上是IPython预先定义好的具备特定功能的函数被收到Jupyter中使用。

  1. %lsmagic
  2. """
  3. Available line magics:
  4. %alias  
  5. %alias_magic  
  6. %autocall  
  7. %automagic  
  8. %autosave  
  9. %bookmark  
  10. %cat  
  11. %cd  
  12. %clear  
  13. %colors  
  14. %config  
  15. %connect_info  
  16. %cp  
  17. %debug  
  18. %dhist  
  19. %dirs  
  20. %doctest_mode  
  21. %ed  
  22. %edit  
  23. %env  
  24. %gui  
  25. %hist  
  26. %history  
  27. %killbgscripts  
  28. %ldir  
  29. %less  
  30. %lf  
  31. %lk  
  32. %ll  
  33. %load  
  34. %load_ext  
  35. %loadpy  
  36. %logoff  
  37. %logon  
  38. %logstart  
  39. %logstate  
  40. %logstop  
  41. %ls  
  42. %lsmagic  
  43. %lx  
  44. %macro  
  45. %magic  
  46. %man  
  47. %matplotlib  
  48. %mkdir  
  49. %more  
  50. %mv  
  51. %notebook  
  52. %page  
  53. %pastebin  
  54. %pdb  
  55. %pdef  
  56. %pdoc  
  57. %pfile  
  58. %pinfo  
  59. %pinfo2  
  60. %popd  
  61. %pprint  
  62. %precision  
  63. %profile  
  64. %prun  
  65. %psearch  
  66. %psource  
  67. %pushd  
  68. %pwd  
  69. %pycat  
  70. %pylab  
  71. %qtconsole  
  72. %quickref  
  73. %recall  
  74. %rehashx  
  75. %reload_ext  
  76. %rep  
  77. %rerun  
  78. %reset  
  79. %reset_selective  
  80. %rm  
  81. %rmdir  
  82. %run  
  83. %save  
  84. %sc  
  85. %set_env  
  86. %store  
  87. %sx  
  88. %system  
  89. %tb  
  90. %time  
  91. %timeit  
  92. %unalias  
  93. %unload_ext  
  94. %who  
  95. %who_ls  
  96. %whos  
  97. %xdel  
  98. %xmode
  100. Available cell magics:
  101. %%!  
  102. %%HTML  
  103. %%SVG  
  104. %%bash  
  105. %%capture  
  106. %%debug  
  107. %%file  
  108. %%html  
  109. %%javascript  
  110. %%js  
  111. %%latex  
  112. %%markdown  
  113. %%perl  
  114. %%prun  
  115. %%pypy  
  116. %%python  
  117. %%python2  
  118. %%python3  
  119. %%ruby  
  120. %%script  
  121. %%sh  
  122. %%svg  
  123. %%sx  
  124. %%system  
  125. %%time  
  126. %%timeit  
  127. %%writefile
  129. Automagic is ON, % prefix IS NOT needed for line magics.
  130. """

常用的%time、%timeit，能为代码提供执行计时服务，区别在于%time返回单次结果，%timeit返回循环N轮次后统计结果。这很适合性能比拼，以下测试sum、np.sum在实现一样的求和功能时，代码耗时情况：

  1. import numpy as np
  2. L = np.linspace(0, 100, 1000000)
  3. f1 = lambda l: sum(l)
  4. f2 = lambda l: np.sum(l)
  6. %time f1(L)
  7. %time f2(L)
  9. %timeit f1(L)
  10. %timeit f2(L)
  11. """
  12. CPU times: user 132 ms, sys: 548 µs, total: 132 ms
  13. Wall time: 133 ms
  14. CPU times: user 810 µs, sys: 479 µs, total: 1.29 ms
  15. Wall time: 1.16 ms
  17. 140 ms ± 3.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
  18. 367 µs ± 39.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
  19. """

显然，此测试中np.sum完胜！

1.3.6.2 Pycharm

虽然，Pycharm也集成了Jupyter的功能，相对而言，Pycharm比较适合开发工程型项目。

1.3.7 Python代码规范

一般来说，我们使用PEP8作为Python代码规范。

1.3.7.1 PEP 8

https://www.python.org/dev/peps/pep-0008/ https://github.com/python/peps/blob/master/pep-0008.txt

要求如：

  • Use 4 spaces per indentation level.
  • Limit all lines to a maximum of 79 characters.
  • …

1.3.7.2 YAPF

可以使用Google开发的格式工具YAPF。

https://github.com/google/yapf https://pypi.org/project/yapf/

（1）安装

  1. pip install yapf

（2）说明

  1. yapf --help
  2. """
  3. usage: yapf [-h] [-v] [-d | -i] [-r | -l START-END] [-e PATTERN]
  4.             [--style STYLE] [--style-help] [--no-local-style] [-p]
  5.             [-vv]
  6.             [files [files ...]]
  8. Formatter for Python code.
  10. positional arguments:
  11.   files
  13. optional arguments:
  14.   -h, --help            show this help message and exit
  15.   -v, --version         show version number and exit
  16.   -d, --diff            print the diff for the fixed source
  17.   -i, --in-place        make changes to files in place
  18.   -r, --recursive       run recursively over directories
  19.   -l START-END, --lines START-END
  20.                         range of lines to reformat, one-based
  21.   -e PATTERN, --exclude PATTERN
  22.                         patterns for files to exclude from formatting
  23.   --style STYLE         specify formatting style: either a style name (for
  24.                         example "pep8" or "google"), or the name of a file
  25.                         with style settings. The default is pep8 unless a
  26.                         .style.yapf or setup.cfg file located in the same
  27.                         directory as the source or one of its parent
  28.                         directories (for stdin, the current directory is
  29.                         used).
  30.   --style-help          show style settings and exit; this output can be saved
  31.                         to .style.yapf to make your settings permanent
  32.   --no-local-style      don't search for local style definition
  33.   -p, --parallel        Run yapf in parallel when formatting multiple files.
  34.                         Requires concurrent.futures in Python 2.X
  35.   -vv, --verbose        Print out file names while processing
  36.  """

  • 应用1

    1. yapf -i run.py

  • 应用2 ```python # from yapf.yapflib.yapf_api import FormatCode code = “”” x = { ‘a’:37,’b’:42,

‘c’:927}

y = ‘hello ‘’world’ z = ‘hello ‘+’world’ a = ‘hello {}’.format(‘world’) class foo ( object ): def f (self ): return 37-+2 def g(self, x,y=42): return y def f ( a ) : return 37+-+a[42-x : y*3] “””

print(FormatCode(code)) “”” (“x = {‘a’: 37, ‘b’: 42, ‘c’: 927}\n\ny = ‘hello ‘ ‘world’\nz = ‘hello ‘ + ‘world’\na = ‘hello {}’.format(‘world’)\n\n\nclass foo(object):\n def f(self):\n return 37 -+2\n\n def g(self, x, y=42):\n return y\n\n\ndef f(a):\n return 37 + -+a[42 - x:y*3]\n”, True) “””

  2. （3）示例<br />以下例举官方例子。
  4. 修改前：
  6. ```python
  7. x = {  'a':37,'b':42,
  9. 'c':927}
  11. y = 'hello ''world'
  12. z = 'hello '+'world'
  13. a = 'hello {}'.format('world')
  14. class foo  (     object  ):
  15.   def f    (self   ):
  16.     return       37*-+2
  17.   def g(self, x,y=42):
  18.       return y
  19. def f  (   a ) :
  20.   return      37+-+a[42-x :  y**3]

修改后：

  1. x = {'a': 37, 'b': 42, 'c': 927}
  3. y = 'hello ' 'world'
  4. z = 'hello ' + 'world'
  5. a = 'hello {}'.format('world')
  8. class foo(object):
  9.     def f(self):
  10.         return 37 * -+2
  12.     def g(self, x, y=42):
  13.         return y
  16. def f(a):
  17.     return 37 + -+a[42 - x:y**3]

1.3.8 Python片段

详见：

https://www.yuque.com/ivanaxu/pytools

示例，009.Test_multiprocessing_run, multiprocessing多线程
截屏2020-12-09 下午3.33.13.png

1.3.9 Python编译

由于Python编译对大部分开发者是无感的，一般来说，我们不需要像SAS那样进一步延伸编译层的PDV概念，进而能编写出更加优质（但也更加难懂）的代码。故而以下只点出三个概念：

  • 其一，基本编译

    Python每次运行都要进行转换成字节码，然后再有虚拟机把字节码转换成机器语言，最后才能在硬件上运行。与编译性语言相比，每次多出了编译和链接的过程，性能肯定会受到影响；而python并不是每次都需要转换字节码，解释器在转换之前会判断代码文件的修改时间是否与上一次转换后的字节码pyc文件的修改时间一致，若不一致才会重新转换。

  • 其二，code object

我们可以通过内置函数compile，将源文件编译成code object，如下：

  1. cmd = """
  2. print(1)
  3. """
  4. code_object = compile(cmd, "", "exec")
  5. print(code_object)
  6. eval(code_object)
  8. """
  9. <code object <module> at 0x7f859d1b79c0, file "", line 2>
  10. 1
  11. """

还可以通过dis进行反编译：

  1. import dis
  2. dis.dis(code_object)
  3. """
  4. 2             0 LOAD_NAME                0 (print)
  5.               2 LOAD_CONST               0 (1)
  6.               4 CALL_FUNCTION            1
  7.               6 POP_TOP
  8.               8 LOAD_CONST               1 (None)
  9.              10 RETURN_VALUE
  10. """

可以看到python字节码其实是模仿的x86的汇编，将代码编译成一条一条的指令交给一个虚拟的cpu去执行的。

有兴趣的同学可拓展学习： https://godbolt.org/(一个多语言在线编辑器) Python代码片段-081.Test_eval, eval的安全攻防

  • 其三，Python部署

这里指Python部署时，将Python转换Pyc/Pyo/So文件，避免直接暴露源码。

  1. # 同一系统环境下
  3. /python3/bin/cython run.py
  4. # /python3/bin/python -m py_compile run.py
  6. gcc -c -fPIC run.c
  8. gcc -shared run.o -o run.so