抖音爬虫

代码地址：https://gitee.com/duxinn/EartipDouyin.git
所在服务器：251
项目地址：/root/projects/EartipDouyin
部署方式：crontab

  1. 3 6-23/1 * * * flock -xn /tmp/douyin.lock -c 'cd /root/projects/EartipDouyin && /root/.virtualenvs/EartipDouyin-d8HskUpw/bin/python -u  run.py  >> "logs/running-$(date +"\%Y-\%m-\%d").log" 2>&1 '

项目环境管理：pipenv
项目环境依赖存储目录：requirements.txt

部署步骤：

  1. cd /root/projects/EartipDouyin
  2. git clone https://gitee.com/duxinn/EartipDouyin.git
  4. pipenv --python 3.7.4  # 创建pipenv环境
  6. pipenv shell # 进入pipenv环境
  8. # 安装依赖包
  9. pip install -r requirements.txt
  11. python run.py  # 试运行
  13. # 查看python路径
  14. pipenv --py
  15. # 显示 /root/.virtualenvs/EartipDouyin-d8HskUpw/bin/python
  16. # 可放入crontab

搜狗爬虫

代码地址：https://gitee.com/duxinn/EartipSougou.git
所在服务器：251
项目地址：/root/projects/EartipSougou

部署方式：crontab

  1. 4 6-23/2 * * * flock -xn /tmp/sougou.lock -c 'cd /root/projects/EartipSougou && /root/.virtualenvs/EartipSougou-MYjfpg56/bin/python -u  run_api.py  >> "logs/running-$(date +"\%Y-\%m-\%d").log" 2>&1 '

项目环境管理：pipenv
项目环境依赖存储目录：requirements.txt

部署步骤：同抖音

B站

代码地址：https://gitee.com/duxinn/EartipBilibili.git
所在服务器：192.168.1.252
项目地址：/root/projects/EartipBilibili
部署方式：crontab

  1. 3 6-23/3 * * * flock -x /tmp/bibi.lock -c  ' cd  /root/projects/EartipBilibili && /root/.local/share/virtualenvs/EartipBilibili-9R72gDmP/bin/python -u run_sel
  2. enium.py  >> "logs/running-$(date +"\%Y-\%m-\%d").log" 2>&1 '

项目环境管理：pipenv
项目环境依赖存储目录：requirements.txt

部署步骤：同抖音

贴吧

代码地址：https://gitee.com/duxinn/EartipTieba.git
所在服务器：192.168.1.252
项目地址：/root/projects/EartipTieba
部署方式：crontab

  1. # 贴子的搜索
  2. 5 6-23/3 * * * flock -x /tmp/tie.lock -c  ' cd  /root/projects/EartipTieba  &&  /root/.local/share/virtualenvs/EartipTieba-A5puD_2X/bin/python  -u run_selenium_tie.py  >> "logs/running_tie-$(date +"\%Y-\%m-\%d").log" 2>&1 '
  4. # 吧的搜索
  5. 7 6-23/6  * * * flock -x /tmp/ba.lock -c  ' cd  /root/projects/EartipTieba  &&  /root/.local/share/virtualenvs/EartipTieba-A5puD_2X/bin/python  -u run_selenium_ba.py  >> "logs/running_ba-$(date +"\%Y-\%m-\%d").log" 2>&1 '

项目环境管理：pipenv
项目环境依赖存储目录：requirements.txt

部署步骤：同抖音

贴吧的旋转识别服务：rotnet

代码地址：https://gitee.com/duxinn/rotnet.git
所在服务器：192.168.1.252
项目地址：/root/projects/EartipTieba
部署方式：docker+conda
项目环境管理：docker+conda

部署步骤：
1、创建容器

  1. 拉取manjaro镜像，注意20210725标签
  2. docker pull docker.io/manjarolinux/base:20210725
  4. 查看manjaro镜像
  5. docker image ls
  6. REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
  7. docker.io/manjarolinux/base   20210725            8fddaa2b2126        2 months ago        1.17 GB
  9. 创建容器
  10. docker run -idt  --privileged=true --restart=always  --name tieba -p 9720:9720  manjaro镜像id
  12. # 进入容器
  13. docker exec -it  -u root  tieba /bin/bash

2、搭建容器内部环境

  1. 选择镜像
  2. sudo pacman-mirrors -i -c China -m rank
  4. 更新源
  5. sudo pacman -Syy
  7. 安装 yay 包管理工具
  8. pacman -S yay
  10. 添加用户name1
  11. useradd -d /home/name1 -m name1
  12. chown name1 -R /home/name1
  14. 修改密码，这里修改为name1
  15. passwd name1
  17. 添加为sudoer
  18. chmod +w /etc/sudoers
  19. vim /etc/sudoers
  20. 在 root ALL=(ALL) ALL 下面
  21. 写入 name1            ALL=(ALL)                ALL
  22. chmod -w /etc/sudoers
  24. 切换到name1
  25. sudo su - name1
  27. 安装conda环境（会输入密码）
  28. yay -S anaconda
  30. conda的配置
  31. ln -s /opt/anaconda/bin/conda  /usr/bin/conda
  32. conda config --set auto_activate_base false
  33. conda config --set show_channel_urls yes

centos8

  1. 编译安装 Anaconda
  2. wget -P /tmp https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh

参考 https://zhuanlan.zhihu.com/p/64930395
https://www.myfreax.com/how-to-install-anaconda-on-centos-8/

3、部署项目环境

  1. /root 下 拉代码
  2. git clone https://gitee.com/duxinn/rotnet.git
  4. 使用conda创建虚拟环境，命名为tieba
  5. conda create -n tieba  python=3.7.4
  7. 查看虚拟环境 有tieba
  8. conda info -e
  10. 进入虚拟环境
  11. conda activate tieba
  13. 按照readme.txt 中的说明安装tensorflow
  14. conda install tensorflow==2.0.0
  15. 按照 requirements.txt 安装pip包
  16. pip install -r requirements.txt  -i https://mirrors.aliyun.com/pypi/simple 
  18. 运行
  19. python run.py

4、测试

  1. curl localhost:9722/index
  2. {"message":"Hello World"}
  4. 说明服务已经起来了

报错和解决

  1. Traceback (most recent call last):
  2.   File "run.py", line 8, in <module>
  3.     from utils import RotNetDataGenerator, angle_error
  4.   File "/root/rotnet/utils/__init__.py", line 1, in <module>
  5.     from .rotnet_utils import *
  6.   File "/root/rotnet/utils/rotnet_utils.py", line 4, in <module>
  7.     import cv2
  8.   File "/opt/anaconda/envs/tieba/lib/python3.7/site-packages/cv2/__init__.py", line 3, in <module>
  9.     from .cv2 import *
  10. ImportError: libSM.so.6: cannot open shared object file: No such file or directory
  12. pacman -S libsm libxext libxrender
  14. centos环境
  15. yum install libSM  libXext  libXrender

CRM

代码地址：https://gitee.com/zpy-git/EartipCrmBackend.git
所在服务器：119.23.211.94
项目地址：docker crm:/root/EartipCrmBackend
部署方式：nginx+docker
项目环境管理：无
项目环境依赖存储目录：requirements.txt

部署步骤：

  1. 拉取manjaro镜像后运行manjaro容器
  2. docker run -idt  --restart=always --name crm -p 9015:9015 docker.io/manjarolinux/base:20210725
  4. # 进入容器
  5. docker exec -it  -u root  crm /bin/bash
  7. 拉取代码
  8. git clone -b master https://gitee.com/zpy-git/EartipCrmBackend.git
  10. 安装
  11. pip install -r requirements.txt
  13. 运行
  14. python manage.py runserver 0:9015

nginx 部署文件：
/usr/local/nginx/conf/nginx.conf

  1. server {
  2.           listen 80;
  3.           listen 443 ssl;
  4.           server_name crm.eartip.cn;
  5.           ssl_certificate /usr/local/nginx/cert/crm/6058513_crm.eartip.cn.pem;
  6.           ssl_certificate_key /usr/local/nginx/cert/crm/6058513_crm.eartip.cn.key;
  7.           ssl_session_cache    shared:SSL:1m;
  8.           ssl_session_timeout  5m;
  10.           fastcgi_param   HTTPS               on;
  11.           fastcgi_param   HTTP_SCHEME         https;
  12.           access_log /usr/local/nginx/logs/httpsaccess.log;
  13.           root /root/project/EartipCrmBackend;
  14.           location / {
  15.               proxy_pass http://127.0.0.1:9015;
  16.               proxy_set_header  Host $host;
  17.               proxy_set_header   X-real-ip $remote_addr;
  18.               proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  19.           }
  20.       }

FoodCrm

代码地址：https://gitee.com/duxinn/FoodCrm.git
所在服务器：119.23.211.94
项目地址：/root/project/FoodCrm
部署方式：nginx+pipenv
项目环境管理：无
项目环境依赖存储目录：requirements.txt

部署步骤：

  1. 拉取代码
  2. git clone -b master https://gitee.com/duxinn/FoodCrm.git
  4. pipenv --python 3.7
  5. pipenv shell
  7. 安装
  8. pip install django
  9. pip install mysqlclient
  10. pip install cacheout
  11. pip install django-cors-headers
  13. 建库
  14. CREATE DATABASE `food_crmdb` DEFAULT CHARACTER SET utf8 
  16. 运行
  17. python manage.py runserver 0:9020
  1. server {
  2.         listen 9021;
  3.         fastcgi_param   HTTPS               on;
  4.         fastcgi_param   HTTP_SCHEME         https;
  5.         access_log /usr/local/nginx/logs/httpsaccess.log;
  6.         root /root/project/FoodCrm/dist;
  7.         index  index.html index.htm;
  8.         location / {
  9.             try_files $uri /index.html;
  10.         }
  11.         location  /api/ {
  12.             proxy_pass http://127.0.0.1:9020;
  13.             proxy_set_header  Host $host;
  14.             proxy_set_header   X-real-ip $remote_addr;
  15.             proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  16.         }
  17. }