1.简介和安装

文档将介绍Tensorflow目标检测的API（link）。API可以用来检测图片或者视频流中的物体，可是是预先训练好的识别模型来检测物体，也可以是自己定义的需要分类的物体。首先，我们需要安装Tensorflow和相关的依赖。对于CPU的机器，直接安装 pip3 install tensorflow 就可以。对于使用GPU计算的机器，计算会更快，安装如下：

  • Ubuntu下安装 link
  • Windows下安装link

其他，可能需要提前安装的python3.7以及python 依赖库：

  1. pip3 install pillow
  2. pip3 install lxml
  3. pip3 install jupyter
  4. pip3 install matplotlib

接下来，需要从github上把Tensorflow 目标检测API项目克隆下来，或者直接下载zip包：

  1. git clone https://github.com/tensorflow/models.git

模型文件下载下来以后，到指定目录，运行cmd:

windows机器需要下载 protoc-3.4.0-win32.zip(link），解压以后，bin目录下会有protoc的执行文件。

接下来，我们就可以在model目录下，用protoc来编译模型文件.proto：

  1. "C:/Program Files/protoc/bin/protoc" object_detection/protos/*.proto --python_out=.

注意编译的目录位置

最后，在models/object_detection目录，打开终端terminal/cmd.exe，使用jupyter notebook，选择object_detection_tutorial.ipynb。选择run all，就可以得到如下结果：
Tensorflow目标检测（一） - 图1

2.视频流目标检测

本部分将包含如何基于示例代码，将其改造成基于摄像头识别的目标检测程序。首先，在object_detection目录下新建detect1.py文件。如果你就想要在notebook里面实现，也没有问题。
检测的机制是我们下载Google的预先训练模型ssd_mobilenet_v1_coco_11_06_2017 link，本模型预训练了大概90个分类，有人，茶杯，手提袋等物体的识别。下面的代码是从opencv来截取视频流，使用tensorflow加载ssd模型进行识别。
接下来，我们将使用opencv包来获取视频信息。如果还没有安装python opencv，可以参考link:

  1. import cv2
  2. cap = cv2.VideoCapture(0)

完整代码：

  1. import os
  3. import cv2
  4. import numpy as np
  5. import tensorflow as tf
  6. from utils import label_map_util
  7. from utils import visualization_utils as vis_util
  9. cap = cv2.VideoCapture(0)
  11. # http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz
  12. MODEL_NAME = 'ssd_mobilenet_v1_coco_11_06_2017'
  13. MODEL_FILE = MODEL_NAME + '.tar.gz'
  14. # Path to frozen detection graph. This is the actual model that is used for the object detection.
  15. PATH_TO_CKPT = MODEL_NAME + '/frozen_inference_graph.pb'
  17. # List of the strings that is used to add correct label for each box.
  18. PATH_TO_LABELS = os.path.join('data', 'mscoco_label_map.pbtxt')
  20. detection_graph = tf.Graph()
  21. with detection_graph.as_default():
  22.     od_graph_def = tf.GraphDef()
  23.     with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
  24.         serialized_graph = fid.read()
  25.         od_graph_def.ParseFromString(serialized_graph)
  26.         tf.import_graph_def(od_graph_def, name='')
  28. category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)
  30. with detection_graph.as_default():
  31.     with tf.Session(graph=detection_graph) as sess:
  32.         while True:
  33.             ret, image_np = cap.read()
  34.             # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
  35.             image_np_expanded = np.expand_dims(image_np, axis=0)
  36.             image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
  37.             # Each box represents a part of the image where a particular object was detected.
  38.             boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
  39.             # Each score represent how level of confidence for each of the objects.
  40.             # Score is shown on the result image, together with the class label.
  41.             scores = detection_graph.get_tensor_by_name('detection_scores:0')
  42.             classes = detection_graph.get_tensor_by_name('detection_classes:0')
  43.             num_detections = detection_graph.get_tensor_by_name('num_detections:0')
  44.             # Actual detection.
  45.             (boxes, scores, classes, num_detections) = sess.run(
  46.                 [boxes, scores, classes, num_detections],
  47.                 feed_dict={image_tensor: image_np_expanded})
  48.             # Visualization of the results of a detection.
  49.             vis_util.visualize_boxes_and_labels_on_image_array(
  50.                 image_np,
  51.                 np.squeeze(boxes),
  52.                 np.squeeze(classes).astype(np.int32),
  53.                 np.squeeze(scores),
  54.                 category_index,
  55.                 use_normalized_coordinates=True,
  56.                 line_thickness=8)
  58.             cv2.imshow('object detection', cv2.resize(image_np, (800, 600)))
  59.             # 按q退出程序
  60.             if cv2.waitKey(25) & 0xFF == ord('q'):
  61.                 cv2.destroyAllWindows()
  62.                 break

3.自定义目标检测

本示例，我们将训练模型来识别吊车(crane)。首先可以先从百度上搜到一些吊车的图片样本。样本越多，模型识别当然越准确，本例只是演示，只下载十几张图片来说明。首先，进入models/research目录，本地安装object_detection的api:

  1. 进入models/research目录，执行python setup.py install 本地安装object_detection api库。

新建一个目录object_detection，然后我们将训练相关的内容都暂时放在这里面：
本部分全部完成以后目录结构如下：
image.png

images里面的图片可以选取感兴趣的需要分类的图片，比如吊车：

1.jpg

3.1 标注数据集

本文是基于有监督的分类学习算法，自然要对样本进行分类。图片下载好以后，我们要在图片上来标注哪个是吊车，也就是打上label。打label的工具用的是LabelImagelink。将标注的xml文件和图片保存到images目录。
安装labelimage这个工具的时候，你可能还需要安装一些依赖库：

  1. pip3 install sip
  2. apt-get install pyqt5-dev-tools
  3. pip3 install -r requirements/requirements-linux-python3.txt
  4. make qt5py3
  5. pip3 install pandas
  6. python3 labelImg.py
  8. pip3 install pycocotools

打开文件夹为图片打标签：
image.png

3.2 构建训练的tfrecord文件

3.2.1.下载工具xml_to_csv工具

下载 https://github.com/datitran/raccoon_dataset 里面的xml_to_csv.py文件，即将xml转化为csv文件的工具基础代码
简单改造一下，指定本地的data目录：

  1. import os
  2. import glob
  3. import pandas as pd
  4. import xml.etree.ElementTree as ET
  7. def xml_to_csv(path):
  8.     xml_list = []
  9.     for xml_file in glob.glob(path + '/*.xml'):
  10.         tree = ET.parse(xml_file)
  11.         root = tree.getroot()
  12.         for member in root.findall('object'):
  13.             value = (root.find('filename').text,
  14.                      int(root.find('size')[0].text),
  15.                      int(root.find('size')[1].text),
  16.                      member[0].text,
  17.                      int(member[4][0].text),
  18.                      int(member[4][1].text),
  19.                      int(member[4][2].text),
  20.                      int(member[4][3].text)
  21.                      )
  22.             xml_list.append(value)
  23.     column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']
  24.     xml_df = pd.DataFrame(xml_list, columns=column_name)
  25.     return xml_df
  28. def main():
  29.     for directory in ['train', 'test']:
  30.         image_path = os.path.join(os.getcwd(),'images/{}'.format(directory))
  31.         xml_df = xml_to_csv(image_path)
  32.         xml_df.to_csv('data/{}_labels.csv'.format(directory), index=None)
  33.         print('Successfully converted xml to csv.')
  36. main()

image.png

3.2.2 将csv文件转化为tfrecord文件

python3 generate_tfrecord.py —csv_input=data/test_labels.csv —output_path=data/test.record —image_dir=images/test

结果如下图：
image.png

3.3 模型训练

本部分，我们开始来训练自定义的模型算法。为此，我们需要图片训练集TFRecords和测试集TFRecords，其次我们需要模型配置文件和基础模型算法文件。这样，我们就可以利用tensorflow的转移学习基于一些预训练好的模型来继续训练出我们需要的算法(链接link)。转移学习可以快速得到训练算法，所需要的训练数据也相对较少。Tensorflow有不少预训练模型和模型对应的配置文件，本示例就选用了ssd_mobilenet_v1_coco_11_06_2017模型和相应的配置文件ssd_mobilenet_v1_pets.config：

3.3.1 预训练好的model模型文件

模型文件封装了识别用的基础算法，基于基础算法，我们利用object_detection api来训练我们感兴趣的算法模型 。直接下载模型：

  1. wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz

下载下来，解压到ssd_mobilenet_v1_coco_11_06_2017目录下：

3.3.2 模型配置文件 configuration file

  1. wget https://raw.githubusercontent.com/tensorflow/models/master/object_detection/samples/configs/ssd_mobilenet_v1_pets.config

将模型文件下载下来放到training目录下：

原版的配置文件中，首先要搜索PATH_TO_BE_CONFIGURED，并且修改掉。
还需要修改batch size，这里设为24，其他的模型可能需要不同大小的批处理大小。如果出现内存错误，你可能就需要减小batch size。最后，还需要修改checkpoint的路径，num_classes修改为1，num_examples修改为12，以及一个label_map_path: “training/object-detect.pbtxt”

  1. # SSD with Mobilenet v1, configured for Oxford-IIIT Pets Dataset.
  2. # Users should configure the fine_tune_checkpoint field in the train config as
  3. # well as the label_map_path and input_path fields in the train_input_reader and
  4. # eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
  5. # should be configured.
  7. model {
  8.   ssd {
  9.       # 将分类的数目设置为1
  10.     num_classes: 1
  11.     box_coder {
  12.       faster_rcnn_box_coder {
  13.         y_scale: 10.0
  14.         x_scale: 10.0
  15.         height_scale: 5.0
  16.         width_scale: 5.0
  17.       }
  18.     }
  19.     matcher {
  20.       argmax_matcher {
  21.         matched_threshold: 0.5
  22.         unmatched_threshold: 0.5
  23.         ignore_thresholds: false
  24.         negatives_lower_than_unmatched: true
  25.         force_match_for_each_row: true
  26.       }
  27.     }
  28.     similarity_calculator {
  29.       iou_similarity {
  30.       }
  31.     }
  32.     anchor_generator {
  33.       ssd_anchor_generator {
  34.         num_layers: 6
  35.         min_scale: 0.2
  36.         max_scale: 0.95
  37.         aspect_ratios: 1.0
  38.         aspect_ratios: 2.0
  39.         aspect_ratios: 0.5
  40.         aspect_ratios: 3.0
  41.         aspect_ratios: 0.3333
  42.       }
  43.     }
  44.     image_resizer {
  45.       fixed_shape_resizer {
  46.         height: 300
  47.         width: 300
  48.       }
  49.     }
  50.     box_predictor {
  51.       convolutional_box_predictor {
  52.         min_depth: 0
  53.         max_depth: 0
  54.         num_layers_before_predictor: 0
  55.         use_dropout: false
  56.         dropout_keep_probability: 0.8
  57.         kernel_size: 1
  58.         box_code_size: 4
  59.         apply_sigmoid_to_scores: false
  60.         conv_hyperparams {
  61.           activation: RELU_6,
  62.           regularizer {
  63.             l2_regularizer {
  64.               weight: 0.00004
  65.             }
  66.           }
  67.           initializer {
  68.             truncated_normal_initializer {
  69.               stddev: 0.03
  70.               mean: 0.0
  71.             }
  72.           }
  73.           batch_norm {
  74.             train: true,
  75.             scale: true,
  76.             center: true,
  77.             decay: 0.9997,
  78.             epsilon: 0.001,
  79.           }
  80.         }
  81.       }
  82.     }
  83.     feature_extractor {
  84.       type: 'ssd_mobilenet_v1'
  85.       min_depth: 16
  86.       depth_multiplier: 1.0
  87.       conv_hyperparams {
  88.         activation: RELU_6,
  89.         regularizer {
  90.           l2_regularizer {
  91.             weight: 0.00004
  92.           }
  93.         }
  94.         initializer {
  95.           truncated_normal_initializer {
  96.             stddev: 0.03
  97.             mean: 0.0
  98.           }
  99.         }
  100.         batch_norm {
  101.           train: true,
  102.           scale: true,
  103.           center: true,
  104.           decay: 0.9997,
  105.           epsilon: 0.001,
  106.         }
  107.       }
  108.     }
  109.     loss {
  110.       classification_loss {
  111.         weighted_sigmoid {
  112.         }
  113.       }
  114.       localization_loss {
  115.         weighted_smooth_l1 {
  116.         }
  117.       }
  118.       hard_example_miner {
  119.         num_hard_examples: 3000
  120.         iou_threshold: 0.99
  121.         loss_type: CLASSIFICATION
  122.         max_negatives_per_positive: 3
  123.         min_negatives_per_image: 0
  124.       }
  125.       classification_weight: 1.0
  126.       localization_weight: 1.0
  127.     }
  128.     normalize_loss_by_num_matches: true
  129.     post_processing {
  130.       batch_non_max_suppression {
  131.         score_threshold: 1e-8
  132.         iou_threshold: 0.6
  133.         max_detections_per_class: 100
  134.         max_total_detections: 100
  135.       }
  136.       score_converter: SIGMOID
  137.     }
  138.   }
  139. }
  141. train_config: {
  142.   batch_size: 24
  143.   optimizer {
  144.     rms_prop_optimizer: {
  145.       learning_rate: {
  146.         exponential_decay_learning_rate {
  147.           initial_learning_rate: 0.004
  148.           decay_steps: 800720
  149.           decay_factor: 0.95
  150.         }
  151.       }
  152.       momentum_optimizer_value: 0.9
  153.       decay: 0.9
  154.       epsilon: 1.0
  155.     }
  156.   }
  157.   fine_tune_checkpoint: "ssd_mobilenet_v1_coco_11_06_2017/model.ckpt"
  158.   from_detection_checkpoint: true
  159.   load_all_detection_checkpoint_vars: true
  160.   # Note: The below line limits the training process to 200K steps, which we
  161.   # empirically found to be sufficient enough to train the pets dataset. This
  162.   # effectively bypasses the learning rate schedule (the learning rate will
  163.   # never decay). Remove the below line to train indefinitely.
  164.   num_steps: 200000
  165.   data_augmentation_options {
  166.     random_horizontal_flip {
  167.     }
  168.   }
  169.   data_augmentation_options {
  170.     ssd_random_crop {
  171.     }
  172.   }
  173. }
  175. train_input_reader: {
  176.   tf_record_input_reader {
  177.     input_path: "data/train.record"
  178.   }
  179.   label_map_path: "training/object-detection.pbtxt"
  180. }
  182. eval_config: {
  183.   metrics_set: "coco_detection_metrics"
  184.   num_examples: 12
  185. }
  187. eval_input_reader: {
  188.   tf_record_input_reader {
  189.     input_path: "data/test.record"
  190.   }
  191.   label_map_path: "training/object-detection.pbtxt"
  192.   shuffle: false
  193.   num_readers: 1
  194. }

training目录下的object-detect.pbtxt的内容：

  1. item {
  2.   id: 1
  3.   name: 'macncheese'
  4. }

models/research目录下：

  1. sudo python3 setup.py build
  2. sudo python3 setup.py install
  3. export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

3.3.3 训练

接下来，激动人心的时刻到了，开始训练！

  1. python3 model_main.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config

见到如下画面说明成功训练中:

image.png
训练一段时间，输入如下:

image.png

从Tensorboard看一下训练结果的total_loss:
image.png

3.4 模型测试

本部分，我们将测试训练好的模型，来看看是否和预期的一致。首先我们要导出模型计算图graph文件，以便测试程序可以直接引入使用。我们可以利用object_detection里面的export_inference_graph.py，只需要传递checkpoint和pipeline config文件就可以导出graph文件了。

  1. python3 export_inference_graph.py \
  2.     --input_type image_tensor \
  3.     --pipeline_config_path training/ssd_mobilenet_v1_pets.config \
  4.     --trained_checkpoint_prefix training/model.ckpt-10856 \
  5.     --output_directory crane_inference_graph