一、TensorFlow Keras
二、TensorFlow基础API
metric使用
- 三、TensorFlow dataset
- 四、TensorFlow Estimator
对两个离散特征做笛卡尔积；交叉特征可以更细节地刻画样本
cross feature: age: [1,2,3,4,5], gender:[male, female]
age_x_gender: [(1, male), (2, male), …, (5, male), …, (5, female)]
100000: 100 -> hash(100000 values) % 100
- 附录

文 @nikkyoya(u2331987) 2021.1.14

一、TensorFlow Keras

Tf-keras简介

keras是什么

基于python的高级神经网络API
Francois Chollet于2014-2015年编写
以 Tensorflow、CTk或 Theano者为后端运行， keras必须有后端才可以运行
- 后端可以切换，现在多用 tensorflow
极方便于快速实验，帮助用户以最少的时间验证自己的想法

Tensorflow-keras是什么

Tensorflow对keras API规范的实现
相对于以 tensorflow为后端的keras，Tensorflow-keras与 Tensorflow结合更加紧密
实现在tf.keras空间下

Tf-keras和keras比较

联系

基于同一套API
- keras程序可以通过改导入方式轻松转为tf. keras程序
- 反之可能不成立，因为tf.keras有其他特性
相同的JSON和HDF5模型序列化格式和语义

区别

Tf.keras全面支持eager mode
- 只是用 keras. Sequentialkeras和. Model时没影响
- 自定义 Model内部运算逻辑的时候会有影响
  - Tf低层API可以 kerasmodel使用的.fit等抽象
  - 适用于研究人员
Tf.keras支持基于tf.data的模型训练
Tf.keras支持TPU训练
Tf.keras支持tf.distribution中的分布式策略
其他特性
- Tf.keras可以与Tensorflow中的estimator集成
- Tf.keras可以保存为SavedModel

如何选择

如果想用tf.keras的任何一个特性，那么选tf.keras
如果后端互换性很重要，那么选 keras
~~如果都不重要，那随便~~

Tf.keras实现分类模型和回归模型

（一）基础概念

1.分类问题与回归问题

分类问题预测的是类别，模型的输出是概率分布
- 什么是概率分布——三分类问题输出例子:［0.2, 0.7, 0.1] 类别即分类中概率最大者
- 为什么使模型输出是概率分布——目标函数
回归问题预测的是值，模型的输出是一个实数值

2.目标函数

为什么需要目标函数？
——机器学习对问题的建模特性决定

机器学习模型的参数是逐步调整的（区别于求解方程）
目标函数可以精确地帮助衡量模型的好坏
- 举例：分类问题
  - Model A：[0.1，0.4，0.5］
  - Model B：[0.1，0.2，0.7]
  - 准确率指标：只有两个状态（对/错），Model A和B没有区别

如何用目标函数求解模型

模型的训练就是调整参数，使得目标函数逐渐变小的过程

3.分类问题的目标函数

需要衡量目标类别与当前预测分布的差距
- 三分类问题输出例子:［0.2，0.7，0.1]
- 三分类真实类别：2->one＿hot->［0, 0, 1] 类别离散值→类别分布
One-hot编码，把正整数变为向量表达
- 生成一个长度不小于正整数（对分类问题即类别个数）的向量，只有正整数的位置处为1，其余位置都为0

4.分类问题的常用损失函数

平方差损失

TensorFlow基础组件 - 图1
举例
预测值： TensorFlow基础组件 - 图2
真实值： TensorFlow基础组件 - 图3
损失函数值= TensorFlow基础组件 - 图4
n为样本数，乘 TensorFlow基础组件 - 图5 →求导时可以抵偿平方系数

交叉熵损失

TensorFlow基础组件 - 图6

5.回归问题的目标函数

预测值与真实值的差距

常用损失函数

平方差损失
绝对值损失

（二）Tf.keras搭建分类模型

分类模型搭建流程

（0）数据准备

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf
from tensorflow import keras
# 获取所有GPU组成list
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only use the first GPU
    try:
        # 设置按需申请
        tf.config.experimental.set_memory_growth(gpus[0], True)
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU")
    except RuntimeError as e:
            # Visible devices must be set before GPUs have been initialized
        print(e)
 # 划分数据集
fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

（1）基于fashion_mnist数据集的分类模型

API文档：Tf.keras.Sequential

# 构建图
model = keras.models.Sequential([
    keras.layers.Flatten(input_shape=[28, 28]),
    keras.layers.Dense(300, activation='relu'),
    keras.layers.Dense(100, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])
model.compile(loss="sparse_categorical_crossentropy",
              optimizer = keras.optimizers.SGD(0.001),
              metrics = ["accuracy"])
# 训练模型
history = model.fit(x_train_scaled, y_train, epochs=10,
                    validation_data=(x_valid_scaled, y_valid), 
                    callbacks = callbacks)

（2）训练数据集的归一化

# x = (x - u) / std
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# x_train: [None, 28, 28] -> [None, 784] 
x_train_scaled = scaler.fit_transform(
    x_train.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)
x_valid_scaled = scaler.transform(
    x_valid.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)
x_test_scaled = scaler.transform(
    x_test.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

（3）回调函数：对训练过程进行监听

API文档：Tf.keras.callbacks

Callbacks: utilities called at certain points during model training.

三个重要的回调函数

EarlyStopping：Stop training when a monitored metric has stopped improving.
- 当loss不再下降时提前停止训练
ModelCheckpoint：Callback to save the Keras model or model weights at some frequency.
- 每隔段时间保存参数的训练状态
TensorBoard：Enable visualizations for TensorBoard.
- 对模型训练过程中指标随着迭代次数推进的变化趋势的自动化可视工具

用 tensorboard 模块进行可视化

打开Terminal ，切换到tensorflow-gpu环境，将路径定位到Tensorboard文件夹（此处是callbacks）的上一层，输入

tensorboard --logdir=callbacks

本地TensorBoard服务器
再通过浏览器访问即可

SCALARS：指标随迭代次数的变化趋势

GRAPH：模型图结构

TIME SERIES：指标随时间的变化趋势

（三）Tf.keras搭建回归模型

回归模型的搭建流程同分类模型，主要区别：1、损失函数不同（均方差）；2、图输出层是一维的（结果是离散值）

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf
from tensorflow import keras
#下载数据集
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
#划分数据集
from sklearn.model_selection import train_test_split
    # 拆分训练集和测试集，test_size默认0.25（3:1）
x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state = 7) 
    # 拆分训练集和验证集
x_train, x_valid, y_train, y_valid = train_test_split(
    x_train_all, y_train_all, random_state = 11)
#归一化
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x_train_scaled = scaler.fit_transform(x_train) 
x_valid_scaled = scaler.transform(x_valid) 
x_test_scaled = scaler.transform(x_test) 
#构建模型图结构
model = keras.models.Sequential([
    keras.layers.Dense(30, activation='relu',
                       input_shape=x_train.shape[1:]),
    keras.layers.Dense(1),
])
model.summary()
#编译模型
model.compile(loss="mean_squared_error", optimizer="sgd")
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]
#训练模型
history = model.fit(x_train_scaled, y_train,
                    validation_data = (x_valid_scaled, y_valid),
                    epochs = 100,
                    callbacks = callbacks)
# 绘制学习曲线
def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 1)
    plt.show()
plot_learning_curves(history)
# 用模型预测
model.evaluate(x_test_scaled, y_test, verbose=0)

Tf.keras实现深度神经网络

（一）基础概念

1.神经网络

神经网络正向计算
keras.layers.Dense

神经网络训练
optimizer（sgd）
下山算法——梯度下降

求导（找到方形）
更新参数（走一步）

深度神经网络——层次非常深的神经网络

2.激活函数

为什么要用激活函数？

3.归一化与批归一化

归一化

Min-Max归一化

TensorFlow基础组件 - 图13

Z-score归一化

TensorFlow基础组件 - 图14
批归一化
（将归一化的思想从输入数据扩展到网络每层的激活值上）

每层激活值都做归一化

为什么归一化会有效？

目标函数等高线（未归一化）：椭圆

目标函数等高线（归一化）：正圆
——加快训练速度

4.Dropout

（随机）弃用部分神经单元

正常全连接的神经网络→子网络

作用

防止过拟合
- 过拟合：训练集上很好，但测试集上不好
- 原因：参数太多，记住样本，不能泛化
- 随机弃用：防止任意两结点的组合学习
  
  Dropout为什么有效？

（二）Tf.keras实现深度神经网络

keras实现DNN

# tf.keras.models.Sequential()

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28, 28]))
for _ in range(20):
    model.add(keras.layers.Dense(100, activation="relu"))
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",
              optimizer = keras.optimizers.SGD(0.01),
              metrics = ["accuracy"])

DNN的问题：
# 1. 参数众多，训练不充分
# 2. 梯度消失 -> 链式法则 -> 复合函数f(g(x))

keras实现批归一化

批归一化能缓解DNN的梯度消失问题

# tf.keras.models.Sequential()

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28, 28]))
for _ in range(20):
# 批归一化在激活函数之后
    model.add(keras.layers.Dense(100, activation="relu"))
    model.add(keras.layers.BatchNormalization()) 
# 批归一化在激活函数之前
    """
    model.add(keras.layers.Dense(100))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Activation('relu'))
    """
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",
              optimizer = keras.optimizers.SGD(0.01),
              metrics = ["accuracy"])

keras更改激活函数

# tf.keras.models.Sequential()

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28, 28]))
for _ in range(20):
    model.add(keras.layers.Dense(100, activation="selu"))
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",
              optimizer = keras.optimizers.SGD(0.01),
              metrics = ["accuracy"])

selu：自带归一化的激活函数，能一定程度上缓解DNN的梯度消失问题

selu激活函数的机制？

keras实现dropout

# tf.keras.models.Sequential()

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28, 28]))
for _ in range(20):
    model.add(keras.layers.Dense(100, activation="selu"))
# 一般情况下，只给最后几层添加dropout，用来防止过拟合
# rate 丢弃的单元数目比例
model.add(keras.layers.AlphaDropout(rate=0.5))
# AlphaDropout: 1. 均值和方差不变 2. 归一化性质也不变
# model.add(keras.layers.Dropout(rate=0.5)) 普通的dropout
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",
              optimizer = keras.optimizers.SGD(0.001),
              metrics = ["accuracy"])

AlphaDropout的Dropout实现方法?

Tf.keras实现Wide&deep模型

（一）基础知识

Google于16年发布，用于分类和回归
应用到了Google Play中的应用推荐
原始论文

将数据特征用稀疏特征与密集特征表示

稀疏特征

离散值特征（类别特征）
用One-hot编码表示

e.g. 专业={计算机, 人文, 其他}. 人文=[0,1,0]
e.g. 词表={你, 我, 他, …}, 他=[0,0,1,0,..,0]

稀疏特征的叉乘
稀疏特征之间可做叉乘（组合）
叉乘={(计算机,人工智能),(计算机,你),…}
稀疏特征的叉乘可以刻画样本
叉乘之后

稀疏特征做叉乘获取共现信息
实现记忆的效果

优缺点

优点
- 有效（重复样本），广泛用于工业界（广告点击率预估，推荐算法）
缺点
- 需要人工设计（选择部分特征做叉乘）
- 可能过拟合，所有特征都叉乘，相当于记住每一个样本
  - 现实中一个样本的所有信息很难全部用离散特征表达，即使可以离散特征也可能有意义上的重合（词表近义词）
- 泛化能力差，没出现过就不会起效果

密集特征

向量表达
- e.g. 词表={你, 我, 他, …}, 他=[0.3,0.2,0.6,(n维向量)]
典型应用：Word2vec工具
- 预训练好的将词转化成向量的工具
- 用向量间距离衡量词间距离
  - 男-女=国王-王后
优缺点
- 优点
  - 带有语义信息，不同向量之间有相关性
  - 兼容没有出现过的特征组合
  - 更少人工参与
- 缺点
  - 过度泛化，推荐不怎么相关的产品

Wide&deep v.s. Wide

Wide模型**

只有一层
所有输入直接连接到输出
输入为稀疏特征（One-hot表达）

Wide&deep模型

左半=Wide模型
右半=Deep模型
- 多层神经网络
- 输入数据（稀疏特征）→密集向量表达（密集特征）→多层全连接NN（深度模型隐含层：除输入输出外的中间层）→输出

google play应用商店推荐算法模型

（二）Tf.keras实现Wide&deep模型

Tf.keras函数式API实现Wide&deep模型

wide&deep模型不再是严格层级结构，而是由两部分不同的层级结构组成：
Sequential→函数式API（功能API）

# 函数式API 功能API
input = keras.layers.Input(shape=x_train.shape[1:])
# 两层NN实现deep model
# 像使用函数一样使用定义好的层次，类复合函数: f(x) = h(g(x))
hidden1 = keras.layers.Dense(30, activation='relu')(input)
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)

# deep模型输出与wide模型输入合并，这里假设wide模型和deep模型的输入是一样的
concat = keras.layers.concatenate([input, hidden2])
output = keras.layers.Dense(1)(concat)

#固化模型
model = keras.models.Model(inputs = [input],
                           outputs = [output])

model.summary()
model.compile(loss="mean_squared_error",
              optimizer = keras.optimizers.SGD(0.001))
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

Tf.keras子类API实现Wide&deep模型

# 子类API
class WideDeepModel(keras.models.Model):
    def __init__(self):
        super(WideDeepModel, self).__init__()
        """定义模型的层次"""
        self.hidden1_layer = keras.layers.Dense(30, activation='relu')
        self.hidden2_layer = keras.layers.Dense(30, activation='relu')
        self.output_layer = keras.layers.Dense(1)

    def call(self, input):
        """完成模型的正向计算"""
        hidden1 = self.hidden1_layer(input)
        hidden2 = self.hidden2_layer(hidden1)
        concat = keras.layers.concatenate([input, hidden2])
        output = self.output_layer(concat)
        return output

# model = WideDeepModel()
model = keras.models.Sequential([
    WideDeepModel(),
])

model.build(input_shape=(None, 8))

model.summary()
model.compile(loss="mean_squared_error",
              optimizer = keras.optimizers.SGD(0.001))
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

多输入与多输出

多输入

# 多输入
input_wide = keras.layers.Input(shape=[5]) #前5个feature作为wide模型输入
input_deep = keras.layers.Input(shape=[6]) #后6个feature作为deep模型输入
hidden1 = keras.layers.Dense(30, activation='relu')(input_deep)
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)
concat = keras.layers.concatenate([input_wide, hidden2])
output = keras.layers.Dense(1)(concat)
model = keras.models.Model(inputs = [input_wide, input_deep],
                           outputs = [output])

model.summary()
model.compile(loss="mean_squared_error",
              optimizer = keras.optimizers.SGD(0.001))
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

模型结构修改

#训练数据拆分
x_train_scaled_wide = x_train_scaled[:, :5]
x_train_scaled_deep = x_train_scaled[:, 2:]

x_valid_scaled_wide = x_valid_scaled[:, :5]
x_valid_scaled_deep = x_valid_scaled[:, 2:]

x_test_scaled_wide = x_test_scaled[:, :5]
x_test_scaled_deep = x_test_scaled[:, 2:]

history = model.fit([x_train_scaled_wide, x_train_scaled_deep], 
                    y_train,
                    validation_data = (
                        [x_valid_scaled_wide, x_valid_scaled_deep],
                        y_valid),
                    epochs = 100,
                    callbacks = callbacks)

训练数据拆分和传入数据修改

model.evaluate([x_test_scaled_wide, x_test_scaled_deep], y_test, verbose=0)

模型预测传入数据修改

多输出
（并不是wide&deep模型所需要的，主要针对多任务学习问题）
房价预测：当前房价与一年后房价（两个预测任务）

# 多输入
input_wide = keras.layers.Input(shape=[5]) #前5个feature作为wide模型输入
input_deep = keras.layers.Input(shape=[6]) #后6个feature作为deep模型输入
hidden1 = keras.layers.Dense(30, activation='relu')(input_deep)
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)
concat = keras.layers.concatenate([input_wide, hidden2])
output = keras.layers.Dense(1)(concat)
output2 = keras.layers.Dense(1)(hideen2)
model = keras.models.Model(inputs = [input_wide, input_deep],
                           outputs = [output, putput2])

model.summary()
model.compile(loss="mean_squared_error",
              optimizer = keras.optimizers.SGD(0.001))
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

模型结构修改

#训练数据拆分
x_train_scaled_wide = x_train_scaled[:, :5]
x_train_scaled_deep = x_train_scaled[:, 2:]

x_valid_scaled_wide = x_valid_scaled[:, :5]
x_valid_scaled_deep = x_valid_scaled[:, 2:]

x_test_scaled_wide = x_test_scaled[:, :5]
x_test_scaled_deep = x_test_scaled[:, 2:]

history = model.fit([x_train_scaled_wide, x_train_scaled_deep], 
                    [y_train, y_train],
                    validation_data = (
                        [x_valid_scaled_wide, x_valid_scaled_deep],
                        [y_valid, y_valid]),
                    epochs = 100,
                    callbacks = callbacks)

传入数据修改

model.evaluate([x_test_scaled_wide, x_test_scaled_deep], [y_test, y_test], verbose=0)

模型预测传入数据修改

超参数搜索

神经网络有很多训练过程中不变的参数（超参数）
- 网络结构参数：几层，每层宽度，每层激活函数等
- 训练参数：batch_size（在一次训练中传入NN的训练集中数据量），学习率（梯度下降中在导数方向前进多少），学习率衰减算法（学习率的变化策略）等
手工去试耗费人力

（一）搜索策略

网格搜索

将超参数离散化形成组合，再一一尝试

定义n维方格
每个方格对应一组超参数
一组一组参数尝试

——暴力，但可以并行化运算

随机搜索

网格搜索缺点：只能取固定离散值，可能难以找到中间的最优解
随机搜索：可能覆盖到最优解，但随机点数可能很多

参数的生成方式为随机
可探索的空间更大

遗传算法搜索
- 对自然界的模拟
- A.初始化候选参数集合->训练->得到模型指标作为生存概率
- B.选择->交叉->变异->产生下一代集合
- C.重新到A
启发式搜索
- 研究热点-AutoML
- 使用循环神经网络来生成参数
- 使用强化学习来进行反馈，使用模型来训练生成参数

（二）超参数搜索实现

手动实现超参数搜索

# learning_rate: [1e-4, 3e-4, 1e-3, 3e-3, 1e-2, 3e-2]用于参数更新
# W = W + grad * learning_rate

learning_rates = [1e-4, 3e-4, 1e-3, 3e-3, 1e-2, 3e-2]
histories = []
for lr in learning_rates:
    model = keras.models.Sequential([
        keras.layers.Dense(30, activation='relu',
                           input_shape=x_train.shape[1:]),
        keras.layers.Dense(1),
    ])
#model.summary()
    optimizer = keras.optimizers.SGD(lr)
    model.compile(loss="mean_squared_error", optimizer=optimizer)
    callbacks = [keras.callbacks.EarlyStopping(
        patience=5, min_delta=1e-2)]
    history = model.fit(x_train_scaled, y_train,
                        validation_data = (x_valid_scaled, y_valid),
                        epochs = 100,
                        callbacks = callbacks)
    histories.append(history)

手动设置超参数（Learning Rate）

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 1)
    plt.show()
for lr, history in zip(learning_rates, histories):
    print("Learning rate: ", lr)
    plot_learning_curves(history)

绘制每个Learning Rate下的学习曲线

各Learning Rate下的学习曲线
手动实现的问题

一个超参数就需要一层for循环；
for循环不能并行化处理，复杂度较大；

sklearn封装keras模型

API文档：tf.keras.wrappers.scikit_learn

# RandomizedSearchCV
# 1. 转化为sklearn的model
# 2. 定义参数集合
# 3. 搜索参数

#tf.keras.wrappers

def build_model(hidden_layers = 1,
                layer_size = 30,
                learning_rate = 3e-3):
    model = keras.models.Sequential()
    model.add(keras.layers.Dense(layer_size, activation='relu',
                                 input_shape=x_train.shape[1:]))
    for _ in range(hidden_layers - 1):
        model.add(keras.layers.Dense(layer_size,
                                     activation = 'relu'))
    model.add(keras.layers.Dense(1))
    optimizer = keras.optimizers.SGD(learning_rate)
    model.compile(loss = 'mse', optimizer = optimizer)
    return model

sklearn_model = tf.keras.wrappers.scikit_learn.KerasRegressor(
    build_fn = build_model)
callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-2)]
history = sklearn_model.fit(x_train_scaled, y_train,
                            epochs = 10,
                            validation_data = (x_valid_scaled, y_valid),
                            callbacks = callbacks)

转化为sklearn的model

from scipy.stats import reciprocal
# f(x) = 1/(x*log(b/a)) , a <= x <= b

param_distribution = {
    "hidden_layers":[1, 2, 3, 4],
    "layer_size": list(np.arange(1, 100)),
    "learning_rate": reciprocal(1e-4, 1e-2),
}

定义参数集合


from sklearn.model_selection import RandomizedSearchCV

random_search_cv = RandomizedSearchCV(sklearn_model,
                                      param_distribution,
                                      n_iter = 10, #从参数集合中选取多少组
                                      #cv = 3,
                                      n_jobs = 1) #并行任务数
random_search_cv.fit(x_train_scaled, y_train, epochs = 100,
                     validation_data = (x_valid_scaled, y_valid),
                     callbacks = callbacks)

# cross_validation机制: 训练集分成n份，n-1训练，最后一份验证.
# n默认值为3，可通过修改参数cv的值来修改它

搜索超参数

print(random_search_cv.best_params_) #最好的参数
print(random_search_cv.best_score_) #最好的分值
print(random_search_cv.best_estimator_) #最好的模型

查看搜索结果

model = random_search_cv.best_estimator_.model
model.evaluate(x_test_scaled, y_test)

用最佳模型预测

二、TensorFlow基础API

基础API
基础API与keras的集成
- 自定义损失函数
- 自定义层次
@tf.function
- 将普通python代码转化成图结构
自定义求导

@tf.function

将python函数编译成图
易于将模型导出成为GraphDef+checkpoint或者SavedModel
使得eager execution可以默认打开
1.0代码可以通过tf.function来继续在2.0里使用
- 替代session

API列表

基础数据类型
- tf.constant, tf.string
- tf.ragged.constant, tf.SparseTensor, tf.Variable
自定义损失函数——tf.reduce_mean
自定义层次——Keras.layers.Lambda和继承法
tf.function
- tf.function， tf.autograph.to_code, get_concrete_function
GrachDef
- det_operations, get_operation_by_name
- get_ttensor_by_name, as_graph_def
自动求导
- tf.GradientTape
- Optimizer.apply_gradients

（一）基础API

tf.constant

t = tf.constant([[1., 2., 3.], [4., 5., 6.]])

索引

# index
print(t)
print(t[:, 1:])
print(t[..., 1

tf.Tensor( [[1. 2. 3.] [4. 5. 6.]], shape=(2, 3), dtype=float32) tf.Tensor( [[2. 3.] [5. 6.]], shape=(2, 2), dtype=float32) tf.Tensor([2. 5.], shape=(2,), dtype=float32)

操作

# ops
print(t+10)
print(tf.square(t)) #平方
print(t @ tf.transpose(t)) #乘自身转置

tf.Tensor( [[11. 12. 13.] [14. 15. 16.]], shape=(2, 3), dtype=float32) tf.Tensor( [[ 1. 4. 9.] [16. 25. 36.]], shape=(2, 3), dtype=float32) tf.Tensor( [[14. 32.] [32. 77.]], shape=(2, 2), dtype=float32)

与numpy转化

# numpy conversion
print(t.numpy())
    #作为numpy的输入
print(np.square(t)) 
    # 将np对象转成tf.tensor
np_t = np.array([[1., 2., 3.], [4., 5., 6.]]) 
print(tf.constant(np_t))

[[1. 2. 3.] [4. 5. 6.]] [[ 1. 4. 9.] [16. 25. 36.]] tf.Tensor( [[1. 2. 3.] [4. 5. 6.]], shape=(2, 3), dtype=float64)

# Scalars 0维向量（数）
t = tf.constant(2.718)
print(t.numpy())
print(t.shape)

2.718 ()

tf.string

单字符

# strings
t = tf.constant("cafe")
print(t)
print(tf.strings.length(t))
print(tf.strings.length(t, unit="UTF8_CHAR"))
print(tf.strings.unicode_decode(t, "UTF8"))

tf.Tensor(b’cafe’, shape=(), dtype=string) tf.Tensor(4, shape=(), dtype=int32) tf.Tensor(4, shape=(), dtype=int32) tf.Tensor([ 99 97 102 101], shape=(4,), dtype=int32)

字符数组

# string array
t = tf.constant(["cafe", "coffee", "咖啡"])
print(tf.strings.length(t, unit="UTF8_CHAR"))
r = tf.strings.unicode_decode(t, "UTF8")
print(r)

tf.Tensor([4 6 2], shape=(3,), dtype=int32)

RaggedTensor

不完整的n维矩阵（每维长度不同）

# ragged tensor
r = tf.ragged.constant([[11, 12], [21, 22, 23], [], [41]])
# index op
print(r)
print(r[1])
print(r[1:2])

tf.Tensor([21 22 23], shape=(3,), dtype=int32)

# ops on ragged tensor
r2 = tf.ragged.constant([[51, 52], [], [71]])
print(tf.concat([r, r2], axis = 0)) #拼接操作，按行拼接

r3 = tf.ragged.constant([[13, 14], [15], [], [42, 43]])
print(tf.concat([r, r3], axis = 1)) #r,r3需要行数相同，按列拼接

print(r.to_tensor()) #转为普通tensor

tf.Tensor( [[11 12 0] [21 22 23] [ 0 0 0] [41 0 0]], shape=(4, 3), dtype=int32) #RaggedTensor，所有0值在正常值之后

SparseTensor

# sparse tensor
s = tf.SparseTensor(indices = [[0, 1], [1, 0], [2, 3]], #正常值indice，定义时indices必须是排好序的
                    values = [1., 2., 3.], #位置上的值
                    dense_shape = [3, 4]) #矩阵大小
print(s)
print(tf.sparse.to_dense(s)) #转为正常矩阵

SparseTensor(indices=tf.Tensor( [[0 1] [1 0] [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64)) tf.Tensor( [[0. 1. 0. 0.] [2. 0. 0. 0.] [0. 0. 0. 3.]], shape=(3, 4), dtype=float32)

# ops on sparse tensors

s2 = s * 2.0
print(s2)

# 不能执行加法
try:
    s3 = s + 1
except TypeError as ex:
    print(ex)

s4 = tf.constant([[10., 20.],
                  [30., 40.],
                  [50., 60.],
                  [70., 80.]])
print(tf.sparse.sparse_dense_matmul(s, s4))

SparseTensor(indices=tf.Tensor( [[0 1] [1 0] [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([2. 4. 6.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64)) unsupported operand type(s) for +: ‘SparseTensor’ and ‘int’ tf.Tensor( [[ 30. 40.] [ 20. 40.] [210. 240.]], shape=(3, 2), dtype=float32)

# sparse tensor
s5 = tf.SparseTensor(indices = [[0, 2], [0, 1], [2, 3]],
                    values = [1., 2., 3.],
                    dense_shape = [3, 4])
print(s5)
s6 = tf.sparse.reorder(s5)
print(tf.sparse.to_dense(s6))

SparseTensor(indices=tf.Tensor( [[0 2] [0 1] [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64)) tf.Tensor( [[0. 2. 1. 0.] [0. 0. 0. 0.] [0. 0. 0. 3.]], shape=(3, 4), dtype=float32)

Variables

# Variables
v = tf.Variable([[1., 2., 3.], [4., 5., 6.]])
print(v)
print(v.value())
print(v.numpy())

tf.Tensor( [[1. 2. 3.] [4. 5. 6.]], shape=(2, 3), dtype=float32) [[1. 2. 3.] [4. 5. 6.]]

# assign value 变量的重新赋值
v.assign(2*v)
print(v.numpy())
v[0, 1].assign(42) # 对位置重新赋值
print(v.numpy())
v[1].assign([7., 8., 9.]) #对行重新赋值
print(v.numpy())

[[ 2. 4. 6.] [ 8. 10. 12.]] [[ 2. 42. 6.] [ 8. 10. 12.]] [[ 2. 42. 6.] [ 7. 8. 9.]]

try:
    v[1] = [7., 8., 9.] # =不能重新赋值，必须用assign函数
except TypeError as ex:
    print(ex)

‘ResourceVariable’ object does not support item assignment

（二）基础API与keras的集成

1 自定义损失函数

def customized_mse(y_true, y_pred):
    return tf.reduce_mean(tf.square(y_pred - y_true))

model = keras.models.Sequential([
    keras.layers.Dense(30, activation='relu',
                       input_shape=x_train.shape[1:]),
    keras.layers.Dense(1),
])
model.summary()
model.compile(loss=customized_mse, optimizer="sgd",
              metrics=["mean_squared_error"])
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

2 自定义层次

layer = tf.keras.layers.Dense(100) #全连接层的神经单元数
layer = tf.keras.layers.Dense(100, input_shape=(None, 5)) #input_shape一般在第一层指定dense_layer输入（样本数*5，None：不定值），输出100（None*100矩阵）
layer(tf.zeros([10, 5])) #像使用函数一样使用layer,tf2.*限定

获得layer中包含的所有参数

layer.variables

[<tf.Variable ‘dense_1/kernel:0’ shape=(5, 100) dtype=float32, numpy=

array([[ 0.15976147, -0.21502799, -0.07832763, -0.07678932, 0.15619548,

     -0.13655013,  0.20808806,  0.01320128, -0.05075614, -0.12650774,

      0.17424123,  0.15503065, -0.01738872,  0.17415772,  0.14157884,

      0.12198548, -0.13682888, -0.00431134, -0.11316137, -0.0941491 ,

      0.19477858, -0.15650672, -0.04523107,  0.11980413,  0.0392478 ,

     -0.03217201, -0.05695881, -0.09442574, -0.1769021 ,  0.13404123,

      0.22155072,  0.00532427, -0.20503883, -0.07037227,  0.16178603,

      0.15753557,  0.20727096,  0.23379399,  0.04048635, -0.23585105,

     -0.10203241,  0.06950574, -0.03137924, -0.19482769,  0.08389793,

     -0.0394402 , -0.00789529, -0.03464368, -0.12009872, -0.10856129,

     -0.00894293, -0.15162197, -0.032401  , -0.00453915,  0.05632652,

      0.17267881, -0.06191218, -0.05819538,  0.17946924, -0.15665913,

     -0.16304854,  0.00419821, -0.23776583, -0.16124432, -0.09681283,

     -0.21562152,  0.06472899,  0.22464116, -0.00569429,  0.08298178,

      0.11549039,  0.05924587,  0.03496118, -0.1051037 , -0.0621435 ,

      0.03481527, -0.07776555, -0.20981023,  0.0910445 , -0.0354397 ,

      0.16490476, -0.00227122,  0.0864424 , -0.13491175, -0.1645971 ,

      0.01050323,  0.14815761, -0.16781896,  0.13963409, -0.18121573,

     -0.23153992, -0.12473008,  0.1756901 ,  0.22858872,  0.13873275,

     -0.07768981, -0.00570698, -0.21528007,  0.00757009,  0.18363057],

    [ 0.11877824,  0.1168793 , -0.16063112, -0.17231184, -0.13409498,

      0.19905145, -0.2014717 ,  0.06295772, -0.04477178,  0.13132559,

      0.02939065,  0.03398912, -0.18317087, -0.03417668, -0.0107834 ,

      0.23002048,  0.06328867,  0.22760455, -0.19573069,  0.1198089 ,

     -0.1555649 , -0.02448821, -0.1734567 ,  0.12476839,  0.06721567,

     -0.1859357 ,  0.16312553, -0.1407745 ,  0.04455628, -0.22905497,

     -0.19142778, -0.22951172,  0.12258504,  0.1085989 ,  0.11974679,

     -0.03365536,  0.04648311, -0.03597827, -0.17160097, -0.12544727,

      0.06869908, -0.19143125,  0.22244458,  0.01437087, -0.03775081,

      0.12372582, -0.12253619,  0.11636381,  0.0799488 ,  0.12603168,

      0.03701521, -0.08781768, -0.11272953, -0.19216225,  0.13539644,

      0.20037527,  0.01965778, -0.03328758,  0.19321935,  0.08564301,

     -0.08135879,  0.19894253,  0.05638538, -0.23345044, -0.03519736,

      0.08768143, -0.08977255,  0.19856672,  0.21901618, -0.17389253,

      0.08617128, -0.14592302, -0.08758481,  0.13933013, -0.10471724,

     -0.05098224,  0.14235462,  0.15303789, -0.23809639, -0.08832294,

      0.02031069,  0.0559523 , -0.12055045,  0.06665261,  0.0479513 ,

     -0.08828138,  0.20095529,  0.02761547,  0.03795286, -0.08495584,

     -0.20158786,  0.02046029,  0.00867324,  0.11415212,  0.07045172,

     -0.15951589, -0.04511054,  0.10843597,  0.0239722 ,  0.19611128],

    [ 0.06575288,  0.2319472 ,  0.02630915,  0.06555082,  0.10341518,

     -0.22588201, -0.09562983, -0.06301573,  0.0936505 , -0.01802544,

      0.10455464, -0.22898191,  0.15039228, -0.16919808,  0.1916085 ,

     -0.12750028, -0.01717289,  0.20635293,  0.03781982, -0.14620668,

     -0.04488444, -0.06710777, -0.05447578, -0.04199792,  0.06323545,

     -0.17838557, -0.06185564, -0.06520277,  0.03831242, -0.03136453,

      0.01833086,  0.04521824,  0.12240265, -0.22481441, -0.23007765,

     -0.02078243, -0.2205326 ,  0.12952165,  0.02959962, -0.00605129,

     -0.02509233, -0.01998001,  0.21180545, -0.17241916, -0.03981026,

      0.21422301,  0.1693532 , -0.16202216,  0.07298566, -0.02760105,

      0.06959338, -0.21102753, -0.08139749,  0.04676224, -0.05416957,

      0.04917015, -0.01942708,  0.00196809,  0.12898101,  0.10945933,

     -0.00723343, -0.09607568, -0.10398597,  0.17198323,  0.03353049,

     -0.22060822, -0.05254391,  0.1702088 , -0.06497537,  0.06605254,

     -0.10460377,  0.01676236, -0.12414208, -0.04885554, -0.06743388,

     -0.10943221, -0.10569546, -0.20162518, -0.18120456,  0.23769595,

      0.01256581, -0.15475243,  0.16375937, -0.04159869, -0.14625306,

      0.02401985, -0.03575583, -0.04931861,  0.19226141, -0.22309631,

     -0.20494531,  0.11331509,  0.17416649,  0.06172945, -0.09567474,

     -0.09036852,  0.23214902,  0.0489447 ,  0.09294797,  0.15390666],

    [-0.17737857, -0.18906884,  0.20800714,  0.18031667, -0.04482706,

      0.11783125,  0.23290856,  0.13584308, -0.2087704 ,  0.03467382,

     -0.1224075 , -0.0173824 ,  0.08969595,  0.11672856, -0.16228962,

      0.08299066,  0.1263643 ,  0.20883657,  0.11351739, -0.11898713,

     -0.01544076, -0.10768828, -0.12749064, -0.21114449,  0.11841713,

     -0.12943742,  0.21584527,  0.15418832,  0.18037207,  0.13766332,

      0.21342348, -0.21019743,  0.134468  ,  0.17600726, -0.06488413,

      0.21859638, -0.20597608, -0.18852752,  0.06003846, -0.18052879,

     -0.21775812,  0.01654042,  0.14494626, -0.0470764 ,  0.19375344,

     -0.08053894, -0.08301774,  0.18195082, -0.10353127,  0.1721399 ,

      0.22610258,  0.15497144, -0.155101  ,  0.22665949, -0.21150993,

      0.13490684, -0.1813324 , -0.05590421,  0.20388336, -0.14803404,

     -0.12166522, -0.03127021,  0.01856966,  0.12462242, -0.18116277,

      0.07210417,  0.1368192 ,  0.09557848,  0.20420377,  0.22029643,

     -0.04349463,  0.20889862, -0.09501664, -0.20409772,  0.22081412,

     -0.19286382, -0.13874817, -0.14168417, -0.01535691, -0.06024285,

     -0.20417163,  0.08094741, -0.16762799, -0.03637671,  0.02360807,

      0.05724873,  0.17640142,  0.23869203, -0.09119059, -0.10150877,

      0.00279078, -0.13638958, -0.17503062, -0.04303041,  0.21908353,

      0.02966647,  0.11748509,  0.00320597, -0.1525138 , -0.09930906],

    [ 0.14521824,  0.00987744, -0.06205305,  0.07410519,  0.12307341,

      0.10008018,  0.18523215, -0.04782329, -0.0641716 , -0.22103709,

      0.07627855, -0.01990074,  0.14073421, -0.16386804, -0.02798302,

      0.17231806,  0.11191709,  0.12163399, -0.03887653,  0.09699525,

     -0.23051292,  0.07509579, -0.19534229,  0.18384896, -0.00816447,

      0.21739645, -0.03420636,  0.03474154,  0.04589911, -0.05675083,

      0.00387518,  0.10147156,  0.16241352,  0.2050067 ,  0.09604742,

      0.14956607,  0.213484  ,  0.06119736,  0.06285249,  0.02077337,

      0.13911705, -0.10113329, -0.1210945 , -0.13392931, -0.18992984,

     -0.01247907, -0.11904646, -0.2268505 , -0.23361304, -0.09504554,

     -0.15068164,  0.23496388,  0.08663981,  0.16571404,  0.23014064,

     -0.08652052,  0.09993   ,  0.08749835,  0.06988804, -0.0851372 ,

      0.22989945,  0.160183  ,  0.23885925, -0.02504036, -0.10181008,

      0.2021188 , -0.21528782, -0.2222352 , -0.15230861, -0.00771257,

      0.22475721, -0.12518431, -0.01185596,  0.05325596,  0.20660283,

      0.12990983, -0.04936986,  0.01129554, -0.2356372 ,  0.1903253 ,

     -0.2333641 ,  0.18106572,  0.11063577, -0.05270833,  0.22134696,

     -0.15732771,  0.10120948, -0.05766672, -0.10450801,  0.19721611,

      0.14783986,  0.21742307,  0.02052437, -0.22242983, -0.07544668,

     -0.15735045, -0.03890607,  0.00672534, -0.18067765, -0.06355779]],

   dtype=float32)>,

<tf.Variable ‘dense_1/bias:0’ shape=(100,) dtype=float32, numpy=

array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],

   dtype=float32)>]

dense_layer

x * w（矩阵w，即此处的kernal）+ b（此处的dense）

获得layer中所有可训练的变量

layer.trainable_variables

[<tf.Variable ‘dense_1/kernel:0’ shape=(5, 100) dtype=float32, numpy=

array([[ 0.15976147, -0.21502799, -0.07832763, -0.07678932, 0.15619548,

     -0.13655013,  0.20808806,  0.01320128, -0.05075614, -0.12650774,

      0.17424123,  0.15503065, -0.01738872,  0.17415772,  0.14157884,

      0.12198548, -0.13682888, -0.00431134, -0.11316137, -0.0941491 ,

      0.19477858, -0.15650672, -0.04523107,  0.11980413,  0.0392478 ,

     -0.03217201, -0.05695881, -0.09442574, -0.1769021 ,  0.13404123,

      0.22155072,  0.00532427, -0.20503883, -0.07037227,  0.16178603,

      0.15753557,  0.20727096,  0.23379399,  0.04048635, -0.23585105,

     -0.10203241,  0.06950574, -0.03137924, -0.19482769,  0.08389793,

     -0.0394402 , -0.00789529, -0.03464368, -0.12009872, -0.10856129,

     -0.00894293, -0.15162197, -0.032401  , -0.00453915,  0.05632652,

      0.17267881, -0.06191218, -0.05819538,  0.17946924, -0.15665913,

     -0.16304854,  0.00419821, -0.23776583, -0.16124432, -0.09681283,

     -0.21562152,  0.06472899,  0.22464116, -0.00569429,  0.08298178,

      0.11549039,  0.05924587,  0.03496118, -0.1051037 , -0.0621435 ,

      0.03481527, -0.07776555, -0.20981023,  0.0910445 , -0.0354397 ,

      0.16490476, -0.00227122,  0.0864424 , -0.13491175, -0.1645971 ,

      0.01050323,  0.14815761, -0.16781896,  0.13963409, -0.18121573,

     -0.23153992, -0.12473008,  0.1756901 ,  0.22858872,  0.13873275,

     -0.07768981, -0.00570698, -0.21528007,  0.00757009,  0.18363057],

    [ 0.11877824,  0.1168793 , -0.16063112, -0.17231184, -0.13409498,

      0.19905145, -0.2014717 ,  0.06295772, -0.04477178,  0.13132559,

      0.02939065,  0.03398912, -0.18317087, -0.03417668, -0.0107834 ,

      0.23002048,  0.06328867,  0.22760455, -0.19573069,  0.1198089 ,

     -0.1555649 , -0.02448821, -0.1734567 ,  0.12476839,  0.06721567,

     -0.1859357 ,  0.16312553, -0.1407745 ,  0.04455628, -0.22905497,

     -0.19142778, -0.22951172,  0.12258504,  0.1085989 ,  0.11974679,

     -0.03365536,  0.04648311, -0.03597827, -0.17160097, -0.12544727,

      0.06869908, -0.19143125,  0.22244458,  0.01437087, -0.03775081,

      0.12372582, -0.12253619,  0.11636381,  0.0799488 ,  0.12603168,

      0.03701521, -0.08781768, -0.11272953, -0.19216225,  0.13539644,

      0.20037527,  0.01965778, -0.03328758,  0.19321935,  0.08564301,

     -0.08135879,  0.19894253,  0.05638538, -0.23345044, -0.03519736,

      0.08768143, -0.08977255,  0.19856672,  0.21901618, -0.17389253,

      0.08617128, -0.14592302, -0.08758481,  0.13933013, -0.10471724,

     -0.05098224,  0.14235462,  0.15303789, -0.23809639, -0.08832294,

      0.02031069,  0.0559523 , -0.12055045,  0.06665261,  0.0479513 ,

     -0.08828138,  0.20095529,  0.02761547,  0.03795286, -0.08495584,

     -0.20158786,  0.02046029,  0.00867324,  0.11415212,  0.07045172,

     -0.15951589, -0.04511054,  0.10843597,  0.0239722 ,  0.19611128],

    [ 0.06575288,  0.2319472 ,  0.02630915,  0.06555082,  0.10341518,

     -0.22588201, -0.09562983, -0.06301573,  0.0936505 , -0.01802544,

      0.10455464, -0.22898191,  0.15039228, -0.16919808,  0.1916085 ,

     -0.12750028, -0.01717289,  0.20635293,  0.03781982, -0.14620668,

     -0.04488444, -0.06710777, -0.05447578, -0.04199792,  0.06323545,

     -0.17838557, -0.06185564, -0.06520277,  0.03831242, -0.03136453,

      0.01833086,  0.04521824,  0.12240265, -0.22481441, -0.23007765,

     -0.02078243, -0.2205326 ,  0.12952165,  0.02959962, -0.00605129,

     -0.02509233, -0.01998001,  0.21180545, -0.17241916, -0.03981026,

      0.21422301,  0.1693532 , -0.16202216,  0.07298566, -0.02760105,

      0.06959338, -0.21102753, -0.08139749,  0.04676224, -0.05416957,

      0.04917015, -0.01942708,  0.00196809,  0.12898101,  0.10945933,

     -0.00723343, -0.09607568, -0.10398597,  0.17198323,  0.03353049,

     -0.22060822, -0.05254391,  0.1702088 , -0.06497537,  0.06605254,

     -0.10460377,  0.01676236, -0.12414208, -0.04885554, -0.06743388,

     -0.10943221, -0.10569546, -0.20162518, -0.18120456,  0.23769595,

      0.01256581, -0.15475243,  0.16375937, -0.04159869, -0.14625306,

      0.02401985, -0.03575583, -0.04931861,  0.19226141, -0.22309631,

     -0.20494531,  0.11331509,  0.17416649,  0.06172945, -0.09567474,

     -0.09036852,  0.23214902,  0.0489447 ,  0.09294797,  0.15390666],

    [-0.17737857, -0.18906884,  0.20800714,  0.18031667, -0.04482706,

      0.11783125,  0.23290856,  0.13584308, -0.2087704 ,  0.03467382,

     -0.1224075 , -0.0173824 ,  0.08969595,  0.11672856, -0.16228962,

      0.08299066,  0.1263643 ,  0.20883657,  0.11351739, -0.11898713,

     -0.01544076, -0.10768828, -0.12749064, -0.21114449,  0.11841713,

     -0.12943742,  0.21584527,  0.15418832,  0.18037207,  0.13766332,

      0.21342348, -0.21019743,  0.134468  ,  0.17600726, -0.06488413,

      0.21859638, -0.20597608, -0.18852752,  0.06003846, -0.18052879,

     -0.21775812,  0.01654042,  0.14494626, -0.0470764 ,  0.19375344,

     -0.08053894, -0.08301774,  0.18195082, -0.10353127,  0.1721399 ,

      0.22610258,  0.15497144, -0.155101  ,  0.22665949, -0.21150993,

      0.13490684, -0.1813324 , -0.05590421,  0.20388336, -0.14803404,

     -0.12166522, -0.03127021,  0.01856966,  0.12462242, -0.18116277,

      0.07210417,  0.1368192 ,  0.09557848,  0.20420377,  0.22029643,

     -0.04349463,  0.20889862, -0.09501664, -0.20409772,  0.22081412,

     -0.19286382, -0.13874817, -0.14168417, -0.01535691, -0.06024285,

     -0.20417163,  0.08094741, -0.16762799, -0.03637671,  0.02360807,

      0.05724873,  0.17640142,  0.23869203, -0.09119059, -0.10150877,

      0.00279078, -0.13638958, -0.17503062, -0.04303041,  0.21908353,

      0.02966647,  0.11748509,  0.00320597, -0.1525138 , -0.09930906],

    [ 0.14521824,  0.00987744, -0.06205305,  0.07410519,  0.12307341,

      0.10008018,  0.18523215, -0.04782329, -0.0641716 , -0.22103709,

      0.07627855, -0.01990074,  0.14073421, -0.16386804, -0.02798302,

      0.17231806,  0.11191709,  0.12163399, -0.03887653,  0.09699525,

     -0.23051292,  0.07509579, -0.19534229,  0.18384896, -0.00816447,

      0.21739645, -0.03420636,  0.03474154,  0.04589911, -0.05675083,

      0.00387518,  0.10147156,  0.16241352,  0.2050067 ,  0.09604742,

      0.14956607,  0.213484  ,  0.06119736,  0.06285249,  0.02077337,

      0.13911705, -0.10113329, -0.1210945 , -0.13392931, -0.18992984,

     -0.01247907, -0.11904646, -0.2268505 , -0.23361304, -0.09504554,

     -0.15068164,  0.23496388,  0.08663981,  0.16571404,  0.23014064,

     -0.08652052,  0.09993   ,  0.08749835,  0.06988804, -0.0851372 ,

      0.22989945,  0.160183  ,  0.23885925, -0.02504036, -0.10181008,

      0.2021188 , -0.21528782, -0.2222352 , -0.15230861, -0.00771257,

      0.22475721, -0.12518431, -0.01185596,  0.05325596,  0.20660283,

      0.12990983, -0.04936986,  0.01129554, -0.2356372 ,  0.1903253 ,

     -0.2333641 ,  0.18106572,  0.11063577, -0.05270833,  0.22134696,

     -0.15732771,  0.10120948, -0.05766672, -0.10450801,  0.19721611,

      0.14783986,  0.21742307,  0.02052437, -0.22242983, -0.07544668,

     -0.15735045, -0.03890607,  0.00672534, -0.18067765, -0.06355779]],

   dtype=float32)>,

<tf.Variable ‘dense_1/bias:0’ shape=(100,) dtype=float32, numpy=

array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],

   dtype=float32)>]

其他方法及变量可通过help(*)进行查看

2.1 子类实现自定义层次

# customized dense layer.
class CustomizedDenseLayer(keras.layers.Layer):
    def __init__(self, units, activation=None, **kwargs):
        self.units = units #输出有多少个单元
        self.activation = keras.layers.Activation(activation) #使用的激活函数
        super(CustomizedDenseLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        """构建所需要的参数"""
        # x * w + b. input_shape:[None, a] ,w:[a,b],output_shape: [None, b]，b即units
        self.kernel = self.add_weight(name = 'kernel',
                                      shape = (input_shape[1], self.units), #矩阵大小
                                      initializer = 'uniform', #初始化参数矩阵的方法
                                      trainable = True) #参数kernal是否可训练（变）
        self.bias = self.add_weight(name = 'bias',
                                    shape = (self.units, ),
                                    initializer = 'zeros',
                                    trainable = True)
        super(CustomizedDenseLayer, self).build(input_shape)

    def call(self, x):
        """完成正向计算"""
        return self.activation(x @ self.kernel + self.bias)

model = keras.models.Sequential([
    CustomizedDenseLayer(30, activation='relu',
                         input_shape=x_train.shape[1:]),
    CustomizedDenseLayer(1),
])
model.summary()
model.compile(loss="mean_squared_error", optimizer="sgd")
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

子类实现自定义层次

model summary

2.2 lambda实现自定义层次

使用简单函数（无参数）定义层次

# tf.nn.softplus : log(1+e^x) ，平滑版relu
customized_softplus = keras.layers.Lambda(lambda x : tf.nn.softplus(x))
print(customized_softplus([-10., -5., 0., 5., 10.]))

lambda实现自定义层次

model = keras.models.Sequential([
    CustomizedDenseLayer(30, activation='relu',
                         input_shape=x_train.shape[1:]),
    CustomizedDenseLayer(1),
    customized_softplus,
    # keras.layers.Dense(1, activation="softplus"),
    # keras.layers.Dense(1), keras.layers.Activation('softplus'),
])
model.summary()
model.compile(loss="mean_squared_error", optimizer="sgd")
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

用自定义层次构建模型

model summary

（三）tf.function

tf库函数可像python函数一样被调用→layer(tf.zeros([10, 5]))
python函数可像tf库函数一样被调用→tf.function, autograph
@tf.function： py func -> tf graph

tf.function：普通python语法写的函数或代码块变成tf图
autograph：tf.fucction依赖的函数转换机制

普通python函数转tf图实现的函数→快（tf优化）

%timeit scaled_elu(tf.random.normal((100, 100)))
%timeit scaled_elu_tf(tf.random.normal((100, 100)))

2.39 ms ± 147 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) 384 µs ± 64.9 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

3.1 py函数转换成tf图

方式一：`tf.function(*)`

# tf.function and auto-graph.
def scaled_elu(z, scale=1.0, alpha=1.0):
    # z >= 0 ? scale * z : scale * alpha * tf.nn.elu(z)
    is_positive = tf.greater_equal(z, 0.0)
    return scale * tf.where(is_positive, z, alpha * tf.nn.elu(z))

print(scaled_elu(tf.constant(-3.)))
print(scaled_elu(tf.constant([-3., -2.5])))

scaled_elu_tf = tf.function(scaled_elu) #tf.function转换
print(scaled_elu_tf(tf.constant(-3.)))
print(scaled_elu_tf(tf.constant([-3., -2.5])))

print(scaled_elu_tf.python_function is scaled_elu) #返回原先的python函数

tf.Tensor(-0.95021296, shape=(), dtype=float32)

tf.Tensor([-0.95021296 -0.917915 ], shape=(2,), dtype=float32)

tf.Tensor(-0.95021296, shape=(), dtype=float32)

tf.Tensor([-0.95021296 -0.917915 ], shape=(2,), dtype=float32)

True

方式二：`@tf.function`

# 1 + 1/2 + 1/2^2 + ... + 1/2^n

@tf.function
def converge_to_2(n_iters):
    total = tf.constant(0.)
    increment = tf.constant(1.)
    for _ in range(n_iters):
        total += increment
        increment /= 2.0
    return total

print(converge_to_2(20))

将普通python代码转为tf图结构的中间代码

def display_tf_code(func):
    code = tf.autograph.to_code(func)
    from IPython.display import display, Markdown
    display(Markdown('```python\n{}\n```'.format(code)))

display_tf_code(scaled_elu)

display_tf_code(converge_to_22)

变量声明为tf.Variable，必须放在@tf.function外面

var = tf.Variable(0.)

@tf.function
def add_21():
    return var.assign_add(21) # += 

print(add_21())

3.2 函数签名与图结构

【模型的保存、载入】
函数签名：函数中的参数类型
input_signature限定函数的输入类型→tf中可保存为SavedModel

@tf.function(input_signature=[tf.TensorSpec([None], tf.int32, name='x')])
def cube(z):
    return tf.pow(z, 3)

try:
    print(cube(tf.constant([1., 2., 3.])))
except ValueError as ex:
    print(ex)

print(cube(tf.constant([1, 2, 3])))

get_concrete_function：为由@tf.function标注的py func加上input_signature -> SavedModel

cube_func_int32 = cube.get_concrete_function(
    tf.TensorSpec([None], tf.int32)) #传入输入类型，同input_signatur定义
print(cube_func_int32)

ConcreteFunction cube(z)

Args:
z: int32 Tensor, shape=(None,)
Returns:
int32 Tensor, shape=(None,)

除传入类型外，也可传入具体参数值

print(cube_func_int32 is cube.get_concrete_function(
    tf.TensorSpec([5], tf.int32)))
print(cube_func_int32 is cube.get_concrete_function(
    tf.constant([1, 2, 3])))

对get_concrete_function的操作

cube_func_int32.graph

cube_func_int32.graph.get_operations()

[,

,

,

]

pow_op = cube_func_int32.graph.get_operations()[2]
print(pow_op)

name: “Pow”

op: “Pow”

input: “x”

input: “Pow/y”

attr {

key: “T”

value {
type: DT_INT32
}

}

操作2(pow)的图定义

print(list(pow_op.inputs))
print(list(pow_op.outputs))

[, ]

[]

在graph中通过名称获得operation或tensor

cube_func_int32.graph.get_operation_by_name("x")

#Placeholder放输入

cube_func_int32.graph.get_tensor_by_name("x:0")

tensor名

cube_func_int32.graph.as_graph_def()

node {

name: “x”

op: “Placeholder”

attr {
key: "_user_specified_name"

value {

  s: "x"

}
}

attr {
key: "dtype"

value {

  type: DT_INT32

}
}

attr {
key: "shape"

value {

  shape {

    dim {

      size: -1

    }

  }

}
}

}

node {

name: “Pow/y”

op: “Const”

attr {
key: "dtype"

value {

  type: DT_INT32

}
}

attr {
key: "value"

value {

  tensor {

    dtype: DT_INT32

    tensor_shape {

    }

    int_val: 3

  }

}
}

}

node {

name: “Pow”

op: “Pow”

input: “x”

input: “Pow/y”

attr {
key: "T"

value {

  type: DT_INT32

}
}

}

node {

name: “Identity”

op: “Identity”

input: “Pow”

attr {
key: "T"

value {

  type: DT_INT32

}
}

}

versions {

producer: 561

}

（四）自定义求导

4.1 近似求导

def f(x):
    return 3. * x ** 2 + 2. * x - 1

def approximate_derivative(f, x, eps=1e-3):
    return (f(x + eps) - f(x - eps)) / (2. * eps)

print(approximate_derivative(f, 1.))

7.999999999999119

def g(x1, x2):
    return (x1 + 5) * (x2 ** 2)

def approximate_gradient(g, x1, x2, eps=1e-3):
    dg_x1 = approximate_derivative(lambda x: g(x, x2), x1, eps)
    dg_x2 = approximate_derivative(lambda x: g(x1, x), x2, eps)
    return dg_x1, dg_x2

print(approximate_gradient(g, 2., 3.))

(8.999999999993236, 41.999999999994486)

4.2 tf.GradientTape的使用

4.2.1 tf.FradientTape求导

# 定义变量
x1 = tf.Variable(2.0)
x2 = tf.Variable(3.0)

with tf.GradientTape() as tape:
# 定义函数
    z = g(x1, x2)

dz_x1 = tape.gradient(z, x1) 
print(dz_x1)

# 未保存的tape只能执行一次(对象会被系统自动释放)，不能执行第二次
try:
    dz_x2 = tape.gradient(z, x2)
except RuntimeError as ex:
    print(ex)

tf.Tensor(9.0, shape=(), dtype=float32)

A non-persistent GradientTape can only be used tocompute one set of gradients (or jacobians)

4.2.2 同时求x1,x2的偏导

方法一

声明tape时设置其为可保存，但注意使用完后需要自行删除

x1 = tf.Variable(2.0)
x2 = tf.Variable(3.0)
with tf.GradientTape(persistent = True) as tape:
    z = g(x1, x2)

dz_x1 = tape.gradient(z, x1)
dz_x2 = tape.gradient(z, x2)
print(dz_x1, dz_x2)

del tape

tf.Tensor(9.0, shape=(), dtype=float32) tf.Tensor(42.0, shape=(), dtype=float32)

方法二

x1 = tf.Variable(2.0)
x2 = tf.Variable(3.0)
with tf.GradientTape() as tape:
    z = g(x1, x2)

dz_x1x2 = tape.gradient(z, [x1, x2])

print(dz_x1x2)

[, ]

4.2.3 求常量的导数

直接求解不行

x1 = tf.constant(2.0)
x2 = tf.constant(3.0)
with tf.GradientTape() as tape:
    z = g(x1, x2)

dz_x1x2 = tape.gradient(z, [x1, x2])

print(dz_x1x2)

[None, None]

x1 = tf.constant(2.0)
x2 = tf.constant(3.0)
with tf.GradientTape() as tape:
    tape.watch(x1)
    tape.watch(x2)
    z = g(x1, x2)

dz_x1x2 = tape.gradient(z, [x1, x2])

print(dz_x1x2)

[, ]

4.2.4 两个目标函数对一个变量求导

x = tf.Variable(5.0)
with tf.GradientTape() as tape:
    z1 = 3 * x
    z2 = x ** 2
tape.gradient([z1, z2], x)

#两导数值之和

4.2.5 求二阶导数

tape嵌套实现

x1 = tf.Variable(2.0)
x2 = tf.Variable(3.0)
with tf.GradientTape(persistent=True) as outer_tape:
    with tf.GradientTape(persistent=True) as inner_tape:
        z = g(x1, x2)
    inner_grads = inner_tape.gradient(z, [x1, x2])
outer_grads = [outer_tape.gradient(inner_grad, [x1, x2])
               for inner_grad in inner_grads]
print(outer_grads)
del inner_tape
del outer_tape

[[None, ], [, ]]

TensorFlow基础组件 - 图33

4.2.6 梯度下降方法模拟

learning_rate = 0.1
x = tf.Variable(0.0)

for _ in range(100):
    # 求导
    with tf.GradientTape() as tape:
        z = f(x)
    dz_dx = tape.gradient(z, x)
    # 更新x
    x.assign_sub(learning_rate * dz_dx)
print(x)

4.3 tf.GradientTape与optimzier的结合使用

learning_rate = 0.1
x = tf.Variable(0.0)

optimizer = keras.optimizers.SGD(lr = learning_rate)

for _ in range(100):
    with tf.GradientTape() as tape:
        z = f(x)
    dz_dx = tape.gradient(z, x)
    optimizer.apply_gradients([(dz_dx, x)]) #[(梯度，变量)]
print(x)

举例
对模型fit部分的调整

以batch形式遍历训练集，得到metirc
1. 自动求导
每次epoch结束后，在验证集上验证，得到验证集上的metric ```python
metric使用

metric = keras.metrics.MeanSquaredError() print(metric([5.], [2.])) print(metric([0.], [1.])) print(metric.result()) #keras中的metirc具有累加功能

metric.reset_states() #清空之前的记录结果 metric([1.], [3.]) print(metric.result())

> tf.Tensor(9.0, shape=(), dtype=float32)

> tf.Tensor(5.0, shape=(), dtype=float32)

> tf.Tensor(5.0, shape=(), dtype=float32)

> tf.Tensor(4.0, shape=(), dtype=float32)

手动训练模型（自定义求导）
```python
epochs = 100 #遍历次数
batch_size = 32 
steps_per_epoch = len(x_train_scaled) // batch_size #每个epoch的训练次数
optimizer = keras.optimizers.SGD()
metric = keras.metrics.MeanSquaredError()

def random_batch(x, y, batch_size=32):
    idx = np.random.randint(0, len(x), size=batch_size)
    return x[idx], y[idx]

model = keras.models.Sequential([
    keras.layers.Dense(30, activation='relu',
                       input_shape=x_train.shape[1:]),
    keras.layers.Dense(1),
])

for epoch in range(epochs):
    metric.reset_states()
    for step in range(steps_per_epoch):
        x_batch, y_batch = random_batch(x_train_scaled, y_train,
                                        batch_size)
        with tf.GradientTape() as tape:
            y_pred = model(x_batch)
            y_pred = tf.squeeze(y_pred, 1)
            loss = keras.losses.mean_squared_error(y_batch, y_pred)
            metric(y_batch, y_pred)
        grads = tape.gradient(loss, model.variables)
        grads_and_vars = zip(grads, model.variables)
        optimizer.apply_gradients(grads_and_vars)
        print("\rEpoch", epoch, " train mse:",
              metric.result().numpy(), end="")
    y_valid_pred = model(x_valid_scaled)
    y_valid_pred = tf.squeeze(y_valid_pred, 1)
    valid_loss = keras.losses.mean_squared_error(y_valid_pred, y_valid)
    print("\t", "valid mse: ", valid_loss.numpy())

三、TensorFlow dataset

TF.data API的功能

读取数据
数据预处理
将数据传给训练程序

（一）dataset基础API的使用

API列表

tf.data.Dataset.from_tensor_slices repeat,batch,interleave,map,shuffle,list_files dataset——数据集，tf抽象出的一个概念，也是tf.data API下的主要概念
从数据集中产生batch，对每个样本进行处理

1.1 从内存中构建数据集

tf.data.Dataset.from_tensor_slices

dataset = tf.data.Dataset.from_tensor_slices(np.arange(10))
print(dataset)

1.2 遍历dataset

for item in dataset:
    print(item)

tf.Tensor(0, shape=(), dtype=int32)

tf.Tensor(1, shape=(), dtype=int32)

tf.Tensor(2, shape=(), dtype=int32)

tf.Tensor(3, shape=(), dtype=int32)

tf.Tensor(4, shape=(), dtype=int32)

tf.Tensor(5, shape=(), dtype=int32)

tf.Tensor(6, shape=(), dtype=int32)

tf.Tensor(7, shape=(), dtype=int32)

tf.Tensor(8, shape=(), dtype=int32)

tf.Tensor(9, shape=(), dtype=int32)

1.3 epoch&get batch

——机器学习问题对dataset常用的两个操作
epoch重复读取（遍历一次数据集称为1个epoch）**
batch size 每次训练网络参数时从整个数据集中取得那一小部分数据的量

dataset = dataset.repeat(3).batch(7) #遍历三次，batch_size为7；连环调用
for item in dataset:
    print(item)

tf.Tensor([0 1 2 3 4 5 6], shape=(7,), dtype=int32)

tf.Tensor([7 8 9 0 1 2 3], shape=(7,), dtype=int32)

tf.Tensor([4 5 6 7 8 9 0], shape=(7,), dtype=int32)

tf.Tensor([1 2 3 4 5 6 7], shape=(7,), dtype=int32)

tf.Tensor([8 9], shape=(2,), dtype=int32)

1.4 `interleave`

对现有dataset中每个元素做一些操作，并将处理完的新结果合并形成新的dataset

# interleave: 
# case: 文件dataset -> 具体数据集

dataset2 = dataset.interleave(
    lambda v: tf.data.Dataset.from_tensor_slices(v), # map_fn
    cycle_length = 5, # cycle_length
    block_length = 5, # block_length,每次取出的数量,可以达到均匀混合的效果
)
for item in dataset2:
    print(item)

tf.Tensor(0, shape=(), dtype=int32)

tf.Tensor(1, shape=(), dtype=int32)

tf.Tensor(2, shape=(), dtype=int32)

tf.Tensor(3, shape=(), dtype=int32)

tf.Tensor(4, shape=(), dtype=int32)

tf.Tensor(7, shape=(), dtype=int32)

tf.Tensor(8, shape=(), dtype=int32)

tf.Tensor(9, shape=(), dtype=int32)

tf.Tensor(0, shape=(), dtype=int32)

tf.Tensor(1, shape=(), dtype=int32)

tf.Tensor(4, shape=(), dtype=int32)

tf.Tensor(5, shape=(), dtype=int32)

tf.Tensor(6, shape=(), dtype=int32)

tf.Tensor(7, shape=(), dtype=int32)

tf.Tensor(8, shape=(), dtype=int32)

tf.Tensor(1, shape=(), dtype=int32)

tf.Tensor(2, shape=(), dtype=int32)

tf.Tensor(3, shape=(), dtype=int32)

tf.Tensor(4, shape=(), dtype=int32)

tf.Tensor(5, shape=(), dtype=int32)

tf.Tensor(8, shape=(), dtype=int32)

tf.Tensor(9, shape=(), dtype=int32)

tf.Tensor(5, shape=(), dtype=int32)

tf.Tensor(6, shape=(), dtype=int32)

tf.Tensor(2, shape=(), dtype=int32)

tf.Tensor(3, shape=(), dtype=int32)

tf.Tensor(9, shape=(), dtype=int32)

tf.Tensor(0, shape=(), dtype=int32)

tf.Tensor(6, shape=(), dtype=int32)

tf.Tensor(7, shape=(), dtype=int32)

1.5 元组初始化dataset

x = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array(['cat', 'dog', 'fox'])
dataset3 = tf.data.Dataset.from_tensor_slices((x, y))
print(dataset3)

for item_x, item_y in dataset3:
    print(item)
for item_x, item_y in dataset3:
    print(item_x.numpy(), item_y.numpy())

{‘feature’: , ‘label’: }

{‘feature’: , ‘label’: }

{‘feature’: , ‘label’: }

[1 2] b’cat’

[3 4] b’dog’

[5 6] b’fox’

1.6 字典初始化dataset

dataset4 = tf.data.Dataset.from_tensor_slices({"feature": x,
                                               "label": y})
for item in dataset4:
    print(item)
for item_x, item_y in dataset3:
    print(item["feature"].numpy(), item["label"].numpy())

{‘feature’: , ‘label’: }

{‘feature’: , ‘label’: }

{‘feature’: , ‘label’: }

[5 6] b’fox’

[5 6] b’fox’

[5 6] b’fox’

（二）dataset读取csv文件

API列表

tf.data.TextLineDataset,tf.io,decode_csv

将sklearn的数据集存为外部csv文件

from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing()

from sklearn.model_selection import train_test_split

x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state = 7)
x_train, x_valid, y_train, y_valid = train_test_split(
    x_train_all, y_train_all, random_state = 11)

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
x_train_scaled = scaler.fit_transform(x_train)
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

output_dir = "generate_csv"
if not os.path.exists(output_dir):
    os.mkdir(output_dir)

def save_to_csv(output_dir, data, name_prefix,
                header=None, n_parts=10):# 存成10个文件
    path_format = os.path.join(output_dir, "{}_{:02d}.csv")
    filenames = []

    for file_idx, row_indices in enumerate(
        np.array_split(np.arange(len(data)), n_parts)):
        part_csv = path_format.format(name_prefix, file_idx)
        filenames.append(part_csv)
        with open(part_csv, "wt", encoding="utf-8") as f:
            if header is not None:
                f.write(header + "\n")
            for row_index in row_indices:
                f.write(",".join(
                    [repr(col) for col in data[row_index]]))
                f.write('\n')
    return filenames

# 按行merge
train_data = np.c_[x_train_scaled, y_train]
valid_data = np.c_[x_valid_scaled, y_valid]
test_data = np.c_[x_test_scaled, y_test]

header_cols = housing.feature_names + ["MidianHouseValue"]
header_str = ",".join(header_cols)

train_filenames = save_to_csv(output_dir, train_data, "train",
                              header_str, n_parts=20)
valid_filenames = save_to_csv(output_dir, valid_data, "valid",
                              header_str, n_parts=10)
test_filenames = save_to_csv(output_dir, test_data, "test",
                             header_str, n_parts=10)

读取系列csv并形成一个dataset

filename→dataset
read file according to filenames in dataset → datasets → merged dataset
parse csv

2.1 filename→dataset

filename_dataset = tf.data.Dataset.list_files(train_filenames)
for filename in filename_dataset:
    print(filename)

tf.Tensor(b’generate_csv\train_05.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_07.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_18.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_14.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_13.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_04.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_12.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_08.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_17.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_19.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_16.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_10.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_11.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_03.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_06.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_00.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_15.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_02.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_09.csv’, shape=(), dtype=string)

tf.Tensor(b’generate_csv\train_01.csv’, shape=(), dtype=string)

2.2 read file → datasets → merged dataset

n_readers = 5 # 读取文件的并行度
dataset = filename_dataset.interleave(
    lambda filename: tf.data.TextLineDataset(filename).skip(1),
    cycle_length = n_readers
)
for line in dataset.take(15):
    print(line.numpy())

b’-0.09719300311107498,-1.249743071766074,0.36232962250170797,0.026906080250728295,1.033811814747154,0.045881586971778555,1.3418334617377423,-1.6353869745909178,1.832’

b’0.09734603446040174,0.7527628439249472,-0.20218964416999152,-0.1954700015215477,-0.4060513603629498,0.006785531677655949,-0.813715166526018,0.656614793197258,1.119’

b’2.51504373119231,1.0731637904355105,0.5574401201546321,-0.17273513019187772,-0.612912610473286,-0.01909156503651574,-0.5710993036045546,-0.027490309606616956,5.00001’

b’0.401276648075221,-0.9293421252555106,-0.05333050451405854,-0.1865945262276826,0.6545661895448709,0.026434465728210874,0.9312527706398824,-1.4406417263474771,2.512’

b’-0.6672227549433569,-0.04823952235146133,0.34529405473316743,0.5382668657200925,1.8521839533415545,-0.0611253832474835,-0.8417093045554153,1.520484740533546,1.59’

b’1.0534699704183814,-0.1283397589791022,0.13509497508586193,-0.2852867771449356,-0.37066719915986596,-0.017744041396267323,0.7586222527919203,-1.1510205879341566,2.674’

b’-1.4803330571456954,-0.6890414153725881,-0.35624704887282904,-0.1725588908792445,-0.8215884329530113,-0.1382309124854157,1.9157132913404298,-1.0211904224385344,0.928’

b’1.8444675088321243,0.5124621340420246,0.505783649224786,-0.20645711406004988,-0.021362018052499883,-0.05811312281214649,0.8332732875369839,-1.2658703497187516,4.513’

b’-0.8757754235423053,1.874166156711919,-0.9487499555702599,-0.09657184824705009,-0.7163432355284542,-0.07790191228558485,0.9825753570271144,-1.4206678547327694,2.75’

b’1.5180511450515526,-0.5288409421173064,0.8102470510967439,-0.1921416982863481,0.44135393614167334,0.027335058055345158,-0.8183808561975836,0.8563535093443789,2.898’

b’-0.8698076415077927,-0.44874070548966555,0.9621267572121975,3.9409717092762584,-0.9740125119816802,-0.09383082108319943,-0.6690787867074531,1.6752822455475638,0.425’

b’0.21174628471128154,1.1532640270631513,-0.2507761334605016,-0.2564987121705146,-0.6473894854916754,0.017590216427099285,0.7959477701644521,-1.1510205879341566,1.935’

b’-0.8109823614533115,0.43236189741438374,-0.09614708870040699,-0.011052490243107498,-0.5884158834865357,-0.15275615510545787,-1.3036125820405071,1.15096811566138,4.889’

b’0.7751155655229017,1.874166156711919,0.15645971958808144,-0.18905190538070707,-0.6292437617977863,-0.08791603438866835,-0.7483955111240856,0.5717258388347319,4.851’

b’-0.3295635160799086,0.9930635538078697,-0.8771740525217612,-0.3636710820906513,-1.1164564429787098,-0.08510593365640572,1.0665577711153127,-1.38571357940702,1.563’

2.3 parse csv

tf.io.decode_csv

# tf.io.decode_csv(str, record_defaults定义字符串中各field类型)

sample_str = '1,2,3,4,5'
record_defaluts = [tf.constant(0, dtype=tf.int32)] * 5 
parsed_fields = tf.io.decode_csv(sample_str, record_defaults)
print(parsed_fields)

[, , , , ]

sample_str = '1,2,3,4,5'
record_defaluts = [tf.constant(0, dtype=tf.int32)] * 5 
record_defaults = [
    tf.constant(0, dtype=tf.int32),
    0,
    np.nan,
    "hello",
    tf.constant([]) #tf会将其转换为tf类型
]
parsed_fields = tf.io.decode_csv(sample_str, record_defaults)
print(parsed_fields)

[, , , , ]

解析行

def parse_csv_line(line, n_fields = 9):
    defs = [tf.constant(np.nan)] * n_fields
    parsed_fields = tf.io.decode_csv(line, record_defaults=defs)
    x = tf.stack(parsed_fields[0:-1])
    y = tf.stack(parsed_fields[-1:])
    return x, y

parse_csv_line(b'-0.9868720801669367,0.832863080552588,-0.18684708416901633,-0.14888949288707784,-0.4532302419670616,-0.11504995754593579,1.6730974284189664,-0.7465496877362412,1.138',
               n_fields=9)

(<tf.Tensor: shape=(8,), dtype=float32, numpy=

array([-0.9868721 , 0.8328631 , -0.18684709, -0.1488895 , -0.45323023,
    -0.11504996,  1.6730974 , -0.74654967], dtype=float32)>,
)

三合一
interleave：一→多
map：一→一

# 1. filename -> dataset
# 2. read file -> dataset -> datasets -> merge(filename dataset -> 文本dataset)
# 3. parse csv
def csv_reader_dataset(filenames, n_readers=5,
                       batch_size=32, n_parse_threads=5,# 解析并行度
                       shuffle_buffer_size=10000): 
    dataset = tf.data.Dataset.list_files(filenames)
    dataset = dataset.repeat()
    dataset = dataset.interleave(
        lambda filename: tf.data.TextLineDataset(filename).skip(1),
        cycle_length = n_readers
    )
    dataset.shuffle(shuffle_buffer_size) # 混排
    dataset = dataset.map(parse_csv_line,
                          num_parallel_calls=n_parse_threads)
    dataset = dataset.batch(batch_size)
    return dataset

train_set = csv_reader_dataset(train_filenames, batch_size=3)
for x_batch, y_batch in train_set.take(2):
    print("x:")
    pprint.pprint(x_batch)
    print("y:")
    pprint.pprint(y_batch)

x:

y:

x:

y:

2.4 与tf.keras的结合使用

batch_size = 32
train_set = csv_reader_dataset(train_filenames,
                               batch_size = batch_size)
valid_set = csv_reader_dataset(valid_filenames,
                               batch_size = batch_size)
test_set = csv_reader_dataset(test_filenames,
                              batch_size = batch_size)

model = keras.models.Sequential([
    keras.layers.Dense(30, activation='relu',
                       input_shape=[8]),
    keras.layers.Dense(1),
])
model.compile(loss="mean_squared_error", optimizer="sgd")
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

history = model.fit(train_set,
                    validation_data = valid_set,
                    steps_per_epoch = 11160 // batch_size, # len(train_set)/batch_size
                    validation_steps = 3870 // batch_size,
                    epochs = 100,
                    callbacks = callbacks)

model.evaluate(test_set, steps = 5160 // batch_size)

（三）dataset读取tfrecord文件

tfrecord：tf独有的存储数据的格式，有很多tf优化
API列表

tf.train.FloatList, tf.train.Int64List, tf.train.BytesList tf.train.Feature, tf.tain.Features, tf.train.Example example.SerializeToString tf.io.ParseSingleExample tf.io.VarLenFeature, tf.io.FixedLenFeature tf.data.TFRecordDataset, tf.io.TFRecordOptions

3.1 tfrecord基础API使用

tfrecord→文件格式
创建features

# BytesList
favorite_books = [name.encode('utf-8')
                  for name in ["machine learning", "cc150"]]
favorite_books_bytelist = tf.train.BytesList(value = favorite_books)
print(favorite_books_bytelist)

# FloatList
hours_floatlist = tf.train.FloatList(value = [15.5, 9.5, 7.0, 8.0])
print(hours_floatlist)

# Int64List
age_int64list = tf.train.Int64List(value = [42])
print(age_int64list)

features = tf.train.Features(
    feature = {
        "favorite_books": tf.train.Feature(
            bytes_list = favorite_books_bytelist),
        "hours": tf.train.Feature(
            float_list = hours_floatlist),
        "age": tf.train.Feature(int64_list = age_int64list),
    }
)
print(features)

value: “machine learning”

value: “cc150”

value: 15.5

value: 9.5

value: 7.0

value: 8.0

value: 42

feature {

key: “age”

value {
int64_list {

  value: 42

}
}

}

feature {

key: “favorite_books”

value {
bytes_list {

  value: "machine learning"

  value: "cc150"

}
}

}

feature {

key: “hours”

value {
float_list {

  value: 15.5

  value: 9.5

  value: 7.0

  value: 8.0

}
}

}

用features组建example

example = tf.train.Example(features=features)
print(example)

serialized_example = example.SerializeToString() #序列化
print(serialized_example)

features {

feature {
key: "age"

value {

  int64_list {

    value: 42

  }

}
}

feature {
key: "favorite_books"

value {

  bytes_list {

    value: "machine learning"

    value: "cc150"

  }

}
}

feature {
key: "hours"

value {

  float_list {

    value: 15.5

    value: 9.5

    value: 7.0

    value: 8.0

  }

}
}

}

b’\n\\n-\n\x0efavorite_books\x12\x1b\n\x19\n\x10machine learning\n\x05cc150\n\x1d\n\x05hours\x12\x14\x12\x12\n\x10\x00\x00xA\x00\x00\x18A\x00\x00\xe0@\x00\x00\x00A\n\x0c\n\x03age\x12\x05\x1a\x03\n\x01*’

将example储存为tfrecord外部文件

output_dir = 'tfrecord_basic'
if not os.path.exists(output_dir):
    os.mkdir(output_dir)
filename = "test.tfrecords"
filename_fullpath = os.path.join(output_dir, filename)
with tf.io.TFRecordWriter(filename_fullpath) as writer:
    for i in range(3):
        writer.write(serialized_example)

读取tfrecord文件

dataset = tf.data.TFRecordDataset([filename_fullpath])
for serialized_example_tensor in dataset:
    print(serialized_example_tensor)

tf.Tensor(b’\n\\n-\n\x0efavorite_books\x12\x1b\n\x19\n\x10machine learning\n\x05cc150\n\x1d\n\x05hours\x12\x14\x12\x12\n\x10\x00\x00xA\x00\x00\x18A\x00\x00\xe0@\x00\x00\x00A\n\x0c\n\x03age\x12\x05\x1a\x03\n\x01*’, shape=(), dtype=string)

tf.Tensor(b’\n\\n-\n\x0efavorite_books\x12\x1b\n\x19\n\x10machine learning\n\x05cc150\n\x1d\n\x05hours\x12\x14\x12\x12\n\x10\x00\x00xA\x00\x00\x18A\x00\x00\xe0@\x00\x00\x00A\n\x0c\n\x03age\x12\x05\x1a\x03\n\x01*’, shape=(), dtype=string)

tf.Tensor(b’\n\\n-\n\x0efavorite_books\x12\x1b\n\x19\n\x10machine learning\n\x05cc150\n\x1d\n\x05hours\x12\x14\x12\x12\n\x10\x00\x00xA\x00\x00\x18A\x00\x00\xe0@\x00\x00\x00A\n\x0c\n\x03age\x12\x05\x1a\x03\n\x01*’, shape=(), dtype=string)

# 解析序列化features
expected_features = {
    "favorite_books": tf.io.VarLenFeature(dtype = tf.string),
    "hours": tf.io.VarLenFeature(dtype = tf.float32),
    "age": tf.io.FixedLenFeature([], dtype = tf.int64),
}
dataset = tf.data.TFRecordDataset([filename_fullpath])
for serialized_example_tensor in dataset:
    example = tf.io.parse_single_example(
        serialized_example_tensor,
        expected_features)
    books = tf.sparse.to_dense(example["favorite_books"],
                               default_value=b"")
    for book in books:
        print(book.numpy().decode("UTF-8"))

{‘favorite_books’: , ‘hours’: , ‘age’: }

machine learning

cc150

{‘favorite_books’: , ‘hours’: , ‘age’: }

machine learning

cc150

{‘favorite_books’: , ‘hours’: , ‘age’: }

machine learning

cc150

将tfrecord存成压缩文件

filename_fullpath_zip = filename_fullpath + '.zip'
options = tf.io.TFRecordOptions(compression_type = "GZIP")
with tf.io.TFRecordWriter(filename_fullpath_zip, options) as writer:
    for i in range(3):
        writer.write(serialized_example)

读取压缩后的tfrecord文件

dataset_zip = tf.data.TFRecordDataset([filename_fullpath_zip], 
                                      compression_type= "GZIP")
for serialized_example_tensor in dataset_zip:
    example = tf.io.parse_single_example(
        serialized_example_tensor,
        expected_features)
    books = tf.sparse.to_dense(example["favorite_books"],
                               default_value=b"")
    for book in books:
        print(book.numpy().decode("UTF-8"))

machine learning

cc150

machine learning

cc150

machine learning

cc150

3.2 生成tfrecord文件

区分训练集、验证集和测试集文件

source_dir = "./generate_csv/"

def get_filenames_by_prefix(source_dir, prefix_name):
    all_files = os.listdir(source_dir)
    results = []
    for filename in all_files:
        if filename.startswith(prefix_name):
            results.append(os.path.join(source_dir, filename))
    return results

train_filenames = get_filenames_by_prefix(source_dir, "train")
valid_filenames = get_filenames_by_prefix(source_dir, "valid")
test_filenames = get_filenames_by_prefix(source_dir, "test")

import pprint
pprint.pprint(train_filenames)
pprint.pprint(valid_filenames)
pprint.pprint(test_filenames)

使用tf.data API从csv文件中获取训练集、验证集和测试集**

def parse_csv_line(line, n_fields = 9):
    defs = [tf.constant(np.nan)] * n_fields
    parsed_fields = tf.io.decode_csv(line, record_defaults=defs)
    x = tf.stack(parsed_fields[0:-1])
    y = tf.stack(parsed_fields[-1:])
    return x, y

def csv_reader_dataset(filenames, n_readers=5,
                       batch_size=32, n_parse_threads=5,
                       shuffle_buffer_size=10000):
    dataset = tf.data.Dataset.list_files(filenames)
    dataset = dataset.repeat() #dataset的遍历是无限次的
    dataset = dataset.interleave(
        lambda filename: tf.data.TextLineDataset(filename).skip(1),
        cycle_length = n_readers
    )
    dataset.shuffle(shuffle_buffer_size)
    dataset = dataset.map(parse_csv_line,
                          num_parallel_calls=n_parse_threads)
    dataset = dataset.batch(batch_size)
    return dataset

batch_size = 32
train_set = csv_reader_dataset(train_filenames,
                               batch_size = batch_size)
valid_set = csv_reader_dataset(valid_filenames,
                               batch_size = batch_size)
test_set = csv_reader_dataset(test_filenames,
                              batch_size = batch_size)

分别遍历三个数据集，写入tfrecord文件**

def serialize_example(x, y):
    """Converts x, y to tf.train.Example and serialize"""
    input_feautres = tf.train.FloatList(value = x)
    label = tf.train.FloatList(value = y)
    features = tf.train.Features(
        feature = {
            "input_features": tf.train.Feature(
                float_list = input_feautres),
            "label": tf.train.Feature(float_list = label)
        }
    )
    example = tf.train.Example(features=features)
    return example.SerializeToString()

def csv_dataset_to_tfrecords(base_filename, dataset,
                             n_shards, steps_per_shard,
                             compression_type = None):
    options = tf.io.TFRecordOptions(
        compression_type = compression_type)
    all_filenames = []

    for shard_id in range(n_shards):
        filename_fullpath = '{}_{:05d}-of-{:05d}'.format(
            base_filename, shard_id, n_shards)
        with tf.io.TFRecordWriter(filename_fullpath, options) as writer:
            for x_batch, y_batch in dataset.skip(shard_id * steps_per_shard).take(steps_per_shard):
                for x_example, y_example in zip(x_batch, y_batch):
                    writer.write(
                        serialize_example(x_example.numpy(), y_example.numpy()))
        all_filenames.append(filename_fullpath)
    return all_filenames

生成tfrecord**

n_shards = 20
train_steps_per_shard = 11610 // batch_size // n_shards
valid_steps_per_shard = 3880 // batch_size // n_shards
test_steps_per_shard = 5170 // batch_size // n_shards

output_dir = "generate_tfrecords"
if not os.path.exists(output_dir):
    os.mkdir(output_dir)

train_basename = os.path.join(output_dir, "train")
valid_basename = os.path.join(output_dir, "valid")
test_basename = os.path.join(output_dir, "test")

train_tfrecord_filenames = csv_dataset_to_tfrecords(
    train_basename, train_set, n_shards, train_steps_per_shard, None)
valid_tfrecord_filenames = csv_dataset_to_tfrecords(
    valid_basename, valid_set, n_shards, valid_steps_per_shard, None)
test_tfrecord_fielnames = csv_dataset_to_tfrecords(
    test_basename, test_set, n_shards, test_steps_per_shard, None)

生成压缩的tfrecord**

n_shards = 20
train_steps_per_shard = 11610 // batch_size // n_shards
valid_steps_per_shard = 3880 // batch_size // n_shards
test_steps_per_shard = 5170 // batch_size // n_shards

output_dir = "generate_tfrecords_zip"
if not os.path.exists(output_dir):
    os.mkdir(output_dir)

train_basename = os.path.join(output_dir, "train")
valid_basename = os.path.join(output_dir, "valid")
test_basename = os.path.join(output_dir, "test")

train_tfrecord_filenames = csv_dataset_to_tfrecords(
    train_basename, train_set, n_shards, train_steps_per_shard,
    compression_type = "GZIP")
valid_tfrecord_filenames = csv_dataset_to_tfrecords(
    valid_basename, valid_set, n_shards, valid_steps_per_shard,
    compression_type = "GZIP")
test_tfrecord_fielnames = csv_dataset_to_tfrecords(
    test_basename, test_set, n_shards, test_steps_per_shard,
    compression_type = "GZIP")

3.3 tf.data读取tfrecord并与tf.keras结合使用

读取和解析example

expected_features = {
    "input_features": tf.io.FixedLenFeature([8], dtype=tf.float32),
    "label": tf.io.FixedLenFeature([1], dtype=tf.float32)
}

def parse_example(serialized_example):
    example = tf.io.parse_single_example(serialized_example,
                                         expected_features)
    return example["input_features"], example["label"]

# 从tfrecord文件中读取数据并形成dataset
def tfrecords_reader_dataset(filenames, n_readers=5,
                             batch_size=32, n_parse_threads=5,
                             shuffle_buffer_size=10000):
    dataset = tf.data.Dataset.list_files(filenames)
    dataset = dataset.repeat()
    dataset = dataset.interleave(
        lambda filename: tf.data.TFRecordDataset(
            filename, compression_type = "GZIP"),
        cycle_length = n_readers
    )
    dataset.shuffle(shuffle_buffer_size)
    dataset = dataset.map(parse_example,
                          num_parallel_calls=n_parse_threads)
    dataset = dataset.batch(batch_size)
    return dataset

tfrecords_train = tfrecords_reader_dataset(train_tfrecord_filenames,
                                           batch_size = 3)
for x_batch, y_batch in tfrecords_train.take(10):
    print(x_batch)
    print(y_batch)

tf.Tensor(

[[ 1.4843758 0.11196095 0.2451948 0.11828088 -0.7199724 -0.13889429

-0.9536859 0.8563535 ]

[-0.8272339 -0.92934215 -0.44595614 -0.11337186 -0.91140974 -0.21430673

-1.1356478 1.1210073 ]

[-1.2616565 0.27216142 -0.3939014 -0.06565835 0.286208 0.14643465

0.3247131 0.13230066]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[5.00001]

[0.858 ]

[0.541 ]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[ 0.33392596 -0.8492419 -0.32878366 0.10122446 0.41504264 -0.1665643

-0.7437298 0.56673235]

[ 0.35129648 -1.0895426 0.26549947 -0.16705142 -0.6927538 0.02216902

1.2905109 -1.1809814 ]

[-0.82094646 -0.4487407 -0.45438793 -0.24141093 -0.10029592 0.05197262

-1.4295862 1.2358571 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[5.00001]

[2.609 ]

[1.208 ]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[ 0.6853324 -1.8104447 -0.18240152 -0.12475578 1.1336133 -0.11667841

-0.8790348 0.6366409 ]

[-0.24628098 1.2333642 -0.41765466 0.02003763 0.16009521 0.15687561

-0.7250671 0.6965625 ]

[-0.71741605 0.03186071 -0.1506887 -0.23676488 -0.5122039 -0.06149909

1.7524141 -1.0861055 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[2.852]

[1.849]

[0.819]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[-1.2985821 0.19206119 -0.7858234 -0.0713922 -0.06400447 -0.18166348

-0.16984999 -0.5418175 ]

[ 1.6231267 -1.8104447 0.79948825 -0.13621119 0.8396526 -0.06950033

1.3371677 -0.7415562 ]

[-1.20411 0.5925624 -0.3522208 -0.21028471 -0.81433016 0.1312104

0.98257536 -1.3108115 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[2.096]

[2.631]

[0.7 ]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[ 0.21808705 0.35226166 0.13502641 -0.14116886 -0.84789973 -0.14950703

0.5440005 -0.1073858 ]

[-0.436877 -1.3298433 0.2768636 0.22049394 -1.1064763 -0.09777523

-0.7950524 1.4306023 ]

[ 0.6471812 -2.1308458 0.32654023 0.10183216 -0.45867395 -0.04176512

-0.36114326 0.6915691 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[1.322]

[0.663]

[1.111]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[-1.1442723 0.6726626 -0.30919224 0.01544129 -0.5330714 -0.09136026

1.3511648 -0.95627534]

[-0.6129799 0.4323619 0.12575388 -0.07846551 0.11745277 0.09121013

-0.12785879 0.32205245]

[-0.51456475 -0.36864048 -0.03090033 0.08479609 -0.3570579 0.19547077

0.50200933 -0.09240539]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[0.784]

[0.628]

[0.606]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[-0.6720183 1.0731637 -0.6061425 0.06724407 -1.2471056 -0.16936974

1.1132146 -1.2808508 ]

[-0.01609511 1.8741661 0.27636048 -0.09006003 -0.2853823 -0.10189375

0.52533776 -0.11237926]

[ 0.18393216 -1.8104447 0.13225727 -0.24872285 -0.5947669 -0.006972

0.87526447 -0.6267065 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[2.25 ]

[0.876]

[1.5 ]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[ 0.3735158 -0.6089412 0.19678533 -0.10472994 -0.08940848 -0.01627934

-0.650416 0.551752 ]

[ 0.97935224 1.8741661 0.14258558 -0.16552539 0.18549924 -0.07590286

0.9779097 -1.4456352 ]

[ 0.6276261 -0.92934215 0.04200385 -0.2889256 0.43137378 0.09972834

0.7586222 -1.1310468 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[2.389]

[3.636]

[2.508]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[ 0.01193215 -0.92934215 -0.7273498 0.08191014 -0.0422296 -0.10956222

-0.85104066 0.8463666 ]

[-0.36680886 -1.1696428 0.31091136 0.12876572 -0.46411768 -0.01830396

1.0898862 -0.32709837]

[ 0.5139185 -1.2497431 -0.15809242 -0.13876112 2.8030198 -0.06659425

0.8939273 -1.1959618 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[2.25 ]

[0.933]

[2.54 ]], shape=(3, 1), dtype=float32)

tf.Tensor(

[[-0.8604297 -0.36864048 -0.9447822 -0.11035065 0.3442743 -0.01955136

0.77728504 -1.1410336 ]

[-0.40037763 -0.28854024 -0.6460236 0.05778906 -0.61109805 -0.1500565

-0.82771224 0.8413731 ]

[-0.19741978 -0.28854024 -0.33716527 -0.06148583 0.8813877 0.02256449

-0.7997181 0.8114123 ]], shape=(3, 8), dtype=float32)

tf.Tensor(

[[1.125]

[1.882]

[1.619]], shape=(3, 1), dtype=float32)

生成训练用数据集

batch_size = 32
tfrecords_train_set = tfrecords_reader_dataset(
    train_tfrecord_filenames, batch_size = batch_size)
tfrecords_valid_set = tfrecords_reader_dataset(
    valid_tfrecord_filenames, batch_size = batch_size)
tfrecords_test_set = tfrecords_reader_dataset(
    test_tfrecord_fielnames, batch_size = batch_size)

训练模型

model = keras.models.Sequential([
    keras.layers.Dense(30, activation='relu',
                       input_shape=[8]),
    keras.layers.Dense(1),
])
model.compile(loss="mean_squared_error", optimizer="sgd")
callbacks = [keras.callbacks.EarlyStopping(
    patience=5, min_delta=1e-2)]

history = model.fit(tfrecords_train_set,
                    validation_data = tfrecords_valid_set,
                    steps_per_epoch = 11160 // batch_size,
                    validation_steps = 3870 // batch_size,
                    epochs = 100,
                    callbacks = callbacks)

model.evaluate(tfrecords_test_set, steps = 5160 // batch_size)

四、TensorFlow Estimator

（一）Keras转estimator

API列表

Tf.keras.estimator.to_estimator
- train, evaluate

导入数据

# https://storage.googleapis.com/tf-datasets/titanic/train.csv
# https://storage.googleapis.com/tf-datasets/titanic/eval.csv
train_file = "./data/titanic/train.csv"
eval_file = "./data/titanic/eval.csv"

train_df = pd.read_csv(train_file)
eval_df = pd.read_csv(eval_file)

y_train = train_df.pop('survived')
y_eval = eval_df.pop('survived')

# print(train_df.shape, eval_df.shape)
# train_df.age.hist(bins = 20)
# train_df.sex.value_counts().plot(kind = 'barh')
# train_df['class'].value_counts().plot(kind = 'barh')
# pd.concat([train_df, y_train], axis = 1).groupby('sex').survived.mean().plot(kind='barh'

feature_columns

# 离散特征
categorical_columns = ['sex', 'n_siblings_spouses', 'parch', 'class',
                       'deck', 'embark_town', 'alone']
#连续特征
numeric_columns = ['age', 'fare']

feature_columns = []

for categorical_column in categorical_columns:
    vocab = train_df[categorical_column].unique() #离散特征所在列所有可能值
    print(categorical_column, vocab)
    feature_columns.append(
        tf.feature_column.indicator_column( #one-hot编码
            tf.feature_column.categorical_column_with_vocabulary_list(
                categorical_column, vocab)))

for categorical_column in numeric_columns:
    feature_columns.append(
        tf.feature_column.numeric_column(
            categorical_column, dtype=tf.float32))

sex [‘male’ ‘female’] n_siblings_spouses [1 0 3 4 2 5 8] parch [0 1 2 5 3 4] class [‘Third’ ‘First’ ‘Second’] deck [‘unknown’ ‘C’ ‘G’ ‘A’ ‘B’ ‘D’ ‘F’ ‘E’] embark_town [‘Southampton’ ‘Cherbourg’ ‘Queenstown’ ‘unknown’] alone [‘n’ ‘y’]

构建dataset

def make_dataset(data_df, label_df, epochs = 10, shuffle = True,
                 batch_size = 32):
    dataset = tf.data.Dataset.from_tensor_slices(
        (dict(data_df), label_df))
    if shuffle:
        dataset = dataset.shuffle(10000)
    dataset = dataset.repeat(epochs).batch(batch_size)
    return dataset

train_dataset = make_dataset(train_df, y_train, batch_size = 5)

for x, y in train_dataset.take(1):
    print(x, y)

{‘sex’: , ‘age’: , ‘n_siblings_spouses’: , ‘parch’: , ‘fare’: , ‘class’: , ‘deck’: , ‘embark_town’: <tf.Tensor: shape=(5,), dtype=string, numpy=

array([b’Southampton’, b’Cherbourg’, b’Southampton’, b’Southampton’,
   b'Southampton'], dtype=object)>, 'alone': <tf.Tensor: shape=(5,), dtype=string, numpy=array([b'y', b'n', b'y', b'y', b'n'], dtype=object)>} tf.Tensor([0 1 1 0 1], shape=(5,), dtype=int64)

# keras.layers.DenseFeature
for x, y in train_dataset.take(1):
    age_column = feature_columns[7]
    gender_column = feature_columns[0]
    print(keras.layers.DenseFeatures(age_column)(x).numpy())
    print(keras.layers.DenseFeatures(gender_column)(x).numpy())

[[28.]

[49.]

[14.]

[28.]

[28.]]

[[1. 0.]

[0. 1.]

[0. 1.]

[1. 0.]

[1. 0.]]

# keras.layers.DenseFeature
for x, y in train_dataset.take(1):
    print(keras.layers.DenseFeatures(feature_columns)(x).numpy())

[[ 28. 0. 1. 1. 0. 0. 1. 0.

0.       0.       0.       0.       0.       0.       1.       0.

0.       0.       8.05     0.       1.       0.       0.       0.

0.       0.       1.       0.       0.       0.       0.       0.

1.       0.    ]

[ 60. 1. 0. 0. 1. 0. 0. 0.

0.       0.       1.       0.       0.       0.       0.       1.

0.       0.      79.2      1.       0.       0.       0.       0.

0.       0.       0.       1.       0.       0.       0.       0.

1.       0.    ]

[ 21. 0. 1. 1. 0. 0. 1. 0.

0.       0.       0.       0.       0.       0.       1.       0.

0.       0.       7.775    0.       1.       0.       0.       0.

0.       0.       1.       0.       0.       0.       0.       0.

1.       0.    ]

[ 28. 0. 1. 1. 0. 0. 1. 0.

0.       0.       0.       0.       0.       0.       1.       0.

0.       0.       7.7958   0.       1.       0.       0.       0.

0.       0.       1.       0.       0.       0.       0.       0.

1.       0.    ]

[ 36. 1. 0. 0. 1. 0. 0. 0.

0.       0.       1.       0.       0.       0.       1.       0.

0.       0.     120.       1.       0.       0.       0.       0.

0.       0.       0.       0.       1.       0.       0.       0.

1.       0.    ]]

keras_to_estimator

构建模型

model = keras.models.Sequential([
    keras.layers.DenseFeatures(feature_columns),
    keras.layers.Dense(100, activation='relu'),
    keras.layers.Dense(100, activation='relu'),
    keras.layers.Dense(2, activation='softmax'),
])
model.compile(loss='sparse_categorical_crossentropy',
              optimizer = keras.optimizers.SGD(lr=0.01),
              metrics = ['accuracy'])

训练模型
1. model.fit
2. model -> estimator -> train

1. model.fit

train_dataset = make_dataset(train_df, y_train, epochs = 100)
eval_dataset = make_dataset(eval_df, y_eval, epochs = 1, shuffle = False)
history = model.fit(train_dataset,
                    validation_data = eval_dataset,
                    steps_per_epoch = 19,
                    validation_steps = 8,
                    epochs = 100)

2. keras model -> estimator -> train

报错：Unexpectedly found an instance of type <class 'dict'>. Expected a symbolic tensor instance

estimator = keras.estimator.model_to_estimator(model)
# input_fn:
# 1. function
# 2. return a. (features, labels) b. dataset -> (feature, label)
estimator.train(input_fn = lambda : make_dataset(
    train_df, y_train, epochs=100))

（二）用预定义的estimator

API列表

Tf.estimator.BaseLineClassifier
Tf.estimator.LinearClassifier
Tf.estimator.DNNClassifier

会自动保存tensorboard，可在训练完成后用命令行打开tensorboard查看训练过程

BaseLineClassifier

报错：FailedPreconditionError: 2 root error(s) found. (0) Failed precondition: GetNext() failed because the iterator has not been initialized. Ensure that you have run the initializer operation for this iterator before getting the next element.

LinearClassifier

linear_output_dir = 'linear_model'
if not os.path.exists(linear_output_dir):
    os.mkdir(linear_output_dir)
linear_estimator = tf.estimator.LinearClassifier(
    model_dir = linear_output_dir,
    n_classes = 2,
    feature_columns = feature_columns)
linear_estimator.train(input_fn = lambda : make_dataset(
    train_df, y_train, epochs = 100))

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {‘_model_dir’: ‘linear_model’, ‘_tf_random_seed’: None, ‘_save_summary_steps’: 100, ‘_save_checkpoints_steps’: None, ‘_save_checkpoints_secs’: 600, ‘_session_config’: allow_soft_placement: true

graph_options {

rewrite_options {
meta_optimizer_iterations: ONE
}

}

, ‘_keep_checkpoint_max’: 5, ‘_keep_checkpoint_every_n_hours’: 10000, ‘_log_step_count_steps’: 100, ‘_train_distribute’: None, ‘_device_fn’: None, ‘_protocol’: None, ‘_eval_distribute’: None, ‘_experimental_distribute’: None, ‘_experimental_max_worker_delay_secs’: None, ‘_session_creation_timeout_secs’: 7200, ‘_checkpoint_save_graph_def’: True, ‘_service’: None, ‘_cluster_spec’: ClusterSpec({}), ‘_task_type’: ‘worker’, ‘_task_id’: 0, ‘_global_id_in_cluster’: 0, ‘_master’: ‘’, ‘_evaluation_master’: ‘’, ‘_is_chief’: True, ‘_num_ps_replicas’: 0, ‘_num_worker_replicas’: 1}

INFO:tensorflow:Calling model_fn.

D:\Python\Anaconda\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:1727: UserWarning: layer.add_variable is deprecated and will be removed in a future version. Please use layer.add_weight method instead.

warnings.warn(‘layer.add_variable is deprecated and ‘

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0…

INFO:tensorflow:Saving checkpoints for 0 into linear_model\model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0…

INFO:tensorflow:loss = 0.6931472, step = 0

INFO:tensorflow:global_step/sec: 137.919

INFO:tensorflow:loss = 0.36079478, step = 100 (0.726 sec)

INFO:tensorflow:global_step/sec: 146.804

INFO:tensorflow:loss = 0.35266042, step = 200 (0.682 sec)

INFO:tensorflow:global_step/sec: 154.02

INFO:tensorflow:loss = 0.45829007, step = 300 (0.648 sec)

INFO:tensorflow:global_step/sec: 153.314

INFO:tensorflow:loss = 0.54480356, step = 400 (0.652 sec)

INFO:tensorflow:global_step/sec: 156.419

INFO:tensorflow:loss = 0.43118882, step = 500 (0.640 sec)

INFO:tensorflow:global_step/sec: 156.672

INFO:tensorflow:loss = 0.53275156, step = 600 (0.637 sec)

INFO:tensorflow:global_step/sec: 156.423

INFO:tensorflow:loss = 0.44017065, step = 700 (0.639 sec)

INFO:tensorflow:global_step/sec: 152.614

INFO:tensorflow:loss = 0.5682069, step = 800 (0.655 sec)

INFO:tensorflow:global_step/sec: 155.695

INFO:tensorflow:loss = 0.5055707, step = 900 (0.643 sec)

INFO:tensorflow:global_step/sec: 150.552

INFO:tensorflow:loss = 0.36983708, step = 1000 (0.663 sec)

INFO:tensorflow:global_step/sec: 155.453

INFO:tensorflow:loss = 0.41094226, step = 1100 (0.644 sec)

INFO:tensorflow:global_step/sec: 147.452

INFO:tensorflow:loss = 0.32404315, step = 1200 (0.677 sec)

INFO:tensorflow:global_step/sec: 156.668

INFO:tensorflow:loss = 0.46529418, step = 1300 (0.638 sec)

INFO:tensorflow:global_step/sec: 157.653

INFO:tensorflow:loss = 0.29717875, step = 1400 (0.635 sec)

INFO:tensorflow:global_step/sec: 148.544

INFO:tensorflow:loss = 0.6039681, step = 1500 (0.672 sec)

INFO:tensorflow:global_step/sec: 146.804

INFO:tensorflow:loss = 0.32268602, step = 1600 (0.681 sec)

INFO:tensorflow:global_step/sec: 152.151

INFO:tensorflow:loss = 0.28891423, step = 1700 (0.658 sec)

INFO:tensorflow:global_step/sec: 155.937

INFO:tensorflow:loss = 0.5024715, step = 1800 (0.640 sec)

INFO:tensorflow:global_step/sec: 154.495

INFO:tensorflow:loss = 0.34215456, step = 1900 (0.647 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1960…

INFO:tensorflow:Saving checkpoints for 1960 into linear_model\model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1960…

INFO:tensorflow:Loss for final step: 0.32699785.

[10]:

linear_estimator.evaluate(input_fn = lambda : make_dataset(
    eval_df, y_eval, epochs = 1, shuffle = False))

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2021-01-23T18:13:58Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from linear_model\model.ckpt-1960

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 0.56539s

INFO:tensorflow:Finished evaluation at 2021-01-23-18:13:58

INFO:tensorflow:Saving dict for global step 1960: accuracy = 0.7878788, accuracy_baseline = 0.625, auc = 0.83516985, auc_precision_recall = 0.7807338, average_loss = 0.46775854, global_step = 1960, label/mean = 0.375, loss = 0.45215982, precision = 0.7171717, prediction/mean = 0.38183394, recall = 0.7171717

INFO:tensorflow:Saving ‘checkpoint_path’ summary for global step 1960: linear_model\model.ckpt-1960

[11]:

{‘accuracy’: 0.7878788,

‘accuracy_baseline’: 0.625,

‘auc’: 0.83516985,

‘auc_precision_recall’: 0.7807338,

‘average_loss’: 0.46775854,

‘label/mean’: 0.375,

‘loss’: 0.45215982,

‘precision’: 0.7171717,

‘prediction/mean’: 0.38183394,

‘recall’: 0.7171717,

‘global_step’: 1960}

DNNClassifier

dnn_output_dir = './dnn_model'
if not os.path.exists(dnn_output_dir):
    os.mkdir(dnn_output_dir)
dnn_estimator = tf.estimator.DNNClassifier(
    model_dir = dnn_output_dir,
    n_classes = 2,
    feature_columns=feature_columns,
    hidden_units = [128, 128], #两层，每层128个单元
    activation_fn = tf.nn.relu,
    optimizer = 'Adam')
dnn_estimator.train(input_fn = lambda : make_dataset(
    train_df, y_train, epochs = 100))

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {‘_model_dir’: ‘./dnn_model’, ‘_tf_random_seed’: None, ‘_save_summary_steps’: 100, ‘_save_checkpoints_steps’: None, ‘_save_checkpoints_secs’: 600, ‘_session_config’: allow_soft_placement: true

graph_options {

rewrite_options {
meta_optimizer_iterations: ONE
}

}

, ‘_keep_checkpoint_max’: 5, ‘_keep_checkpoint_every_n_hours’: 10000, ‘_log_step_count_steps’: 100, ‘_train_distribute’: None, ‘_device_fn’: None, ‘_protocol’: None, ‘_eval_distribute’: None, ‘_experimental_distribute’: None, ‘_experimental_max_worker_delay_secs’: None, ‘_session_creation_timeout_secs’: 7200, ‘_checkpoint_save_graph_def’: True, ‘_service’: None, ‘_cluster_spec’: ClusterSpec({}), ‘_task_type’: ‘worker’, ‘_task_id’: 0, ‘_global_id_in_cluster’: 0, ‘_master’: ‘’, ‘_evaluation_master’: ‘’, ‘_is_chief’: True, ‘_num_ps_replicas’: 0, ‘_num_worker_replicas’: 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0…

INFO:tensorflow:Saving checkpoints for 0 into ./dnn_model\model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0…

INFO:tensorflow:loss = 1.621333, step = 0

INFO:tensorflow:global_step/sec: 144.195

INFO:tensorflow:loss = 0.5959885, step = 100 (0.694 sec)

INFO:tensorflow:global_step/sec: 167.149

INFO:tensorflow:loss = 0.51040477, step = 200 (0.599 sec)

INFO:tensorflow:global_step/sec: 165.591

INFO:tensorflow:loss = 0.6042056, step = 300 (0.603 sec)

INFO:tensorflow:global_step/sec: 166.137

INFO:tensorflow:loss = 0.4114812, step = 400 (0.602 sec)

INFO:tensorflow:global_step/sec: 166.808

INFO:tensorflow:loss = 0.40442616, step = 500 (0.599 sec)

INFO:tensorflow:global_step/sec: 167.106

INFO:tensorflow:loss = 0.3285016, step = 600 (0.600 sec)

INFO:tensorflow:global_step/sec: 166.632

INFO:tensorflow:loss = 0.51703423, step = 700 (0.600 sec)

INFO:tensorflow:global_step/sec: 167.74

INFO:tensorflow:loss = 0.31758735, step = 800 (0.595 sec)

INFO:tensorflow:global_step/sec: 166.952

INFO:tensorflow:loss = 0.47445992, step = 900 (0.599 sec)

INFO:tensorflow:global_step/sec: 168.4

INFO:tensorflow:loss = 0.24317755, step = 1000 (0.595 sec)

INFO:tensorflow:global_step/sec: 167.271

INFO:tensorflow:loss = 0.1966712, step = 1100 (0.597 sec)

INFO:tensorflow:global_step/sec: 166.393

INFO:tensorflow:loss = 0.29284936, step = 1200 (0.601 sec)

INFO:tensorflow:global_step/sec: 167.447

INFO:tensorflow:loss = 0.36962345, step = 1300 (0.597 sec)

INFO:tensorflow:global_step/sec: 167.092

INFO:tensorflow:loss = 0.3014151, step = 1400 (0.599 sec)

INFO:tensorflow:global_step/sec: 167.331

INFO:tensorflow:loss = 0.30321366, step = 1500 (0.597 sec)

INFO:tensorflow:global_step/sec: 165.998

INFO:tensorflow:loss = 0.46479976, step = 1600 (0.602 sec)

INFO:tensorflow:global_step/sec: 168.953

INFO:tensorflow:loss = 0.38980004, step = 1700 (0.592 sec)

INFO:tensorflow:global_step/sec: 167.575

INFO:tensorflow:loss = 0.51549876, step = 1800 (0.597 sec)

INFO:tensorflow:global_step/sec: 166.996

INFO:tensorflow:loss = 0.2591187, step = 1900 (0.600 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1960…

INFO:tensorflow:Saving checkpoints for 1960 into ./dnn_model\model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1960…

INFO:tensorflow:Loss for final step: 0.2688936.

[12]:

dnn_estimator.evaluate(input_fn = lambda : make_dataset(
    eval_df, y_eval, epochs = 1, shuffle = False))

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2021-01-23T18:15:31Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from ./dnn_model\model.ckpt-1960

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 0.50210s

INFO:tensorflow:Finished evaluation at 2021-01-23-18:15:32

INFO:tensorflow:Saving dict for global step 1960: accuracy = 0.8030303, accuracy_baseline = 0.625, auc = 0.84471995, auc_precision_recall = 0.8039435, average_loss = 0.4739524, global_step = 1960, label/mean = 0.375, loss = 0.45056942, precision = 0.75268817, prediction/mean = 0.3647281, recall = 0.7070707

INFO:tensorflow:Saving ‘checkpoint_path’ summary for global step 1960: ./dnn_model\model.ckpt-1960

[13]:

{‘accuracy’: 0.8030303,

‘accuracy_baseline’: 0.625,

‘auc’: 0.84471995,

‘auc_precision_recall’: 0.8039435,

‘average_loss’: 0.4739524,

‘label/mean’: 0.375,

‘loss’: 0.45056942,

‘precision’: 0.75268817,

‘prediction/mean’: 0.3647281,

‘recall’: 0.7070707,

‘global_step’: 1960}

（三）Tf.feature_column做特征工程

API列表

Tf.feature_column
- categorical_column_with_vocabulary_list
- numeric_column
- indicator_column
- cross_column
keras.layers.DenseFeatures ```python categorical_columns = [‘sex’, ‘n_siblings_spouses’, ‘parch’, ‘class’,
```
                 'deck', 'embark_town', 'alone']
```
numeric_columns = [‘age’, ‘fare’]

feature_columns = [] for categorical_column in categorical_columns: vocab = train_df[categorical_column].unique() print(categorical_column, vocab) feature_columns.append( tf.feature_column.indicator_column( tf.feature_column.categorical_column_with_vocabulary_list( categorical_column, vocab)))

for numeric_column in numeric_columns: feature_columns.append( tf.feature_column.numeric_column( numeric_column, dtype=tf.float32))

对两个离散特征做笛卡尔积；交叉特征可以更细节地刻画样本

cross feature: age: [1,2,3,4,5], gender:[male, female]

age_x_gender: [(1, male), (2, male), …, (5, male), …, (5, female)]

100000: 100 -> hash(100000 values) % 100

feature_columns.append( tf.feature_column.indicator_column( tf.feature_column.crossed_column( [‘age’, ‘sex’], hash_bucket_size = 100)))

```

附录

TensorFlow v2.4官方API文档

TensorFlow基础组件

一、TensorFlow Keras

Tf-keras简介

keras是什么

Tensorflow-keras是什么

Tf-keras和keras比较

联系

区别

如何选择

Tf.keras实现分类模型和回归模型

（一）基础概念

1.分类问题与回归问题

2.目标函数

3.分类问题的目标函数

4.分类问题的常用损失函数

5.回归问题的目标函数

（二）Tf.keras搭建分类模型

分类模型搭建流程

（0）数据准备

（1）基于fashion_mnist数据集的分类模型

（2）训练数据集的归一化

（3）回调函数：对训练过程进行监听

用 tensorboard 模块进行可视化

（三）Tf.keras搭建回归模型

Tf.keras实现深度神经网络

（一）基础概念

1.神经网络

2.激活函数

3.归一化与批归一化

4.Dropout

（二）Tf.keras实现深度神经网络

keras实现DNN

keras实现批归一化

keras更改激活函数

keras实现dropout

Tf.keras实现Wide&deep模型

（一）基础知识

（二）Tf.keras实现Wide&deep模型

Tf.keras函数式API实现Wide&deep模型

Tf.keras子类API实现Wide&deep模型

多输入与多输出

超参数搜索

（一）搜索策略

（二）超参数搜索实现

手动实现超参数搜索

sklearn封装keras模型

二、TensorFlow基础API

（一）基础API

tf.constant

tf.string

RaggedTensor

SparseTensor

Variables

（二）基础API与keras的集成

1 自定义损失函数

2 自定义层次

2.1 子类实现自定义层次

2.2 lambda实现自定义层次

（三）tf.function

3.1 py函数转换成tf图

方式一：tf.function(*)

方式二：@tf.function

将普通python代码转为tf图结构的中间代码

3.2 函数签名与图结构

（四）自定义求导

4.1 近似求导

4.2 tf.GradientTape的使用

4.2.1 tf.FradientTape求导

4.2.2 同时求x1,x2的偏导

方法一

方法二

4.2.3 求常量的导数

4.2.4 两个目标函数对一个变量求导

4.2.5 求二阶导数

4.2.6 梯度下降方法模拟

4.3 tf.GradientTape与optimzier的结合使用

metric使用

三、TensorFlow dataset

（一）dataset基础API的使用

1.1 从内存中构建数据集

方式一：`tf.function(*)`

方式二：`@tf.function`

1.4 `interleave`