在Wide&Deep之后，诸多模型延续了双网络组合的结构，DeepFM就是其中之一。DeepFM对Wide&Deep的改进之处在于，它用FM替换掉了原来的Wide部分，加强了浅层网络部分特征组合的能力。事实上，由于FM本身就是由一阶部分和二阶部分组成的，DeepFM相当于同时组合了原Wide部分+二阶特征交叉部分+Deep部分三种结构，无疑进一步增强了模型的表达能力。
DeepFM.jpg
为了学习多阶交叉特征，DeepFM由FM和Deep两部分组成，两部分共用相同的输入，模型预测输出为

DeepFM - 图2

其中DeepFM - 图3，是预测CTR的结果，DeepFM - 图4是FM部分的输出，DeepFM - 图5是Deep部分的输出。

FM部分

FM.png
FM是由一阶部分和二阶部分组成的，其输出是一个Addition unit和一系列Inner Product units的和：

DeepFM - 图7
其中DeepFM - 图8(DeepFM - 图9为特征one hot后的维度数)，DeepFM - 图10(DeepFM - 图11为设置的Dense Embedding的维度数)。Addition unit（DeepFM - 图12）表达一阶特征，Inner Product表达二阶。

Deep部分

DNN.png
Deep部分用来学习高阶特征，其输入是连续稠密向量，由原始的超高维稀疏且具有组(field)属性(性别、地点、年龄等)数据由Dense Embedding层映射至低维稠密向量。

Dense Embedding

embedding.png
如上图所示，embedding层的输出为：

DeepFM - 图15

其中DeepFM - 图16为第i个特征向量，DeepFM - 图17为特征类别数(比如只有年龄，性别，地点3种类特征则m=3)。

Code

  1. # deepFM.py
  2. import tensorflow as tf
  3. from tensorflow.python.ops import embedding_ops
  4. from tensorflow.python.layers import normalization
  7. class EmbeddingTable:
  8.     def __init__(self):
  9.         self._weights = {}
  11.     def add_weights(self, vocab_name, vocab_size, embed_dim):
  12.         """
  13.         :param vocab_name: 一个field拥有两个权重矩阵，一个用于线性连接，另一个用于非线性（二阶或更高阶交叉）连接
  14.         :param vocab_size: 字典总长度
  15.         :param embed_dim: 二阶权重矩阵shape=[vocab_size, order2dim]，映射成的embedding, 既用于接入DNN的第一屋，也是用于FM二阶交互的隐向量
  16.         """
  17.         linear_weight = tf.get_variable(name='{}_linear_weight'.format(vocab_name),
  18.                                         shape=[vocab_size, 1],
  19.                                         initializer=tf.glorot_normal_initializer(),
  20.                                         dtype=tf.float32)
  22.         # 二阶（FM）与高阶（DNN）的特征交互，共享embedding矩阵
  23.         embed_weight = tf.get_variable(name='{}_embed_weight'.format(vocab_name),
  24.                                        shape=[vocab_size, embed_dim],
  25.                                        initializer=tf.glorot_normal_initializer(),
  26.                                        dtype=tf.float32)
  28.         self._weights[vocab_name] = (linear_weight, embed_weight)
  30.     def get_linear_weights(self, vocab_name): return self._weights[vocab_name][0]
  32.     def get_embed_weights(self, vocab_name): return self._weights[vocab_name][1]
  35. def build_embedding_table(params):
  36.     embed_dim = params['embed_dim']  # 必须有统一的embedding长度
  38.     embedding_table = EmbeddingTable()
  39.     for vocab_name, vocab_size in params['vocab_sizes'].items():
  40.         embedding_table.add_weights(vocab_name=vocab_name, vocab_size=vocab_size, embed_dim=embed_dim)
  42.     return embedding_table
  45. def output_logits_from_linear(features, embedding_table, params):
  46.     field2vocab_mapping = params['field_vocab_mapping']
  47.     combiner = params.get('multi_embed_combiner', 'sum')
  49.     fields_outputs = []
  50.     # 当前field下有一系列的<tag:value>对，每个tag对应一个bias（待优化），
  51.     # 将所有tag对应的bias，按照其value进行加权平均，得到这个field对应的bias
  52.     for fieldname, vocabname in field2vocab_mapping.items():
  53.         sp_ids = features[fieldname + "_ids"]
  54.         sp_values = features[fieldname + "_values"]
  56.         linear_weights = embedding_table.get_linear_weights(vocab_name=vocabname)
  58.         # weights: [vocab_size,1]
  59.         # sp_ids: [batch_size, max_tags_per_example]
  60.         # sp_weights: [batch_size, max_tags_per_example]
  61.         # output: [batch_size, 1]
  62.         output = embedding_ops.safe_embedding_lookup_sparse(linear_weights, sp_ids, sp_values,
  63.                                                             combiner=combiner,
  64.                                                             name='{}_linear_output'.format(fieldname))
  66.         fields_outputs.append(output)
  68.     # 因为不同field可以共享同一个vocab的linear weight，所以将各个field的output相加，会损失大量的信息
  69.     # 因此，所有field对应的output拼接起来，反正每个field的output都是[batch_size,1]，拼接起来，并不占多少空间
  70.     # whole_linear_output: [batch_size, total_fields]
  71.     whole_linear_output = tf.concat(fields_outputs, axis=1)
  72.     tf.logging.info("linear output, shape={}".format(whole_linear_output.shape))
  74.     # 再映射到final logits（二分类，也是[batch_size,1]）
  75.     # 这时，就不要用任何activation了，特别是ReLU
  76.     return tf.layers.dense(whole_linear_output, units=1, use_bias=True, activation=None)
  77.     #return tf.layers.dense(whole_linear_output, units=6, use_bias=True, activation=None)
  79. def output_logits_from_bi_interaction(features, embedding_table, params):
  80.     field2vocab_mapping = params['field_vocab_mapping']
  82.     # 论文上的公式就是要求sum，而且我也试过mean和sqrtn，都比用mean要差上很多
  83.     # 但是，这种情况，仅仅是针对criteo数据的，还是理论上就必须用sum，而不能用mean和sqrtn
  84.     # 我还不太确定，所以保留一个接口能指定其他combiner的方法
  85.     combiner = params.get('multi_embed_combiner', 'sum')
  87.     # 见《Neural Factorization Machines for Sparse Predictive Analytics》论文的公式(4)
  88.     fields_embeddings = []
  89.     fields_squared_embeddings = []
  91.     for fieldname, vocabname in field2vocab_mapping.items():
  92.         sp_ids = features[fieldname + "_ids"]
  93.         sp_values = features[fieldname + "_values"]
  95.         # --------- embedding
  96.         embed_weights = embedding_table.get_embed_weights(vocabname)
  97.         # embedding: [batch_size, embed_dim]
  98.         embedding = embedding_ops.safe_embedding_lookup_sparse(embed_weights, sp_ids, sp_values,
  99.                                                                combiner=combiner,
  100.                                                                name='{}_embedding'.format(fieldname))
  101.         fields_embeddings.append(embedding)
  103.         # --------- square of embedding
  104.         squared_emb_weights = tf.square(embed_weights)
  106.         squared_sp_values = tf.SparseTensor(indices=sp_values.indices,
  107.                                             values=tf.square(sp_values.values),
  108.                                             dense_shape=sp_values.dense_shape)
  110.         # squared_embedding: [batch_size, embed_dim]
  111.         squared_embedding = embedding_ops.safe_embedding_lookup_sparse(squared_emb_weights, sp_ids, squared_sp_values,
  112.                                                                        combiner=combiner,
  113.                                                                        name='{}_squared_embedding'.format(fieldname))
  114.         fields_squared_embeddings.append(squared_embedding)
  116.     # calculate bi-interaction
  117.     sum_embedding_then_square = tf.square(tf.add_n(fields_embeddings))  # [batch_size, embed_dim]
  118.     square_embedding_then_sum = tf.add_n(fields_squared_embeddings)  # [batch_size, embed_dim]
  119.     bi_interaction = 0.5 * (sum_embedding_then_square - square_embedding_then_sum)  # [batch_size, embed_dim]
  120.     tf.logging.info("bi-interaction, shape={}".format(bi_interaction.shape))
  122.     # calculate logits
  123.     logits = tf.layers.dense(bi_interaction, units=1, use_bias=True, activation=None)
  124.     #logits = tf.layers.dense(bi_interaction, units=6, use_bias=True, activation=None)
  125.     # 因为FM与DNN共享embedding，所以除了logits，还返回各field的embedding，方便搭建DNN
  126.     return logits, fields_embeddings
  129. def output_logits_from_dnn(fields_embeddings, params, is_training):
  130.     dropout_rate = params['dropout_rate']
  131.     do_batch_norm = params['batch_norm']
  133.     X = tf.concat(fields_embeddings, axis=1)
  134.     tf.logging.info("initial input to DNN, shape={}".format(X.shape))
  136.     for idx, n_units in enumerate(params['hidden_units'], start=1):
  137.         X = tf.layers.dense(X, units=n_units, activation=tf.nn.relu)
  138.         tf.logging.info("layer[{}] output shape={}".format(idx, X.shape))
  140.         X = tf.layers.dropout(inputs=X, rate=dropout_rate, training=is_training)
  141.         if is_training:
  142.             tf.logging.info("layer[{}] dropout {}".format(idx, dropout_rate))
  144.         if do_batch_norm:
  145.             # BatchNormalization的调用、参数，是从DNNLinearCombinedClassifier源码中拷贝过来的
  146.             batch_norm_layer = normalization.BatchNormalization(momentum=0.999, trainable=True,
  147.                                                                 name='batchnorm_{}'.format(idx))
  148.             X = batch_norm_layer(X, training=is_training)
  150.             if is_training:
  151.                 tf.logging.info("layer[{}] batch-normalize".format(idx))
  153.     # connect to final logits, [batch_size,1]
  154.     #connect to final logits, [batch_size,6] 6类
  155.     #return tf.layers.dense(X, units=1, use_bias=True, activation=None)
  156.     return tf.layers.dense(X, units=1, use_bias=True, activation=None)
  159. def model_fn(features, labels, mode, params):
  160.     for featname, featvalues in features.items():
  161.         if not isinstance(featvalues, tf.SparseTensor):
  162.             raise TypeError("feature[{}] isn't SparseTensor".format(featname))
  164.     # ============= build the graph
  165.     embedding_table = build_embedding_table(params)
  167.     linear_logits = output_logits_from_linear(features, embedding_table, params)
  169.     bi_interact_logits, fields_embeddings = output_logits_from_bi_interaction(features, embedding_table, params)
  171.     dnn_logits = output_logits_from_dnn(fields_embeddings, params, (mode == tf.estimator.ModeKeys.TRAIN))
  173.     general_bias = tf.get_variable(name='general_bias', shape=[1], initializer=tf.constant_initializer(0.0))
  175.     logits = linear_logits + bi_interact_logits + dnn_logits
  176.     logits = tf.nn.bias_add(logits, general_bias)  # bias_add，获取broadcasting的便利
  178.     # reshape [batch_size,1] to [batch_size], to match the shape of 'labels'
  179.     logits = tf.reshape(logits, shape=[-1])
  180.     probabilities = tf.sigmoid(logits)
  182.     # ============= predict spec
  183.     if mode == tf.estimator.ModeKeys.PREDICT:
  184.         return tf.estimator.EstimatorSpec(
  185.             mode=mode,
  186.             predictions={'uids':tf.sparse_tensor_to_dense(features['uids']),'probabilities': probabilities})
  188.     # ============= evaluate spec
  189.     # 这里不设置regularization，模仿DNNLinearCombinedClassifier的做法, L1/L2 regularization通过设置optimizer=
  190.     # tf.train.ProximalAdagradOptimizer(learning_rate=0.1,
  191.     #     l1_regularization_strength=0.001,
  192.     #     l2_regularization_strength=0.001)来实现
  193.     # STUPID TENSORFLOW CANNOT AUTO-CAST THE LABELS FOR ME
  194.     loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))
  195.     #predictions=tf.greater_equal(probabilities,0.5)
  196.     eval_metric_ops = {'auc': tf.metrics.auc(labels, probabilities)}
  197.     if mode == tf.estimator.ModeKeys.EVAL:
  198.         return tf.estimator.EstimatorSpec(
  199.             mode=mode,
  200.             loss=loss,
  201.             eval_metric_ops=eval_metric_ops)
  203.     # ============= train spec
  204.     assert mode == tf.estimator.ModeKeys.TRAIN
  205.     train_op = params['optimizer'].minimize(loss, global_step=tf.train.get_global_step())
  206.     return tf.estimator.EstimatorSpec(mode,
  207.                                       loss=loss,
  208.                                       train_op=train_op,
  209.                                       eval_metric_ops=eval_metric_ops)

Source