高层框架：TFLearn和Keras

上一节我们学习了Tensorflow的高层API封装，可以通过简单的几步就生成一个DNN分类器来解决MNIST手写识别问题。

尽管Tensorflow也在不断推进Estimator API。但是，这并不是工具的全部。在Tensorflow官方的API方外，我们还有强大的工具，比如TFLearn和Keras。

这节我们就做一个武器库的展示，看看专门为Tensorflow做的高层框架TFLearn和跨Tensorflow和CNTK几种后端的Keras为我们做了哪些强大的功能封装。

机器来写莎士比亚的戏剧

之前我们简单介绍了强大的用于处理序列数据的RNN。RNN比起其它网络的重要优点是可以学习了序列数据之后进行自生成。
比如，学习《唐诗三百首》可以写诗，学习了Linux Kernel源代码就能写C代码（虽然基本上编译不过）。

我们首先来一个自动写莎士比亚戏剧的例子吧。
在看代码之前我先唠叨几句。深度学习对于数据量的要求还是比较高的，像训练自动生成的这种，一般得几百万到几千万量级的训练数据下才能有好的效果。比如只用几篇小说来训练肯定生成不知所云的小说。就算是人类也做不到只学几首诗就会写诗么。
另外一点就是，训练数据量上来了，对于时间和算力的要求也是指数级提高的。
比如我们用莎翁的戏剧来训练，虽然数据量也不是特别的大，也就16万多行，但是在CPU上训练的话也不是一两个小时能搞定的，大约是天为单位。
后面我们举图像或视频的例子，在CPU上训，论月也是并不意外的。

那么，这个需要训一天左右的例子，代码会多复杂呢？答案是核心代码不过10几行，总共加上数据处理和测试代码也不过50行左右。

  1. from __future__ import absolute_import, division, print_function
  3. import os
  4. import pickle
  5. from six.moves import urllib
  7. import tflearn
  8. from tflearn.data_utils import *
  10. path = "shakespeare_input.txt"
  11. char_idx_file = 'char_idx.pickle'
  13. if not os.path.isfile(path):
  14.     urllib.request.urlretrieve("https://raw.githubusercontent.com/tflearn/tflearn.github.io/master/resources/shakespeare_input.txt", path)
  16. maxlen = 25
  18. char_idx = None
  19. if os.path.isfile(char_idx_file):
  20.   print('Loading previous char_idx')
  21.   char_idx = pickle.load(open(char_idx_file, 'rb'))
  23. X, Y, char_idx = \
  24.     textfile_to_semi_redundant_sequences(path, seq_maxlen=maxlen, redun_step=3,
  25.                                          pre_defined_char_idx=char_idx)
  27. pickle.dump(char_idx, open(char_idx_file,'wb'))
  29. g = tflearn.input_data([None, maxlen, len(char_idx)])
  30. g = tflearn.lstm(g, 512, return_seq=True)
  31. g = tflearn.dropout(g, 0.5)
  32. g = tflearn.lstm(g, 512, return_seq=True)
  33. g = tflearn.dropout(g, 0.5)
  34. g = tflearn.lstm(g, 512)
  35. g = tflearn.dropout(g, 0.5)
  36. g = tflearn.fully_connected(g, len(char_idx), activation='softmax')
  37. g = tflearn.regression(g, optimizer='adam', loss='categorical_crossentropy',
  38.                        learning_rate=0.001)
  40. m = tflearn.SequenceGenerator(g, dictionary=char_idx,
  41.                               seq_maxlen=maxlen,
  42.                               clip_gradients=5.0,
  43.                               checkpoint_path='model_shakespeare')
  45. for i in range(50):
  46.     seed = random_sequence_from_textfile(path, maxlen)
  47.     m.fit(X, Y, validation_set=0.1, batch_size=128,
  48.           n_epoch=1, run_id='shakespeare')
  49.     print("-- TESTING...")
  50.     print("-- Test with temperature of 1.0 --")
  51.     print(m.generate(600, temperature=1.0, seq_seed=seed))
  52.     print("-- Test with temperature of 0.5 --")
  53.     print(m.generate(600, temperature=0.5, seq_seed=seed))

上面的例子需要使用TFLearn框架，可以通过

  1. pip install tflearn

来安装。

TFLearn是专门为Tensorflow开发的高层次API框架。
用TFLearn API的主要好处是可读性更好，比如刚才的核心代码：

  1. g = tflearn.input_data([None, maxlen, len(char_idx)])
  2. g = tflearn.lstm(g, 512, return_seq=True)
  3. g = tflearn.dropout(g, 0.5)
  4. g = tflearn.lstm(g, 512, return_seq=True)
  5. g = tflearn.dropout(g, 0.5)
  6. g = tflearn.lstm(g, 512)
  7. g = tflearn.dropout(g, 0.5)
  8. g = tflearn.fully_connected(g, len(char_idx), activation='softmax')
  9. g = tflearn.regression(g, optimizer='adam', loss='categorical_crossentropy',
  10.                        learning_rate=0.001)
  12. m = tflearn.SequenceGenerator(g, dictionary=char_idx,
  13.                               seq_maxlen=maxlen,
  14.                               clip_gradients=5.0,
  15.                               checkpoint_path='model_shakespeare')

从输入数据，三层LSTM，三层Dropout，最后是一个softmax的全连接层。

我们再来看一个预测泰坦尼克号幸存概率的网络的结构：

  1. # Build neural network
  2. net = tflearn.input_data(shape=[None, 6])
  3. net = tflearn.fully_connected(net, 32)
  4. net = tflearn.fully_connected(net, 32)
  5. net = tflearn.fully_connected(net, 2, activation='softmax')
  6. net = tflearn.regression(net)
  8. # Define model
  9. model = tflearn.DNN(net)
  10. # Start training (apply gradient descent algorithm)
  11. model.fit(data, labels, n_epoch=10, batch_size=16, show_metric=True)

我们来看看训练第一轮之后生成的戏剧，这个阶段肯定还是有点惨，不过基本的意思有了：

  1. THAISA:
  2. Why, sir, say if becel; sunthy alot but of
  3. coos rytermelt, buy -
  4. bived with wond I saTt fas,'? You and grigper.
  6. FIENDANS:
  7. By my wordhand!
  9. KING RECENTEN:
  10. Wish sterest expeun The siops so his fuurs,
  11. And emour so, ane stamn.
  12. she wealiwe muke britgie; I dafs tpichicon, bist,
  13. Turch ose be fast wirpest neerenler.
  15. NONTo:
  16. So befac, sels at, Blove and rackity;
  17. The senent stran spard: and, this not you so the wount
  18. hor hould batil's toor wate
  19. What if a poostit's of bust contot;
  20. Whit twetemes, Game ifon I am
  21. Ures the fast to been'd matter:
  22. To and lause. Tiess her jittarss,
  23. Let concertaet ar: and not!
  24. Not fearle her g

我们再看看训练10轮之后的结果：

  1. PEMBROKE:
  2. There tell the elder pieres,
  3. Would our pestilent shapeing sebaricity. So have partned in me, Project of Yorle
  4. again, and then when you set man
  5. make plash'd of her too sparent
  6. upon this father be dangerous puny or house;
  7. Born is now been left of himself,
  8. This true compary nor no stretches, back that
  9. Horses had hand or question!
  11. POLIXENES:
  12. I have unproach the strangest
  13. padely carry neerful young Yir,
  14. Or hope not fall-a a cause of banque.
  16. JESSICA:
  17. He that comes to find the just,
  18. And eyes gold, substrovious;
  19. Yea pity a god on a foul rioness, these tebles and purish new head meet again?

20轮之后的结果：

  1. y prison,
  2. Fatal and ominous children and the foot, it will
  3. hear with you: it is my pace comprite
  4. To come my soldiers, if I were dread,
  5. Of breath as what I charge with I well;
  6. Her palace and every tailor, the house of wondrous sweet mark!
  8. STANLEY:
  9. Take that spirit, thou hast
  10. 'no whore he did eyes, and what men damned, and
  11. I had evils; by lap, or so,
  12. But wholow'st thy report subject,
  13. Had my rabble against thee;
  14. And no rassians which he secure
  15. of genslications; when I have move undertake-inward, into Bertounce;
  16. Upon a shift, meet as we are. He beggars thing
  17. Have for it will, but joy with the minute cannot whom we prarem
  18. -- Test with temperature of 0.5 --
  19. y prison,
  20. Fatal and ominous rein here,
  21. The princess have all to prince and the marsh of his company
  22. To prove brother in the world,
  23. And we the forest prove more than the heavens on the false report of the fools,
  24. Depose the body of my wits.
  26. DUKE SENIOR:
  27. The night better appelled my part.
  29. ANGELO:
  30. Care you that may you understand your grace
  31. I may speak of a point, and seems as in the heart
  32. Who be deeds to show and sale for so
  33. unhouses me of her soul, and the heart of them from the corder black to stand about up.
  35. CLAUDIO:
  36. The place of the world shall be married his love.

从生成城市名字说起

大家的莎士比亚模型应该正在训练过程中吧，咱们闲着也是闲着，不如从一个更简单的例子来看看这个生成过程。
我们还是取TFLearn的官方例子，通过读取美国主要城市名字列表来生成一些新的城市名字。

我们以Z开头的城市为例：

  1. Zachary
  2. Zafra
  3. Zag
  4. Zahl
  5. Zaleski
  6. Zalma
  7. Zama
  8. Zanesfield
  9. Zanesville
  10. Zap
  11. Zapata
  12. Zarah
  13. Zavalla
  14. Zearing
  15. Zebina
  16. Zebulon
  17. Zeeland
  18. Zeigler
  19. Zela
  20. Zelienople
  21. Zell
  22. Zellwood
  23. Zemple
  24. Zena
  25. Zenda
  26. Zenith
  27. Zephyr
  28. Zephyr Cove
  29. Zephyrhills
  30. Zia Pueblo
  31. Zillah
  32. Zilwaukee
  33. Zim
  34. Zimmerman
  35. Zinc
  36. Zion
  37. Zionsville
  38. Zita
  39. Zoar
  40. Zolfo Springs
  41. Zona
  42. Zumbro Falls
  43. Zumbrota
  44. Zuni
  45. Zurich
  46. Zwingle
  47. Zwolle

一共20580个城市。这个训练就快多了，在纯CPU上训练，大约5到6分钟可以训练一轮。

代码如下，跟上面写莎翁的戏剧的如出一辙：

  1. from __future__ import absolute_import, division, print_function
  3. import os
  4. from six import moves
  5. import ssl
  7. import tflearn
  8. from tflearn.data_utils import *
  10. path = "US_Cities.txt"
  11. if not os.path.isfile(path):
  12.     context = ssl._create_unverified_context()
  13.     moves.urllib.request.urlretrieve("https://raw.githubusercontent.com/tflearn/tflearn.github.io/master/resources/US_Cities.txt", path, context=context)
  15. maxlen = 20
  17. string_utf8 = open(path, "r").read().decode('utf-8')
  18. X, Y, char_idx = \
  19.     string_to_semi_redundant_sequences(string_utf8, seq_maxlen=maxlen, redun_step=3)
  21. g = tflearn.input_data(shape=[None, maxlen, len(char_idx)])
  22. g = tflearn.lstm(g, 512, return_seq=True)
  23. g = tflearn.dropout(g, 0.5)
  24. g = tflearn.lstm(g, 512)
  25. g = tflearn.dropout(g, 0.5)
  26. g = tflearn.fully_connected(g, len(char_idx), activation='softmax')
  27. g = tflearn.regression(g, optimizer='adam', loss='categorical_crossentropy',
  28.                        learning_rate=0.001)
  30. m = tflearn.SequenceGenerator(g, dictionary=char_idx,
  31.                               seq_maxlen=maxlen,
  32.                               clip_gradients=5.0,
  33.                               checkpoint_path='model_us_cities')
  35. for i in range(40):
  36.     seed = random_sequence_from_string(string_utf8, maxlen)
  37.     m.fit(X, Y, validation_set=0.1, batch_size=128,
  38.           n_epoch=1, run_id='us_cities')
  39.     print("-- TESTING...")
  40.     print("-- Test with temperature of 1.2 --")
  41.     print(m.generate(30, temperature=1.2, seq_seed=seed).encode('utf-8'))
  42.     print("-- Test with temperature of 1.0 --")
  43.     print(m.generate(30, temperature=1.0, seq_seed=seed).encode('utf-8'))
  44.     print("-- Test with temperature of 0.5 --")
  45.     print(m.generate(30, temperature=0.5, seq_seed=seed).encode('utf-8'))

我们看下第一轮训练完生成的城市名：

  1. t and Shoot
  2. Cuthbertd
  3. Lettfrecv
  4. El
  5. Ceoneel Sutd
  6. Sa

第二轮：

  1. stle
  2. Finchford
  3. Finch Dasthond
  4. madloogd
  5. Wlaycoyarfw

第三轮：

  1. averal
  2. Cape Carteret
  3. Acbiropa Heowar Sor Dittoy
  4. Do

第十轮：

  1. hoenchen
  2. Schofield
  3. Stcojos
  4. Schabell
  5. StcaKnerum Cri

第二十轮，好像开始有点意思了：

  1. Hill
  2. Cherry Hills Village
  3. Hillfood Pork
  4. Hillbrook

第三十轮，又有点退化：

  1. ckitat
  2. Kline
  3. Klondike
  4. Klonsder
  5. Klansburg
  6. Dlandon
  7. D

第四十轮：

  1. Branch
  2. Villages of Ocite
  3. Sidaydaton
  4. Sidway
  5. Siddade

第100轮：

  1. Atlasburg
  2. Atmautluak
  3. Attion
  4. Attul
  5. Atta
  6. Aque Creek

tflearn.SequenceGenerator的好处和坏处都是将细节都封装起来了，我们难以看到它的背后发生了什么。

温度参数

其实在前面的结果中，我们只是节选了一种温度下的结果。输出的结果一般都输出几种温度的值。那么这个温度是什么意思呢？

温度是表征概念变化的量。如果温度高，比如大于1，就代表希望输出结果更稳定。稳定的结果就是可能每次生成的句子都是一样的。如果等于1，那么对结果没有影响。如果小于1，那么就会让每次生成的结果变化比较大。
像做诗这样比较浪漫的事情，我们一般希望温度值在0.1以下，有变化才好玩，不是吗？

下面是城市生成值，在三种不同温度下的生成结果：

  1. -- Test with temperature of 1.2 --
  3. Atlasburg
  4. Atmautluak
  5. Attion
  6. Attul
  7. Atta
  8. Aque Creek
  9. -- Test with temperature of 1.0 --
  11. Atlasburg
  12. Atmautluak
  13. Attila
  14. Attaville
  15. Atteville
  16. -- Test with temperature of 0.5 --
  18. Atlasburg
  19. Atmautluak
  20. Attigua
  21. Attinword
  22. Attrove

跨后端高层API - Keras，生成尼采的文章

Keras是可以跨Tensorflow，微软的CNTK等多种后端的API。
可以通过

  1. pip install keras

来安装keras。我们安装了Tensorflow之后，Keras会选用Tensorflow来做它的后端。

我们也看下Keras上文本生成的例子。官方例子是用来生成尼采的句子。

核心语句就6句话：

  1. model = Sequential()
  2. model.add(LSTM(128, input_shape=(maxlen, len(chars))))
  3. model.add(Dense(len(chars)))
  4. model.add(Activation('softmax'))
  6. optimizer = RMSprop(lr=0.01)
  7. model.compile(loss='categorical_crossentropy', optimizer=optimizer)

下面是完整的代码，大家跑来玩玩吧。如果对尼采不感兴趣，也可以换成别的文章。不过请注意，正如注释中所说的，文本随便换，但是要保持在10万字符以上。最好是100万字符以上。

  1. '''Example script to generate text from Nietzsche's writings.
  2. At least 20 epochs are required before the generated text
  3. starts sounding coherent.
  4. It is recommended to run this script on GPU, as recurrent
  5. networks are quite computationally intensive.
  6. If you try this script on new data, make sure your corpus
  7. has at least ~100k characters. ~1M is better.
  8. '''
  10. from __future__ import print_function
  11. from keras.callbacks import LambdaCallback
  12. from keras.models import Sequential
  13. from keras.layers import Dense, Activation
  14. from keras.layers import LSTM
  15. from keras.optimizers import RMSprop
  16. from keras.utils.data_utils import get_file
  17. import numpy as np
  18. import random
  19. import sys
  20. import io
  22. path = get_file('nietzsche.txt', origin='https://s3.amazonaws.com/text-datasets/nietzsche.txt')
  23. with io.open(path, encoding='utf-8') as f:
  24.     text = f.read().lower()
  25. print('corpus length:', len(text))
  27. chars = sorted(list(set(text)))
  28. print('total chars:', len(chars))
  29. char_indices = dict((c, i) for i, c in enumerate(chars))
  30. indices_char = dict((i, c) for i, c in enumerate(chars))
  32. # cut the text in semi-redundant sequences of maxlen characters
  33. maxlen = 40
  34. step = 3
  35. sentences = []
  36. next_chars = []
  37. for i in range(0, len(text) - maxlen, step):
  38.     sentences.append(text[i: i + maxlen])
  39.     next_chars.append(text[i + maxlen])
  40. print('nb sequences:', len(sentences))
  42. print('Vectorization...')
  43. x = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)
  44. y = np.zeros((len(sentences), len(chars)), dtype=np.bool)
  45. for i, sentence in enumerate(sentences):
  46.     for t, char in enumerate(sentence):
  47.         x[i, t, char_indices[char]] = 1
  48.     y[i, char_indices[next_chars[i]]] = 1
  51. # build the model: a single LSTM
  52. print('Build model...')
  53. model = Sequential()
  54. model.add(LSTM(128, input_shape=(maxlen, len(chars))))
  55. model.add(Dense(len(chars)))
  56. model.add(Activation('softmax'))
  58. optimizer = RMSprop(lr=0.01)
  59. model.compile(loss='categorical_crossentropy', optimizer=optimizer)
  62. def sample(preds, temperature=1.0):
  63.     # helper function to sample an index from a probability array
  64.     preds = np.asarray(preds).astype('float64')
  65.     preds = np.log(preds) / temperature
  66.     exp_preds = np.exp(preds)
  67.     preds = exp_preds / np.sum(exp_preds)
  68.     probas = np.random.multinomial(1, preds, 1)
  69.     return np.argmax(probas)
  72. def on_epoch_end(epoch, logs):
  73.     # Function invoked at end of each epoch. Prints generated text.
  74.     print()
  75.     print('----- Generating text after Epoch: %d' % epoch)
  77.     start_index = random.randint(0, len(text) - maxlen - 1)
  78.     for diversity in [0.2, 0.5, 1.0, 1.2]:
  79.         print('----- diversity:', diversity)
  81.         generated = ''
  82.         sentence = text[start_index: start_index + maxlen]
  83.         generated += sentence
  84.         print('----- Generating with seed: "' + sentence + '"')
  85.         sys.stdout.write(generated)
  87.         for i in range(400):
  88.             x_pred = np.zeros((1, maxlen, len(chars)))
  89.             for t, char in enumerate(sentence):
  90.                 x_pred[0, t, char_indices[char]] = 1.
  92.             preds = model.predict(x_pred, verbose=0)[0]
  93.             next_index = sample(preds, diversity)
  94.             next_char = indices_char[next_index]
  96.             generated += next_char
  97.             sentence = sentence[1:] + next_char
  99.             sys.stdout.write(next_char)
  100.             sys.stdout.flush()
  101.         print()
  103. print_callback = LambdaCallback(on_epoch_end=on_epoch_end)
  105. model.fit(x, y,
  106.           batch_size=128,
  107.           epochs=60,
  108.           callbacks=[print_callback])

文本生成背后的原理 - 只不过是概率的预测而己

TFLearn的封装做得太好，我们看不到细节。所以我们参看一下Keras的同功能实现的代码:

  1.         for i in range(400):
  2.             x_pred = np.zeros((1, maxlen, len(chars)))
  3.             for t, char in enumerate(sentence):
  4.                 x_pred[0, t, char_indices[char]] = 1.
  6.             preds = model.predict(x_pred, verbose=0)[0]
  7.             next_index = sample(preds, diversity)
  8.             next_char = indices_char[next_index]
  10.             generated += next_char
  11.             sentence = sentence[1:] + next_char
  13.             sys.stdout.write(next_char)
  14.             sys.stdout.flush()

我们可以看到，本质上是调用model.predict来预测在当前序列下最可能出现的字符是什么。

  1. preds = model.predict(x_pred, verbose=0)[0]

从sample的代码中，我们可以看到温度值背后的原理：

  1. def sample(preds, temperature=1.0):
  2.     # helper function to sample an index from a probability array
  3.     preds = np.asarray(preds).astype('float64')
  4.     preds = np.log(preds) / temperature
  5.     exp_preds = np.exp(preds)
  6.     preds = exp_preds / np.sum(exp_preds)
  7.     probas = np.random.multinomial(1, preds, 1)
  8.     return np.argmax(probas)

了解了这些原理之后，使用不同的文本，加上不同的温度值，享受您的机器创作之旅吧！