数据描述

波士顿房价数据集（Boston House Price Dataset）（https://www.kaggle.com/c/boston-housing）

boston_housing数据集对房价数据进行回归分析，数据来自1970年代，波斯顿周边地区的房价，是用于机器学习的经典数据集。该数据集很小，共计506条数据

每条数据包含房屋以及房屋周围的详细信息。其中包含城镇犯罪率，一氧化氮浓度，住宅平均房间数，到中心区域的加权距离以及自住房平均房价等等。

  • CRIM：城镇人均犯罪率。
  • ZN：住宅用地超过 25000 sq.ft. 的比例。
  • INDUS：城镇非零售商用土地的比例。
  • CHAS：查理斯河空变量（如果边界是河流，则为1；否则为0）。
  • NOX：一氧化氮浓度。
  • RM：住宅平均房间数。
  • AGE：1940 年之前建成的自用房屋比例。
  • DIS：到波士顿五个中心区域的加权距离。
  • RAD：辐射性公路的接近指数。
  • TAX：每 10000 美元的全值财产税率。
  • PTRATIO：城镇师生比例。
  • B：1000（Bk-0.63）^ 2，其中 Bk 指代城镇中黑人的比例。
  • LSTAT：人口中地位低下者的比例。
  • MEDV：自住房的平均房价，以千美元计。

keras普通实现

  1. import numpy as np
  2. import matplotlib.pyplot as plt
  4. from keras.datasets import boston_housing
  5. from keras.models import Sequential
  6. from keras.layers import Dense
  7. from keras.callbacks import EarlyStopping
  9. from sklearn.preprocessing import StandardScaler
  12. def load_data():
  13.     (x_train, y_train), (x_test, y_test) = boston_housing.load_data()
  14.     x = np.vstack((x_train,x_test))
  15.     y = np.concatenate((y_train, y_test))
  16.     x = StandardScaler().fit_transform(x)
  17.     y = StandardScaler().fit_transform(y.reshape(-1, 1))
  18.     x_train = x[1:401, :]
  19.     x_test = x[401:, :]
  20.     y_train = y[1:401, :]
  21.     y_test = y[401:, :]
  22.     return (x_train, y_train), (x_test, y_test)
  25. def draw_train_history(history):
  26.     # summarize history for loss
  27.     plt.plot(history.history['loss'])
  28.     plt.plot(history.history['val_loss'])
  29.     plt.title('model loss')
  30.     plt.ylabel('loss')
  31.     plt.xlabel('epoch')
  32.     plt.legend(['train', 'validation'], loc='upper left')
  33.     plt.show()
  36. def build_model():
  37.     model = Sequential()
  38.     model.add(Dense(64, activation='relu', input_shape=(13,)))
  39.     model.add(Dense(64, activation='relu'))
  40.     model.add(Dense(1, activation='linear'))
  41.     model.compile(optimizer='adam',
  42.                   loss='mean_squared_error')
  43.     return model
  46. if __name__ == '__main__':
  47.     (x_train, y_train), (x_test, y_test) = load_data()
  49.     model = build_model()
  50.     early_stopping = EarlyStopping(monitor='loss', patience=10)
  51.     history = model.fit(x_train, y_train,
  52.                         epochs=500,
  53.                         batch_size=64,
  54.                         validation_split=0.2,
  55.                         callbacks=[early_stopping])
  56.     draw_train_history(history)
  58.     loss = model.evaluate(x_test, y_test, batch_size=64)
  59.     print("test loss: {}".format(loss))

模型结构

回归-keras - 图1

模型输出

  1. test loss: 0.20772670450664701

模型损失曲线

回归-keras - 图2

keras集成学习-平均法

  1. import numpy as np
  2. import matplotlib.pyplot as plt
  4. from keras.datasets import boston_housing
  5. from keras.models import Model
  6. from keras.layers import Input, Dense, average
  7. from keras.callbacks import EarlyStopping
  9. from sklearn.preprocessing import StandardScaler
  12. def load_data():
  13.     (x_train, y_train), (x_test, y_test) = boston_housing.load_data()
  14.     x = np.vstack((x_train,x_test))
  15.     y = np.concatenate((y_train, y_test))
  16.     x = StandardScaler().fit_transform(x)
  17.     y = StandardScaler().fit_transform(y.reshape(-1, 1))
  18.     x_train = x[1:401, :]
  19.     x_test = x[401:, :]
  20.     y_train = y[1:401, :]
  21.     y_test = y[401:, :]
  22.     return (x_train, y_train), (x_test, y_test)
  25. def draw_train_history(history):
  26.     # summarize history for loss
  27.     plt.plot(history.history['loss'])
  28.     plt.plot(history.history['val_loss'])
  29.     plt.title('model loss')
  30.     plt.ylabel('loss')
  31.     plt.xlabel('epoch')
  32.     plt.legend(['train', 'validation'], loc='upper left')
  33.     plt.show()
  36. def build_model():
  37.     inputs = Input(shape=(13, ))
  38.     model1_1 = Dense(64, activation='relu')(inputs)
  39.     model2_1 = Dense(128, activation='relu')(inputs)
  40.     model3_1 = Dense(32, activation='relu')(inputs)
  41.     model1_2 = Dense(32, activation='relu')(model1_1)
  42.     model2_2 = Dense(64, activation='relu')(model2_1)
  43.     model3_2 = Dense(16, activation='relu')(model3_1)
  44.     model1_out = Dense(1, activation='linear')(model1_2)
  45.     model2_out = Dense(1, activation='linear')(model2_2)
  46.     model3_out = Dense(1, activation='linear')(model3_2)
  47.     out = average([model1_out, model2_out, model3_out])
  48.     model = Model(inputs=inputs, outputs=out)
  49.     model.compile(optimizer='adam',
  50.                   loss='mean_squared_error')
  51.     return model
  54. if __name__ == '__main__':
  55.     (x_train, y_train), (x_test, y_test) = load_data()
  57.     model = build_model()
  58.     early_stopping = EarlyStopping(monitor='loss', patience=10)
  59.     history = model.fit(x_train, y_train,
  60.                         epochs=500,
  61.                         batch_size=64,
  62.                         validation_split=0.2,
  63.                         callbacks=[early_stopping])
  64.     draw_train_history(history)
  65.     model.save("regression-average-ensemble.h5")
  67.     loss = model.evaluate(x_test, y_test, batch_size=64)
  68.     print("test loss: {}".format(loss))

模型结构

回归-keras - 图3

模型输出

  1. test loss: 0.14132633038929532

模型损失曲线

回归-keras - 图4

keras-scikit_learn集成学习

  1. import numpy as np
  3. from keras.datasets import boston_housing
  4. from keras.wrappers.scikit_learn import KerasRegressor
  5. from keras.models import Sequential
  6. from keras.layers import Dense
  8. from sklearn.preprocessing import StandardScaler
  9. from sklearn.ensemble import VotingRegressor
  10. from sklearn.externals import joblib
  12. def load_data():
  13.     (x_train, y_train), (x_test, y_test) = boston_housing.load_data()
  14.     x = np.vstack((x_train,x_test))
  15.     y = np.concatenate((y_train, y_test))
  16.     x = StandardScaler().fit_transform(x)
  17.     y = StandardScaler().fit_transform(y.reshape(-1, 1))
  18.     x_train = x[1:401, :]
  19.     x_test = x[401:, :]
  20.     y_train = y[1:401, :]
  21.     y_test = y[401:, :]
  22.     return (x_train, y_train), (x_test, y_test)
  25. def build_model1():
  26.     model = Sequential()
  27.     model.add(Dense(128, activation='relu', input_shape=(13, )))
  28.     model.add(Dense(64, activation='relu'))
  29.     model.add(Dense(1, activation='linear'))
  30.     model.compile(optimizer='adam',
  31.                   loss='mean_squared_error')
  32.     return model
  34. def build_model2():
  35.     model = Sequential()
  36.     model.add(Dense(64, activation='relu', input_shape=(13, )))
  37.     model.add(Dense(32, activation='relu'))
  38.     model.add(Dense(1, activation='linear'))
  39.     model.compile(optimizer='adam',
  40.                   loss='mean_squared_error')
  41.     return model
  43. def build_model3():
  44.     model = Sequential()
  45.     model.add(Dense(32, activation='relu', input_shape=(13, )))
  46.     model.add(Dense(16, activation='relu'))
  47.     model.add(Dense(1, activation='linear'))
  48.     model.compile(optimizer='adam',
  49.                   loss='mean_squared_error')
  50.     return model
  53. if __name__ == '__main__':
  54.     (x_train, y_train), (x_test, y_test) = load_data()
  56.     model1 = KerasRegressor(build_fn=build_model1, epochs=100, batch_size=64)
  57.     model1._estimator_type = "regressor"
  58.     model2 = KerasRegressor(build_fn=build_model2, epochs=100, batch_size=64)
  59.     model2._estimator_type = "regressor"
  60.     model3 = KerasRegressor(build_fn=build_model3, epochs=100, batch_size=64)
  61.     model3._estimator_type = "regressor"
  63.     cls = VotingRegressor(estimators=[
  64.                                       ('model1', model1),
  65.                                       ('model2', model2),
  66.                                       ('model3', model3)
  67.                                       ])
  68.     cls.fit(x_train, y_train)
  69.     joblib.dump(cls, "sklearn-regressor.h5")
  71.     print("score: ", cls.score(x_test, y_test))

这里我们使用scikit_learn中的VotingRegressor作为模型集成的工具，在sklearn的文档中是这样描述VotingRegressor的

A voting regressor is an ensemble meta-estimator that fits several base regressors, each on the whole dataset. Then it averages the individual predictions to form a final prediction.

也就是它会将每个基学习器的输出结果做平均作为最后的输出结果。

如果我们使用StackingRegressor做模型集成工具的话，他会将每个基学习器的输出再放进一个regressor中做最后的预测，也就是下面我们要实现的学习法。我们来看一下文档中的描述

Stacked generalization consists in stacking the output of individual estimator and use a regressor to compute the final prediction. Stacking allows to use the strength of each individual estimator by using their output as input of a final estimator.

模型输出

  1. score:  0.8482947319781606

keras集成学习-学习法

  1. import numpy as np
  2. import matplotlib.pyplot as plt
  4. from keras.datasets import boston_housing
  5. from keras.models import Model
  6. from keras.layers import Input, Dense, concatenate
  7. from keras.callbacks import EarlyStopping
  9. from sklearn.preprocessing import StandardScaler
  12. def load_data():
  13.     (x_train, y_train), (x_test, y_test) = boston_housing.load_data()
  14.     x = np.vstack((x_train,x_test))
  15.     y = np.concatenate((y_train, y_test))
  16.     x = StandardScaler().fit_transform(x)
  17.     y = StandardScaler().fit_transform(y.reshape(-1, 1))
  18.     x_train = x[1:401, :]
  19.     x_test = x[401:, :]
  20.     y_train = y[1:401, :]
  21.     y_test = y[401:, :]
  22.     return (x_train, y_train), (x_test, y_test)
  25. def draw_train_history(history):
  26.     # summarize history for loss
  27.     plt.plot(history.history['loss'])
  28.     plt.plot(history.history['val_loss'])
  29.     plt.title('model loss')
  30.     plt.ylabel('loss')
  31.     plt.xlabel('epoch')
  32.     plt.legend(['train', 'validation'], loc='upper left')
  33.     plt.show()
  36. def build_model():
  37.     inputs = Input(shape=(13, ))
  38.     model1_1 = Dense(64, activation='relu')(inputs)
  39.     model2_1 = Dense(128, activation='relu')(inputs)
  40.     model3_1 = Dense(32, activation='relu')(inputs)
  41.     model1_2 = Dense(32, activation='relu')(model1_1)
  42.     model2_2 = Dense(64, activation='relu')(model2_1)
  43.     model3_2 = Dense(16, activation='relu')(model3_1)
  44.     model1_3 = Dense(1, activation='linear')(model1_2)
  45.     model2_3 = Dense(1, activation='linear')(model2_2)
  46.     model3_3 = Dense(1, activation='linear')(model3_2)
  47.     con = concatenate([model1_3, model2_3, model3_3])
  48.     output = Dense(1, activation='linear')(con)
  49.     model = Model(inputs=inputs, outputs=output)
  50.     model.compile(optimizer='adam',
  51.                   loss='mean_squared_error')
  52.     return model
  54. if __name__ == '__main__':
  55.     (x_train, y_train), (x_test, y_test) = load_data()
  57.     model = build_model()
  58.     early_stopping = EarlyStopping(monitor='loss', patience=10)
  59.     history = model.fit(x_train, y_train,
  60.                         epochs=500,
  61.                         batch_size=64,
  62.                         validation_split=0.2,
  63.                         callbacks=[early_stopping])
  64.     draw_train_history(history)
  65.     model.save("regression-ensemble.h5")
  67.     loss = model.evaluate(x_test, y_test, batch_size=64)
  68.     print("test loss: {}".format(loss))

模型结构

回归-keras - 图5

模型输出

  1. test loss: 0.16616634471075875

模型损失曲线

回归-keras - 图6

代码位置

https://github.com/Knowledge-Precipitation-Tribe/Neural-network/tree/master/code/Ensemble-Learning