使用频谱共聚算法对文档进行聚合

翻译者：@N!no 校验者：待校验

这个例子演示了20个新闻组数据集上的光谱协聚类算法。‘comp.os.ms-windows.misc’ 类别被排除在外，因为它包含许多只包含数据的帖子。

TF-IDF 矢量帖构成一个词频矩阵，然后使用 Dhillon 光谱协聚算法对其进行重组。由此产生的文档词双聚类表明在这些子集文档中被使用频率更高的子集词。

对于一些最好的双聚类来说，它最常见的文档类别和十个最重要的单词会被打印出来。最佳双类别由其归一化的切割决定。最好的单词是通过比较它们在两区内和两区外的总和来确定的。

为了进行比较，我们还使用 MiniBatchKMeans 对文档进行集群。从双聚类衍生出的文档聚类比使用 MiniBatchKMeans 得到的聚类具有更好的 V-measure。

  1. Vectorizing...
  2. Coclustering...
  3. Done in 2.75s. V-measure: 0.4387
  4. MiniBatchKMeans...
  5. Done in 5.69s. V-measure: 0.3344
  7. Best biclusters:
  8. ----------------
  9. bicluster 0 : 1829 documents, 2524 words
  10. categories   : 22% comp.sys.ibm.pc.hardware, 19% comp.sys.mac.hardware, 18% comp.graphics
  11. words        : card, pc, ram, drive, bus, mac, motherboard, port, windows, floppy
  13. bicluster 1 : 2391 documents, 3275 words
  14. categories   : 18% rec.motorcycles, 17% rec.autos, 15% sci.electronics
  15. words        : bike, engine, car, dod, bmw, honda, oil, motorcycle, behanna, ysu
  17. bicluster 2 : 1887 documents, 4232 words
  18. categories   : 23% talk.politics.guns, 19% talk.politics.misc, 13% sci.med
  19. words        : gun, guns, firearms, geb, drugs, banks, dyer, amendment, clinton, cdt
  21. bicluster 3 : 1146 documents, 3263 words
  22. categories   : 29% talk.politics.mideast, 26% soc.religion.christian, 25% alt.atheism
  23. words        : god, jesus, christians, atheists, kent, sin, morality, belief, resurrection, marriage
  25. bicluster 4 : 1732 documents, 3967 words
  26. categories   : 26% sci.crypt, 23% sci.space, 17% sci.med
  27. words        : clipper, encryption, key, escrow, nsa, crypto, keys, intercon, secure, wiretap
  1. from collections import defaultdict
  2. import operator
  3. from time import time
  5. import numpy as np
  7. from sklearn.cluster import SpectralCoclustering
  8. from sklearn.cluster import MiniBatchKMeans
  9. from sklearn.datasets import fetch_20newsgroups
  10. from sklearn.feature_extraction.text import TfidfVectorizer
  11. from sklearn.metrics.cluster import v_measure_score
  13. print(__doc__)
  16. def number_normalizer(tokens):
  17.     """ 将所有数字标记映射到占位符。
  19.     对于许多应用程序来说，以数字开头的令牌并没有直接的用处，但是这样的令牌存在的事实可能是相关的。通过应用这种降维形式，一些方法可能会表现得更好。
  20.     """
  21.     return ("#NUMBER" if token[0].isdigit() else token for token in tokens)
  24. class NumberNormalizingVectorizer(TfidfVectorizer):
  25.     def build_tokenizer(self):
  26.         tokenize = super().build_tokenizer()
  27.         return lambda doc: list(number_normalizer(tokenize(doc)))
  30. # 不包含 'comp.os.ms-windows.misc' 类别
  31. categories = ['alt.atheism', 'comp.graphics',
  32.               'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware',
  33.               'comp.windows.x', 'misc.forsale', 'rec.autos',
  34.               'rec.motorcycles', 'rec.sport.baseball',
  35.               'rec.sport.hockey', 'sci.crypt', 'sci.electronics',
  36.               'sci.med', 'sci.space', 'soc.religion.christian',
  37.               'talk.politics.guns', 'talk.politics.mideast',
  38.               'talk.politics.misc', 'talk.religion.misc']
  39. newsgroups = fetch_20newsgroups(categories=categories)
  40. y_true = newsgroups.target
  42. vectorizer = NumberNormalizingVectorizer(stop_words='english', min_df=5)
  43. cocluster = SpectralCoclustering(n_clusters=len(categories),
  44.                                  svd_method='arpack', random_state=0)
  45. kmeans = MiniBatchKMeans(n_clusters=len(categories), batch_size=20000,
  46.                          random_state=0)
  48. print("Vectorizing...")
  49. X = vectorizer.fit_transform(newsgroups.data)
  51. print("Coclustering...")
  52. start_time = time()
  53. cocluster.fit(X)
  54. y_cocluster = cocluster.row_labels_
  55. print("Done in {:.2f}s. V-measure: {:.4f}".format(
  56.     time() - start_time,
  57.     v_measure_score(y_cocluster, y_true)))
  59. print("MiniBatchKMeans...")
  60. start_time = time()
  61. y_kmeans = kmeans.fit_predict(X)
  62. print("Done in {:.2f}s. V-measure: {:.4f}".format(
  63.     time() - start_time,
  64.     v_measure_score(y_kmeans, y_true)))
  66. feature_names = vectorizer.get_feature_names()
  67. document_names = list(newsgroups.target_names[i] for i in newsgroups.target)
  70. def bicluster_ncut(i):
  71.     rows, cols = cocluster.get_indices(i)
  72.     if not (np.any(rows) and np.any(cols)):
  73.         import sys
  74.         return sys.float_info.max
  75.     row_complement = np.nonzero(np.logical_not(cocluster.rows_[i]))[0]
  76.     col_complement = np.nonzero(np.logical_not(cocluster.columns_[i]))[0]
  77.     # 注意：接下来的操作等同于 X[rows[:, np.newaxis], cols].sum() 
  78.     #      但是会针对于 scipy <= 0.16 的版本更快一些
  79.     weight = X[rows][:, cols].sum()
  80.     cut = (X[row_complement][:, cols].sum() +
  81.            X[rows][:, col_complement].sum())
  82.     return cut / weight
  85. def most_common(d):
  86.     """默认字典有最大值的项。
  88.     在 Python >= 2.7 中类似于 Counter.most_common 。
  89.     """
  90.     return sorted(d.items(), key=operator.itemgetter(1), reverse=True)
  93. bicluster_ncuts = list(bicluster_ncut(i)
  94.                        for i in range(len(newsgroups.target_names)))
  95. best_idx = np.argsort(bicluster_ncuts)[:5]
  97. print()
  98. print("Best biclusters:")
  99. print("----------------")
  100. for idx, cluster in enumerate(best_idx):
  101.     n_rows, n_cols = cocluster.get_shape(cluster)
  102.     cluster_docs, cluster_words = cocluster.get_indices(cluster)
  103.     if not len(cluster_docs) or not len(cluster_words):
  104.         continue
  106.     # 种类
  107.     counter = defaultdict(int)
  108.     for i in cluster_docs:
  109.         counter[document_names[i]] += 1
  110.     cat_string = ", ".join("{:.0f}% {}".format(float(c) / n_rows * 100, name)
  111.                            for name, c in most_common(counter)[:3])
  113.     # 单词
  114.     out_of_cluster_docs = cocluster.row_labels_ != cluster
  115.     out_of_cluster_docs = np.where(out_of_cluster_docs)[0]
  116.     word_col = X[:, cluster_words]
  117.     word_scores = np.array(word_col[cluster_docs, :].sum(axis=0) -
  118.                            word_col[out_of_cluster_docs, :].sum(axis=0))
  119.     word_scores = word_scores.ravel()
  120.     important_words = list(feature_names[cluster_words[i]]
  121.                            for i in word_scores.argsort()[:-11:-1])
  123.     print("bicluster {} : {} documents, {} words".format(
  124.         idx, n_rows, n_cols))
  125.     print("categories   : {}".format(cat_string))
  126.     print("words        : {}\n".format(', '.join(important_words)))