# 使用 doc2vec 進行情緒分析 既然我們知道如何訓練單詞嵌入，我們也可以擴展這些方法以進行文檔嵌入。我們將在以下部分中探討如何執行此操作。 ## 做好準備 在前面關于 word2vec 方法的部分中，我們設法捕獲了單詞之間的位置關系。我們沒有做的是捕捉單詞與它們來自的文檔（或電影評論）之間的關系。 word2vec 的一個擴展來捕獲文檔效果，稱為 doc2vec。 doc2vec 的基本思想是引入文檔嵌入，以及可能有助于捕獲文檔基調的單詞嵌入。例如，只知道單詞`movie`和`love`彼此接近可能無法幫助我們確定評論的情緒。評論可能是談論他們如何熱愛電影或他們如何不愛電影。但是如果評論足夠長并且在文檔中找到了更多否定詞，那么我們可以采用可以幫助我們預測后續詞語的整體語氣。 Doc2vec 只是為文檔添加了一個額外的嵌入矩陣，并使用一個單詞窗口加上文檔索引來預測下一個單詞。文檔中的所有文字窗口都具有相同的文檔索引。值得一提的是，考慮如何將文檔嵌入與單詞嵌入相結合是很重要的。我們通過對它們求和來將單詞嵌入組合在單詞窗口中。將這些嵌入與文檔嵌入相結合有兩種主要方式：通常，文檔嵌入要么添加到單詞嵌入中，要么連接到單詞嵌入的末尾。如果我們添加兩個嵌入，我們將文檔嵌入大小限制為與嵌入字大小相同的大小。如果我們連接，我們解除了這個限制，但增加了邏輯回歸必須處理的變量數量。為了便于說明，我們將向您展示如何處理此秘籍中的串聯。但總的來說，對于較小的數據集，添加是更好的選擇。 第一步是將文檔和單詞嵌入適用于整個電影評論集。然后我們將進行訓練測試分組，訓練邏輯模型，看看我們是否可以更準確地預測評論情緒。 ## 操作步驟 我們將按如下方式處理秘籍： 1. 我們將從加載必要的庫并開始圖會話開始，如下所示： ```py import tensorflow as tf import matplotlib.pyplot as plt import numpy as np import random import os import pickle import string import requests import collections import io import tarfile import urllib.request import text_helpers from nltk.corpus import stopwords sess = tf.Session() ``` 1. 我們將加載電影評論語料庫，就像我們在前兩個秘籍中所做的那樣。使用以下代碼執行此操作： ```py texts, target = text_helpers.load_movie_data() ``` 1. 我們將聲明模型參數，如下所示： ```py batch_size = 500 vocabulary_size = 7500 generations = 100000 model_learning_rate = 0.001 embedding_size = 200 # Word embedding size doc_embedding_size = 100 # Document embedding size concatenated_size = embedding_size + doc_embedding_size num_sampled = int(batch_size/2) window_size = 3 # How many words to consider to the left. # Add checkpoints to training save_embeddings_every = 5000 print_valid_every = 5000 print_loss_every = 100 # Declare stop words stops = stopwords.words('english') # We pick a few test words. valid_words = ['love', 'hate', 'happy', 'sad', 'man', 'woman'] ``` 1. 我們將正則化電影評論，并確保每個電影評論都大于所需的窗口大小。使用以下代碼執行此操作： ```py texts = text_helpers.normalize_text(texts, stops) # Texts must contain at least as much as the prior window size target = [target[ix] for ix, x in enumerate(texts) if len(x.split()) > window_size] texts = [x for x in texts if len(x.split()) > window_size] assert(len(target)==len(texts)) ``` 1. 現在我們將創建我們的單詞字典。請務必注意，我們不必創建文檔字典。文件索引只是文件的索引;每個文檔都有一個唯一的索引： ```py word_dictionary = text_helpers.build_dictionary(texts, vocabulary_size) word_dictionary_rev = dict(zip(word_dictionary.values(), word_dictionary.keys())) text_data = text_helpers.text_to_numbers(texts, word_dictionary) # Get validation word keys valid_examples = [word_dictionary[x] for x in valid_words] ``` 1. 接下來，我們將定義單詞嵌入和文檔嵌入。然后我們將聲明我們的噪聲對比損失參數。使用以下代碼執行此操作： ```py embeddings = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) doc_embeddings = tf.Variable(tf.random_uniform([len(texts), doc_embedding_size], -1.0, 1.0)) # NCE loss parameters nce_weights = tf.Variable(tf.truncated_normal([vocabulary_size, concatenated_size], stddev=1.0 / np.sqrt(concatenated_size))) nce_biases = tf.Variable(tf.zeros([vocabulary_size])) ``` 1. 我們現在將聲明 doc2vec 索引和目標詞索引的占位符。請注意，輸入索引的大小是窗口大小加 1.這是因為我們生成的每個數據窗口都有一個附加的文檔索引，如下所示： ```py x_inputs = tf.placeholder(tf.int32, shape=[None, window_size + 1]) y_target = tf.placeholder(tf.int32, shape=[None, 1]) valid_dataset = tf.constant(valid_examples, dtype=tf.int32) ``` 1. 現在我們必須創建嵌入函數，它將單詞嵌入加在一起，然后在最后連接文檔嵌入。使用以下代碼執行此操作： ```py embed = tf.zeros([batch_size, embedding_size]) for element in range(window_size): embed += tf.nn.embedding_lookup(embeddings, x_inputs[:, element]) doc_indices = tf.slice(x_inputs, [0,window_size],[batch_size,1]) doc_embed = tf.nn.embedding_lookup(doc_embeddings,doc_indices) # concatenate embeddings final_embed = tf.concat(axis=1, values=) ``` 1. 我們還需要聲明一組驗證詞的余弦距離，我們可以經常打印出來以觀察 doc2vec 模型的進度。使用以下代碼執行此操作： ```py loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weights, biases=nce_biases, labels=y_target, inputs=final_embed, num_sampled=num_sampled, num_classes=vocabulary_size)) # Create optimizer optimizer = tf.train.GradientDescentOptimizer(learning_rate=model_learning_rate) train_step = optimizer.minimize(loss) ``` 1. 我們還需要從一組驗證單詞中聲明余弦距離，我們可以經常打印出來以觀察 doc2vec 模型的進度。使用以下代碼執行此操作： ```py norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keep_dims=True)) normalized_embeddings = embeddings / norm valid_embeddings = tf.nn.embedding_lookup(normalized_embeddings, valid_dataset) similarity = tf.matmul(valid_embeddings, normalized_embeddings, transpose_b=True) ``` 1. 為了以后保存我們的嵌入，我們將創建一個模型`saver`函數。然后我們可以初始化變量，這是我們開始訓練單詞嵌入之前的最后一步： ```py saver = tf.train.Saver({"embeddings": embeddings, "doc_embeddings": doc_embeddings}) init = tf.global_variables_initializer() sess.run(init) loss_vec = [] loss_x_vec = [] for i in range(generations): batch_inputs, batch_labels = text_helpers.generate_batch_data(text_data, batch_size, window_size, method='doc2vec') feed_dict = {x_inputs : batch_inputs, y_target : batch_labels} # Run the train step sess.run(train_step, feed_dict=feed_dict) # Return the loss if (i+1) % print_loss_every == 0: loss_val = sess.run(loss, feed_dict=feed_dict) loss_vec.append(loss_val) loss_x_vec.append(i+1) print('Loss at step {} : {}'.format(i+1, loss_val)) # Validation: Print some random words and top 5 related words if (i+1) % print_valid_every == 0: sim = sess.run(similarity, feed_dict=feed_dict) for j in range(len(valid_words)): valid_word = word_dictionary_rev[valid_examples[j]] top_k = 5 # number of nearest neighbors nearest = (-sim[j, :]).argsort()[1:top_k+1] log_str = "Nearest to {}:".format(valid_word) for k in range(top_k): close_word = word_dictionary_rev[nearest[k]] log_str = '{} {},'.format(log_str, close_word) print(log_str) # Save dictionary + embeddings if (i+1) % save_embeddings_every == 0: # Save vocabulary dictionary with open(os.path.join(data_folder_name,'movie_vocab.pkl'), 'wb') as f: pickle.dump(word_dictionary, f) # Save embeddings model_checkpoint_path = os.path.join(os.getcwd(),data_folder_name,'doc2vec_movie_embeddings.ckpt') save_path = saver.save(sess, model_checkpoint_path) print('Model saved in file: {}'.format(save_path)) ``` 1. 這導致以下輸出： ```py Loss at step 100 : 126.176816940307617 Loss at step 200 : 89.608322143554688 ... Loss at step 99900 : 17.733346939086914 Loss at step 100000 : 17.384489059448242 Nearest to love: ride, with, by, its, start, Nearest to hate: redundant, snapshot, from, performances, extravagant, Nearest to happy: queen, chaos, them, succumb, elegance, Nearest to sad: terms, pity, chord, wallet, morality, Nearest to man: of, teen, an, our, physical, Nearest to woman: innocuous, scenes, prove, except, lady, Model saved in file: /.../temp/doc2vec_movie_embeddings.ckpt ``` 1. 現在我們已經訓練了 doc2vec 嵌入，我們可以在邏輯回歸中使用這些嵌入來預測評論情緒。首先，我們為邏輯回歸設置了一些參數。使用以下代碼執行此操作： ```py max_words = 20 # maximum review word length logistic_batch_size = 500 # training batch size ``` 1. 我們現在將數據集拆分為訓練集和測試集： ```py train_indices = np.sort(np.random.choice(len(target), round(0.8*len(target)), replace=False)) test_indices = np.sort(np.array(list(set(range(len(target))) - set(train_indices)))) texts_train = [x for ix, x in enumerate(texts) if ix in train_indices] texts_test = [x for ix, x in enumerate(texts) if ix in test_indices] target_train = np.array([x for ix, x in enumerate(target) if ix in train_indices]) target_test = np.array([x for ix, x in enumerate(target) if ix in test_indices]) ``` 1. 接下來，我們將評論轉換為數字單詞索引，并將每個評論填充或裁剪為 20 個單詞，如下所示： ```py text_data_train = np.array(text_helpers.text_to_numbers(texts_train, word_dictionary)) text_data_test = np.array(text_helpers.text_to_numbers(texts_test, word_dictionary)) # Pad/crop movie reviews to specific length text_data_train = np.array([x[0:max_words] for x in [y+[0]*max_words for y in text_data_train]]) text_data_test = np.array([x[0:max_words] for x in [y+[0]*max_words for y in text_data_test]]) ``` 1. 現在我們將聲明圖中與邏輯回歸模型相關的部分。我們將添加數據占位符，變量，模型操作和損失函數，如下所示： ```py # Define Logistic placeholders log_x_inputs = tf.placeholder(tf.int32, shape=[None, max_words + 1]) log_y_target = tf.placeholder(tf.int32, shape=[None, 1]) A = tf.Variable(tf.random_normal(shape=[concatenated_size,1])) b = tf.Variable(tf.random_normal(shape=[1,1])) # Declare logistic model (sigmoid in loss function) model_output = tf.add(tf.matmul(log_final_embed, A), b) # Declare loss function (Cross Entropy loss) logistic_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=model_output, labels=tf.cast(log_y_target, tf.float32))) ``` 1. 我們需要創建另一個嵌入函數。前半部分中的嵌入函數在三個單詞（和文檔索引）的較小窗口上進行訓練，以預測下一個單詞。在這里，我們將采用相同的方式進行 20 字復習。使用以下代碼執行此操作： ```py # Add together element embeddings in window: log_embed = tf.zeros([logistic_batch_size, embedding_size]) for element in range(max_words): log_embed += tf.nn.embedding_lookup(embeddings, log_x_inputs[:, element]) log_doc_indices = tf.slice(log_x_inputs, [0,max_words],[logistic_batch_size,1]) log_doc_embed = tf.nn.embedding_lookup(doc_embeddings,log_doc_indices) # concatenate embeddings log_final_embed = tf.concat(1, [log_embed, tf.squeeze(log_doc_embed)]) ``` 1. 接下來，我們將在圖上創建預測和準確率函數，以便我們可以在訓練生成過程中評估模型的表現。然后我們將聲明一個優化函數并初始化所有變量： ```py prediction = tf.round(tf.sigmoid(model_output)) predictions_correct = tf.cast(tf.equal(prediction, tf.cast(log_y_target, tf.float32)), tf.float32) accuracy = tf.reduce_mean(predictions_correct) # Declare optimizer logistic_opt = tf.train.GradientDescentOptimizer(learning_rate=0.01) logistic_train_step = logistic_opt.minimize(logistic_loss, var_list=[A, b]) # Intitialize Variables init = tf.global_variables_initializer() sess.run(init) ``` 1. 現在我們可以開始后勤模型訓練了： ```py train_loss = [] test_loss = [] train_acc = [] test_acc = [] i_data = [] for i in range(10000): rand_index = np.random.choice(text_data_train.shape[0], size=logistic_batch_size) rand_x = text_data_train[rand_index] # Append review index at the end of text data rand_x_doc_indices = train_indices[rand_index] rand_x = np.hstack((rand_x, np.transpose([rand_x_doc_indices]))) rand_y = np.transpose([target_train[rand_index]]) feed_dict = {log_x_inputs : rand_x, log_y_target : rand_y} sess.run(logistic_train_step, feed_dict=feed_dict) # Only record loss and accuracy every 100 generations if (i+1)%100==0: rand_index_test = np.random.choice(text_data_test.shape[0], size=logistic_batch_size) rand_x_test = text_data_test[rand_index_test] # Append review index at the end of text data rand_x_doc_indices_test = test_indices[rand_index_test] rand_x_test = np.hstack((rand_x_test, np.transpose([rand_x_doc_indices_test]))) rand_y_test = np.transpose([target_test[rand_index_test]]) test_feed_dict = {log_x_inputs: rand_x_test, log_y_target: rand_y_test} i_data.append(i+1) train_loss_temp = sess.run(logistic_loss, feed_dict=feed_dict) train_loss.append(train_loss_temp) test_loss_temp = sess.run(logistic_loss, feed_dict=test_feed_dict) test_loss.append(test_loss_temp) train_acc_temp = sess.run(accuracy, feed_dict=feed_dict) train_acc.append(train_acc_temp) test_acc_temp = sess.run(accuracy, feed_dict=test_feed_dict) test_acc.append(test_acc_temp) if (i+1)%500==0: acc_and_loss = [i+1, train_loss_temp, test_loss_temp, train_acc_temp, test_acc_temp] acc_and_loss = [np.round(x,2) for x in acc_and_loss] print('Generation # {}. Train Loss (Test Loss): {:.2f} ({:.2f}). Train Acc (Test Acc): {:.2f} ({:.2f})'.format(*acc_and_loss)) ``` 1. 這導致以下輸出： ```py Generation # 500\. Train Loss (Test Loss): 5.62 (7.45). Train Acc (Test Acc): 0.52 (0.48) Generation # 10000\. Train Loss (Test Loss): 2.35 (2.51). Train Acc (Test Acc): 0.59 (0.58) ``` 1. 我們還應該注意到，我們在名為 doc2vec 的`text_helpers.generate_batch_data()`函數中創建了一個單獨的數據批量生成方法，我們在本文的第一部分中使用它來訓練 doc2vec 嵌入。以下是與該方法有關的該函數的摘錄： ```py def generate_batch_data(sentences, batch_size, window_size, method='skip_gram'): # Fill up data batch batch_data = [] label_data = [] while len(batch_data) < batch_size: # select random sentence to start rand_sentence_ix = int(np.random.choice(len(sentences), size=1)) rand_sentence = sentences[rand_sentence_ix] # Generate consecutive windows to look at window_sequences = [rand_sentence[max((ix-window_size),0):(ix+window_size+1)] for ix, x in enumerate(rand_sentence)] # Denote which element of each window is the center word of interest label_indices = [ix if ix<window_size else window_size for ix,x in enumerate(window_sequences)] # Pull out center word of interest for each window and create a tuple for each window if method=='skip_gram': ... elif method=='cbow': ... elif method=='doc2vec': # For doc2vec we keep LHS window only to predict target word batch_and_labels = [(rand_sentence[i:i+window_size], rand_sentence[i+window_size]) for i in range(0, len(rand_sentence)-window_size)] batch, labels = [list(x) for x in zip(*batch_and_labels)] # Add document index to batch!! Remember that we must extract the last index in batch for the doc-index batch = [x + [rand_sentence_ix] for x in batch] else: raise ValueError('Method {} not implmented yet.'.format(method)) # extract batch and labels batch_data.extend(batch[:batch_size]) label_data.extend(labels[:batch_size]) # Trim batch and label at the end batch_data = batch_data[:batch_size] label_data = label_data[:batch_size] # Convert to numpy array batch_data = np.array(batch_data) label_data = np.transpose(np.array([label_data])) return batch_data, label_data ``` ## 工作原理 在這個秘籍中，我們進行了兩個訓練循環。第一個是適合 doc2vec 嵌入，第二個循環是為了適應電影情緒的邏輯回歸。 雖然我們沒有大幅度提高情緒預測準確率（它仍然略低于 60％），但我們在電影語料庫中成功實現了 doc2vec 的連接版本。為了提高我們的準確率，我們應該為 doc2vec 嵌入和可能更復雜的模型嘗試不同的參數，因為邏輯回歸可能無法捕獲自然語言中的所有非線性行為。