On July 31, 2018, FiveThirtyEight released a dataset of nearly 3 million tweets from accounts associated with the Russian government's Internet Research Agency (IRA). The dataset resulted from a research project led by Clemson University researchers Darren Linvill and Patrick Warren, who wrote a research paper describing the project. It presents us with an opportunity to understand the objectives and methods of the people operating these accounts, starting with the basic question: what topics did they tweet about, and how did those topics differ across the audiences targetted by the IRA?

Dataset Summary

The dataset contains 2,973,371 tweets published between February 2012 and May 2018, with almost all tweets occurring after October 2014:

Each tweet record contains the user name of the account that published the tweet, the date and time that the tweet was published, the tweet's language, whether the tweet was a retweet, the number of followers of the account and the number of accounts followed by the account, and the content of the tweet. Each record also contains a category applied by Linvill and Warren:

After some initial exploratory analysis to establish a foundation for analysis, I performed some typical cleanup on the text of each tweet. Specifically, I removed punctuation, URLs, and "mentions" (i.e., @[account name] strings that occur in tweets to mention another account), normalized whitespace, and removed any tweets without contents. This left a dataset with 2,215,817 tweets.

Topic Modeling

I thought it would be interesting to look at the broad themes or topics in the tweets published by the LeftTroll and RightTroll accounts as classified by the Clemson researchers. To do so, I started by taking a 35% random sample of the tweets, just to make the processing go more quickly on my MacBook Pro; this resulted in 143,332 LeftTroll tweets and 241,978 RightTroll tweets. Next I removed common stop words, synthesized "co-located" words (e.g., the words "Donald" and "Trump" are joined to form a two-word term) if they co-occurred more than 50 times across the dataset, and removed all terms from analysis that occurred fewer than five times. I then fit a Latent Dirichlet Allocation (LDA) model, using a prior of 20 topics after experimenting with topic counts of 5, 10, and 30.

The topics learned by the model for the LeftTroll group, along with the 12 most relevant words in the topic and the labels I assigned based on my interpretation of them were:

I thought it would be interesting to attempt to visualize the assignment of tweets to topics. The relevant output from the LDA model is a matrix θ that indicates the probability of each of the topics occurring in each document. Visualizing a 143K+ x 20 matrix is challenging, but it is made much easier by using a dimensionality-reduction technique to attempt to reduce the 20 features (i.e., the topic distribution for each tweet) down to 2. I decided to see what t-Distributed Stochastic Neighbor Embedding (t-SNE) could do here, and it turned out to be quite effective, with one caveat. As it happens, for about half (49.3%) of the sampled LeftTroll tweets, the LDA model could not assign any topic with greater than a probability of .4. In other words, about half of the LeftTroll tweets contained terms prominent in several topics, so that no single topic stood out as predominant. This could be due to a large number of tweets truly addressing multiple issues (or jumbling terms from across topics), or it could be just a consequence of the number of topics selected or some other factor in how the model fitting occurred. In any case, t-SNE had difficulty clustering the tweets with this "no-predominant-topic" tweets included, so we can consider them part of one additional "topic" and exclude them from the visualization. With some experimentation, I found that using a t-SNE perplexity hyperparameter of 30 resulted in the cleanest clustering (I also tried 10, 20, and 50). The result, with each dot being a tweet and the color of each dot representing the likeliest topic for that tweet, is:

Note that there are actually 40 somewhat distinct clusters here, two for each of the 20 topics. The smaller clusters are tweets assigned to a topic with probability=1, while the larger clusters have a lower probability assignment to the likeliest topic. Overall, t-SNE with a perplexity hyperparameter of 30 seems to do a very good job at reducing the 20 columns in the θ matrix down to two for visualization.

The topics learned by the model for the RightTroll group were, not surprisingly, quite different:

I was also not surprised to see that the t-SNE dimensionality reduction resulted in a very similar visualization of the θ matrix based on the 120,075 (49.6%) RightTroll tweets with a topic-assignment probability of at least .4:

One somewhat surprising observation I noticed through experimentation was that applying a Tf-Idf transformation to the document-term matrix resulted in similar clustering via t-SNE, but the interpretability of the topics (in terms of the words in each) was far worse. In fact, after the Tf-Idf transformation, I was unable to discern a clear interpretation for any of them. I have not yet explored the underlying reason for this observation, but I may do so in a subsequent blog post.

R Packages Used...and Code

I used the excellent text2vec R package (webpage, GitHub) for all of the natural language processing (vocabulary construction, determination of co-located terms, building the document-term matrix for each sample of tweets) and fitting of the LDA model underlying the analysis presented above. I used the Rtsne package (GitHub) for applying t-SNE dimensionality reduction to the θ matrix.

The code to create the graphics for the post is available in my GitHub repository.