{"id":273,"date":"2016-06-11T15:47:59","date_gmt":"2016-06-11T15:47:59","guid":{"rendered":"http:\/\/www.twonewthings.com\/gabrielrecchia\/?p=273"},"modified":"2016-06-15T10:36:14","modified_gmt":"2016-06-15T10:36:14","slug":"numberless-degrees-of-similitude-word-vectors","status":"publish","type":"post","link":"http:\/\/www.twonewthings.com\/gabrielrecchia\/2016\/06\/11\/numberless-degrees-of-similitude-word-vectors\/","title":{"rendered":"\u201cNumberless degrees of similitude\u201d: A response to Ryan Heuser\u2019s
\u2018Word Vectors in the Eighteenth Century, Part 1\u2019"},"content":{"rendered":"
\n

As someone who has been involved in the world of word vectors since 2007, it\u2019s been fascinating to watch the innovative applications that they\u2019ve been put to over the last few years. After reading Ryan Heuser\u2019s \u2018Word Vectors in the Eighteenth Century, Part 1<\/a>,\u2019 I was left impressed not only by the thoughtful scholarship but also by the sheer mystery of how such an unusual analogy (\u201criches are to virtue as learning is to genius\u201d) might be encoded in the lexical statistics of a corpus of eighteenth-century texts (ECCO-TCP<\/a>). This post is an attempt to answer that question for myself. My discoveries along the way may be of interest to others curious about the lexical contexts of \u2018virtue,\u2019 \u2018riches,\u2019 \u2018learning,\u2019 and \u2018genius\u2019 in the eighteenth century, or to anyone who finds themselves in the business of interpreting word vectors.<\/p>\n

Very good descriptions of word vectors and vector space models have been provided by others inside<\/a> and outside<\/a> of the digital humanities, but they’re still unknown enough within DH that I thought it might be worth giving a brief overview and history. If you\u2019re already familiar with the mechanics and history of vector space models and want to get straight to the investigation of virtue, riches, learning, <\/em>and genius<\/em> in ECCO-TCP, feel free to skip ahead<\/a><\/a>.<\/p>\n<\/div>\n

\n

Lexical maths and fresh politeness<\/h2>\n

A vector is simply an ordered list of numbers. The vectors in a vector space model need not necessarily represent words, but for\u00a0the models that interest us here, every word is assigned a unique vector that contains information about the lexical contexts in which it appears in a large body of text. There are many different ways in which this information can be encoded. Here\u2019s one particularly simplistic approach: Start by creating a vector that that is intended to represent the contextual usage of a particular word (the so-called target word<\/em>) in some particular corpus, denoted here by V(target<\/em>). Let each component of that vector correspond to a different context word <\/em>that appears in the same sentence as the target word, and instantiate that component with the number of sentences in which the two words co-occur. Suppose our target word is king<\/em>, and our context words are he, his, him, she, her, hers, crown, throne, royal<\/em>, and majesty<\/em>. The resulting ten-dimensional vector might look something like this:<\/p>\n\n\n\n\n
he<\/td>\nhis<\/td>\nhim<\/td>\nshe<\/td>\nher<\/td>\nhers<\/td>\ncrown<\/td>\nthrone<\/td>\nroyal<\/td>\nmajesty<\/td>\n<\/tr>\n
6509<\/td>\n8612<\/td>\n3093<\/td>\n486<\/td>\n1053<\/td>\n1<\/td>\n360<\/td>\n223<\/td>\n335<\/td>\n491<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In other words, king <\/em>appears in the same sentence as he <\/em>6509 times, in the same sentence as his <\/em>8612 times, and so on. A handy thing about vectors is that we can do arithmetic with them. For example, if we assume the following plausible vector for the target word man<\/em>\u2026<\/p>\n\n\n\n\n
he<\/td>\nhis<\/td>\nhim<\/td>\nshe<\/td>\nher<\/td>\nhers<\/td>\ncrown<\/td>\nthrone<\/td>\nroyal<\/td>\nmajesty<\/td>\n<\/tr>\n
10438<\/td>\n10466<\/td>\n3409<\/td>\n1582<\/td>\n1961<\/td>\n12<\/td>\n70<\/td>\n31<\/td>\n42<\/td>\n49<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u2026we can calculate what we get if we subtract V(man<\/em>) from V<\/em>(king<\/em>), by taking the difference between corresponding components:<\/p>\n\n\n\n\n
he<\/td>\nhis<\/td>\nhim<\/td>\nshe<\/td>\nher<\/td>\nhers<\/td>\ncrown<\/td>\nthrone<\/td>\nroyal<\/td>\nmajesty<\/td>\n<\/tr>\n
-3929<\/td>\n-1854<\/td>\n-316<\/td>\n-1096<\/td>\n-908<\/td>\n-11<\/td>\n290<\/td>\n192<\/td>\n293<\/td>\n442<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u00a0<\/em><\/p>\n

Not surprisingly, we end up with a vector V(king<\/em>) \u2013 <\/em>V(man<\/em>) that has low values for words that appear more frequently with man <\/em>than with king<\/em>, and higher values for words that appear more frequently with king <\/em>than with man. <\/em>There are some obvious problems with vectors that have been constructed in this way. For one thing, we\u2019re not controlling for the frequency of the target words. A relatively rare word like Croesus <\/em>might yield a vector of very small numbers across the board. So although we might hope that V(king<\/em>) \u2013 V(Croesus<\/em>) would yield a vector that could tell us something about the differences between the discourse contexts of king <\/em>and Croesus<\/em>, in actuality we\u2019d likely get a vector that is nearly identical to king<\/em>\u2019s, which would not be particularly useful. And we\u2019re not controlling for the frequency of context words, so common words like he<\/em>, his<\/em>, and so on will have undue influence. Even so, if we continue the thought experiment by adding V(woman<\/em>) to V(king<\/em>) \u2013 <\/em>V(man<\/em>), it\u2019s plausible that we\u2019d get something like the following:<\/p>\n\n\n\n\n
he<\/td>\nhis<\/td>\nhim<\/td>\nshe<\/td>\nher<\/td>\nhers<\/td>\ncrown<\/td>\nthrone<\/td>\nroyal<\/td>\nmajesty<\/td>\n<\/tr>\n
-2680<\/td>\n-1003<\/td>\n-4<\/td>\n701<\/td>\n1698<\/td>\n19<\/td>\n306<\/td>\n196<\/td>\n301<\/td>\n448<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u00a0<\/em><\/p>\n

In other words, we’d get a vector that has low values<\/em> for context words which we expect man <\/em>and king <\/em>to share in common (e.g. he, his, him<\/em>), high values <\/em>for context words with which woman <\/em>occurs much more frequently than man <\/em>(e.g. she, her, hers<\/em>), and high values<\/em> for context words with which king <\/em>occurs much more frequently than man <\/em>(e.g. crown, throne, royal, majesty<\/em>). These are all properties that we\u2019d expect V(queen<\/em>) to have. If we\u2019re lucky, it\u2019s even possible that V(queen<\/em>) would come up as the best match if we were to quantify the overall similarity of V(king<\/em>) \u2013 <\/em>V(man<\/em>) + <\/em>V(woman<\/em>) to the vectors of every possible target word in our corpus. If this turned out to be true, then this vector space model could be said to have successfully completed the analogy \u201cman is to king as woman is to ___.<\/em>\u201d<\/p>\n

The above example is not a particularly good one except for illustrative purposes, because of how na\u00efve this particular algorithm for building a vector space model is. Not only does it fail to control for word frequency, it only attends to ten context words that we cherry-picked rather than the thousands of different words that appear in our corpus. This model is not likely to do well on any battery of analogy questions. However, I include the example because I hope it\u2019s intuitive to readers who aren\u2019t intimately familiar with these models, and also to underscore that it\u2019s possible to do analogical vector arithmetic in\u00a0fairly traditional vector spaces, even though this sort of thing has become much more closely associated with popular recent tools such as word2vec<\/em>, which construct a rather different sort of vector that I\u2019ll describe shortly. <\/em>Indeed, Levy, Goldberg, & Dagan (2015) have demonstrated that traditional count-based models can hold their own on analogy tasks against word2vec, with the requisite tweaks.<\/p>\n

A historical aside: Vector space models have a long history in computational linguistics, information retrieval, and cognitive science. One interesting predecessor in the world of cognitive psychology was the famed psychologist Charles Osgood\u2019s semantic differential<\/em>, an experimental technique in which study participants had to rate words on dimensions such as active <\/em>versus passive<\/em>, angular <\/em>versus rounded<\/em>, and small <\/em>versus large<\/em>. On a scale of fresh <\/em>to stale<\/em>, how fresh (stale) is the concept POLITE<\/em>? (Empirical answer below.)<\/p>\n

\"Semantic

Semantic differentials, averaged across two groups of twenty study participants each. From The Nature and Measurement of Meaning<\/em>, pp. 229. Psychological Bulletin, 49(3), May 1952.<\/p><\/div>\n

Tasks to which vector space models have been applied include retrieving documents related to a particular search query, grouping together documents with related meanings, classifying texts by genre, essay grading, modelling how the human brain might represent some aspects of lexical semantics, and many others. As a result, much attention has been paid to building models that optimize performance on various evaluation criteria, such as maximizing correlations with human judgments of word similarity, scores on multiple-choice tests of synonymy, or accuracy on sentence completion tasks. Less attention has been paid to questions of interpretation that are more relevant to the humanities (how best to interpret the dimensions of various species of word vector, what kinds of information can and cannot be retrieved from the statistics of language, what kinds of analogies are most and least successfully represented in vector spaces, etc.), although there has been important research in all of these areas as well.<\/p>\n

Vector space models for which the initial step is to count the number of contexts (sentences, documents, etc.) in which context words co-occur with the target word are generally called count-based <\/em>models; these can be viewed as upgraded versions of the na\u00efve model illustrated earlier in this post. These are generally made more robust by incorporating transformations that control for overall word frequency and\/or make the model more robust to data sparsity. It\u2019s also common for count-based models to use alternative notions of the \u2018context\u2019 of a target word, such as a \u2018window\u2019 that includes only a small number of terms that appear just to its left or right.<\/p>\n

In some count-based models, the dimensions correspond directly to transformed counts of particular words (e.g., Bullinaria & Levy 2007), whereas others use statistical methods to transform high-dimensional vectors to lower-dimensional versions (Landauer & Dumais 1997; Pennington, Socher, & Manning 2014), making the interpretation of individual dimensions a bit more fraught. Depending on who you ask, the class of \u2018count-based models\u2019 may or may not include random vector models <\/em>(Jones, Willits, & Dennis, 2015) such as random indexing (Kanerva, Kristofersson, & Holst 2000; Sahlgren 2005) and BEAGLE (Jones, Kintsch, & Mewhort, 2006), which assign a randomly initialized and unchanging index vector <\/em>to each context word, and an initially empty memory vector <\/em>to each target word. As the algorithm chugs through a corpus of texts, the index vectors of words that appear in the context of a target word are added to the target word\u2019s memory vector. Ultimately, memory vectors that repeatedly have the same index vectors added to them (crown, throne, royal, majesty<\/em>) end up more similar to each other, resulting once again in a space of memory vectors in which words that appear in similar contexts end up having similar vectorial representations.<\/p>\n

\"An

An example of how vector similarity between vectors representing a word and the word’s context changes with context in a random vector model. From Jones & Mewhort (2007), Psychological Review<\/em>, 114(1), p. 18.<\/p><\/div>\n

Count-based models are frequently contrasted with prediction-based[1]<\/a> models, such as those generated by the popular software package word2vec <\/em>(Mikolov, Chen, Corrado, & Dean, 2013). I won\u2019t explain the inner workings of word2vec<\/em> in detail as this has been done by many<\/a> others<\/a>, other than to say that word2vec<\/em> consists of two distinct algorithms. The first, continuous bag of words (CBOW)<\/em>, attempts to predict the target word given its context, while the skip-gram <\/em>algorithm attempts to predict a word\u2019s context given the word itself. Skip-grams seem to achieve equal or superior performance to CBOW for most purposes, and as such this is the algorithm employed most frequently in the literature. To optimize its ability to predict contexts that actually appear in the corpus, and to minimize its tendency to predict contexts that don\u2019t, the algorithm continuously tweaks a large number of parameters according to a partially stochastic process. Ultimately, the learned model parameters corresponding to a given target word are treated as that word\u2019s vector representation. These relatively low-dimensional [2]<\/a> vectors are often described as \u201cword embeddings,\u201d to distinguish them from vectors of models in which every dimension corresponds to one of an extremely large number of context words.<\/p>\n

Levy & Goldberg rocked the world of word embeddings in 2014 when they demonstrated that despite the apparent gulf between the inner workings of prediction-based and count-based models, word2vec<\/em>\u2019s skip-gram algorithm is implicitly factorizing a word-context PMI matrix. In other words, the core of what it\u2019s doing, mathematically speaking, is not much different than the sort of thing that some count-based models have been doing for awhile. This is particularly clear in the case of the Glove model (Pennington, Socher, & Manning, 2014), which explicitly factorizes a word-context PMI matrix, and generally achieves comparable results to word2vec. However, word2vec has two other properties that contribute to its continued popularity:<\/p>\n