Cultural Shft or Lngustc Drft? Comparng Two Computatonal Measures of Semantc Change Wllam L. Hamlton, Jure Leskovec, Dan Jurafsky Department of Computer Scence, Stanford Unversty, Stanford CA, 94305 wlef,jure,jurafsky@stanford.edu Abstract Words shft n meanng for many reasons, ncludng cultural factors lke new technologes and regular lngustc processes lke subjectfcaton. Understandng the evoluton of language and culture requres dsentanglng these underlyng causes. Here we show how two dfferent dstrbutonal measures can be used to detect two dfferent types of semantc change. The frst measure, whch has been used n many prevous works, analyzes global shfts n a word s dstrbutonal semantcs; t s senstve to changes due to regular processes of lngustc drft, such as the semantc generalzaton of promse ( I promse. It promsed to be exctng. ). The second measure, whch we develop here, focuses on local changes to a word s nearest semantc neghbors; t s more senstve to cultural shfts, such as the change n the meanng of cell ( prson cell cell phone ). Comparng measurements made by these two methods allows researchers to determne whether changes are more cultural or lngustc n nature, a dstncton that s essental for work n the dgtal humantes and hstorcal lngustcs. 1 Introducton Dstrbutonal methods of embeddng words n vector spaces accordng to ther co-occurrence statstcs are a promsng new tool for dachronc semantcs (Gulordava and Baron, 2011; Jatowt and Duh, 2014; Kulkarn et al., 2014; Xu and Kemp, 2015; Hamlton et al., 2016). Prevous work, however, does not consder the underlyng causes of semantc change or how to dstentangle dfferent types of change. We show how two computatonal measures can be used to dstngush between semantc changes caused by cultural shfts (e.g., technologcal advancements) and those caused by more regular processes of semantc change (e.g., grammatcalzaton or subjectfcaton). Ths dstncton s essental for research on lngustc and cultural evoluton. Detectng cultural shfts n language use s crucal to computatonal studes of hstory and other dgtal humantes projects. By contrast, for advancng hstorcal lngustcs, cultural shfts amount to nose and only the more regular shfts matter. Our work bulds on two ntutons: that dstrbutonal models can hghlght syntagmatc versus paradgmatc relatons wth neghborng words (Schutze and Pedersen, 1993) and that nouns are more lkely to undergo changes due to rregular cultural shfts whle verbs more readly partcpate n regular processes of semantc change (Gentner and France, 1988; Traugott and Dasher, 2001). We use ths noun vs. verb mappng as a proxy to compare our two measures senstvtes to cultural vs. lngustc shfts. Senstvty to nomnal shfts ndcates a propensty to capture rregular cultural shfts n language, such as those due to technologcal advancements (Traugott and Dasher, 2001). Senstvty to shfts n verbs (and other predcates) ndcates a propensty to capture regular processes of lngustc drft (Gentner and France, 1988; Kntsch, 2000; Traugott and Dasher, 2001). The frst measure we analyze s based upon changes to a word s local semantc neghborhood;
Global measure of change Local neghborhood measure of change frolcsome joyous daft merry gay (1900s) brllant wtty lesban hspanc gay (1990s) homosexual woman queer heterosexual Fgure 1: Two dfferent measures of semantc change. Wth the global measure of change, we measure how far a word has moved n semantc space between two tme-perods. Ths measure s senstve to subtle shfts n usage and also global effects due to the entre semantc space shftng. For example, ths captures how actually underwent subjectfcaton durng the 20th century, shftng from uses n objectve statements about the world ( actually dd try ) to subjectve statements of atttude ( I actually agree ; see Traugott and Dasher, 2001 for detals). In contrast, wth the local neghborhood measure of change, we measure changes n a word s nearest neghbors, whch captures drastc shfts n core meanng, such as gay s shft n meanng over the 20th century. we show that t s more senstve to changes n the nomnal doman and captures changes due to unpredctable cultural shfts. Our second measure reles on a more tradtonal global noton of change; we show that t better captures changes, lke those n verbs, that are the result of regular lngustc drft. Our analyss reles on a large-scale statstcal study of sx hstorcal corpora n multple languages, along wth case-studes that llustrate the fne-graned dfferences between the two measures. 2 Methods We use the dachronc word2vec embeddngs constructed n our prevous work (Hamlton et al., 2016) to measure how word meanngs change between consecutve decades. 1 In these representatons each word w has a vector representaton w (t) (Turney and Pantel, 2010) at each tme pont, whch captures ts co-occurrence statstcs for that tme perod. The vectors are constructed usng the skp-gram wth negatve samplng (SGNS) algorthm (Mkolov et al., 2013) and post-processed to algn the semantc spaces between years. Measurng the dstance between word vectors for consecutve decades allows us to compute the rate at whch the dfferent words 1 http://nlp.stanford.edu/projects/hstwords/. Ths URL also lnks to detaled dataset descrptons and the code needed to replcate the experments n ths paper. change n meanng (Gulordava and Baron, 2011). We analyzed the decades from 1800 to 1990 usng vectors derved from the Google N-gram datasets (Ln et al., 2012) that have large amounts of hstorcal text (Englsh, French, German, and Englsh Fcton). We also used vectors derved from the Corpus of Hstorcal Amercan Englsh (COHA), whch s smaller than Google N-grams but was carefully constructed to be genre balanced and contans word lemmas as well as surface forms (Daves, 2010). We examned all decades from 1850 through 2000 usng the COHA dataset and used the part-of-speech tags provded wth the corpora. 2.1 Measurng semantc change We examne two dfferent ways to measure semantc change (Fgure 1). Global measure The frst measure analyzes global shfts n a word s vector semantcs and s dentcal to the measure used n most prevous works (Gulordava and Baron, 2011; Jatowt and Duh, 2014; Km et al., 2014; Hamlton et al., 2016). We smply take a word s vectors for two consecutve decades and measure the cosne dstance between them,.e. d G (w (t), w (t+1) ) = cos-dst(w (t), w (t+1) ). (1)
(Verb - noun) change 0.2 0.1 0.0 0.1 0.2 0.3 0.4 Global measure Local measure Englsh (All) Englsh (Fc.) German French COHA (word) COHA (lemma) Fgure 2: The global measure s more senstve to semantc changes n verbs whle the local neghborhood measure s more senstve to noun changes. Examnng how much nouns change relatve to verbs (usng coeffcents from mxed-model regressons) reveals that the two measures are senstve to dfferent types of semantc change. Across all languages, the local neghborhood measure always assgns relatvely hgher rates of change to nouns (.e., the rght/green bars are lower than the left/blue bars for all pars), though the results vary by language (e.g., French has hgh noun change-rates overall). 95% confdence ntervals are shown. Local neghborhood measure The second measure s based on the ntuton that only a word s nearest semantc neghbors are relevant. For ths measure, we frst fnd word w s set of k nearest-neghbors (accordng to cosne-smlarty) wthn each decade, whch we denote by the ordered set N k (w (t) ). Next, to measure the change between decades t and t + 1, we compute a second-order smlarty vector for w (t) from these neghbor sets wth entres defned as s (t) (j) = cos-sm(w (t), w (t) j ) w j N k (w (t) ) N k (w (t+1) ), (2) and we compute an analogous vector for w (t+1). The second-order vector, s (t), contans the cosne smlarty of w and the vectors of all w s nearest semantc neghbors n the the tme-perods t and t + 1. Workng wth varants of these second-order vectors has been a popular approach n many recent works, though most of these works defne these vectors aganst the full vocabulary and not just a word s nearest neghbors (del Prado Martn and Brendel, 2016; Eger and Mehler, 2016; Rodda et al., 2016). Fnally, we compute the local neghborhood change as d L (w (t), w (t+1) ) = cos-dst(s (t), s (t+1) ). (3) Ths measures the extent to whch w s smlarty wth ts nearest neghbors has changed. The local neghborhood measure defned n (3) captures strong shfts n a word s paradgmatc relatons but s less senstve to global shfts n syntagmatc contexts (Schutze and Pedersen, 1993). We Dataset # Nouns # Verbs Google Englsh All 5299 2722 Google Englsh Fc. 4941 3128 German 5443 1844 French 2310 4992 COHA (Word) 4077 1267 COHA (Lemma) 3389 783 Table 1: Number of nouns and verbs tested n each dataset. used k = 25 n all experments (though we found the results to be consstent for k [10, 50]). 2.2 Statstcal methodology To test whether nouns or verbs change more accordng to our two measures of change, we buld on our prevous work and used a lnear mxed model approach (Hamlton et al., 2016). Ths approach amounts to a lnear regresson where the model also ncludes random effects to account for the fact that the measurements for ndvdual words wll be correlated across tme (McCulloch and Neuhaus, 2001). We ran two regressons per datatset: one wth the global d G values as the dependent varables (DVs) and one wth the local neghborhood d L values. In both cases we examned the change between all consecutve decades and normalzed the DVs to zeromean and unt varance. We examned nouns/verbs wthn the top-10000 words by frequency rank and removed all words that occurred <500 tmes n the smaller COHA dataset. The ndependent varables are word frequency, the decade of the change (represented categorcally), and varable ndcatng
Word 1850s context 1990s context actually...dnners whch you have actually eaten. Wth that, I actually agree. must O, George, we must have fath. Whch you must have heard ten years ago... promse I promse to pay you......the day promsed to be lovely. gay Gay brdals and other merry-makngs of men....the result of gay rghts demonstratons. vrus Ths young man s...nfected wth the vrus....a rapdly spreadng computer vrus. cell The door of a gloomy cell... They really need ther cell phones. Table 2: Example case-studes of semantc change. The frst three words are examples of regular lngustc shfts, whle the latter three are examples of words that shfted due to exogenous cultural factors. Contexts are from the COHA data (Daves, 2010). Global - local change 0.3 0.1 0.1 0.3 Regular lngustc shfts Irregular cultural shfts actually must promse gay vrus cell Fgure 3: The global measure captures classc examples of lngustc drft whle the local measure captures example cultural shfts. Examnng the semantc dstance between the 1850s and 1990s shows that the global measure s more senstve to regular shfts (and vce-versa for the local measure). The plot shows the dfference between the measurements made by the two methods. whether a word s a noun or a verb (proper nouns are excluded, as n Hamlton et al., 2016). 2 3 Results Our results show that the two seemngly related measures actually result n drastcally dfferent notons of semantc change. 3.1 Nouns vs. verbs The local neghborhood measure assgns far hgher rates of semantc change to nouns across all languages and datasets whle the opposte s true for the global dstance measure, whch tends to assgn hgher rates of change to verbs (Fgure 2). We focused on verbs vs. nouns snce they are the two major parts-of-speech and prevous research has shown that verbs are more semantcally mutable than nouns and thus more lkely to undergo lngustc drft (Gentner and France, 1988), whle nouns are far more lkely to change due to cultural shfts lke new technologes (Traugott and Dasher, 2001). However, some well-known regular lngustc shfts nclude rarer parts of speech lke adverbs (ncluded n our case studes below). Thus we also confrmed 2 Frequency was ncluded snce t s known to strongly nfluence the dstrbutonal measures (Hamlton et al., 2016). that the dfferences shown n Fgure 2 also hold when adverbs and adjectves are ncluded along wth the verbs. Ths modfed analyss showed analogous sgnfcant trends, whch fts wth prevous research argung that adverbal and adjectval modfers are also often the target of regular lngustc changes (Traugott and Dasher, 2001). The results of ths large-scale regresson analyss show that the local measure s more senstve to changes n the nomnal doman, a doman n whch change s known to be drven by cultural factors. In contrast, the global measure s more senstve to changes n verbs, along wth adjectves and adverbs, whch are known to be the targets of many regular processes of lngustc change (Traugott and Dasher, 2001; Hopper and Traugott, 2003) 3.2 Case studes We examned sx case-study words grouped nto two sets. These case studes show that three examples of well-attested regular lngustc shfts (set A) changed more accordng to the global measure, whle three well-known examples of cultural changes (set B) change more accordng to the local neghborhood measure. Table 2 lsts these words wth some representatve hstorcal contexts (Daves, 2010).
Set A contans three words that underwent attested regular lngustc shfts detaled n Traugott and Dasher (2001): actually, must, and promse. These three words represent three dfferent types of regular lngustc shfts: actually s a case of subjectfcaton (detaled n Fgure 1); must shfted from a deontc/oblgaton usage ( you must do X ) to a epstemc one ( X must be the case ), exemplfyng a regular pattern of change common to many modal verbs; and promse represents the class of shftng performatve speech acts that undergo rch changes due to ther pragmatc uses and subjectfcaton (Traugott and Dasher, 2001). The contexts lsted n Table 2 exemplfy these shfts. Set B contans three words that were selected because they underwent well-known cultural shfts over the last 150 years: gay, vrus, and cell. These words ganed new meanngs due to uses n communty-specfc vernacular (gay) or technologcal advances (vrus, cell). The cultural shfts underlyng these changes n usage e.g., the development of the moble cell phone were unpredctable n the sense that they were not the result of regulartes n human lngustc systems. Fgure 3 shows how much the meanng of these word changed from the 1850s to the 1990s accordng to the two dfferent measures on the Englsh Google data. We see that the words n set A changed more when measurements were made usng the global measure, whle the opposte holds for set B. 4 Dscusson Our results show that our novel local neghborhood measure of semantc change s more senstve to changes n nouns, whle the global measure s more senstve to changes n verbs. Ths mappng algns wth the tradtonal dstncton between rregular cultural shfts n nomnals and more regular cases of lngustc drft (Traugott and Dasher, 2001) and s further renforced by our sx case studes. Ths fndng emphaszes that researchers must develop and use measures of semantc change that are tuned to specfc tasks. For example, a cultural change-pont detecton framework would be more successful usng our local neghborhood measure, whle an emprcal study of grammatcalzaton would be better off usng the tradtonal global dstance approach. Comparng measurements made by these two approaches also allows researchers to assess the extent to whch semantc changes are lngustc or cultural n nature. Acknowledgements The authors thank C. Mannng, V. Prabhakaran, S. Kumar, and our anonymous revewers for ther helpful comments. Ths research has been supported n part by NSF CNS-1010921, IIS-1149837, IIS-1514268 NIH BD2K, ARO MURI, DARPA XDATA, DARPA SIMPLEX, Stanford Data Scence Intatve, SAP Stanford Graduate Fellowshp, NSERC PGS-D, Boeng, Lghtspeed, and Volkswagen. References Mark Daves. 2010. The Corpus of Hstorcal Amercan Englsh: 400 mllon words, 1810-2009. http://corpus.byu.edu/coha/. Fermn Moscoso del Prado Martn and Chrstan Brendel. 2016. Case and Cause n Icelandc: Reconstructng Causal Networks of Cascaded Language Changes. In Proc. ACL. Steffen Eger and Alexander Mehler. 2016. On the Lnearty of Semantc Change: Investgatng Meanng Varaton va Dynamc Graph Models. In Proc. ACL. Dedre Gentner and Ilene M. France. 1988. The verb mutablty effect: Studes of the combnatoral semantcs of nouns and verbs. Lexcal ambguty resoluton: Perspectves from psycholngustcs, neuropsychology, and artfcal ntellgence, pages 343 382. Krstna Gulordava and Marco Baron. 2011. A dstrbutonal smlarty approach to the detecton of semantc change n the Google Books Ngram corpus. In Proc. GEMS 2011 Workshop on Geometrcal Models of Natural Language Semantcs, pages 67 71. Assocaton for Computatonal Lngustcs. Wllam L. Hamlton, Jure Leskovec, and Dan Jurafsky. 2016. Dachronc Word Embeddngs Reveal Statstcal Laws of Semantc Change. In Proc. ACL. Paul J Hopper and Elzabeth Closs Traugott. 2003. Grammatcalzaton. Cambrdge Unversty Press, Cambrdge, UK. Adam Jatowt and Kevn Duh. 2014. A framework for analyzng semantc change of words across tme. In Proc. 14th ACM/IEEE-CS Conf. on Dgtal Lbrares, pages 229 238. IEEE Press. Yoon Km, Y-I. Chu, Kentaro Hanak, Darshan Hegde, and Slav Petrov. 2014. Temporal analyss of language through neural language models. arxv preprnt arxv:1405.3515.
Walter Kntsch. 2000. Metaphor comprehenson: A computatonal theory. Psychon. Bull. Rev., 7(2):257 266. Vvek Kulkarn, Ram Al-Rfou, Bryan Perozz, and Steven Skena. 2014. Statstcally sgnfcant detecton of lngustc change. In Proc. 24th WWW Conf., pages 625 635. Yur Ln, Jean-Baptste Mchel, Erez Leberman Aden, Jon Orwant, Wll Brockman, and Slav Petrov. 2012. Syntactc annotatons for the google books ngram corpus. In Proc. ACL System Demonstratons. Charles E McCulloch and John M Neuhaus. 2001. Generalzed lnear mxed models. Wley-Interscence, Hoboken, NJ. Tomas Mkolov, Ilya Sutskever, Ka Chen, Greg S. Corrado, and Jeff Dean. 2013. Dstrbuted representatons of words and phrases and ther compostonalty. In NIPS. Martna Rodda, Marco Senald, and Alessandro Lenc. 2016. Panta re: Trackng Semantc Change wth Dstrbutonal Semantcs n Ancent Greek. In Italan Conference of Computatonal Lngustcs. Hnrch Schutze and Jan Pedersen. 1993. A vector model for syntagmatc and paradgmatc relatedness. In Proc. 9th Annu. Conf. of the UW Centre for the New OED and Text Research, pages 104 113. Cteseer. Elzabeth Closs Traugott and Rchard B Dasher. 2001. Regularty n Semantc Change. Cambrdge Unversty Press, Cambrdge, UK. Peter D. Turney and Patrck Pantel. 2010. From frequency to meanng: Vector space models of semantcs. J. Artf. Intell. Res., 37(1):141 188. Yang Xu and Charles Kemp. 2015. A computatonal evaluaton of two laws of semantc change. In Proc. 37th Annu. Conf. Cogn. Sc. Soc.