Till startsida
Webbkarta
Till innehåll Läs mer om hur kakor används på gu.se

Using Deep Neural Networks to Learn Syntactic Agreement

Artikel i vetenskaplig tidskrift
Författare Jean-Philippe Bernardy
Shalom Lappin
Publicerad i Linguistic Issues in Language Technology
Volym 15
Nummer/häfte 2
Sidor 1-15
ISSN 1945-3590
Publiceringsår 2017
Publicerad vid Institutionen för filosofi, lingvistik och vetenskapsteori
Sidor 1-15
Språk en
Länkar csli-lilt.stanford.edu/ojs/index.ph...
Ämnesord deep neural networks ; syntactic agreement
Ämneskategorier Språk och litteratur, Kognitionsforskning, Elektroteknik och elektronik

Sammanfattning

We consider the extent to which different deep neural network (DNN) configurations can learn syntactic relations, by taking up Linzen et al.’s (2016) work on subject-verb agreement with LSTM RNNs. We test their methods on a much larger corpus than they used (a ~24 million example part of the WaCky corpus, instead of their ~1.35 million example corpus, both drawn from Wikipedia). We experiment with several different DNN architectures (LSTM RNNs, GRUs, and CNNs), and alternative parameter settings for these systems (vocabulary size, training to test ratio, number of layers, memory size, drop out rate, and lexical embedding dimension size). We also try out our own unsupervised DNN language model. Our results are broadly compatible with those that Linzen et al. report. However, we discovered some interesting, and in some cases, surprising features of DNNs and language models in their performance of the agreement learning task. In particular, we found that DNNs require large vocabularies to form substantive lexical embeddings in order to learn structural patterns. This finding has interesting consequences for our understanding of the way in which DNNs represent syntactic information. It suggests that DNNs learn syntactic patterns more efficiently through rich lexical embeddings, with semantic as well as syntactic cues, than from training on lexically impoverished strings that highlight structural patterns.

Sidansvarig: Webbredaktion|Sidan uppdaterades: 2012-09-11
Dela:

På Göteborgs universitet använder vi kakor (cookies) för att webbplatsen ska fungera på ett bra sätt för dig. Genom att surfa vidare godkänner du att vi använder kakor.  Vad är kakor?