11

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on several occasions that trigram models outperform PCFGs.

What would be a good source to cite for trigram models being something of a standard?

What would be a good source to cite for the claim that trigram models outperform more complex models such as PCFGs?

I've gone ahead and started a bounty for this question: even just one good reference will help!

hippietrail
  • 14,687
  • 7
  • 61
  • 146
Julie
  • 377
  • 2
  • 9
  • 1
    you might see if your library has this book: http://www-nlp.stanford.edu/fsnlp/ –  Oct 06 '12 at 17:35
  • @jlovegren Thanks, although I'm not sure that they actually compare trigram models to the others empirically in that book, or even argue that they outperform more complex language models. I'll read again more closely. – Julie Oct 10 '12 at 21:52
  • I've never read the book, so you've probably gotten it right. –  Oct 11 '12 at 00:17

1 Answers1

7

The reason why trigrams can be considered powerful compared to n-grams of higher order, lies in the problem of data sparsity; when n is higher, data becomes increasingly sparse. Because of this, using trigrams is a good compromise which often yields good results.

In a study by Chen and Goodman (1998), the effect of varying n-gram orders on the performance of a language model was compared. As expected, they found that 4-grams and 5-grams significantly outperform trigram models when using a very large data set (approximately 1e+06 sentences).

I'm not sure where you could find a source stating that trigrams outperform other models, but if you cite Chen and Goodman you can at least say that higher order n-grams are only viable with large data sets.

(Sorry I can't provide you with a link to this article, but here's the citation at least!)

Chen, Stanley F. and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. Harvard University, Center for Research in Computing Technology, TR-10-98:1–63, 1998.

Bjerva
  • 332
  • 1
  • 5