11

Is there a list that contains every two letter combination that is not found in any English words? I have searched for a very long time and found nothing. It would also be useful if I had three letter combinations that do not exist. I have to search through a large text file looking for strings that can contain actual words. (And another list of things that only a few words are apart of, like qa)

EDIT: I have compiled a list myself, here it is for anyone who may need it:

bk  fq  jc  jt  mj  qh  qx  vj  wz  zh
bq  fv  jd  jv  mq  qj  qy  vk  xb  zj
bx  fx  jf  jw  mx  qk  qz  vm  xg  zn
cb  fz  jg  jx  mz  ql  sx  vn  xj  zq
cf  gq  jh  jy  pq  qm  sz  vp  xk  zr
cg  gv  jk  jz  pv  qn  tq  vq  xv  zs
cj  gx  jl  kq  px  qo  tx  vt  xz  zx
cp  hk  jm  kv  qb  qp  vb  vw  yq
cv  hv  jn  kx  qc  qr  vc  vx  yv
cw  hx  jp  kz  qd  qs  vd  vz  yz
cx  hz  jq  lq  qe  qt  vf  wq  zb
dx  iy  jr  lx  qf  qv  vg  wv  zc
fk  jb  js  mg  qg  qw  vh  wx  zg

I used 70,000 words, but there are more so some other bigrams could exit. Doubtful though. (A commenter mentions teriyaki, so it's not foolproof, but the above list still solved my personal problem of finding the needle in the haystack that I was looking for)

Nicholas Pipitone
  • 221
  • 2
  • 7
  • 3
    Given the vagaries of English spelling, and its even vaguer implementations, I doubt that there are any impossible bigrams in English, and damn few trigrams. – jlawler Jul 30 '13 at 19:00
  • @jlawler How about a list that contains the percent of words that use that bigram? That would be useful because I can list them from most likely an English word to probably German. :) – Nicholas Pipitone Jul 30 '13 at 19:03
  • Feel free to compile one. – jlawler Jul 30 '13 at 19:07
  • But I cannot find a list that contains these impossible bigrams. Or a list that contains bigrams in order by usage. If I were to make one myself, I would need a list of every English word, and where would I find that? – Nicholas Pipitone Jul 30 '13 at 20:10
  • "q" is usually followed by "u" in English spelling, so impossible bigrams might include such spellings as "qg." What is more, not all letters can be doubled in written English. So "hh," "qq," and "uu" might be candidates. But I have no clue about how to compile a definitive list. – James Grossmann Jul 30 '13 at 20:21
  • It seems that with a dictionary as a reference (and there are several available on the internet), it shouldn't take long to search out a list of bigrams, and not much longer to do trigrams (if doing this through a computer program, then this is very easy). – Ryno Jul 31 '13 at 03:44
  • 2
    To get a list of English word you will first have to decide which definition of "English word" you want. People who build English corpora will tell you that no matter how big a corpus is that previously unrecorded words appear in new texts constantly. – hippietrail Aug 02 '13 at 05:04
  • 11
    bk: lambkin, subkingdom; cb: Macbeth; cp: Macpherson; fk: Kafkaesque; gv: Longview; hk: fishkill, latchkey, Poughkeepsie; iy: multiyear, teriyaki; I'm listing these to underscore other commenters' points. What counts as an English word? What corpus do you use? I personally would accept all of these as English words. By doing a search in a dictionary with 130,000 entries, I found many other bigrams from your list, as well, although most of them were more peripheral proper nouns and loanwords. – musicallinguist Aug 02 '13 at 18:59
  • Any chance you could share your dataset? Some of those bigrams seem intuitively wrong (such as bq or qx) and I'd be interested to see which words they appear in. – acattle Aug 03 '13 at 03:49
  • 2
    bk subkingdom bq subquery bx subxiphisternal cb ecbolic cf cf. cg ecgonine cj calcjarlite etc etc. There's exceptions to every rule and nothing is impossible. These may be rare bigrams but they're not impossible bigrams. I found these examples in Wiktionary. – hippietrail Aug 03 '13 at 04:19
  • D'oh @musicallinguist and I had similar ideas apparently but I missed that comment. – hippietrail Aug 03 '13 at 04:25
  • @acattle I used the freely available CMU pronouncing dictionary. – musicallinguist Aug 05 '13 at 13:46
  • 1
    Since English is a language which borrows so heavily from so many languages, I doubt that the bi-gram list would be as long as this. (Food for thought -> would you consider Zhuang (A branch of Tai languague) to be an English word) – WiccanKarnak Oct 02 '17 at 03:22
  • Further counterexamples: Rajput, kvetch, calque, calx, slumgum, ramjet, kumquat, hamza, upquiver, upvote, postquestionnaire, avgas, avtur, cowquake... – Vikki Apr 18 '22 at 04:28
  • Oh, and mixblood and taxgathering... – Vikki Apr 18 '22 at 04:36
  • ...arcweld... – Vikki Apr 18 '22 at 04:42

3 Answers3

13

Of the 676 total possible bigrams "there are only seven bigrams that do not occur among the 2.8 trillion mentions: JQ, QG, QK, QY, QZ, WQ, and WZ."

Norvig also produced data for trigrams through 9-grams "by position within word ... and also by word length." Of the 17576 (26**3) possible trigrams, Norvig found 8,653 (see types count in ngrams3 fusion table), so 8923 (50.768%) trigrams did not occur.

From http://norvig.com/mayzner.html English Letter Frequency Counts: Mayzner Revisited or ETAOIN SRHLDCU

Norvig is a Google research scientist. Analysis was based on the Google Books corpus mid 2012. Norvig normalized words (so word=Word=WORD) and then excluded those that occurred fewer than 100,000 times. Thus these 7 bigrams might legitimately exist (e.g., "howzat" per Ken Grace in the comments) but are very rare (unless you are interested in cricket).

Similarly, some bigrams might not occur depending upon what you mean by "English words." Note that some Roman numerals and abbreviations were included (e.g., no Scrabble word contains "qc" but "QC" for "quality control" was common in my technical books). The list also contains some names and some clearly foreign words (e.g., "forschungsgemeinschaft" in the longest words list must have appeared more than 100,000 times, probably due to funding acknowledgements of the German equivalent of the NSF in scientific papers per Christopher Orr in the comments). I'm unaware of an English "dictionary" word that contains "xx" but the bigram appeared 79 million times in "Exxon" (formed in 1972), "ExxonMobile" (in 1999), "XXX" (whether strike-through, placeholder, hardcore sex rating, movie or album name, Roman numeral 30, genetic abnormality, kisses, extra strong, elided XXXL size, poison/dangerous label, ... [and now a top level domain]), etc.

This list excluded all words that contained numbers or punctuation (presumably also therefore excluding contractions, possessives, and hyphenations).

In addition to Norvig's post, Google Books, and the Brown Corpus (mentioned by acattle), see the Corpus of Contemporary American English (or British or International or others) for easy to use tools and datasets that could be used to extract data patterns that might be of more interest. Google and Microsoft also provide tools and ngram datasets based on web content. Wiktionary might also be useful when considering specific bigrams. Scrabble, word finder, or other game aids might be useful as well.

http://corpus.byu.edu/coca/ http://corpus.byu.edu/ http://ngrams.googlelabs.com/ http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html http://research.microsoft.com/web-ngram https://en.wiktionary.org/wiki/Appendix:Two-letter_combinations

Note that this answer addresses the limited question of "two [and three] letter combination[s] ... not found in any English words" but not how best to identify "English words" mixed with other text (or vice versa).

Edit: Primarily added Trigram count, links list, and last paragraph.

BillR
  • 231
  • 2
  • 5
  • 26*26=676 Good catch, Agnius Vasiliauskas. I tried to reinstate your edit but this comment is the best I can do. – BillR Dec 16 '17 at 02:25
11

To elaborate on jlawler and Ryno's comments, you are very very very unlikely to find the kind of data you want precompiled and available on the web. The reason is A) it's pretty easy to do yourself, and B) it isn't particularly useful once you get past an introductory level, i.e. there are probably better ways to judge the "Englishness" of words than just looking at their bigrams.

First, you need a corpus. Obviously not every corpus will have every word in it which might leave some rare, but valid, bigrams out. You could use multiple corpora but realistically it won't make much difference. The Brown Corpus is freely available and is used a lot in this sort of thing. Alternatively, you could grab a bunch of books off of Project Gutenberg.

I'm assuming you have at least some programming knowledge since you talk about needing to scan a large text, so once you have your reference corpus it should be trivial to scan through it and create a list of all unique bigrams or compute bigram probabilities. Typically when computing bigram probabilities you should use Naive Bayes (P(AB) = P(B|A) = C(AB)/C(A), where C() is the count) instead of raw probabilities (count of the bigram / total number of bigrams). This is because the independence assumption (P(AB) = P(B|A)) means you can compute probabilities one [additional] letter at a time.

Beyond bigram or full-word based solutions, there is a similar question over on StackOverflow about English-like word generation (instead of detection) which takes a syllabic approach. Essentially you have 2 lists; valid onset/nucleus (or onset/vowel, i.e. the first half of a syllable) pairs and valid nucleus/coda (i.e. the second half of the syllable) pairs. These are then stitched together in a Markov chain resulting in valid English syllables. The important part is that these pairs are not a fixed length, for example "-a" + "at" = "at" vs. "-stre" + "ength" = "strength". From here building valid English words is trivial (off the top of my head I can't think of invalid coda/onset combinations nor any valid syllable-initial/final pairs that cannot also be word-initial/final). Of course, this still leaves the problem of data since syllabifying itself is non-trivial.

Community
  • 1
acattle
  • 2,898
  • 1
  • 14
  • 26
  • I got a dictionary containing 70k words and compiled a list myself. It made a huge difference, because checking if a word is valid by going through the entire dictionary takes like 1/4 a second, while checking through invalid bigrams was done in a matter of milliseconds. I am looking to fish through thousands of words, most of which contain completely valid bigrams, so what other ways can I test the "Englishness" of words? – Nicholas Pipitone Aug 02 '13 at 17:33
  • 3
    One of the problems is that deciding if a word is valid in a language is a phonological process and although English spelling is relatively phonemic, it isn't perfect. Beyond that, we have loanwords that are valid words but may introduce rare bigrams (such as "qa" from "Qatar"). Even then, pure bigrams alone can't account for English consonant-vowel patterns. Your program might consider a word with 0 vowels as valid. Hand-coded rules may help but lack coverage. Similarly trigrams or 4-grams would better capture vowel patterns but your data would be too sparse to be useful, requiring smoothing. – acattle Aug 03 '13 at 03:38
  • My point is that it's actually a non-trivial problem. I did a bit more googling and I found a question over on StackOverflow that you should be able to adapt to suit your needs. It seems to work at the syllable-level (which is not a fixed length in English), which should help tremendously. – acattle Aug 03 '13 at 03:44
  • Well, one field where it would be pretty useful information would be typography. Maybe typographers have compiled some of this information. – brass tacks Aug 02 '15 at 17:45
1

Really, the answer is this list: [], the list without any bigrams because of these words (short definitions are given for difficult ones): (though CX, DX, FX, JX, PX, QJ, QK, QN, QX, VF and VX are places and QG is an event)

subkingdom (BK) subquarter (BQ) subxerophilous (BX) ecbolic (CB) frolicful (CF) ecgonine (CG) arcjet (CJ) ecphonesis (CP) cvarci, a type of pork cracklings (CV) rusticwork (CW) Chicxulub, Mexico (CX) Muradxan, Azerbaijan (DX) offkey (FK) fqih, alternate spelling of faqih (FQ) arfvedstonite, a type of rock (FV) Efxeinoupoli, Greece (FX) styfziekte, a disease (FZ) gqom, a style of music (GQ) dogvane, a small windvane (GV) hangxiety, post-drinking anxiety (GX) latchkey (HK) boschvark, a type of bushpig (HV) cuauhxicalli, a type of Aztec bowl (HX) machzor, a Jewish prayer book (HZ) teriyaki (IY) rajbari, a rajah's house (JB) tajch, a type of resevoir in Slovakia (JC) slojd, a teaching method (JD) bamischijf, a Dutch snack (JF) rajgira, the amaranth plant (JG) jhala, a genre of music (JH) rijksdaalder, a type of coin (JK) majlis, a Muslim assembly (JL) sejmik, a Polish parliament (JM) jnana, knowledge in Hinduism (JN) tejpat, a type of tree (JP) Hujqan, Azerbaijan (JQ) hijri, the Islamic calendar (JR) rijsttafel, a type of dish (JS) ijtihad, reasoning by an expert in Islamic law (JT) ajvar, a bell pepper relish (JV) ajwain, an herb (JW) Wajxaklajun, a Mayan ruin in Guatemala (JX) jynx, a type of bird (JY) ijzerkoekje, a type of cookie (JZ) cuckquean, a woman with an unfaithful husband (KQ) kvass, a type of drink (KV) kickxia, a genus of plantains (KX) ekzema, an old spelling of eczema (KZ) pulque, a drink (LQ) calx, a metallic oxide (LX) slumgum (MG) circumjacent (MJ) kumquat (MQ) womxn, feminist spelling of woman (MX) hamza, an Arabic letter (MZ) upquiver, to quiver upwards (PQ) upvote (PV) Qw'apx, a Native American Nuxalk village in Canada (PX) maqbara, a Muslim cemetery (QB) qcepo, a skin disease from Peru (QC) taluqdar, an Indian aristocrat (QD) sheqel, a coin of Israel (QE) waqf, Islamic charitable endowment (QF) Iraqgate, a scandal from the 1980s (QG) fiqh, Islamic law (QH) Saqvaqjuac, a research station in Canada (QJ) Aqkand, Iran (QK) taqlid, conforming to the teachings of another (QL) iqmik, a type of tobacco Farooqnagar, India (QN) qoph, a Hebrew letter (QO) aqpik, the cloudberry in Alaska (QP) miqra, the Hebrew Bible kamotiqs, plural of kamotiq, a type of sled (QS) maqta, the last couplet in a set of two couplets in Urdu poetry (QT) qvevri, a type of jug (QV) Qwerty, taqwa (QW) Laqxija, Malta (QX) iqyax, a type of kayak (QY) sharqzadegi, the loss of Iranian manufacture due to importing from China (QZ) transxiphoid, across the xiphoid process (SX) grosz, a Polish unit of money outquote, to quote better (TQ) txalaparta, a Basque instrument (TX) navbar, a navigation bar on a webpage (VB) novcic, a Montenegrin coin (VC) havdalah, a Jewish religious ceremony (VD) Gunneklevfjord, Norway (VF) avgolemono, a Greek sauce (VG) evhoe, a Bacchic interjection (VH) navjote, the ritual in which one becomes Zoroastrian (VJ) sovkhoz, a state-owned Soviet farm (VK) tradevman, a type of officer in the US navy (VM) czarevna, a czar's daughter (VN) sovprene, Soviet rubber (VP) servqual, a marketing concept (VQ) chavtastic, related to chavs (VT) lavway, a Caribbean call and response chant (VW) Vxelodvnvm, a Roman fort in England (VX) evzone, a Greek soldier (VZ) cawquaw, a type of porcupine (WQ) lowveld, a low-altitude south African plain (WV) lewxern, archaic form of lucern, or the lynx (WX) frowzy (WZ) foxberry, the lingonberry or bearberry (XB) foxglove (XG) alexjejewite, a type of hydrocarbon boxkeeper, a person who keeps theater boxes (XK) poxvirus (XV) ethoxzolamide (XZ) triptyque, a customs pass for importing cars (YQ) wyvern (YV) analyze (YZ) fuzzball (ZB) vizcacha, a type of South American rodent (ZC) wayzgoose, a party given to lower-level printers by the master printer (ZG) quinzhee, a type of snow shelter (ZH) muzjik, a Russian peasant (ZJ) biznaga, a type of cactus (ZN) mezquite, an old form of mesquite (ZQ) mizrah, the direction Jews pray (ZR) britzska, a type of Polish carraige (ZS) eschscholtzxanthin, Mestizx (ZX)

Ethan Weyn
  • 11
  • 1