How is vocabulary size measured? Five methods
Nobody counts the words a person knows one by one. Every vocabulary-size test works by sampling: take a list of words, test a small random sample of it, find what share of the sample the person knows, and multiply that share back up to the size of the list. The methods differ in three choices: which list to sample from, what counts as "knowing" a word, and how to stop people from guessing. Those choices are why published estimates for the same kind of person range from under 10,000 words to over 200,000.
The estimate itself, how many words a typical adult actually knows, is in how many words does the average person know. This post is about how the counting is done.
Method 1: sample a dictionary
The oldest method takes a big dictionary, picks words from it at regular intervals, and asks people about them. Its weakness is the dictionary itself, and a pair of studies on the same dictionary shows it clearly.
In 1946 the psychologist G. W. Hartmann took 50 words from Merriam-Webster's unabridged New International Dictionary, one from the same position on every fortieth page, and asked 106 undergraduates to give their meanings. On average they defined 26.9 of the 50. The dictionary had about 400,000 entries, so Hartmann concluded that a student knew about 215,000 words (26.9 ÷ 50 × 400,000).
In 1990 Goulden, Nation and Read went back to a later edition, Webster's Third, and first asked what its entries were. Its preface claimed over 450,000 words. Without inflected forms, that fell to 267,000 lemmas. Of those, 19,000 were names and 40,000 were second entries for words that already had one (bat, fan, spring), leaving 208,000. Of those, 64,000 were derived words and 67,000 compounds, which a person who knows the base word can often work out, and 22,000 could not be classified. That left 54,000 base words. Goulden and his colleagues tested 250 of them on 20 graduates, who only had to say whether they knew each word. The average was 17,200, with a range from 13,200 to 20,700.
Same kind of dictionary, similar groups of students, and a twelvefold difference. It cannot come from the task, because Hartmann's students had the harder one: they had to define each word. It comes from what was counted as a word. Hartmann counted names, inflected forms, derived words and compounds, and Goulden's team counted none of them.
Method 2: decide what a word is, then how well it must be known
That is why every serious test now starts by fixing its unit. A word family is a base word with its inflections and derivations, so nation, national and nationally are one; a lemma is a dictionary headword with its inflections only. The difference between the two is set out in the average-person post, and it is large. Paul Nation's notes on his own test give one figure: the 1,000 commonest English word families contain 6,857 different word forms, just under seven per family.
The second choice is what the person has to do, and a harder task gives a smaller number. In 2013 Milton and Treffers-Daller used Goulden's own word sample, but asked for a synonym or an explanation of each word instead of a simple "I know it". Their estimate, from mainly first-year university students, was 9,800 word families: well below Goulden's 17,200 from the same words.
Method 3: yes or no, with made-up words mixed in
The quickest task is to show a word and ask "do you know it?". It has an obvious problem: people say yes to words they half-recognise, or say yes to look good. In 1987 P. Meara and B. Buxton proposed the yes/no test as an alternative to multiple choice, with the fix that LexTALE and the large online studies below also use: mix in made-up words that look real. Anyone who says yes to a word that does not exist is guessing, and how often they do it tells you how much to discount the rest of their yeses.
A very large study of this kind ran online. Marc Brysbaert and colleagues at Ghent University built a list of 61,800 English lemmas, following the design of an earlier Dutch study (Keuleers and colleagues, 2015). Each visitor saw 67 real words and 33 made-up ones, generated by a program called Wuggy, and was warned that a yes to a made-up word would be penalised. The score was simple: the share of yeses to made-up words was subtracted from the share of yeses to real words. 221,268 people took part, in 265,346 sessions, and only the first three sessions per person were counted, because some people took the test more than a hundred times.
The size of the crowd is what makes this work. No one person sees all 61,800 words, but across a crowd every word is seen by enough people to know what share of them know it. The authors are clear about the limit of a yes: they call their figures "upper estimates of word knowledge, going for width rather than depth".
There is more than one way to correct for guessing. Brysbaert's team subtracted. Meara proposed a score called ΔM, with its own penalty for yeses to made-up words, and a later method by Huibregtse, Admiraal and Meara, called I-SDT, uses signal detection theory to correct for a person's general tendency to say yes or no as well as for guessing.
Method 4: LexTALE, a five-minute proficiency check
LexTALE (Lexical Test for Advanced Learners of English), published by Kristin Lemhöfer and Mirjam Broersma in 2012, uses the same yes/no format for a different purpose. It does not estimate a number of words. It gives researchers a quick, comparable measure of how good a participant's English is.
It has 60 items, 40 words and 20 made-up words, chosen from a 240-item test by Meara. The real words are rare, between 1 and 26 occurrences per million words, so even good learners will not know them all. That is why there are twice as many words as made-up words. The made-up words are pronounceable and follow English spelling: proom, made by changing letters of a real word, or rebondicate, made by joining real word parts. The score is the average of the percentage correct on the words and the percentage correct on the made-up words, so that a habit of saying yes and a habit of saying no are penalised equally. In tests with Dutch and Korean learners of English, it predicted their vocabulary knowledge well, correlated substantially with their general proficiency, and was generally a better predictor than the learners' own ratings of themselves. It can be taken online in English, Dutch and German.
Method 5: Nation's Vocabulary Size Test
Paul Nation's Vocabulary Size Test takes the opposite approach: no made-up words, and you have to show you know the meaning. Each item is a word in a short, simple sentence followed by four possible meanings. The words are sampled evenly from frequency bands of 1,000 word families each, built from the British National Corpus:
- The 14,000 version has 140 items, 10 from each band. Your score multiplied by 100 is your estimated vocabulary in word families.
- Two 20,000 versions have 100 items each, and your score is multiplied by 200.
Three design choices stand out. There is no "I don't know" option, because Nation wants learners to make informed guesses, which "are likely to draw on sub-conscious knowledge". Loanwords and cognates from the learner's own language are kept, because they are a real part of what the learner knows. And because each band has so few items, the test is not meant to show how well any one band is known: "the total score for the test is what matters." The 140-item version takes about 40 minutes.
The five methods side by side
| Method | What you do | What is counted | How guessing is handled | Time |
|---|---|---|---|---|
| Dictionary sampling (Hartmann) | Define words | Every dictionary entry | Not at all | Untimed |
| Base words (Goulden et al.) | Say whether you know each word | Base words only | Not at all | Short |
| Crowd yes/no (Brysbaert et al.) | Yes or no, made-up words mixed in | Lemmas | Subtract the made-up-word yes rate | A few minutes |
| LexTALE | Yes or no, made-up words mixed in | A proficiency score, not a count | Averaged accuracy on both kinds of item | 5 minutes |
| Vocabulary Size Test (Nation) | Choose the meaning from four | Word families, by frequency band | Informed guessing is encouraged | About 40 minutes |
How our own vocabulary tests work
Verbavia (our own course) has a free two-minute test for each of its languages, such as the Spanish vocabulary test; all of them are on the vocabulary tests page. They combine two of the methods above: the yes/no format with made-up words, and Nation's sampling by frequency band. Here is exactly what the Spanish test does.
The list. The words come from the course's own Spanish word list, which is in frequency order and reaches past rank 18,000. Names, rows that are only a conjugated form of another word, plurals and feminines of a word already on the list, and grammatical particles are left out, because they are not separate words you could be asked about. What is left is cut into six bands, from ranks 1–1,000 to everything past rank 11,000. The bands get wider further down the list, because a word at rank 9,000 and one at rank 10,000 are about equally rare.
The sample. Each attempt shows 5 random words from every band and 6 made-up words, 36 in all, shuffled together. A retake draws different words. Every made-up word was written by hand to look Spanish (bartinar, fendosa) and checked against the course's word list and against the subtitle word counts, where it must not appear even once. In Mandarin, a made-up word is two common characters that do not form a word.
The correction. For each band, your yes rate on real words is corrected by your yes rate on made-up words:
known share = (yes rate on the band − yes rate on made-up words) ÷ (1 − yes rate on made-up words)
That is the standard correction for guessing. It assumes that for a word you do not know, you say yes as often as you do for a word that does not exist. It differs from Brysbaert's plain subtraction by the division, which scales the result back up. Say a band stands for 1,000 words, you say yes to 4 of its 5, and to 1 of the 6 made-up words. Plain subtraction gives 0.80 − 0.17 = 0.63, or 633 words. The test gives 0.63 ÷ 0.83 = 0.76, or 760. If you say yes to none of the made-up words, your answers are taken as they are.
The number. Each band's known share is multiplied by the number of words it stands for, and the bands are added up. The total is rounded to the nearest step of a fixed scale. If you say yes to three or more of the six made-up words, no number is given at all, because knowledge can then no longer be told apart from guessing.
The coverage line. The result also says roughly what share of everything said in Spanish films and TV your words make up. That comes from the same subtitle frequency curve used across this blog. It is an approximation, and the page says so: the curve counts word forms (habla and hablo are two), the test counts dictionary words, and one known word usually covers several forms, so the real figure is, if anything, a little higher.
What it does not claim. It does not say how you compare with other learners, because there are no such data, and nothing you answer is sent anywhere or stored. With 5 words a band it is quick but coarse, and like every yes/no test it measures whether you recognise words, not whether you can use them. More demanding tests, which ask for the meaning, give smaller estimates.
So which number should you trust?
- Check the unit first. Word families, lemmas, word forms and dictionary entries can differ by a factor of several, as the Hartmann and Goulden studies show.
- Then check the task. Ticking a word you recognise gives a higher number than showing that you know its meaning, as Goulden's and Milton's results on the same words show.
- For a learner, a frequency-band test is the useful one. It tells you not only how many words you know but where your knowledge runs out, which is the next part of the list to learn. Why the top of that list matters most is in how many words do you need to learn a language and, for Spanish, how many words you need to speak Spanish. The Spanish course teaches the list in that order.
Which language has the most words, and why the question has no single answer, is in which language has the most words.
Sources
- Brysbaert, M., Stevens, M., Mandera, P. & Keuleers, E. (2016). How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant's age. Frontiers in Psychology, 7, 1116. frontiersin.org. The source here for the studies by Hartmann (1946), Goulden, Nation and Read (1990) and Milton and Treffers-Daller (2013), with their figures, and that more demanding tests give smaller estimates; for the crowdsourced test (61,800 lemmas, 67 words and 33 made-up words generated by Wuggy, the subtraction, the number of participants and sessions, and the first-three-sessions rule) and its design following Keuleers et al. (2015); and for "upper estimates of word knowledge".
- Lemhöfer, K. & Broersma, M. (2012). Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English. Behavior Research Methods, 44, 325–343. link.springer.com. The 60 items and their source in Meara's 10K test, the word frequencies, the made-up words proom and rebondicate, the averaged-percentage score, ΔM and I-SDT, and Meara and Buxton's (1987) proposal of yes/no tests as an alternative to multiple choice. The test itself: lextale.com.
- Nation, P. (2012). The Vocabulary Size Test: information and specifications. Victoria University of Wellington. PDF, from the vocabulary tests page. The versions and their multipliers, the 1,000-family bands from the British National Corpus, word families as the unit (6,857 word types in the first 1,000 families), no "I don't know" option, loanwords and cognates kept, and the time to take it.
- Our own test: the scoring is in
scripts/vocab_test.js, which every page at /test/ runs; which words are included, the bands and the size of the sample are inscripts/build_test_pages.py; the made-up words and the checks they must pass are inscripts/test_pseudowords.py. The coverage line uses the OpenSubtitles 2018 frequency lists of Dave, H., FrequencyWords, content licensed CC BY-SA 4.0, from Lison, P. & Tiedemann, J. (2016), OpenSubtitles2016, Proceedings of LREC 2016.