Don't Panic — Natural Language Processing

Natural language processing is the set of techniques that let machines read, generate, and reason about human language — the same stuff you use every time you search the web, ask a voice assistant a question, or receive an auto-generated summary. Most of the world's data is unstructured text: emails, documents, medical records, social media posts. NLP exists to make that data computable.

Before NLP, a machine treated language as opaque character sequences. With NLP, it can detect entities, resolve references, classify documents, translate between languages, and produce coherent text. The field has been through three eras, and knowing which one you are in saves you from the mistakes each one made.

Rule-based systems (1950s–1990s) were grammars and dictionaries written by hand. They worked in the small worlds their authors specified and failed completely outside them. Statistical methods (1990s–2013) replaced hand-crafted rules with probabilities learned from large corpora. Coverage improved, but someone still had to design the features. Neural and pretrained models (2013–present) — Word2Vec, BERT, GPT, and the Transformer architecture behind them — learn representations directly from raw text, and that is where the field has been ever since.

The Transformer (2017) is the idea everything else hangs off. Instead of reading text one word at a time like older recurrent networks, a Transformer uses self-attention to let every word attend to every other word simultaneously. This enables massive parallelism during training and handles long-range dependencies naturally. Every major model you will encounter — BERT, GPT, T5, LLaMA — is a Transformer.

Two paradigms emerged. BERT reads text bidirectionally and excels at understanding: classification, entity recognition, question answering. GPT reads left to right and excels at generation: text completion, instruction following, creative tasks. They differ in training objective and attention mask, not in architecture. Knowing which one you need saves you from fine-tuning a generator to do classification or prompting an encoder to write prose.

The one thing that will surprise you: fluency is not truthfulness. Generative models produce text that reads as if a human wrote it, and that includes sentences that are factually wrong, hallucinated, or subtly biased by their training data. Evaluation is harder in NLP than in many other fields — automated metrics are imperfect proxies, and human evaluation remains the gold standard for generation tasks.

What to read next: the Slides tab maps the ecosystem and the decision points. The Reference tab is the study path. The Field Notes tab tells you what it actually costs when teams get tokenization, evaluation, or model selection wrong. Start there if you want to skip the theory and go straight to the parts that bite.

Don't Panic — Natural Language Processing

Relevant careers