Natural Language Processing

Bridging human language and machine understanding. Learn how raw, messy text is cleaned, tokenized, turned into numbers, and used to model sequences — the foundation of every LLM.

9.1 Why NLP needs numbers

Computers only understand numbers. Human language — words, sentences, paragraphs — is meaningless to a machine until it is converted into numerical vectors. The whole NLP pipeline is about that conversion: raw text → clean text → tokens → numbers → vectors that preserve meaning.

Applications: virtual assistants (Siri, Alexa), machine translation, sentiment analysis, chatbots, spam filtering, autocomplete.

9.2 Text Preprocessing

To a computer, "Apple", "apple" and "apple!" are three different words. Preprocessing normalises text so the model is not confused by trivial differences.

Lowercasing

Converts all text to lowercase ("HELLO" → "hello"). Reduces vocabulary size and prevents the same word being treated as different tokens. Caution: case can matter sometimes ("Apple" the company vs "apple" the fruit).

Punctuation removal

Removes symbols like ! , . ? #. Reduces noise and lets the model focus on core words. Caution: punctuation occasionally carries meaning.

Stop word removal

Stemming vs Lemmatization

We want "waiting", "waited", "waits" to map to one root so the model recognises them as related.

import string
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

text = "Ugh... The deliveries were DELAYED!! I hate waiting"

# 1. Lowercase
text = text.lower()
# 2. Remove punctuation
text = text.translate(str.maketrans('', '', string.punctuation))
# 3. Remove stop words
tokens = [w for w in text.split()
          if w not in stopwords.words('english')]
print("Cleaned tokens:", tokens)

# 4. Stem vs Lemmatize
print("studies ->", PorterStemmer().stem("studies"),
      "|", WordNetLemmatizer().lemmatize("studies"))

9.3 Tokenization

Two strategies

• Word tokenization (simple split) — split on spaces. "I love AI" → ["I", "love", "AI"]. Simple but fails on rare/unknown words.
• Sub-word tokenization (BERT/GPT style) — breaks rare words into known chunks. "microtransactional" → ["micro", "##tra", "##ns", "##act", "##ional"]. This handles any word, even made-up ones.

9.4 Word Embeddings

Tokens are still text. Models need numbers. There are three ways to turn words into numbers — each better than the last.

Naive approach — number the words

Apple=1, Banana=2, Carrot=3… Fails, because it invents a fake ordering (Carrot > Apple?) the same way label-encoding nominal data does.

One-Hot Encoding

Each word becomes a binary vector the size of the whole vocabulary — one position is 1, the rest 0. Problems: (1) huge & sparse — a 50,000-word vocabulary gives 50,000-length vectors that are almost all zeros; (2) "dumb" — it treats "happy" and "joy" as completely unrelated.

Bag of Words (BoW)

Represents a sentence by word counts over the vocabulary. "Good movie" and "Movie is good" get similar vectors because they share words. Still sparse and ignores word order and meaning.

Word Embeddings — the modern solution

Key advantages over one-hot:

• Dense & compact — 50–300 dimensions instead of 50,000.
• Semantic — "king" and "queen" sit close together in vector space; "king" and "apple" sit far apart.
• Learned — the vectors are learned automatically during training.

from sklearn.feature_extraction.text import CountVectorizer

corpus = ["Win free money", "Win huge money"]
bow = CountVectorizer().fit_transform(corpus)
print("BoW matrix:\n", bow.toarray())

# Modern: pre-trained sentence embeddings
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
vecs = model.encode(["King", "Queen", "Apple"])
print("Embedding length:", len(vecs[0]))   # 384 dense numbers

Padding

• padding='pre' → zeros added at the beginning.
• padding='post' → zeros added at the end.
• maxlen → the final fixed length.

Example: [90, 400, 22, 10, 5] padded to length 9 (post) → [90, 400, 22, 10, 5, 0, 0, 0, 0]. The zeros act like "empty desks" the model ignores.

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

sentences = ["I love AI", "The deliveries were delayed", "Win free money"]

tokenizer = Tokenizer(oov_token="<OOV>")
tokenizer.fit_on_texts(sentences)
seqs = tokenizer.texts_to_sequences(sentences)

padded = pad_sequences(seqs, padding='post')
print(padded)

9.5 Sequential Modelling & Language Foundations

Next-word prediction

The core task behind text generation and autocomplete: given a sequence of words, predict the most likely next word. Given "The cat sat on the", a good model predicts "mat" or "floor".

The next-word-prediction workflow

1. Tokenize & integer-encode the text.
2. Generate N-gram sequences (progressively longer chunks).
3. Pad the sequences to equal length.
4. Split into features X (the context) and target y (the next word).
5. Apply an Embedding layer.
6. Process through an RNN.
7. A Dense output layer with Softmax picks the next word.

Text generation

Once trained, the model generates text iteratively: start with a seed ("The Sky"), predict the next word ("is"), append it ("The Sky is"), and repeat. Modern LLMs like GPT (Lecture 10) do exactly this — but with Transformer architectures and billions of parameters, producing remarkably human-like text.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, SimpleRNN, Dense

model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=16))  # word -> dense vector
model.add(SimpleRNN(32))                             # read the sequence
model.add(Dense(1, activation='sigmoid'))            # spam vs not-spam

model.compile(loss='binary_crossentropy',
              optimizer='adam', metrics=['accuracy'])
model.fit(padded_data, labels, epochs=5, validation_split=0.2)

NLP preprocessing details are heavily tested — work through every one.

import string

text = "Ugh... The DELIVERIES were Delayed!!"

# Lowercase
text = text.lower()
# Remove punctuation: maketrans builds a translation table
text = text.translate(str.maketrans('', '', string.punctuation))

print(text)

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

sentences = ["I love NLP", "Win free money now"]

tokenizer = Tokenizer(oov_token="<OOV>")
tokenizer.fit_on_texts(sentences)
seqs = tokenizer.texts_to_sequences(sentences)

padded = pad_sequences(seqs, maxlen=5, padding='post')
print(padded)