NLP Essentials

Tokenization, TF-IDF vs embeddings vs transformers, attention intuition.

Start here

Computers only understand numbers, so step one for any NLP task is turning words into numbers — that conversion has gone through three generations, and the progression matters more than memorizing any one of them.

— first decide the "units": whole words, word pieces (subwords), or characters.
(oldest) — just counts word frequency, zero understanding of meaning.
Word embeddings () — a dense vector per word, similar words land close together — but one fixed vector regardless of context ("bank" the river = "bank" the company).
embeddings (current) — gives each word a *different* vector depending on the words around it — truly contextual. Powers , , and virtually all modern NLP.

Keep that arc — count-based → fixed vectors → contextual vectors — in mind and the rest of this module reasons through easily instead of needing memorization.

Core threads

: splitting text into units (word, subword/, character). (used by /) handles rare/unseen words gracefully by breaking them into known pieces.
: sparse, frequency-based representation — good , no semantic meaning ("bank" the river vs "bank" the company look identical).
Word embeddings (/GloVe): dense vectors where semantically similar words are close in vector space — but one fixed vector per word regardless of context.
Transformers/: lets each token "look at" every other token and weigh their relevance — this gives *contextual* embeddings (same word, different vector depending on sentence). is the mechanism behind /.
Why transformers won: parallelizable (unlike RNNs which are sequential), captures long-range dependencies better, scales well with data and compute.

Fintech relevance

An MFS company like bKash uses NLP for: KYC document + text extraction, chatbot/IVR intent classification, complaint/ticket categorization, and Bangla-English code-mixed text handling (important — BD text is rarely "clean" English or Bangla).

Viva angle

" vs embeddings vs transformers — when would you use which?" is a very likely question — answer with a tradeoff (speed/interpretability vs semantic power vs compute cost).

Visual reference

Self-attention (simplified)

While encoding "bank," the model looks at every other token and weighs how relevant each is — here, "raised" and "rates" matter most, correctly hinting at the financial sense of "bank."

Cheatsheet

TF-IDF

TF(t,d) × log(N / DF(t))

Term frequency × inverse document frequency.

Attention weight

softmax(Q·Kᵀ / √d)·V

Query-Key similarity → weighted sum of Values.

Text representation evolution

Method	Understands context?	Notes
TF-IDF	No	Sparse, frequency counts only
Word2Vec / GloVe	No (fixed per word)	Dense, similar words are close
Transformer embeddings	Yes	Same word, different vector by sentence

BERT vs GPT

	BERT	GPT
Pretraining	Masked language modeling	Next-token prediction
Context	Bidirectional	Left-to-right (causal)
Best at	Understanding (classification, NER)	Generation (chat, completion)

▸ () means the never truly hits an "unknown word."
▸ is crude/fast; is grammar-aware/correct.
▸: no training needed, use a pretrained with a prompt.

Further study

Jay Alammar — The Illustrated Transformer ↗The single most-cited visual explainer for self-attention and the transformer architecture — read this before the attention questions.Stanford CS224n — NLP with Deep Learning ↗Free lecture notes and slides covering embeddings through transformers in real depth, if you want to go beyond viva level.

Question bank

19 questions

#1Easytokenization

What is , and why do modern models use instead of word-level ?

#2Mediumrepresentations

Compare , word embeddings (), and -based contextual embeddings.

#3Hardattentiontransformers

At an intuition level, what does compute, and why is it useful for language?

#4MediumfintechscenariobKash

How would an NLP pipeline for KYC document processing typically work?

#5Easyclassic-nlp

What is vs ?

#6Mediumcode-mixedfintechbKash

Why is Bangla-English code-mixed text ("banglish") a special challenge for NLP models in a Bangladeshi context?

#7Mediumclassification

What is and when would you reach for it?

#8Mediumtransformerspretraining

What is the difference between -style and -style pretraining objectives, at a high level?

#9Mediummetrics

How would you evaluate a text classification versus a text generation ?

#10MediumfintechchatbotbKash

How would you design intent classification for a customer support chatbot?

#11Mediumpositional-encodingtransformers

has no built-in sense of word order — so how do transformers know the order of tokens in a sequence?

#12Mediumattentiontransformers

Why do transformers use multiple heads instead of just one, larger computation?

#13Easyseq2seqtransformers

What do the encoder and decoder halves of a seq2seq each do? Give an example use case.

#14Mediumdecodinggeneration

What's the tradeoff between greedy decoding and beam search when generating text from a language ?

#15Mediummetricslanguage-models

What does perplexity measure as a language evaluation metric?

#16Hardragllm

What problem does Retrieval-Augmented Generation (RAG) solve for LLMs, and how does it work at a high level?

#17Mediumllmfine-tuning

With a modern LLM, when would you reach for prompting versus fine-tuning?

#18Easymetricsgeneration

What do BLEU and ROUGE actually compare, and what's their key limitation?

#19MediumllmhallucinationfintechbKash

What does it mean when an LLM "," and how do production systems reduce it?