A GPT (Generative Pre-trained Transformer) is a computer program that reads text and predicts what letter comes next. “Generative” means it creates new text. “Pre-trained” means it has been taught on lots of examples. “Transformer” is the specific type of math formula it uses.
The core idea is simple: if you show a model millions of sentences and keep asking “what character comes next?”, it eventually learns the patterns — which letters tend to follow which, where punctuation goes, how sentences are structured. Once trained, it can generate original text by predicting one character at a time, feeding each prediction back in as input for the next one.
In this post, we build a complete, minimal GPT from scratch in about 250 lines of Python + PyTorch. It trains on Shakespeare and generates text character-by-character. Every component is explained, every line of code is accounted for, and interactive demos let you see each piece in action.
Most modern LLMs (GPT-4, Claude, Gemini) use subword tokenizers like BPE or SentencePiece that split text into chunks like “ing” or “tion”. We use something even simpler: one token per character.
Sentence: T o b e o r n o t t o b e
─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─
Characters: T o b e o r n o t t o b e
IDs: 46 41 0 38 35 0 41 44 0 40 41 50 0 50 41 0 38 35Every character — letters, numbers, spaces, punctuation — gets a unique integer ID. The Shakespeare dataset has 65 unique characters, so our vocabulary size is exactly 65. No subword splitting, no special tokens, no complexity.
This is the simplest possible tokenizer and it is perfect for learning. The model must learn spelling, grammar, and sentence structure purely from individual character patterns.
Type any text above to see how it gets encoded into integer IDs. Notice the input/target shift: for training, the model sees positions 0..n-1 and must predict positions 1..n. This shift-by-one is the entire learning task.
A character ID like 46 (for ‘F’) is just a label. It carries no intrinsic meaning. Before the model can reason about characters, it must convert each ID into a vector — a list of numbers that can represent meaning.
We use two separate embedding tables:
Token Embedding (the "what")
A lookup table of shape [65, 128]
65 rows (one per character), each row is 128 numbers
Char ' ' (ID 0) --> [-0.12, 0.54, 0.33, ... 128 numbers ... ]
Char 'T' (ID 46) --> [ 0.89, -0.01, 0.72, ... 128 numbers ... ]
Char 'o' (ID 41) --> [-0.34, 0.67, -0.11, ... 128 numbers ... ]
During training, these numbers are slowly tuned.
Similar characters end up with similar vectors.
Position Embedding (the "where")
A lookup table of shape [256, 128]
256 rows (one per position in context), each row is 128 numbers
Position 0 --> [-0.01, 0.22, 0.15, ... ]
Position 1 --> [ 0.05, -0.11, 0.42, ... ]
Position 2 --> [-0.08, 0.33, -0.07, ... ]
Combined: token_embedding[T] + position_embedding[0]
= representation of 'T' at the start of the sequenceWhy both? Because the same character at different positions means different things. The ‘o’ at position 4 in “hello world” is the end of “hello”, while the ‘o’ at position 7 is the start of “world”. Same letter, different positions, different meaning — the position embedding lets the model distinguish them.
Type a word above to see how each character gets a token embedding (what it means) and a position embedding (where it sits), which are added together to form the full representation.
Once characters are embedded, they pass through transformer blocks. Each block does two things: gossip (attention) and think (feed-forward), with safety rails in between.
The complete recipe for one block:
Input [64, 256, 128]
|
├── LayerNorm ──> Multi-Head Self-Attention ──> + (residual) ──>
│ |
└───────────────────────────────────────────────────────────┘
|
├── LayerNorm ──> Feed-Forward Network ──> + (residual) ──>
│ |
└────────────────────────────────────────────────────────┘
|
Output [64, 256, 128]Self-attention is the core innovation of the Transformer. Every character looks at every previous character and asks: “how relevant are you to me?” It then blends information from the most relevant characters.
Each character generates three vectors:
| Vector | Role | Question | |--------|------|----------| | Query (Q) | What am I looking for? | “Does anyone have info I need?” | | Key (K) | What do I contain? | “I have this kind of information” | | Value (V) | What should I share? | “Here is my actual content” |
A character’s Query is compared against every previous character’s Key. The match score determines how much of that character’s Value gets blended in. Characters attending to themselves keeps their own identity.
The attention formula:
If this formula looks unfamiliar, we have a full dedicated post on the Attention Mechanism Explained that walks through every term with real numbers, interactive heatmaps, and step-by-step matrix multiplication.
A critical detail: during generation (and during training), tokens must NOT be able to see future tokens. That would be cheating — like knowing the answer before the test.
The causal mask is a lower-triangular matrix applied to the attention scores:
T o _ b e _ = allowed, X = blocked
T [ _ X X X X ] T can only see T
o [ _ _ X X X ] o can see T, o
_ [ _ _ _ X X ] _ can see T, o, _
b [ _ _ _ _ X ] b can see T, o, _, b
e [ _ _ _ _ _ ] e can see all (last position)Each position can attend to itself and all previous positions. Future positions are masked out with negative infinity (which becomes 0 after softmax).
Hover over any cell above to see whether a token can attend to another. The triangular pattern is the causal mask — the fundamental constraint that makes generation work.
After gathering information from other characters (attention), each character thinks independently about what it learned. A simple 2-layer neural network:
embedding_dim (128) --> 4x embedding_dim (512) --> ReLU --> embedding_dim (128)The network expands to 4x its size (more “thinking room”), applies ReLU (cutting off negative values to focus on useful signals), then contracts back. No communication between positions here — each character processes its own blended information alone.
Two more pieces make the block work:
Residual connections add the original input back after each sub-layer. This is like a musician improvising — they can go wild, but the original melody is always there as a safety net. If the attention or feed-forward steps are not helpful, the model can fall back to the original input unchanged.
LayerNorm prevents numbers from growing too large as they pass through layers. It normalizes values to have a stable mean and variance, keeping training stable.
Step through the pipeline above to see data flow through each stage of one transformer block. Watch how residuals create shortcuts around the main computation path.
The complete GPT model stacks these pieces into a pipeline:
token_indices [64, 256] (batches of character IDs)
|
├── token_embedding --> [64, 256, 128] (what each char means)
├── position_embedding --> [64, 256, 128] (where each char sits)
|
+ --> [64, 256, 128] (combined representation)
|
├── TransformerBlock × N (N=1 in our minimal version)
| ├── LayerNorm + Attention + Residual
| └── LayerNorm + Feed-Forward + Residual
|
├── LayerNorm
|
└── Linear(vocab_size=65) --> logits [64, 256, 65]
|
For each of 256 positions, 65 scores (one per character)
|
softmax --> probabilities --> pick next characterWith 1 block and our default settings, the model has about 248,000 parameters (adjustable numbers). Each parameter is a weight in one of the linear layers or embedding tables.
Training is the process of slowly adjusting the model’s 248,000 weights to make better predictions. Each iteration follows the same recipe:
The loss is a single number that tells us how well the model is doing:
| Loss | Meaning | |------|---------| | 4.17 | Random guessing (uniform over 65 characters) | | 2.00 | Model knows common patterns like “th”, “he”, “an” | | 1.60 | Model has learned many character-level patterns |
Our actual training run on an NVIDIA GTX 1650 Ti (4GB VRAM) took about 5.5 minutes for 5,000 iterations:
Step 0 | Train Loss: 4.1800 | Val Loss: 4.1787
Step 500 | Train Loss: 2.2403 | Val Loss: 2.2852
Step 1000 | Train Loss: 1.9906 | Val Loss: 2.0768
Step 2000 | Train Loss: 1.7416 | Val Loss: 1.8931
Step 3000 | Train Loss: 1.6592 | Val Loss: 1.8255
Step 4000 | Train Loss: 1.6238 | Val Loss: 1.8006
Step 4999 | Train Loss: 1.5989 | Val Loss: 1.7792We track two losses: training (on text the model has seen) and validation (on held-back text the model has never seen). If training loss goes down but validation loss stagnates, the model is overfitting — memorizing instead of learning general rules. Our gap of ~0.18 is mild overfitting, expected with only 1 block.
Watch the loss curve animate above. The model starts at random guessing (4.18) and steadily improves. The gap between train and validation lines shows how well the model generalizes.
Once trained, we can generate new text by sampling one character at a time:
Temperature controls how “creative” the model is:
| Temp | Effect | Example | |------|--------|---------| | 0.1 | Boring, always picks the most likely char | “to be or not to be” | | 0.8 | Balanced creativity | “to be or not to be that is the point” | | 1.5 | Wild, picks unlikely chars | “to breath or not to breath the air” |
It works by adjusting the probability distribution: dividing logits by temperature sharpens or flattens the curve before sampling.
Watch text generate character by character above. Adjust the temperature slider to see how it affects the output — from predictable at 0.1 to creative at 1.5. The probability bars at the bottom show which characters are being considered for the next position.
The entire model is implemented in a single file of about 250 lines of Python (plus ~350 lines of comments). Here is the complete code structure:
# gpt.py — a minimal char-level GPT
import torch
import torch.nn as nn
import torch.nn.functional as F
# Hyperparameters
BATCH_SIZE = 64
MAXIMUM_CONTEXT_LENGTH = 256
EMBEDDING_DIMENSION = 128
NUMBER_OF_ATTENTION_HEADS = 4
NUMBER_OF_TRANSFORMER_BLOCKS = 1
DROPOUT_PROBABILITY = 0.2
LEARNING_RATE = 3e-4
# The model components (see each section above):
class SingleAttentionHead(nn.Module): ... # one "gossip circle"
class MultiHeadSelfAttention(nn.Module): ... # 4 parallel heads
class FeedForwardLayer(nn.Module): ... # the "thinking" step
class TransformerBlock(nn.Module): ... # gossip + think + residuals
class CharacterLevelGPT(nn.Module): ... # full pipeline
# Training loop for each iteration:
# batch = get_random_batch('train')
# logits, loss = model(batch.input, batch.target)
# loss.backward()
# optimizer.step()
# Generation:
# output = model.generate_text(prompt, max_length, temperature)The full code is available at /home/mysyntax/Documents/llm/gpt.py along with a comprehensive README that documents every section with line-number references.
With a single transformer block and 5,000 training iterations, here is what our model produced:
To dearn's that his like sunboar that
With fairs, not mee to pleace.
COMIZABETH:
Sir in the is dead, hang namend swear
And oved with nefor she nand body well methe very truth,
And the ched his thanged herefore w up whise blody lawsThe model has clearly learned:
the, and, his, thatdoth, 'tis, thouIt has NOT yet learned:
With more transformer blocks (4-6), deeper embeddings, and longer training, the output becomes increasingly coherent. But even with 1 block, the model demonstrates that the transformer architecture works — it finds patterns in character sequences.
PyTorch makes GPU usage trivial. A single line moves the entire model to the GPU:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)The same code works on CPU, but training is about 5x faster on an NVIDIA GPU. For our run on a GTX 1650 Ti (4GB VRAM, CUDA 13.2 driver), the 5,000 iterations completed in 5.5 minutes. On CPU only, the same workload would take 25-30 minutes.
The GPU memory usage is modest — about 1.5GB for our 248K parameter model, leaving plenty of room for larger models.
This minimal GPT is a foundation. Here are natural next steps:
NUMBER_OF_TRANSFORMER_BLOCKS from 1 to 4 or 6 for more coherent outputMAXIMUM_TRAINING_ITERATIONS to 20,000+ for lower lossMAXIMUM_CONTEXT_LENGTH to 512 for longer-range patternsinput.txt with any text file — the model learns your styleEvery large language model — GPT-4, Claude, Gemini, Llama — builds on the same principles we have covered here. The numbers are bigger (billions of parameters instead of 248K), the tokenizers are more sophisticated (subword instead of character-level), and the training data is massive (trillions of tokens instead of 1M characters). But the core mechanism is identical: embed tokens, run transformer blocks, predict the next token.