First, a question: why can't we just feed an image into a fully-connected net?
Before YOLO, answer something more basic. An image is just a pile of numbers (three 0–255 values per pixel), so why not flatten it into one long vector and feed it to an ordinary fully-connected network (MLP), the way we handle tabular data?
The answer hides in two facts that would crush the system outright. First, parameter explosion. A common factory-floor 640×640 color image flattens to 640×640×3 ≈ 1.23M dimensions. If the first hidden layer has just 1000 neurons, that first layer alone holds 1.23M × 1000 ≈ 1.2 billion weights. And that's only layer one. The model hasn't learned anything yet and memory has already blown up.
The second problem is more lethal: spatial structure is destroyed. The instant you flatten a 2D image into 1D, the information "which pixel neighbors which pixel" is gone forever. A crack on the surface of an MLCC (multilayer ceramic capacitor) is formed by a group of adjacent pixels with a brightness gap; after flattening, those pixels are scattered across the vector, and the network must learn from scratch that "these dimensions happen to be spatially adjacent." You've thrown away the most important prior and then forced the network to reinvent it with mountains of data.
So image processing needs a network with two assumptions baked in: spatial adjacency and parameter sharing. That is the reason the Convolutional Neural Network (CNN) exists. Hold onto this motivation — everything in YOLO grows out of it.
What convolution actually computes: sliding window, weight sharing, receptive field
The core action of convolution is almost boringly simple: take a small weight matrix (called a kernel or filter, e.g. 3×3), slide it across the whole image one position at a time, and at each position, take the element-wise product of the kernel's 9 weights with the 9 pixels it covers, and sum them all into one number placed in the output. Slide across the whole image and you get a new 2D array called a feature map.
Input patch 3x3 kernel One output value
┌───────────┐ ┌──────────┐
│ 12 8 3 │ │ -1 -1 -1 │ (12*-1)+(8*-1)+(3*-1)
│ 90 85 4 │ × │ 0 0 0 │ = +(90*0)+...+(4*0)
│ 88 91 2 │ │ 1 1 1 │ +(88*1)+(91*1)+(2*1)
└───────────┘ └──────────┘ = 158 → a horizontal edge here
This simple action hides three key design choices, each answering a pain point from the last section:
Weight sharing: the kernel's 9 weights are reused at every position across the whole image; they don't change. This alone drops the parameter count from "one weight set per pixel" to "9 weights per kernel." A kernel that detects "vertical edges" works in the top-left corner and equally in the bottom-right — which is exactly the nature of images: features are translation-invariant. A crack is the same crack whether it sits on the left or right of the capacitor. Weight sharing writes that prior straight into the architecture.
Locality: each output value looks only at a small 3×3 neighborhood of the input, not the whole image. This preserves spatial structure — only adjacent pixels get computed together.
Receptive field: this is the core term for understanding why CNNs stack many layers, so unpack it fully. A single 3×3 conv layer's output point "sees" only a 3×3 input region. But stack another 3×3 on top, and a point in layer 2 sees a 3×3 of layer 1, while each layer-1 point already saw a 3×3 of the input — so that layer-2 point indirectly sees a 5×5 of the input. Stack again and it becomes 7×7. The receptive field grows with depth. That's why shallow CNN layers catch only local features like edges and textures, while deep layers catch semantics like "this is a whole capacitor" that require a large view to recognize. Shallow sees detail, deep sees the whole — this rule is used directly when we explain why YOLO pulls detection heads from multiple depths.
From feature maps to a pyramid: downsampling, Backbone / Neck / Head, FPN
A CNN doesn't keep computing at full resolution the whole way — that's too expensive. It periodically downsamples: via stride-2 convolutions or pooling, it halves the feature map's height and width. The core action of pooling is also direct — max pooling, for instance, keeps only the maximum of each 2×2 block. This does two things: compute drops (area becomes 1/4), and the receptive field effectively enlarges relative to the original image, letting deep layers cover a larger portion of the input with a same-size kernel.
So a modern detection network takes a three-stage shape. Keep these three terms straight — they are the skeleton of everything that follows:
Input ──► Backbone ──► Neck ──► Head ──► predicted boxes
(extract) (fuse) (output)
Backbone: convs + downsampling; deeper = stronger semantics, lower res
P3(80x80) → P4(40x40) → P5(20x20) ← three scales at 640 input
Neck: upsample deep "strong-semantic, low-res" and fuse with
shallow "weak-semantic, high-res" features (FPN/PAN)
Head: on the fused multi-scale maps, per-cell output
"is there an object, which class, where is the box"
The Backbone does pure feature extraction — the stack of convs from the last section; deeper means more abstract semantics and lower resolution. The Head translates features into final answers (box coordinates, class, confidence). The Neck in between is the key part, resolving a concrete tension: deep features have strong semantics ("this is a capacitor") but low resolution (imprecise boxes, small objects vanish entirely); shallow features have high resolution (precise localization) but weak semantics ("there's an edge here").
The Feature Pyramid Network (FPN) [6] is the Neck's classic solution. Its mechanism: upsample the deep low-resolution features and element-wise add them to same-size shallow features. Each fused level then holds both strong semantics and high resolution. Why bother? Because object scale varies wildly on a factory floor — on one PCB, a large capacitor spans hundreds of pixels while a 0201 resistor may span only 6×3. A single-scale feature map cannot box both accurately. FPN's multi-scale outputs (P3 for small objects, P5 for large) exist precisely for this. Remember this line: small-object misses almost always trace back to this multi-scale design.
The fundamental difficulty of detection, and the road before YOLO
Classification only answers "what is this image," outputting one label. But detection must simultaneously answer "how many objects, what is each, where is each" — the number of outputs is itself variable. That's the hard part: you don't know if there are 0 or 50 objects, yet the network's output structure must be fixed.
The dominant pre-YOLO solution was two-stage, exemplified by Faster R-CNN [1]. Its logic: first a "Region Proposal Network (RPN)" sweeps out thousands of "something might be here" candidate boxes, then each candidate is cropped and individually classified and refined. This road is accurate but slow, and slow by mechanism — it splits "find location" and "recognize class" into two chained stages, running the back-end network many times per image. For factory production lines that need real-time, even frame-rate detection, two-stage is often too heavy.
That is YOLO's reason to exist.
YOLO's core bet: turn detection into a single regression
YOLO (You Only Look Once) [2] is named for its principle: the whole image passes through the network once (one-stage), and a single forward pass emits all boxes and classes at once. How does it turn "variable output count" into "fixed structure"? Through one key invention: the grid.
YOLOv1 slices the input into an S×S grid (S=7 in the original). The rule: whichever grid cell the object's center falls into is responsible for predicting that object. Each cell outputs a fixed number of box coordinates, a confidence per box, and class probabilities for the cell. The output structure is thus fixed at S×S×(prediction length per cell), regardless of how many objects the image holds.
7x7 grid, a capacitor's center falls in cell (3,4)
┌──┬──┬──┬──┬──┬──┬──┐
│ │ │ │ │ │ │ │
├──┼──┼──┼──┼──┼──┼──┤
│ │ │ │ │ │ │ │
├──┼──┼──┼──┼──┼──┼──┤
│ │ │ │ │ │ │ │
├──┼──┼──┼──╆━━╅──┼──┤ ← cell (3,4) owns this capacitor:
│ │ │ │ ┃██┃ │ │ outputs box(x,y,w,h)+conf+class
├──┼──┼──┼──╄━━╃──┼──┤
│ │ │ │ │ │ │ │
└──┴──┴──┴──┴──┴──┴──┘
The price of this bet derives YOLOv1's innate flaw directly: because "one cell owns one center," when two object centers fall in the same cell, one is necessarily missed — which is exactly why YOLOv1 struggles with dense small objects (tightly packed parts on a line). Understanding where this flaw comes from is how you understand what each later version is fixing.
Where bounding boxes come from: the anchor mechanism and its demise
YOLOv1 had the grid regress box dimensions "from nothing," which trains unstably — the network must guess how big a box can be from scratch, and gradients thrash. YOLOv2 [3] introduced the anchor box (a.k.a. prior box) to fix this.
The anchor's underlying logic: instead of predicting a box's absolute size from zero, pre-stage a few "typical shapes" of reference boxes (a tall-thin one, a wide-flat one, a square one) and have the network predict only how much to scale and shift relative to that reference. These typical shapes aren't guessed — YOLOv2 runs k-means on the widths/heights of all ground-truth boxes in the training set and takes the cluster centers as anchors — so if your dataset is all tall-thin part labels, the anchors automatically skew tall-thin. The network outputs not the box itself but four offsets:
box center bx = σ(tx) + cx cx,cy = cell top-left coords
by = σ(ty) + cy σ = sigmoid, locks offset inside the cell
box size bw = pw · e^(tw) pw,ph = anchor width/height (prior)
bh = ph · e^(th) tw,th = raw network outputs
The net only learns tx,ty,tw,th — four "corrections" —
starting from a sensible anchor, far more stable than guessing from zero.
Why e^(tw) instead of plain addition? Because width/height must stay positive; the exponential guarantees that and makes "2× larger" and "2× smaller" symmetric in the loss. Why sigmoid on the center? To lock the center offset inside the responsible cell so it can't drift into a neighbor. Every term — sigmoid, exponential, prior — solves a concrete numerical-stability problem, not decoration.
But anchors bring a new burden: their count, aspect ratios, and scales all become hyperparameters to tune, and tuning them wrong costs accuracy directly. So from YOLOv8 onward the mainstream shifted to anchor-free: have each cell directly predict "distance from center to the four box edges," dropping all anchor hyperparameters. This is a clean evolutionary chain — anchors were born to stabilize training → but became a tuning burden → once the network is strong enough and the positive/negative assignment strategy is good enough, remove them.
The three-part loss: this is where YOLO actually "learns"
Everything above is mechanism (architecture layer). But what the weights get sculpted into is decided by the loss (training-objective layer). YOLO's loss is a sum of three parts, each governing one thing; confusing them is the classic beginner error:
1. Objectness / confidence loss: governs "is there an object here at all." A binary judgment. It exists so the network first learns to filter out the vast sea of background cells — most cells in any image are background, which plants the class-imbalance problem discussed later.
2. Classification loss: computed only on cells that actually contain an object, for "which class." Note the condition — background cells don't participate in classification loss, or background would drown everything.
3. Box regression loss: governs "is the box drawn accurately." Early versions used mean-squared error on coordinates, but this has a mechanistic flaw: computing error on (x,y,w,h) separately does not align with the metric we actually care about — IoU (Intersection over Union). You can have small error on all four numbers yet a poor IoU.
IoU = intersection area / union area pred box ┌─────┐
│ ┌──┼──┐
IoU=1 perfect overlap └──┼──┘ │
IoU=0 no overlap └─────┘ GT box
intersection = overlap, union = total coverage
So modern YOLO uses CIoU loss (Complete-IoU) [7]. It writes three IoU-related geometric factors straight into the loss: overlap area, center-point distance between the two boxes, and aspect-ratio consistency. Why all three? Because with IoU alone, when two boxes don't overlap at all, IoU=0 and the gradient is 0 too — the network has no idea which way to fix. Adding the "center-distance" term means even before overlap, the loss pushes the predicted box toward the ground-truth center. This is a patch derived purely by asking "under what conditions did the original loss have no gradient." When boxes are broadly offset — right position, wrong size — the answer often lives in this loss design.
NMS: why one object gets boxed many times, and how to clean it up
After the forward pass, one capacitor often spawns several high-score boxes — neighboring cells all think "there's a capacitor here." Non-Maximum Suppression (NMS) is the cleanup crew: sort all boxes by confidence, keep the highest, delete every other box whose IoU with it exceeds a threshold (e.g. 0.45) as a duplicate, then repeat on the rest.
NMS has two practical traps hidden in that threshold, mapping directly to line problems: set it too high and two tightly adjacent parts get mistaken for one, deleting a real detection (a miss); set it too low and multiple boxes on one part aren't cleaned up (duplicates). That's why dense-packing scenes demand care with this knob.
YOLOv10 [9] goes further, achieving NMS-free via consistent dual assignments — during training it uses both a one-to-many and a one-to-one label assignment, teaching the network to emit a single high-score box per object, so inference drops the NMS post-processing entirely. Dropping NMS isn't just faster; it makes the pipeline truly end-to-end and cleaner to deploy. This is another "a mechanism (NMS) became the bottleneck → absorb it via training-objective design" evolution line.
The evolution chain: what hole does each version actually patch
String the YOLO generations together and you see it's not random feature-stacking but a chain where each version's flaw forces the next:
v1 [2] single regression + grid, pioneers one-stage. Flaw: one
cell/one object, misses small/dense objects, unstable boxes.
v2 [3] adds anchors (k-means priors) + BatchNorm + high-res
finetune; boxes stabilize, recall rises.
v3 [4] three-scale prediction (FPN idea) + independent logistic
multi-label classification; clear small-object gains.
v4 [5] CSPDarknet backbone + PAN neck + "Bag of Freebies"
(Mosaic aug, CIoU — tricks that add training-only cost).
v5 Ultralytics engineering (PyTorch, usability, auto-anchor);
no paper but became the industry de-facto standard.
v6/v7[8] re-parameterization, ELAN efficient aggregation; faster,
higher.
v9 [10] PGI (programmable gradient info) + GELAN; fixes info loss
in deep networks.
v10[9] consistent dual assignments → NMS-free, end-to-end.
v11 C3k2, C2PSA attention; strengthens small objects & features.
v12[11] attention-centric (Area Attention + R-ELAN); brings
attention's accuracy to real-time speed.
A practical call for the factory floor: don't blindly chase the newest version. Versions like v5/v8 — heavily validated in industry, with mature ecosystems and complete deployment toolchains — are usually more production-ready than the latest generation. The newest version's paper mAP may be a fraction higher, bought at the cost of an unstable ecosystem and export (ONNX/TensorRT) landmines. First ask "is my bottleneck speed, accuracy, or small objects," then pick the version whose strength matches — not the largest version number.
Data handling: this is actually where the ceiling sits
A brutal fact: in real projects, YOLO's success or failure is usually decided not by architecture but by data. This section covers the mechanism.
Annotation quality is the hard ceiling. YOLO's annotation format is one txt per image, each line class cx cy w h (the last four are 0–1 values normalized to image width/height). The easiest mistakes: boxes drawn too loose (enclosing lots of background, diluting the localization signal); missed labels (some real objects unboxed — the network gets punished by objectness loss for "you shouldn't see anything here," effectively teaching it to ignore real objects); wrong class labels. A missed label is more poisonous than a loose box — because it delivers a gradient in the opposite direction.
Data augmentation's mechanism is "force the network to stay invariant to irrelevant changes, without extra annotation cost." Geometric augmentation (flip, rotate, scale) teaches "an object at another angle is still the same object"; photometric augmentation (brightness, contrast, HSV jitter) teaches "under another factory's lighting it's still the same capacitor." The most representative is Mosaic augmentation, introduced in YOLOv4 [5]: stitch four training images into one before training. Its two mechanistic benefits — one forward pass sees four images' content, effectively enlarging batch diversity; and originally-large objects get shrunk into the collage, artificially manufacturing lots of small-object samples and directly strengthening small-object detection. That's why "confirm Mosaic is on" is the front-line check when small objects are missed.
Mosaic: four images randomly stitched + randomly scaled
┌─────────┬───────┐
│ imgA(↓) │ imgB │ one training sample holds 4 scenes,
│ ├───────┤ objects of mixed sizes at once
├────┬────┘ imgC │ → natural multi-scale + small-object boost
│imgD│ (↓) │
└────┴────────────┘
Class imbalance is the norm in manufacturing: ten thousand good units, five images of some rare defect. The network learns directly that "always guess good" drives loss very low — this is the loss honestly reflecting the data distribution, not a bug. RetinaNet's Focal Loss [12] fixes it at the loss level: it down-weights the loss of easy, confident, correctly-classified samples, focusing training on the few hard, rare ones. In practice, pair it with oversampling rare classes and more aggressive augmentation on them.
Data splitting must hold one iron rule: photos of the same batch, same board, same part must not appear in both train and val. Otherwise the validation score is fake — the network merely memorized that specific part's appearance and collapses in production on an unseen lot. This is especially lethal in manufacturing, where same-batch parts look highly alike.
Tuning the details: the mechanism behind each knob
Tuning isn't mysticism — every knob maps to a mechanism. A few of the highest-impact ones:
Learning rate and warmup: lr is the step size of each weight update. Too large and loss oscillates or even diverges (NaN); too small and convergence is slow, stuck in a bad local solution. Warmup (letting lr ramp linearly from a tiny value over the first few epochs) has a mechanistic reason: early in training the weights are random and the gradient direction is unreliable; a large lr then would be dragged off by a bad direction. Walk slowly with small steps first, accelerate once the gradient direction stabilizes.
Batch size and BatchNorm coupling: an often-overlooked linkage. BatchNorm [mentioned in 3] normalizes each layer's output using the mean and variance of "the current batch" during training. If the batch is too small (say VRAM only allows 4), that batch's statistics jitter wildly and are unreliable, BN breaks, training destabilizes. So when you shrink the batch, you often must lower lr with it, or switch to a batch-insensitive normalization.
Input resolution (imgsz): directly decides small objects' fate. At 640 input, an 8-pixel defect occupies less than one cell on the P5 (32× downsampled) feature map — effectively nonexistent. Pushing imgsz to 1280 doubles the small object's effective pixels and is often the most direct (if most expensive) fix for small-object problems.
Loss weights (the box / cls / obj weighting): if your boxes are well-placed but often mis-classify, raise the cls weight; if the class is right but the box is skewed, raise the box weight. This translates "which layer are you currently wrong at" directly into "which loss term to weight up."
Epoch count and early stopping: more is not better. When val loss starts rising while train loss still falls, that's the signal of overfitting — the network is memorizing the training set rather than learning the general rule. Early stopping halts at the moment the val metric peaks.
A troubleshooting tree: treat the right symptom
When the model performs poorly, don't tune randomly. First identify which kind of poor, then return to the corresponding mechanism. A practical tree:
Symptom: train loss won't drop (can't even fit the training set). This is underfitting or a broken data/pipeline. Check: is lr too small or too large (is the loss curve flat or oscillating), is the annotation format correct (normalized? class id starting at 0?), is data actually loading, is augmentation cranked so hard it destroys the images. First confirm the model can overfit a tiny subset (train on 10 images to near-0 loss) — if it can't, there's a pipeline bug, not a tuning problem.
Symptom: good train, poor val (overfitting). The network memorized the training set. Fixes: more data, more augmentation, more regularization (weight decay / dropout), early stopping, or a smaller model. First check the data split from the previous section — many "overfits" are actually train/val leakage or distribution mismatch.
Symptom: small objects missed. Return to the multi-scale line: confirm a high-resolution head like P3 is in use, raise imgsz, confirm Mosaic is on, and if needed add anchors or a detection head at the small object's scale.
Symptom: one class is terrible, others fine. Almost certainly class imbalance or poor annotation quality for that class. Check its sample count and annotation consistency; apply Focal Loss / oversampling / targeted augmentation.
Symptom: many false positives (background recognized as objects). Usually not enough negative samples (pure background images) or too low a confidence threshold. Add more "definitely no object" background images to training so the network learns what background is; raise the conf threshold at inference appropriately.
Symptom: same object boxed repeatedly / adjacent objects swallowed. This is an NMS threshold issue — flip back to the NMS section. For duplicates, lower the IoU threshold; for adjacent misses, raise it, or switch to Soft-NMS.
Symptom: boxes systematically offset or wrong-sized. Look at the localization loss — confirm you use CIoU, not old coordinate MSE, raise the box loss weight, and (if not anchor-free) recheck whether anchors match your dataset's aspect ratios.
Symptom: loss becomes NaN mid-training. Too-large lr is the most common; next is BN collapse from a too-small batch, or bad annotations in the data (coordinates outside 0–1, negatives). First cut lr, enable gradient clipping, and scrub the annotations.
Closing: one logic chain tying the whole piece together
Wrap the whole line up: an MLP on images explodes parameters and destroys spatial structure → so use convolution, writing "translation invariance" into the architecture via weight sharing and locality, and accumulating receptive field by stacking layers so shallow sees detail and deep sees semantics → but objects come in many sizes and a single scale can't box them all → so FPN fuses strong semantics with high resolution across scales → detection's difficulty is variable output count, and YOLO's grid turns it into a fixed-structure single regression, the root of its speed → box stability rests on anchor priors, and the training objective is the three-part loss (objectness/cls/CIoU) that sculpts the weights to both localize and classify → duplicate boxes are cleaned by NMS, or absorbed by dual assignment to go NMS-free → and the ceiling over all of this is data: annotation quality, augmentation strategy, class balance, clean splits → when something breaks, first identify which kind of poor, then walk this mechanism chain back to the right link and fix it.
The pros and cons of each link are not memorized — they are derived from the concrete problem that link solves. Once you understand this chain, when you hit an unfamiliar situation you can reason upstream yourself to which knob to turn.
Further points worth digging into: the DETR / RT-DETR line where Transformers enter vision as a truly end-to-end, anchor-free, NMS-free parallel road; diagnosing accuracy drops during quantization and TensorRT deployment; and wiring detection to tracking (ByteTrack) for real-time counting on the line. Say which one and I'll expand it separately.
References
[1] Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. — https://arxiv.org/abs/1506.01497
[2] Redmon, J., Divvala, S., Girshick, R., Farhadi, A. (2016). You Only Look Once: Unified, Real-Time Object Detection. — https://arxiv.org/abs/1506.02640
[3] Redmon, J., Farhadi, A. (2017). YOLO9000: Better, Faster, Stronger (YOLOv2). — https://arxiv.org/abs/1612.08242
[4] Redmon, J., Farhadi, A. (2018). YOLOv3: An Incremental Improvement. — https://arxiv.org/abs/1804.02767
[5] Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y. M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. — https://arxiv.org/abs/2004.10934
[6] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S. (2017). Feature Pyramid Networks for Object Detection. — https://arxiv.org/abs/1612.03144
[7] Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., Ren, D. (2020). Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression (DIoU/CIoU). — https://arxiv.org/abs/1911.08287
[8] Wang, C.-Y., Bochkovskiy, A., Liao, H.-Y. M. (2022). YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors. — https://arxiv.org/abs/2207.02696
[9] Wang, A., Chen, H., Liu, L., et al. (2024). YOLOv10: Real-Time End-to-End Object Detection. — https://arxiv.org/abs/2405.14458
[10] Wang, C.-Y., Yeh, I.-H., Liao, H.-Y. M. (2024). YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information. — https://arxiv.org/abs/2402.13616
[11] Tian, Y., Ye, Q., Doermann, D. (2025). YOLOv12: Attention-Centric Real-Time Object Detectors. — https://arxiv.org/abs/2502.12524
[12] Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P. (2017). Focal Loss for Dense Object Detection. — https://arxiv.org/abs/1708.02002