Image Classification: From MNIST to CNNs

You just built a neural network for MNIST that works. It recognizes handwritten digits with 95%+ accuracy using fully connected layers. Let’s try applying that same architecture to a real image classification problem.

ImageNet contains 1000 object categories: dogs, cats, cars, airplanes, furniture, food, everything. The images are 224×224 pixels in color (RGB). That’s 224 × 224 × 3 = 150,528 input values.

If we build a fully connected network like we did for MNIST:

How many weights do we need?

Total: ~77 million parameters, almost all in that first layer.

At 4 bytes per float, that’s 308 MB just for the first layer’s weights. For a single image forward pass, you’re doing 77 million multiply-adds in that layer alone. And that’s a tiny network—ResNet-50, a popular architecture, has 25 million parameters total and processes images much more effectively.

But the parameter count isn’t even the worst problem. The real issue is that fully connected layers ignore the structure of images.

When you look at a photo, nearby pixels are related. The pixels forming a cat’s ear are right next to each other. The pixels 200 pixels apart are probably unrelated—one might be part of the sky, the other part of grass.

In a fully connected layer, every input connects to every output with equal treatment. The weight connecting pixel (0,0) to output neuron 5 is just as important as the weight connecting pixel (100,150) to that same neuron. The network has to learn from scratch that spatial proximity matters.

Think about what MNIST taught us: edges and corners in the first layer, curves and loops in later layers, complete digits at the end. That hierarchical structure worked because even in tiny 28×28 images, spatial relationships matter. A horizontal line segment at the top plus a vertical line segment on the right suggests the digit "7" or "1". The network learned this, but it had to figure it out by adjusting millions of weights.

For 224×224 images, that approach breaks down:

We need an architecture that respects how images work.

Look at any photograph. You’ll find the same types of patterns all over the image:

These patterns aren’t position-specific. A vertical edge works the same way whether it’s at the left side of the image or the right. Detecting "fur texture" uses the same logic regardless of where the fur appears.

What if, instead of learning 150,000 separate weights for each position in the image, we learned one edge detector and applied it everywhere? Learn once, use everywhere.

That’s convolution.

Imagine you have a 3×3 grid of weights—a tiny neural network fragment that looks at a 3×3 patch of image. Here’s a concrete example with a vertical edge detector.

The filter has weights arranged like this: the left column is all -1, the middle column is all 0, and the right column is all +1. When you apply this to an image patch where the left side is dark (pixel values around 50) and the right side is bright (pixel values around 200), you get:

Left side contribution: 50 × (-1) + 50 × (-1) + 50 × (-1) = -150 Middle contribution: 50 × 0 + 50 × 0 + 50 × 0 = 0 Right side contribution: 200 × (+1) + 200 × (+1) + 200 × (+1) = +600

Total response: -150 + 0 + 600 = +450

This large positive value tells the network "I found a vertical edge here!" If the edge went the other way (bright to dark), you’d get -450. If the pixels were uniform (no edge), you’d get near zero.

A convolutional layer contains multiple filters (usually 32-128 in early layers, 256-512 in later layers). Each filter learns to detect a different pattern:

When you apply all 32 filters to a 224×224 RGB input, you get 32 output images (called "feature maps"), each 224×224, each highlighting where a specific pattern appears.

Input: 224×224×3 (height × width × RGB channels) ↓ Apply 32 different 3×3×3 filters (3×3 spatial, across all 3 color channels) ↓ Output: 224×224×32 (height × width × 32 feature maps)

Each of those 32 filters has 3×3×3 = 27 weights (plus 1 bias) = 28 parameters per filter. Total: 32 filters × 28 = 896 parameters for the entire layer.

Compare to the 77 million parameters in a fully connected layer doing the same job. This is why convolution works.

Translation invariance: The same filter detects a cat’s ear whether it’s at (50,50) or (150,100). We learned the pattern once.

Parameter efficiency: Instead of millions of separate weights for every position, we have hundreds of shared weights applied everywhere.

Spatial locality: A 3×3 filter naturally respects that nearby pixels are related—it only looks at local regions.

Hierarchical features: Stack multiple convolutional layers and you get hierarchy for free. Layer 1 detects edges. Layer 2 combines edges into shapes. Layer 3 combines shapes into object parts. Layer 4 combines parts into objects.

Let’s trace what happens when you feed a photo of a dog through a convolutional network:

Layer 1 (input: 224×224×3, output: 224×224×64):

Learns edge detectors, color blobs, simple textures. One filter responds to vertical edges, another to horizontal edges, another to red-to-brown gradients. The outputs look like edge maps—abstract, not recognizable as objects yet.

Layer 2 (input: 224×224×64, output: 112×112×128):

Combines edges into shapes. A filter might respond to "corner formed by horizontal and vertical edge". Another responds to "circular boundary". These are still low-level, but starting to look like parts of objects. We also reduce spatial resolution (224→112) because we care more about "what" than "exactly where".

Layer 3 (input: 112×112×128, output: 56×56×256):

Combines shapes into parts. One filter activates for "furry texture bounded by curved edges" (could be part of a dog, cat, or teddy bear). Another responds to "round shape with high contrast" (eye candidate). Another detects "rectangular with parallel lines" (window, door, or book spine).

Layer 4 (input: 56×56×256, output: 28×28×512):

Combines parts into object-specific features. At this level, you have filters that activate specifically for "dog face", "car wheel", "bird wing". The spatial resolution is coarse (28×28) because we don’t need precise localization for classification—we just need to know "this image contains a dog", not "the dog’s nose is at pixel (145, 87)".

Final Layer (global average pooling + fully connected):

Takes the 28×28×512 feature maps and reduces them to 1000 class scores. "I see strong evidence for dog features, weak evidence for cat features, no evidence for airplane features → predict: dog".

This hierarchy isn’t hand-designed. You don’t tell the network "layer 1 learns edges, layer 2 learns shapes". You initialize all the filters with random weights and train with backpropagation, the same as MNIST. The hierarchy emerges because it’s the optimal way to solve the problem given the convolutional structure.

A convolution is a linear operation (multiplication and addition). If you stack ten linear layers together, the result is still just one big linear operation. You could mathematically collapse the whole network into a single layer!

To learn complex, non-linear patterns (like "a dog’s ear"), we must inject non-linearity between the layers. We use an Activation Function.

As data flows through a deep network, the values can drift—some layers might start outputting very large numbers, while others output tiny ones. This is called Internal Covariate Shift, and it makes training very unstable.

Batch Normalization solves this by "re-centering" the data after each convolutional layer. It ensures that the mean is near 0 and the standard deviation is near 1.

A critical concept: receptive field. How much of the original image does a neuron in layer 4 "see"?

In layer 1, a 3×3 filter sees exactly 3×3 pixels of the input.

In layer 2, a 3×3 filter sees 3×3 positions of layer 1’s output. But each of those positions came from 3×3 pixels in the input. So layer 2 effectively sees:

In layer 3, each position sees 3×3 of layer 2, and each layer 2 position sees 5×5 of input:

By layer 5, a single neuron might see 13×13 or more input pixels, depending on the exact architecture.

This is how deep networks build complex patterns from simple ones. Early layers see small regions and detect simple patterns. Later layers see large regions (because they’re seeing outputs from early layers) and detect complex patterns.

Pooling accelerates this. When we reduce 224×224 to 112×112, each position in the smaller map corresponds to 2×2 region in the larger map. This doubles the receptive field growth rate.

You might ask: why not skip the hierarchy and use, say, 11×11 filters in layer 1 to see more context?

Parameters. An 11×11 filter has 121 weights per channel. For a layer with 64 filters processing RGB input:

Compare to two 3×3 layers stacked (which has the same receptive field):

Wait, that’s more parameters! But there’s a critical difference: two layers with ReLU activation in between.

Stack two 3×3 convolutional layers with ReLU between them and you get the same receptive field (5×5 in the input) as one 5×5 layer, but with more non-linearity. The extra ReLU lets the network learn more complex functions. Three 3×3 layers cover the same receptive field as one 7×7, but with three non-linearities instead of one.

This is why modern networks (like VGG, ResNet, MobileNet) use almost exclusively 3×3 and 1×1 convolutions. Larger filters waste parameters without adding representational power.

We mentioned reducing 224→112→56→28. Why?

Computational efficiency: A 224×224 image has 50,176 positions. A 112×112 image has 12,544—4× fewer. Pooling cuts computation dramatically.

Translation invariance: If you shift the input image by 1 pixel, max pooling (taking the maximum in each 2×2 region) might produce the same output. The exact position matters less.

Abstraction: For classification, you don’t care if the dog’s eye is at (45,67) or (47,68). You care that there’s an eye somewhere in the upper-middle of the image. Reducing spatial resolution forces the network to learn position-invariant features.

Max pooling takes the maximum value in each small region (typically 2×2) and outputs it. A 2×2 max pooling operation with stride 2 reduces a 4×4 input to a 2×2 output—you take the max of each non-overlapping 2×2 block.

Modern architectures sometimes use strided convolution instead (convolution with stride=2 skips every other position), which is learnable rather than hard-coded like max pooling. But the principle is the same: progressively reduce spatial dimensions while increasing channel depth.

Almost every CNN follows this pattern:

Spatial dimensions shrink: 224 → 112 → 56 → 28 → 14 → 1

Channel depth grows: 3 → 64 → 128 → 256 → 512

This trades spatial detail for semantic depth. Early layers answer "where are the edges?" Late layers answer "is there a dog?"

Standard convolution is expensive. A 3×3 convolution with 512 input channels and 512 output channels requires:

MobileNetV2 uses a trick called depthwise separable convolution. Instead of one expensive operation, split into two cheaper ones:

Depthwise convolution: Apply 3×3 filter to each channel separately.

This captures spatial patterns within each channel but doesn’t mix information across channels.

Pointwise convolution: Apply 1×1 convolution to mix channels.

This mixes information across channels but doesn’t look at spatial neighbors.

Total cost: 4,608 + 262,144 = 266,752 operations per position

Compare to standard convolution: 2,359,296 operations per position.

Speedup: 8.8× with minimal accuracy loss.

When you’re implementing inference in Vulkan, you need to know which operations dominate runtime. For MobileNetV2:

Pointwise convolutions (those 1×1 convolutions mixing channels) are matrix multiplications. They’re compute-bound. Optimizing them means maximizing arithmetic throughput—using tensor cores, vectorized instructions, large thread blocks.

Depthwise convolutions (the 3×3 per-channel filters) are memory-bound. Each output value requires reading 9 input values and 9 weights. Optimizing them means improving memory access patterns—coalescing reads, using shared memory, minimizing cache misses.

Understanding the architecture tells you where to focus optimization. If you spend days optimizing ReLU (which is just max(0, x)), you’ll see minimal speedup because ReLU is <1% of runtime. If you optimize the pointwise convolutions, you can get 2-3× faster inference.

This is why we teach the theory before the implementation.

We’ve talked about what filters do, but how do they get their values? In MNIST, we had a few thousand weights. In MobileNetV2, we have millions. We can’t tune them by hand.

We use Backpropagation combined with Stochastic Gradient Descent (SGD).

Even the best architecture will fail if the data is poor. If you only show the network photos of dogs on grass, it will learn that "green background = dog." If you show it a dog on a sidewalk, it might predict "grey car."

To prevent this, we use Data Augmentation. During training, we don’t show the network the same image twice. We "mess with it" first:

By the time training is finished, the network has seen millions of variations of dogs, making it incredibly robust to real-world camera noise and lighting conditions.

At the very end of our millions of multiplications, the network spits out 1000 numbers (one for each ImageNet class). These are called Logits. They might look like this:

These aren’t very useful to a human. We want Probabilities that add up to 100% (1.0).

You now understand:

Next, we’ll implement this. You’ll build a complete image classification pipeline: loading images, preprocessing them, running inference with ONNX Runtime, and displaying results. Then we’ll add GPU acceleration, threading for responsiveness, and real-time camera input.

But now you understand what you’re implementing and why it works. The code is just the expression of these concepts.

Previous: Desktop Applications Introduction | Next: CPU Baseline Implementation