Introduction to Building an Inference Engine
You’ve learned about the ML inference pipeline, explored third-party libraries like ONNX Runtime, and implemented basic Vulkan compute operations. Now comes the question: why would you build your own inference engine when libraries like ONNX Runtime already exist and work well?
The honest answer? For most production applications, you probably shouldn’t. ONNX Runtime represents years of optimization work, supports hundreds of operators, and has been battle-tested across countless deployments. Use it when you can.
But understanding how to build an inference engine teaches you something deeper. It demystifies what’s happening under the hood. When you encounter performance bottlenecks, numerical precision issues, or deployment constraints, you’ll understand why they happen and how to address them. You’ll make better decisions about when to use which library, which optimizations matter, and what tradeoffs you’re making.
This section teaches you to build a simple but functional inference engine from scratch. Not production-ready, but real enough to run actual neural networks on your GPU.
The Journey
We’re going to build this in stages, each one revealing a layer of understanding:
Stage 1: Hardcoded Forward Propagation. We start with the simplest thing that could possibly work—a single neural network, hardcoded layer by layer. You load weights from a simple binary file and execute each layer’s compute shader in sequence. No flexibility, no abstraction, just direct execution. This is our baseline. It works, and more importantly, it’s completely understandable.
Stage 2: Identifying the Pattern. Once you see the hardcoded version working, you notice the repetition. Every layer does similar things: allocate buffers, upload data, bind descriptors, dispatch compute shader. The differences are in the details—which shader, what dimensions, which parameters. We identify what’s common and what varies.
Stage 3: Abstraction. With the pattern identified, we build abstractions. A tensor class to represent data. An operation interface for layers. A graph structure to capture connections. This lets us represent any network, not just the one we hardcoded.
Stage 4: Model Formats. Now we need a way to get network definitions and weights into our engine. We could invent our own format, but the industry has already solved this problem. ONNX and TensorFlow Lite are standard formats used across the field. We learn to parse them, extracting the graph structure and learned weights.
Stage 5: Integration. Finally, we connect all the pieces. Load a model file, build our internal representation, allocate GPU resources, and execute inference. The result is a working inference engine that can run real networks.
What We’ll Build
By the end of this section, you’ll have code that:
-
Loads a trained MNIST model (handwritten digit recognition)
-
Parses model structure and weights from ONNX or TensorFlow Lite format
-
Executes the full forward pass on Vulkan compute
-
Produces correct predictions (verifiable against PyTorch/TensorFlow)
The implementation won’t be production-ready. It’ll support a limited set of layer types (dense, convolution, ReLU, max pooling). It won’t have all the optimizations. But it’ll be real, complete, and educational.
Educational vs. Production Code
Throughout this section, we make choices favoring clarity over performance. In production code, you’d use techniques like:
-
Operator fusion (combining multiple operations into one shader, covered in Advanced Topics: Model Optimization)
-
Advanced memory pooling (reusing buffers across non-overlapping lifetimes, covered in Advanced Topics: Memory Management)
-
FP16 or INT8 quantization (trading precision for speed, covered in Advanced Topics: Quantization Strategies)
-
Multi-threaded graph execution (running independent operations in parallel)
We touch on these in later sections, but our inference engine keeps things simple. The goal is understanding, not maximum throughput.
Prerequisites
Before continuing, you should be comfortable with:
-
Vulkan compute shaders (covered in the main tutorial’s compute chapter)
-
Basic neural network concepts (covered in the ML Inference Pipeline section)
-
C++ programming, especially RAII and smart pointers
We’ll use the Vulkan-Hpp RAII bindings throughout, building on patterns from the main tutorial.
The Working Code
All complete, working code lives in the tutorial’s ml_inference directory. The chapters in this section explain the concepts and show key snippets, but we don’t dump thousands of lines into the tutorial itself. When we say "see the attachments," we mean it—the code is there, it compiles, and it runs.
We’re teaching you to think about the problem, not memorizing syntax.
Let’s begin with the simplest version: a hardcoded forward pass.