I trained a neural network on the Apple Neural Engine’s matrix unit. It’s 6.3x faster than PyTorch. [P]

ITT: I demystify the Apple Neural Engine, and provide proof.

If you've spent any time around Apple Silicon ML discussions, you've probably seen the "Neural Engine" referenced as this discrete, mysterious coprocessor sitting on the die — a black box that CoreML talks to, separate from the CPU and GPU. Apple markets it that way. "16-core Neural Engine. 38 TOPS." It's on every spec sheet.

Here's the thing: it's not that simple, and some of the assumptions floating around are just wrong.

What I built:

A bare-metal ARM SME2 bytecode interpreter — custom opcodes, hand-written ARM64 assembly — that drives the M4 Pro Max (or M5) matrix tiles directly. No CoreML. No BNNS. No frameworks. Just raw instructions on the CPU's za tile arrays.

Note: there is a reason for the interpreter approach: these operations require the core to be in streaming mode, I assume to streamline memory load and store operations for z-tile computation efficiency (have to keep the unit fed). You can't inline the smstart or smstop instructions, so by using a simple bytecode interpreter several instructions can be chained together in the same stream session without having to write a new assembly kernel for everything you're trying to do with the matrix unit.

The results?

Performance characteristics that are identical to what Apple markets as the Neural Engine. Same throughput ceilings. Same restrictions (prefers int8, no FP8 support, same bf16/fp32 types). Same documentation (none).

I ran a contention benchmark on M4 Max — GPU (Metal INT8), CPU SME (smopa INT8), Apple's BNNS INT8, and NEON FP32 — both isolated and in every combination, 10 seconds each, with proven-concurrent overlap windows. Every time CoreML is processing a BNNS network, the throughput from the SME2 unit and the CoreML model are halved, proving that they are competing for the same silicon.

Still, I know Apple's marketing mythos is powerful (I still have to convince Claude that the M4 has an SME unit from time to time). For people who still want to believe these are two independent units, I invite you to imagine the following scene:

INTERIOR — APPLE SILICON DESIGN LAB — DAY

ENGINEER: Good news. We taped out the new Scalable Matrix Extension. Four ZA tile arrays, 16KB of new accumulator state, full UMOPA/UMOPS instruction support, outer-product engines, the works. It's on the CPU cores. It does matrix math very fast.

DIRECTOR: Outstanding. Ship it.

ENGINEER: Will do.

DIRECTOR: Oh, one more thing. We also need a second unit. Completely separate. Different part of the die.

ENGINEER: OK. What should it do?

DIRECTOR: Matrix math. Very fast.

ENGINEER: ...the same matrix math?

DIRECTOR: Same operations, same precision constraints, same throughput. But it needs its own name.

ENGINEER: Cramming another one on the die won't be easy, but it will be worth it for the extra performance. Imagine both of them spinning at the same time!

DIRECTOR: Actually, we need to restrict power usage. If one's running, make sure it throttles the other one.

ENGINEER: So you want me to spend transistor budget on a second matrix unit, with identical capabilities to the one we just built, that can't operate concurrently with the first one, on a die where every square millimeter is fought over—

DIRECTOR: Yes. Marketing has a name for it already.

What Apple calls the "Neural Engine" — at least on M4 — appears to be the Scalable Matrix Extension (SME2) built into the CPU cores, accessed through a software stack (CoreML/ANE driver) that abstracts it away. It's genuinely impressive hardware. Apple's marketing department deserves credit for making it sound even more impressive by giving it its own name and its own TOPS line item. But it's not a discrete coprocessor in the way most people assume.

Once you understand that, you can skip CoreML entirely and talk to the hardware directly.

Repo: https://github.com/joshmorgan1000/ane

Includes an all-in-one SME instruction probe script.