The capabilities of various SIMD ISAs don't have enough intersection to be portable outside of relatively trivial cases. Many of the somewhat unique capabilities are load-bearing, so you want to use them on architectures that support them. Taken in whole, someone who cares about performance would be using different data structures and algorithms depending on the specific SIMD architecture and that is nearly impossible to abstract in a library. Too many important but complex details are idiosyncratic to the implementation.
Another way of looking at it is that our programming environments are not sufficiently powerful and expressive to create the necessary abstractions to make SIMD truly portable.
> The capabilities of various SIMD ISAs don't have enough intersection to be portable outside of relatively trivial cases.
I would argue that the "trivial" cases (those relating to linear algebra in 3 dimensions) are also 95% of what people want SIMD for.
If the API can achieve cross-platform and performant vector arithmetic, dot product, and matrix multiplication in the normal ways, that already covers a lot of what people actually need.
Most use cases for SIMD are non-arithmetic in nature and don't assume tidy arrays of homogeneous types. I also use it for some computational geometry but that is the least interesting use case.
SIMD is widely used throughout data infrastructure e.g. parsing data, complex constraint processing, parallel manipulation of heterogeneous data types, compression, etc. I even have an I/O scheduler written in AVX-512 that is many times faster than the scalar equivalent. The ability of SIMD to do complex manipulation of ordinary data structures several times faster than scalar code is under-rated.
While linear algebra is the current thing, database engines have been using SIMD heavily for over a decade and arguably represent the frontier. It is for these use cases that SIMD is non-portable and data infrastructure isn't going away.
It should really be read/advertised as "portabler SIMD". It beats hoping the compiler autovectorizes everything well forever or writing architecture specific code manually again but is going to compromise on average performance vs platform specific SIMD.
Sure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.
Which is why I've never quite understood the appeal of portable SIMD libraries for performance-critical code. If I'm explicitly writing SIMD rather than relying on the auto-vectorizer, it's usually because I want access to the particular capabilities of the target ISA.
For many problems, choosing the right instruction or instruction sequence makes a large difference. Portable SIMD abstractions necessarily expose some common semantic layer, but SIMD ISAs don't actually have equivalent capabilities. Instructions like pshufb, for example, enable algorithmic tricks that don't necessarily have an equally efficient analogue on another architecture.
If maximum performance matters, I generally want intrinsics and architecture-specific implementations; if portability matters more, I'd rather move further up the abstraction stack and use something designed to target multiple architectures, such as ISPC. There are certainly cases where portable SIMD gets close enough to optimal, but I don't think there's a compiler or abstraction that can express every useful SIMD idiom and lower it equally efficiently across fundamentally different ISAs.
Because usually they achieve a very good middle ground, they are useful for when autovectorization isn't good enough, and it is possible to give a little help to the compiler.
There are many ways that performance matters without trying to win a F1 race.
Go isn't alone, .NET, Java have similar portable libraries, and C++ is in the process of getting one.
> They specifies a constant SIMD width so it's non-portable.
This is incorrect, you can use vectors wider than native SIMD width and the compiler will break them down to register size of the target cpu.
In fact it's sometimes better to used wider than native width, in some applications I see 20% better throughput with f32x16 (512 bits) on an AVX2 CPU (256 bits). It is kinda like loop unrolling it.
Except you can't use this in actual code, because either, as is the case in this example with f32x32, you run out of registers and spill all over the place.
Or you aren't using your full vector register or could've gotten better performance by "unrolling" more often for the larger vectors.
If you use f32x16 (the avx-512 wisth), SSE now effectively has 4 registers to work with and will spill when doing anything beyond the most simple stuff.
The default should imo be relative to the native register width, so you can do 1x, 2x or sometimes 4x the native width, depensing on your register preasure.
I can and I do use this is "actual code" and I've got benchmarks to prove that it's got better throughput (for the particular use case, don't extrapolate from there) and the same applies to AVX2 and AVX512: twice the native vector width has ~20% better throughput (ie. using `f32x32` on AVX-512).
I pass in the vector width as a generic parameter like this:
With this I can easily benchmark the same code for any vector width. I can also do some compile time heuristics to choose the vector width based on what's available on the compile target CPU.
> you run out of registers and spill all over the place
As usual when optimizing SIMD code, you should keep an eye on the generated disassembly and the benchmark results and watch for register pressure and the other usual things.
I'm definitely NOT saying that you always get the best perf by using 2x SIMD width, but in this particular case it was so.
This is much much easier to do with portable_simd than if you'd write the same with intrinsics, you can change the SIMD width without having to rewrite all your code (e.g. changing from SSE `_mm_add_ps` to AVX `_mm256_add_ps` etc).
It's still a partial solution, you still need to drop down to intrinsics for some special instructions every now and then (which is easy), but in my projects this accounts for much less than 1% of the lines of code. Not applicable everywhere of course.
> twice the native vector width has ~20% better throughput
Yes, this is what I was saying, but twice the vector width of AVX-512 will perform horrible in SSE, which is why portable SIMD abstractions should make writing code relative to the native vector width simple.
> I pass in the vector width as a generic parameter like this:
My problem is that no portable_simd example code I've seen does this, which causes people to choose one specific N and run with that.
The second part of the problem is how you find the native vector length, so you can instantiate the generic function. IIRC this isn't even exposed in portable_simd and you have to use a seperate crate to get it.
> The second part of the problem is how you find the native vector length, so you can instantiate the generic function. IIRC this isn't even exposed in portable_simd and you have to use a seperate crate to get it.
This is trivial (but not pretty!) to do with something like `#[cfg(target_feature = "avx2")] const SIMD_WIDTH: usize = 8`. You need a few lines of ugly cfg logic to configure this.
A somewhat orthogonal and much more difficult problem is how to select it at runtime. You would either need to have different binaries built with different compiler options, link object files built with different compiler options to same binary, or dynamically link the correct code at runtime.
This is actually one of the (IMO only) cases where intrinsics are more practical: you can use `_mm256_add_ps` from AVX2 intrinsics regardless of whether you've configured your compiler to support AVX2 or not. As long as you check at runtime before calling the code so you don't get illegal instruction exceptions.
> There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width.
What should it choose then? I have a Zen 3 processor, and benchmarking some simd I did recently says 32 byte or 64 byte chunks was fastest. But I'm sure I'd get a different result on a different Zen, and different again on Intel's.
How would the library decide what SIMD width I should use?
It'd need some kind of compile-time hardware-feature-detection, yea? That seems probably feasible since proc macros can do essentially anything they like (worryingly).
Only if the end-user is the one compiling the software, on the same very system they'll be running it on. Which is true of GPU shader kernels, due to how GPU drivers work; but isn't generally true of CPU object code (unless you're on Gentoo.)
What you'd actually want is a matrix of variant implementations burned into the binary, with runtime (or process-boot-time) hardware detection that swaps symbols out to point to the correct variant.
If you want the library to perform that selection, you also need the "correct" / most efficient implementation to be independent of your workload. I'm not that familiar with SIMD performance characteristics, but I wouldn't be surprised if that's not always the case.
From how I understand it, there'd likely only be a single SIMD function impl per uarch that'd actually be fully legally executable without hitting undefined instructions. Plus increasingly-more-generic function impls compiled for lower and lower common-denominator subsets of SIMD functionality. (Ultimately grounding in a non-SIMD impl.)
If that's the case, then the selection logic would be trivial: figure out the full hierarchical ID of the uarch you're running on, then search for the longest prefix match in the table of available impls.
If things work more like you're imagining, though, then I suppose the process-boot impl-selector would narrow down the impl matrix to just the subset that are legal on the running uarch; pick one arbitrarily to be active at first; and then wrap the calls in a handler that gradually re-works the called function in a way reminiscent of a profile-guided JIT, but without the need to actually synthesize any code at runtime — instead, it'd just be a multi-armed bandit passing-through-to and re-ranking competitor impls, with decreasing sampling of the non-first-ranked impls as confidence-in-score-separation increases.
It's definitely a 80% solution where you occasionally need to drop down to intrinsics (at zero runtime perf cost) for CPU specific instructions.
But just having vector types, arithmetic, swizzling, loads and stores will go a long way for basic tasks.
And with generics you can write code that is type and width agnostic. No need to rewrite your code of you want to go from SSE to AVX512, just change from f32x4 to f32x16 (or use generics) and you are done.
Because there are platform vendors. And SIMD performance very much depends on the use-case, which is a balance of practicalities and specifications and intended deployment targets ..
I also think this is a deployment problem, not a build problem, but okay ..
> also think this is a deployment problem, not a build problem ...
This I agree with, deploying and running code for the correct cpu is a problem with no established solution.
As for actually writing the code, portable_simd is great. You need to adjust simd width and compiler config for the cpu you deploy to and fill in the blanks with intrinsics. Which is much less work per target than writing it all with raw intrinsics if you are deploying to more than one target.
I wrote a multi-target SIMD-using synthesizer, and I don't think I would've had as much success if I were just using someone else's library - its been especially important to have the SIMD instructions for both architectures I'm supporting (ARM and x86) directly available in the code, since a synthesizer necessarily involves a working pipeline from one stage to the other. I wonder how much easier/better the code would have been to write with portable_simd ..
> I wonder how much easier/better the code would have been to write with portable_simd ..
I had a quick glance of the code and it seems to use mostly basic arithmetic instructions on fixed width simd vectors using some helper macros like SIMD_MUL(x, y) for _mm_mul_ps, etc. Some explicit simd intrinsics code for stuff like sin.
That would've been pretty easy to write with portable_simd in Rust or the equivalent C/C++ language extensions (or maybe C++26 std::simd).
You'd just use f32x4 (or add a typedef with attribute in C++) and then use x*y instead of SIMD_MUL.
For basic stuff like this, you should get the exact same generated code.
They specifies a constant SIMD width so it's non-portable. Well, not performance portable, but why are we using SIMD again?