With Apple devices you get very fast predictions once it gets going but it is inferior to nvidia precisely during prefetch (processing prompt/context) before it really gets going.
For our code assistant use cases the local inference on Macs will tend to favor workflows where there is a lot of generation and little reading and this is the opposite of how many of use use Claude Code.
Source: I started getting Mac Studios with max ram as soon as the first llama model was released.
> With Apple devices you get very fast predictions once it gets going but it is inferior to nvidia precisely during prefetch (processing prompt/context) before it really gets going
I have a Mac and an nVidia build and I’m not disagreeing
But nobody is building a useful nVidia LLM box for the price of a $500 Mac Mini
You’re also not getting as much RAM as a Mac Studio unless you’re stacking multiple $8,000 nVidia RTX 6000s.
There is always something faster in LLM hardware. Apple is popular for the price points of average consumers.
It depends. This particular model has larger experts with more active parameters so 16GB is likely not enough (at least not without further tricks) but there are much sparser models where an active expert can be in RAM while the weights for all other experts stay on disk. This becomes more and more of a necessity as models get sparser and RAM itself gets tighter. It lowers performance but the end result can still be "useful".
This. It's awful to wait 15 minutes for M3 Ultra to start generating tokens when your coding agent has 100k+ tokens in its context. This can be partially offset by adding DGX Spark to accelerate this phase. M5 Ultra should be like DGX Spark for prefill and M3 Ultra for token generation but who know when it will pop up and for how much? And it still will be at around 3080 GPU levels just with 512GB RAM.
All Apple devices have a NPU which is potentially able to save power for compute bound operations like prefill (at least if you're ok with FP16 FMA/INT8 MADD arithmetic). It's just a matter of hooking up support to the main local AI frameworks. This is not a speedup per se but gives you more headroom wrt. power and thermals for everything else, so should yield higher performance overall.
AFAIK, only CoreML can use Apple's NPU (ANE). Pytorch, MLX and the other kids on the block use MPS (the GPU). I think the limitations you mentioned relate to that (but I might be missing something)
I like Nix as well, you can use this one liner in OSX or Linux to try out arcan/durden/cat9, as it matures you can expect arcan applications to be made available the way html pages/apps and this kind of nix derivation would let you run the "browser":
An article called "A Spreadsheet and a Debugger walk into a Shell" [0] by Bjorn (letoram) is a good showcase of an alternative to cells in a Jupyter notebook (Excel like cells!). Another alternative a bit more similar to Jupyter that also runs on Arcan is Pipeworld.
The scipy/numpy to matlab is a good example. In my opinion it is on its way but in many places the timing is more like 2010-2013 where a lot of people knew python was the future but universities still used only Matlab.
I think the answer depends on the country: In places where the government uses QGIS it is like Blender. In places where ESRI has a stronghold it is like LibreOffice.
Hey AJ, this is almost on topic, do you know of a more up to date version of the dataset you used on the blog post release for H3 v4.0.0 [1]? They stopped updating in Oct 2023. Thanks!
[1] https://data.humdata.org/dataset/kontur-population-dataset
It is a gui framework that allows you create terminals where you can detach any running process into another terminal.
Since it is a complete toolkit, you can have detachable applications where you send both code and state to a server and retrieve it from another device (like Apple's continuity).
In the end it is just a bunch of lua scripts talking to other components via /dev/shm and to other computers using a new protocol called a12://
For our code assistant use cases the local inference on Macs will tend to favor workflows where there is a lot of generation and little reading and this is the opposite of how many of use use Claude Code.
Source: I started getting Mac Studios with max ram as soon as the first llama model was released.
reply