DwarfStar 4 is a really interesting project and I want to go over what makes it so special, how to install it, and give you some demos on some things that I have built with it to give you a real world kind of indication of what to expect from this project. And yes, I was running it on my laptop fan, which if you haven't seen yet, go watch that video. It basically speeds up the model so much. It is basically a tool for serving DeepSeek V4 Flash and V4 Pro, but also GLM 5.2. And it is aimed at getting these models running on consumer hardware. You can run it on Metal. You can run it on Nvidia or you can even run it on ROCM, which is supported by the framework desktop. It is built on top of Llama CPP, so it is really close to the metal we call it. And it is really geared to work on 128 gig or 512 gig workstations, such as my M5 MacBook Pro with 128 gig of RAM or if you have it, the Mac Studio with 512 gig of RAM. Now, looking at DeepSeek V4, it is basically a big ass model. We are talking 150 to 160 gig for the flash version. Now, this is just way too big to run on any consumer hardware, but that didn't stop the DwarfStar team trying to get it to run. To get this going, if we copy the GitHub repo URL here, we knock into terminal. Now, I have navigated to a folder that I have created. You are going to want to go get clone this and put a full stop depending on if you are already inside the folder. I will be doing a masterclass on using the terminal on all this sort of stuff if I haven't already, which might be linked below. Otherwise, you are going to clone the repo. I have already done that. And if we go ls-la, we can see all of the files that were once in the git repo are now on my local machine. And then you want to actually make the model itself. If you are on metal, Linux, CUDA, you do you. I am on Mac here, so I am just going to run make. And it is going to package all of the code up into a set of applications we can use to actually run this model. Next, you want to actually download the model itself. It is going to download it from Hugging Face. And this gives you an indication of what model to download. Basically, if you have got 96 to 128 gig of RAM, then you can download the two-bit quantisation version here. Or if you want to push it a little bit more, we have got our two-bit quantisation with the last six layers being quantised to four bits. So, it is just going to be a little bit bigger. Basically, if you have a workstation that can support it with 256 gig of RAM, you can go full-blown 4-bit quantisation, which is a really nice level to be quantising a model. If you have got the whole hog, then you can go to the Pro 2-bit quantisation, but that is only supported with 512 GB of RAM. So, I have already done it, but let's say you got enough room to push this one here. You want to copy that and actually paste it there. And that is going to download the model using the script here and setting the model in which you want to download. And that is going to download it into the GGUF folder. Now, this is the really interesting part and which is why it caught my attention. GGUF typically does not work as well on Macs as MLX. Now I have done the tests comparing it against MLX and we will get into that later on. But this was again the thing that really interested me about the model, that it is really focusing on the GGUF and that is in part due to the levels of quantisation that is happening in here. So we have got a base level of two bit quantisation. However, we have got different areas of the model that are quantised at different levels, different sizes and stuff like that. Basically, everything is done in order to squeeze as much performance out of the model whilst keeping it as small as possible. This is the stuff that really interests me over just piling in the number of parameters and just straight up quantising something flatly against two bit quantisation or four bit quantisation all around. So really interesting approach and the size of the model shows for it. And this is I will just ask warp terminal here and it will come back to us. It basically is about 80 to 90 gig something like that. There you go. 86 gig. And with that made now we can run DS4 server. And if we pass in the model and a link to gguf and then the actual model name that we downloaded. Pass in metal because I am of course on metal and I am going to run it on host 0.0.0.0. Now, this does not do anything other than allow this model to be served across my network. So, I can actually access it from here, preserving more VRAM on my bigger laptop and just do all my coding on this one. And then I am going to customise the port here for 8765 just for funsies. And that is literally it. This is now serving that model ready for us to consume in whatever coding software we are going to consume it in. Now I will put on the screen now the settings that are on that machine to run it in open code. I do a test here. It is going to load up there and then on screen this. You will start to see it doing its thing. And we are now using this model on that machine or it could be the same machine depending on how your setup is running on localhost port 8765. And you can see that took just under a minute to process. We have our result over here and we now have DwarfStar running with DeepSeek V4 Flash. What are my observations from this model? First of all, it is very very impressive they got this model running on consumer level hardware such as my nearly £10,000 MacBook Pro here. As I mentioned earlier, I tried comparing this to the MLX version of the model and I looked on OMLX here. I downloaded the closest one I could get to the similar size 80 to 90 gig was two bit quantisation of DeepSeek V4 and quite frankly it could not run on this machine. There is an amazing level of different levels of quantisation optimisations that mean having such a large model running on my machine is an incredible feat of engineering. I kept running into problems trying to use the MLX version on this machine. I used the recommended settings which are, if I remember correctly, temperature and top P set to one. Again, it just would not work. They would just run out of space. It just would not do the things I want to do. On the GGUF version, I am getting about 20,000 tokens before it needs to compact. So, I would need to do little pieces of work at a time. So, this is what I actually built. It is the same app. I am just accessing it from different machines. I am pretty proud of this little setup I got going on. The problem is that I worked on this app all day yesterday. And it is a simple app. It just gives me the ability to upload PDFs. It uses a local vision model to infer the document, whether it is handtyped or written out. It is an OCR model. I have done a lot of investigation over this. And so, if you want a deeper dive on local vision models, I would love to share that with you. Just let me know down below. After working on it all day, I mean, first of all, the UI isn't so impressive. Second of all, it actually does break. It does not work. There was no text to extract. I actually got GLM 5.2 to look at the code and it identified the problem with the code. Whereas DeepSeek version 4 flash just could not find the issue with the code. Now these are the actual logs from working on it yesterday. So the first initial run went from 1841 to about 1855 here coming in at 6 minutes and 56 seconds. And that was to get something on the screen and actually loading. But there were a few bugs that I had to work through which took many many hours. Some things were not working correctly and I could not get to a working app. There was something interesting that happened on the MLX version and I am not sure if this is an open code problem or a model problem, but I actually just started looking because it was an empty directory that I wanted to create the app in. I actually just started looking around in other folders to determine what it should build. And I kept trying to tell it not to, that it is an empty folder and it should build the app from scratch. And it gave me an outline which just so happened to be a very similar setup and layout as the GGUF version. I just could not trust whether that was it holding the memory of the listing out the files in its cache or anything like that. The point is the model itself was behaving really badly and that kind of in lies the problem about the DeepSeek V4 flash model. In my opinion, it is an impressive feat of engineering, but if this model cannot do the work that you want it to do, it is not able to do the work, if it is not smart enough to do the work that you need it to do, in my eyes, it is just not a good model. And I see a lot of people using DeepSeek Flash. So here comes my first question: what are you doing with this model that makes it a really appealing model? Are you doing like small little changes here or there that mean you fix little issues? Because at the end of the day, it actually did a better job of architecting the software itself and laying out all the files than it did actually fixing bugs and I made sure I cleared the cache. Sometimes I even restarted the server but like starting a new session on open code. These were smaller tasks that it did worse in than actually again architecting the app. I always thought architecting the app is a much more difficult problem to solve than bug fixing and things like that. So what are you doing with this model that makes it so appealing to you? That is my verdict on using DwarfStar with DeepSeek V4 Flash. I definitely want to see this project gain a bit more traction. Obviously, as the models get better, then this in and of itself, the actual technology around DeepSeek V4 Flash is really, really impressive. But ultimately, even at 128 gig of RAM, the Flash model just is not smart enough for me to do coding work. That will do it for this one. Like, subscribe if you haven't already. Till next time, keep on vibing.