About this video
Cloud AI is a privacy nightmare you can finally escape. This video demonstrates how the new Hermes update allows for one-click local model setups, eliminating the need for complex configuration files and manual hardware guessing. \n\nKey Takeaways:\n- How to set up local models in Hermes with a single click.\n- Understanding the difference between GGUF and MLX performance.\n- Using custom endpoints like OMLX to run 27b models.\n- Real-world hardware testing using complex logic puzzles.
The complexity of local AI was a lie designed to keep you paying for API tokens
For too long, running a decent large language model on your own hardware felt like trying to build a rocket in your garage. You would spend an entire evening wrestling with environment variables, configuration files, and terminal commands just to get a single prompt out of a 7b model. Those days are officially over. With the latest update to the Hermes platform, local AI has reached its 'plug and play' moment.
The one click revolution
The most significant barrier to entry for local AI has always been hardware compatibility. Most users have no idea if their GPU can handle a 27b parameter model or if they should stick to a 4-bit quantised version of Qwen. Hermes now solves this by automatically scanning your system and recommending models that actually fit your VRAM.
You no longer need to guess your context window or manually set your repeat penalty. The programme handles the heavy lifting, allowing you to go from installation to your first local prompt in under 15 mins.
GGUF versus MLX: Speed and compatibility
While GGUF remains the gold standard for broad compatibility, specifically on Apple Silicon, MLX often takes the lead in raw speed. The Hermes interface allows you to bridge these worlds. By using custom endpoints like OMLX, you can tap into the specific performance gains of MLX models while keeping the slick, agentic workflow of the Hermes interface.
We tested this with a 27b Qwen model, and the results were staggering. The model correctly identified the 43 quintillion possible configurations of a 3x3 Rubik's cube while pushing the GPU to 100% efficiency. This is not just a toy; it is a professional-grade tool running entirely offline.
Why you should switch today
- Privacy: Your data never leaves your machine.
- Cost: Zero subscription fees or token costs.
- Simplicity: No more Ollama terminal marathons.
If you have been waiting for a reason to take your AI local, this is it. Download the latest version, check your provider settings, and see what your machine is truly capable of.
Transcript▾
Hermes has just launched local models, solving one of the biggest problems people face: what is the best model to run on my device? Until now, it required an entire evening dealing with setup files and extra processes, running your local model on one app while consuming it inside Hermes, guessing context lengths, and model settings like repetition penalty and Top K. By the end of this video, you will have a one-click setup for local models running on your Hermes agent, with your local models added without needing to reload them.
Chapter 3 of this video will be when I stop using the button Hermes provides, not because it is broken, but because of one number it does not show you in the interface. This is why some local models seem unusable, and most people give up after a week. The cost is about 15 mins of your time and a few extra megabytes to download the runtime, plus the model size.
When you load Hermes, go to settings and under 'Providers', you will find the new local models list. Ensure you are using the latest version. Under 'Providers', you get model recommendations. It often suggests Qwen. I can run a 35b model with 21gb of space. In 'Configuration', it shows a 27b 3.8 model, which is smaller but not a Mixture of Experts (MoE), which agent workflows prefer. You can download other models, but it might say they are too large for the device. It can handle a full context window of 256k.
You can add models from Hugging Face. Check the GGUF files to see if they fit your GPU. Usually, you just click 'Set up for me' and it downloads automatically. If this saves you an evening of setting up Ollama, leave a like.
Once installed, scroll to the 'Local' section to choose the model. Testing a 3x3 Rubik's cube query: the GPU hits 100% and Qwen reports 43 quintillion combinations. For existing models, you can add a model file. Hugging Face usually downloads to your root folder. If you use 'command shift full stop' to show hidden files, check the cache.
One hurdle is that this does not run MLX models natively. GGUF and MLX both work on Mac, but MLX is often faster. If you want MLX, use a server like OMLX. I have a 27b 4-bit Qwen model connected via a custom endpoint on port 8006. Dense models like the 27b version are slower but often smarter than MoE versions.
Hermes local models are the best implementation I have seen. The compatibility check alone solves the biggest decision for beginners. Download Hermes now and see what you can run.