Deepseek referred here seems to be v3, not r1. While the linked article didn’t seem to have info on parameter size, fact that they state it is sparse MoE architecture should suggest it is capable to run pretty quick (compared to other models of similar parameter space), so that’s cool.
Depending on what you want to do with it, and what your expectations are; the smaller distilled versions could work on CPU, but most likely will need extra help on top, just like other similar sized models.
This being a reasoning model, you might get a more well thought out results out of it, but at the end of the day, smaller parameter space (easiest to think as ‘less vocabulary’), smaller capabilities.
If you just want something to very quickly chat back and forth with on a CPU, try IBM’s granite3.1-moe:3b, which is very fast even on a modern CPU, but doesn’t really excel in complex problems without additional support (ie: RAG or tool use).
8B parameter tag is the distilled llama 3.1 model, which should be great for general writing. 7B is distilled qwen 2.5 math, and 14B is distilled qwen 2.5 (general purpose but good at coding). They have the entire table called out on their huggingface page, which is handy to know which one to use for specific purposes.
The full model is 671B and unfortunately not going to work on most consumer hardwares, so it is still tethered to the cloud for most people.
Also, it being a made in China model, there are some degree of censorship mandated. So depending on use case, this may be a point of consideration, too.
Overall, it’s super cool to see something at this level to be generally available, especially with all the technical details out in the open. Hopefully we’ll see more models with this level of capability become available so there are even more choices and competition.
Changing port is security by obscurity and it doesn’t take much time for botnets to scan all of IPV4 space on all ports. See for example the ever updated list that’s available on Shodan.
Disable password login and use certificates as you’ve suggested already, add fail2ban to block random drive-bys, and you’re off to the races.
Yep! Give granite a try. I think that would be perfect for this use case both in terms of able to answer your queries and doing them quickly, without a GPU by just using modern CPU. I was getting above 30 tokens per second on my 10th gen i5, which kind of blew my mind.
Thinking models like r1 will be better at things like troubleshooting a faulty furnace, or user problems, so there’s benefits in pushing those envelopes. However, if all you need is to give basic instructions, have it infer your intent, and finally perform the desired tasks, then smaller mixture of experts models should be passable even without a GPU.