On this submit I am going to share how and why we (OpenFaaS Ltd) bought 4x DGX Sparks for our staff, and what we've realized from it. Having operated an RTX 6000 Professional with Qwen 3.5-3.Eight 27B for a number of months, we've expertise of each quick GPUs, and intensive clusters of unified memory. Qwen is a smaller dense mannequin that's like an attack canine - it's centered but needs careful handling, whereas the larger models reminiscent of DeepSeek or GLM are able to explore problems, largely unsupervised, at a slower rate and bigger context. Caveat: No a part of this weblog submit has been written by AI. It's long-kind as a result of buying 2-four DGX Sparks is a major investment for an individual, not to mention a enterprise.The primary two DGX Sparks purchased linked with a excessive velocity Join-X cable. You'll discover the photograph reveals two, and never 4 devices. The eventual purchase of 2 extra models was painful and arduous to justify, but opened a capability for us that no single consumer GPU could even hold in reminiscence. Whether or not utilizing a cloud model for an off-the-cuff chat, to develop a business plan, or to jot down and adapt code for your corporation, every token is distributed to the cloud supplier.Many retain the inputs and outputs from these exchanges to improve their fashions, and when you've got the opportunity to opt-out, they should be retained for up to 30 days. Conversations which are flagged for potential Terms of Service (ToS) violations could also be retained indefinitely and could end in account suspension. As you learn this submit, I would like to show you that we have concrete causes for deploying native AI for our group, and that it isn't just a matter of desire.It isn't at all about cost per token, and probably never might be. I'm not writing a doctrine or a manifesto, I am not making an attempt to influence you to follow our example. Nevertheless, should you do draw up the metaphorical moat of your castle, it cannot only be enjoyable, but provide tangible advantages and cut back danger. If your data crosses the castle's portcullis, is it really yours? The answer is extra nuanced than that. This put up is a pure successor to my piece in June: Local Qwen is not a worse Opus, it's a special device.There, we make the argument that only local would do for a subset of our core use cases in the business: customer support utilizing unredacted Kubernetes dumps, and some other sort of work where we'd like to use AI, however information-sharing with a third-celebration is simply not an choice. Many will argue that Zero Data Retention (ZDR) arrangements imply owning infrastructure is redundant. Others will argue that the 'tokenomics' (value per token) would take a number of years to break even on the outlay. I'm going to current two counter-arguments, which apply to our use.The primary is that any form of "knowledge sharing" with a third-party continues to be knowledge sharing, whether or not the supplier pinky promises not to process or retain it. The second is a collection of significant penalties of over-reliance on third-occasion inference services. A recent report from Anthropic alleges that at the least two vendors had been routing their requests on to Anthropic and nearly actually hiding the fact from their users, with out consent. Anthropic then went on to learn, digest, and publish data about highly confidential inside info - the full specifications and strategic targets of a flagship AI program, dwell credentials to a nationwide protection database, surveillance archives on a tracked particular person.The most worrying a part of the article for me was the place Anthropic demonstrates unregulated visibility that even governments lack. Past enterprise software program and assist, there are extra refined risks to cloud inference. A latest article in the Washington Put up showed that an individual's ChatGPT transcripts were used against him in court: They confided in ChatGPT. Their secrets and techniques ended up in courtroom. If subpoenaed, transcripts could in principle reveal proof concealment, establish premeditation, intent, or the timelines for a trial. OpenAI claims to have used an unreleased inner AI model with 10,000 autonomous brokers to resolve the Navier-Stokes existence and smoothness drawback in 88 hours, inadvertently beating human mathematicians like Tristan Buckmaster at New York University who had spent a year on related work.This raises the query: if Buckmaster had only used local fashions, would he have made the identical quantity of progress, and would OpenAI have even had this problem on their radar? BOSS88 login can get began with local AI, particularly if you use a effectively trodden path, and lean on a coding agent to configure the system. That said, it's value filling in any gaps in terminology. When i finished the primary publish in June, we had a reasonably properly-tuned Qwen 3.6 27B setup working on the 6000 Pro GPU. It was in a position to do all of the issues we would have liked it to, and in a short time.We're speaking about over one hundred tokens per second decode for a lot of agentic duties and for coding, with very little drop-off for two concurrent streams. We steadily started pushing Qwen to do real work that we would usually give to Claude or Codex. Qwen was much quicker, and much more right than the majority of community Pull Requests. There were two niggles: that despite adopting an up to date chat template, the mannequin would usually pause after a software name - or worse still, loop indefinitely. I remember listening to the followers working hard and I assumed my agent was simply getting on with the task at hand.After half-hour I determined to test in and the terminal's scrollback was completely filled with the same repeating line. So "native AI" for us meant supervision, and on the events I let it run unattended, I might likely come back to 30 minutes of 1000W (at the wall) being burnt for no good cause. DeepSeek presented roughly a 10x increase in parameter rely, and benchmarked very well throughout coding and agentic tasks. Alternatively, its decode speeds have been about half of that seen with Qwen on our RTX 6000 Pro.I wasn't even certain if forty tokens per second could be tolerable, let alone usable. So once i saw that my buddy Jacopo (from the Kubernetes and KubeCon days) had simply bought two units to run this same mannequin, I reached out. He set up a neighborhood recipe for 2x DGX Sparks, and we started collaborating nearly immediately over an authenticated inlets tunnel I hosted for us. I added that to my opencode config and the initial results had been actually encouraging. Though I have to say, we each obtained caught up by the claim of "65-85 tokens per second" by the author of the recipe, this turned out to be "counting to 200" - a approach of benchmaxxing the model's drafter.In actuality, it could hit that pace for short generations of predictable tokens, but was typically round 30-40 tokens per second. Just like with Anthropic, OpenAI, or OpenCode's Zen, I knew there was a really real chance that inputs and outputs were being logged or retained, so I handled the endpoint like any other public endpoint. Trying again, I pushed about 70-100M tokens per day (together with cache hits) and stored it to engaged on my open source initiatives (where IP is already out there), or driving our merchandise using agentic abilities (being careful not to leak credentials).I gave him suggestions over the course of a week, and we improved and updated the serving recipe. Not like our Qwen setup the place we were restricted to a local context length of 256K tokens, the community example for DS4F got here with 1M tokens per session, so we might work on bigger, unsupervised tasks. We did run into looping once, where a Korean character was outputted followed by a new line - a doubtless artifact of the tokenizer. Happily I caught it and that is not been much of a difficulty since. At the time, I believed two items would do all the things we needed.Stay with me, and you will learn how they inadvertently duplicated themselves. Having already purchased a relatively expensive RTX 6000 Pro for round 8500 GBP, then seeing it rise to 16000 GBP in a couple of short months, I felt lots of tension. Two DGX Sparks in the UK with 4TB drives had been going to value us virtually 10000 GBP to purchase, and the prices had elevated the earlier month. I in all probability did extra testing than was necessary with DeepSeek, but by the time I put the order in, I was completely certain this was going to be right for us. Qwen would stay our attack dog - fast, aggressive, extremely focused, and needing cautious dealing with.DeepSeek can be our explorer, planning product modifications, constructing reports, combining a number of abilities to test our products finish-to-finish, and exploring new ideas. On high of that, it could do all the issues we needed from Qwen to do, just a bit slower. At the time, the 1TB Asus GX10 gadget was also accessible at a decrease cost, but we've discovered that the 4TB drive is essential for exploring totally different quantizations and models, as and once they grow to be out there. NVIDIA presents a low cost program for qualifying startups known as Inception, and Scan was one of the vendors that allowed us to claw back about 600 GBP on the acquisition.The Founder's Version Sparks are lovely to behold, made from premium supplies, and intensely quiet whilst serving. The fit, finish, and sizing is very just like a Mac Mini M1 technology system, albeit with almost 10x the RAM. Setup expertise for the Sparks. The setup could be very properly refined, and does not require a screen. The Spark advertises a WiFi hotspot, which you connect to, and kind in the password from the booklet. Once fully updated, its WiFi hotspot will disappear and the machine reboots for about 10-quarter-hour. The confusing part is that the browser stays open with the progress bar making no apparent progress.Jacopo had arrange DeepSeek on his machines utilizing Codex, so I didn't hesitate to do the identical. I know the broad concepts - the model weights, the NCCL, the fabric, KV cache, and the way quantized all the pieces is, but beyond that I haven't pored over every element. For Qwen, we put in vLLM immediately onto the host, however the vast majority of Spark setups are closely tied into Docker, and that i managed to resist the urge to unpackage all of it.Docker's not remotely isolating anything right here, raw gadgets are handed on to the containers, it's extra about packaging assets and builds of the underlying infrastructure. Having faced the refusals and flagged prompts by cloud models, we did try out a group checkpoint that abliterated or "uncensored" the model. It did work as expected: the modified weights accepted prompts the bottom refused. But a move by Claude confirmed the base was already compliant - 11/12 safety prompts vs 12/12 for the modified weights. Nevertheless, we found that the drafter mannequin had a really low acceptance charge as a result of distributional drift between the base model and its abliterated serving path. There are additionally anecdotal reports of these modified weights being extra susceptible to looping.Knowing all this, we wouldn't run them without an excellent motive. In the put up about Qwen, I mention our inner LLM gateway: toilgate. It discovers the self-hosted models across numerous machines, data detailed utilization metrics, authenticates human customers utilizing OAuth, and bots could be provided with API keys. It solved just about all of the operational challenges we had with manual endpoints exposed over tunnels: the machine might change the model it's serving, it could also be down, it may be exhausting to see who is using what, when updates should be deployed, and managing API keys. We developed a plugin for the opencode and pi harnesses, which presents all obtainable models on start-up, together with an "auto" mode for if you simply want a fast response.When a mannequin is used solely by one individual - you might have a fairly good thought about how a lot you need to use it and when, however when it's extended to your team, it turns into harder to manage upkeep, updates, and to check out newer fashions as they arrive. There's even variance between totally different group and vendor quantized fashions. We found that with NVFP4 quantizations for GLM 5.3 Flash (320B complete, 18B lively), NVIDIA's version had a 20% lead on prefill, while it was 7% slower at decode than LibertAI's version.The majority of information for serving models on DGX Spark is cut up between X, and NVIDIA's user boards. And while evaluating fashions, runtimes, recipes, and quantizations, we often discovered claims weren't reproducible, or had been deliberately misleading for clout. Very few people will depart pace on the table, so speculative decoding is almost always used. A smaller drafter model generates a set of 1-7 doubtless next tokens, and the main model accepts or rejects them. It could imply the difference between 30 and one hundred tokens per second in decode, however does not do a lot for prefill. The issue with this space is that you don't know what you do not know. Structured output sounds very very similar to what agents do all day, but once you realise it is counting to 200 (one of the vital predictable duties there is for an LLM), you could also be about to experience purchaser's remorse.And it's also value saying that inside benchmarks, there are methods to sport the numbers, like turning considering off, and solely producing 400 tokens. We wrote RigMark to offer us and the community with a approach to get consistent and consultant figures for coding and agentic tasks. You'll find figures for 2-four DGX Sparks throughout a number of different models, and may send a PR so as to add your individual. Why did we purchase 4?Constructing on the success of Qwen, and DeepSeek, we wished to start out red teaming our code, and had heard that Hugging Face valiantly used GLM 5.2 to defend itself against OpenAI's assaults. They wrote up a particular information on self-hosting the model. That, and realizing how well it scored on CyberGym, running it at a tolerable speed became our aim. You possibly can read OpenAI's write-up on the Hugging Face incident here. The worst fallout of the incident is that many people exterior the tech industry believe that AI one way or the other can wake up one morning and determine, unprompted, to perform malicious cyber attacks of its own volition.Now, while it is completely possible that AI might be misused, in this case the swarm of AI brokers was working in the direction of a goal to get an ExploitGym flag that was set by humans tasking them. There is no such thing as a client GPU that would hold the GLM 5.2's 753B parameters, apart from the unified reminiscence of a cluster of DGX Sparks. Now most of our products are core infrastructure for builders and enterprises.OpenFaaS is a platform for working serverless features on Kubernetes, without lock-in to a single cloud's FaaS mannequin, and more tunability. It's used in a variety of regulated industries, including healthcare, protection, and oil & gas. SlicerVM was built to comprise and isolate AI coding agents, and our Inlets product is a secure community tunnel for entry to particular person endpoints - both privately or publicly. Last time I mentioned the actual work we did, folks accused me of advertising. I've saved the descriptions as terse as possible, so you possibly can perceive the applying of native AI in our actual work.So how do you make a product safe? You do static evaluation, you utilize individuals who know what they're doing, you be taught from the industry, and have responsible disclosure procedures in place. But beyond that you simply actively exploit your own software program to find vulnerabilities, and work with security researchers to patch them. These sorts of audits require in depth resources, excess of the cost of 4x DGX Sparks for a single run over just one product.That is paraphrased, however the point is that cloud providers have very stringent boundaries around pink-teaming, which is not just about cyber protection. It requires energetic cyber assault, creating Proof Of Concepts (PoCs) that can prove an issue, or escalate over a safety boundary. We will just about reside with 30-day retention for business purposes, but we absolutely don't want all of our core IP to be retained and trained upon indefinitely, and that can be a pre-requisite for entry to the massive labs' "Cyber Defense" programs. On the time of writing, to apply to Claude's program, you'll want to disable ZDR on your organization. A summary of our preliminary Purple Team run - each model ran the identical rank R0-R8 throughout six personal repositories, managed by a governor agent that ranked, collected, and verified the outcomes.I believed we would wish abliterated weights as a way to work on pink teaming, but was immediately shocked that both DeepSeek and GLM complied with no lecture or caveats. I also thought we'd need to buy a noisy MikroTik CRS804 DDQ swap (1500 GBP) so as to cluster the 4x items, however we managed to get away without it by patching the NCCL library and giving items that had been circuitously linked a strategy to route via one another. The pace seems to be on par with a switched setup.You will discover our switchless GLM-5.Three Flash TP4 recipe on GitHub, with a separate repository for the switchless NCCL patches, so you'll be able to adapt it to other models. When you place it this fashion, even finding 1-2 extreme security issues may pay for the 4x devices in a healthy enterprise. There are three factors of friction I need to handle as we shut out the How and Why. 3. Should a person purchase these personally? I can utterly see how people who've spent some huge cash on either GPUs or DGX Sparks double down on their purchasing choices, and see the opposite lane as in some way invalidating their very own.The "needs to be a GPU" crowd may be very loud, and "very proper" on X. Until very just lately, the DGX Spark crew had been in all probability surviving on fumes - trying to run dense fashions like Qwen 27B which simply profit from the speed that solely GPUs can provide for each prefill and decode. We've seen both sides and are within the fortunate place to not have to decide on between them, however at this stage, I feel the Sparks have completely won me over for utility, value for money, and performance. Not to say that internet hosting them in a house office requires very little house in comparison with my enormous Fractal Torrent ATX / Ryzen Pc, which if I used to be building it new in 2026 would value considerably greater than it did.I genuinely consider it was a lucky accident that DeepSeek V4 Flash 0731 simply happened to fit completely on two Sparks, and run shortly. The most recent model 4.1 Flash makes use of weights which might be virtually double the dimensions of the earlier generation, and require a further "N-gram" desk to slot in someplace - offloaded to NVMe, or squeezed into RAM. The tables provide virtually on the spot lookup speeds for tokens which might be commonly used.Buying 4x models of the identical generation of product is a risk, however these units will be capable of run immediately's models for a substantial period of time, and a brand new model being launched that's 2-4x the scale, does not all of the sudden take away from our skill to run present models. If a present model is value 10-20k GBP to our business right now, it's going to be value the identical tomorrow, and the day after. NVIDIA is reportedly halting production of the first-edition DGX Spark, however the third-celebration ecosystem (e.g. ASUS, Dell, MSI) will proceed to supply their very own variants. AI is shifting so fast, but we need to be grounded. I've taken to utilizing DeepSeek V4 Flash 0731 or GLM-5.3 Flash for product work instead of the leading SOTA model with Claude or Codex.I wish to see how far they are often pushed, and at instances, bleeding edge intelligence will provide a really related result, but with all the downsides we've explored right here already. Where I might put pause is if you need native AI at house, there actually is not an affordable technique to get into it. Every viable NVIDIA GPU has been rising in value month on month, and older group favourites just like the 3090 are energy hungry and sound like hairdryers.You may ask me how I do know. I ran two for just so long as I might bear it, before jumping to that RTX 6000 Professional. A card like the 5090 GPU (32GB) can serve a 27B model all day long, however leaves very little room for the KV cache. A card no person is talking about is the 5000 Professional Blackwell (48GB) which is available in a 1.5 slot blower format, and is designed for all day use. It has sufficient VRAM for a 27B model at a good high quality, with loads of room for the KV cache. My journey with native AI has been a continuum, that began off with sluggish, and unproductive exploration - and developed into a firm part of how we operate our enterprise and help clients.I purchased an NVIDIA RTX 3090 in 2023, and one other shortly after realising that 24GB was not enough VRAM for the fashions that we needed to run. That sat largely unused until Qwen 3.5 27B got here along in February 24, 2026 and adjusted the landscape for native AI in a everlasting manner. Following on from that, we purchased the RTX 6000 Pro and ran Qwen like our assault dog on specific, supervised duties. DeepSeek V4 Flash 0731 moved the aim posts once more, bringing long-horizon, close to state-of-the-artwork (SOTA) reasoning and coding to native AI. Not long after GLM 5.2 had turn out to be our goal for in-home red-teaming, Z.ai triggered a stir by releasing GLM-5.Three Flash, which had improved benchmarks and ran faster attributable to being a smaller flash mannequin, so we switched over to it.We had been able to pit Qwen, DeepSeek, and GLM-5.Three Flash against one another in a aggressive local AI contest over the course of every week, which discovered real Excessive, Medium, and Low vulnerabilities in our products, which we subsequently fastened. The irony shouldn't be lost on me that after i offered Claude with the findings of our week's work, it instantly flagged our account with Anthropic. Local AI is just not about saving pennies on the price of tokens. It might properly take you years to recoup the prices on that basis compared to coding plans. It's about the info you simply cannot share, the workflows that diverge from the blessed paths, and the sovereignty you gain from running AI with out invasive data assortment. You'll be able to observe me on X: @alexellisuk or reach out for a gathering in order for you to discuss local AI for your business. The switchless NCCL patches, recipes for serving models, together with RigMark for benchmarking can be found on my GitHub account.