Software engineer (video games). Likes dogs, DJing + EDM, running, electronics and loud bangs in Reservoir.
Around 150W for UPS, Mikrotik router, Pi 4, Proxmox/TrueNAS server with 8x SAS drives. Without the SAS drives I’d be closer to 50W, but how else am I going to heat up my cupboard? I got the SAS drives at a bargain price thankfully… I estimate about AUD300/year in electricity.
Strongly recommend looking into Proxmox hypervisor as your base OS. Then install as many Linux VMs/LXCs as you like, with backups/snapshots all taken care for you. I’m on an Intel 8th gen CPU and have 3-4 VMs running, including Jellyfin, Home Assistant OS, an assortment of random docker images and a dedicated headless Debian VM for various AI projects. The CPU is more than enough, your 12th gen target will breeze through it. Get more RAM than you think you need - it will become the bottleneck for what you can do, not your CPU. I’m running 64GB and don’t regret it!
For video encoding/decoding like Jellyfin/Frigate, make sure you stick with Intel CPU for hardware codec support via QuickSync, otherwise you’ll need to add a GPU.
Read my post above to a different commenter, there’s a new type of ASIC being talked about that’s very different to an NPU or GPU, no RAM necessary.
LLM ASICs are different to NPUs. The idea is to encode a specific model’s weights (all billions of them) as a gigantic resistor network on the chip itself. Analog, where ohms law becomes the processor, and no RAM is necessary - the chip IS the model. It would pave the way for cost effective, fast, low-power local inference. Given the analog circuitry, there would be issues like interference, temperature sensitivity and so on, but we’re seeing models get good results even when heavily quantised, so the idea has legs. The big disadvantage is you’re stuck with that one model forever, so we’re likely to see these first appear in non-consumer applications like automotive, defence etc. A pretty exciting concept that solves some of the environmental issues around AI.
More info for the curious: https://taalas.com/the-path-to-ubiquitous-ai/
I’m really interested to see what comes of the LLM ASIC idea - it worked wonders for hardware video codecs. I could see the smartphone industry picking up again on annual releases worthy of upgrades if they included dedicated AI chips not beholden to a cloud subscription. I think we’re reaching the point that the frontier models are becoming “good enough” to use for a while before uploading to the next.
Looking forward to when they ship the S28 with no camera and you just prompt an LLM for what you want a picture of.
Sorry
And even that would be considered absurd by yesterday’s standards. The entire game ROM for Super Mario Bros on NES is 40KB.
How was the experience? And worth the pain?
It’s up to you to make it cloudless, but Home Assistant is the only solution I know of out there that even allows this possibility. I refuse to use anything in my home that requires a third party app or cloud connection (aside from initial pairing so I can flash it with ESPHome or some other local-only firmware). Admittedly it complicates things, but the payoff is so worth it.
And it’s so nice having zero dependence on the cloud. If the internet drops out, everything still works, including the mobile app.
It is, but it’s been branded Linux Subsystem For Copilot 365 Enterprise Service Pack 2.