We live in a world of cattle, not pets, and Kubernetes rules the roost. I’ve been meaning to spend some time getting up to speed on the latest but didn’t have enough hardware to make that happen until now. I recently bought a whole pile of surplus hardware so I will be able to experiment with orchestration and container platforms in the office. Watch this space for more details on “Rabbit”, my personal cloud lab!
Read my entire Rabbit Cloud series
- Introducing Rabbit: I Bought a Cloud!
- Running Rabbits: More About My Cloud NUCs
- Tortoise or Hare? Nvidia Jetson TK1
- Powering Rabbits: The Mean Well LRS-350-12 Power Supply
Note: I haven’t been blogging much, and I know lots of people are subscribed via email. That’s a great way to keep up with my personal blog! If you don’t want emails from me, please go ahead and unsubscribe. It won’t hurt my feelings!
Rabbits or Cattle
Over the last year, I’ve repeatedly repurposed my VMware vSphere lab. First, I used Docker Swarm, then I blew that away and ran Rancher, with a few stints as standalone machines in between. The more I’ve gone down the container orchestration rabbit hole, the more I’ve realized I needed more hardware to truly experiment with Kubernetes.
My lab consisted of just three Intel NUCs: A fourth-generation Core i5 and two fifth-generation Core i3 machines. They’ve been an excellent platform for containers, running a plain version of Ubuntu, and it’s great that they’re x86 machines, since this is much better supported by container images. But I needed a lot more! I soon added in an old Sandy Bridge Core i5 desktop and even my 2013 “trashcan” Mac Pro. But these were all “pets” with their own personality.
I needed “cattle.” Or more precisely, “rabbits!”
My friend Robert Novak has been posting images of some surplus server hardware he recently purchased: 10 Intel NUCs plus 5 Nvidia Jetson bare-board computers. I immediately saw the possibilities of having a dozen or more computers to play with. They had originally been part of a video encoding and streaming platform called “Rabb.it” and came with networking and power already connected!
Using Robert’s link to a long-running eBay auction of 373 such units, I made an offer on three trays of 15 computers each. My rabbits are multiplying! They were almost free at this price, but still need RAM and storage to make them useful. Still, my whole cloud lab will cost less than a few new machines. And they’ll be quieter, cooler, and use less power to boot!
I like the concept of “rabbits” for these systems. They can be pets or cattle, depending on your perspective. So I’m going to call this my “Rabbit Project”!
My Rabbits: Intel NUCs
Each “Rabbit” tray includes ten Intel NUC5PPYB integrated single-board computers and five Nvidia Jetson TK1 single-board computers. The main attraction for me are the Intel NUCs.
Although not much faster than a Jetson (or a Raspberry Pi 4 for that matter), these old fifth-generation NUCs are equipped with “Braswell” generation Pentium N3700 CPUs. Fully x86 compatible, these are ultra-low power quad-core processors based on the “Silvermont” Atom core. They support up to 8 GB of RAM and include Gigabit Ethernet, SATA, USB 3.0, and even a compact M.2 slot.
The surplus NUCs come without RAM or storage, however. Each uses a single DDR3L SO-DIMM and the system supports a maximum of 8 GB. So I headed back to eBay and ordered Samsung RAM in bulk from a company in Hong Kong. I’m not yet sure what I’m going to do about storage, so that will be a topic for a future post. They might boot from SD card (a full-size slot is included), USB, SATA, or M.2. Robert is using some bulk SATA SSDs but these require a special cable for power, adding to the cost. I have also ordered a super-compact (2230) M.2 SSD to test.
Much to my surprise, a closer inspection revealed that one of my Rabbit boards contains sixth-generation NUC6CAYB boards instead of the fifth-generation NUC5PPYB I was expecting! These have Celeron J3455 CPUs which use “Goldmont” Atom cores and are roughly 40% faster despite having a slightly slower clock speed. They also support dual-channel memory and have better graphics with Vulkan 1.2.
Note: Although the specs say it it limited to 8 GB, I have confirmed that the NUC6CAYB works fine with 16 GB (2x 8 GB SO-DIMMs)!
None of these are speedy by any means, but they will do for lab use. Equipped with 8 GB of RAM and an SSD, each will easily run a few containers and associated services. And each NUC or Jetson draws just tens of watts even fully loaded and is barely audible.
My Hares: Nvidia Jetson TK1
The Rabbit boards also include five Nvidia Jetson TK1 single-board computers. These are ARM-based systems that were attractive last decade thanks to their Tegra K1 SoC, which includes a modestly-powerful “Kepler” GPU.
Originally pitched as machine learning systems, the Tegra K1 GPU did not live up to its billing. The 192 ALUs don’t offer much and software support is severely lacking in 2020. Still, despite being just 32-bit, the ARM Cortex-A15 CPU is faster than a Raspberry Pi 4. And the Jetson TK1 board includes 2 GB of RAM and eMMC storage for booting. Plus, it uses even less power than the NUCs: I’m booting one for testing with one of the many power supplies left over after shucking some external hard disk drives.
The NUCs will be the “rabbits” I use to run container workloads, while the Jetsons will be the “hares” I use to support the cluster. I intend to use them as routers, log servers, time servers, and monitors for the cluster. I’m also considering using a few as storage servers, since they also include SATA and M.2 and have plenty of power to support NFS.
Other Components: Networking, Power, and More
Each Rabbit board includes a Netgear ProSafe GS116 16-port Gigabit Ethernet switch. These are simple, un-managed switches but consume little power and are silent and fan-less. Plus, they’re all cabled up to support all 15 nodes on each tray.
The Rabbit trays also include a simple power supply setup, with a 350 Watt 12 Volt power supply and fuse blocks. These are simple and efficient as well, and were designed to support this workload with ease. While they’re nothing flashy, this setup sure beats a pile of power bricks!
One more unusual piece of hardware is included: NewerTech Headless Video Accelerators. These are dummy HDMI plugs that emulate an attached monitor so the NUCs will not disable their GPUs when booted. I actually purchased these for my lab NUCs previously, and they are surprisingly useful. With these in place, the Intel GPUs can be used for Vulkan or OpenCL workloads.
Stephen’s Stance: So Many Rabbits!
I ordered a pile of these computers so I would never again have to scrounge or re-purpose one. Any time I need an x86 machine, I will simply grab a Rabbit NUC from the tray and have it running in minutes. It’s nice to have “enough” systems at hand!
My next challenge is kitting the systems out with RAM and storage and getting them to boot. The previous owner set them for network booting from the looks of it, so I’ll be blowing away the current configuration in favor of a standard Ubuntu or Jetson Linux install. Follow along here on the blog as I select and experiment with RAM and storage, configure the operating system, and get the cloud up and running!
Very cool. My SSDs are arriving today in the mail, so hopefully I will have the NUCs up and running this weekend.
The Jetsons are loaded by Jetpack 3 from NVIDIA with Ubuntu 14.04 (I think it’s .04) but can be upgraded online to 16.04. Maybe newer with some limitations.
Thanks for the tip!
I was excited to learn that the NUC6s support more than 8GB RAM!
Also, I did find a 3320 M.2 SSD so I ordered one to try. If it works, this would be an amazing storage platform for the NUCs!
I’m trying to side-load the Jetsons now but am having issues with them. So far two have refused to boot and I’m just following the Nvidia install instructions. Which OS are you using? Where did you find it?
It took some work to find the right version – TK1 is long in the tooth even for a rabbit and NVIDIA doesn’t make it easy to find the supported Jetpack versions (Jetpack 3.1 is the last one supporting TK1, with L4T r21.5).
Try https://developer.nvidia.com/embedded/dlc/jetpack-l4t-3_1 – requires a free NVIDIA developer program membership, and after you log in you’ll have to go back to that URL to download it.
It will tell you it only supports Ubuntu 14, but I threw Ubuntu 16.04 on an older laptop and it worked fine.
And I think you mean 2230 for the shortest m.2 form factor. Let me know how that works out … if I do a second bundle I might consider going m.2 instead of 2.5.
Yes sorry – 2230 of course. But yeah we shall see! Anyway, SD boot is working fine so for now I’ll just do that. SD cards are cheap!
I’ve got one of these trays on order. From what I’ve researched, you should be able to run Armbian on the TK1, although it requires an updated UBOOT. I’ll be attempting that at some point.
Excellent! I haven’t finished my Jetson TK1 writeup, but the summary so far is “it kinda stinks”. Mainly because it’s painful to install Tegra Linux, and the one you get is painfully outdated. I hope Armbian works better.
One issue though is that the TK1 appears to have a buggy USB controller. I get typing issues, and the network is slow. Apparently there’s some mojo about uploading firmware on boot or something.
One thing I don’t understand about building an experimental lab out of a bunch of small computers is why you would do it in the first place rather than buying a single used previous generation PC with as much RAM and storage as you can afford. Then, using your favorite virtualization package, you can simulate having as many computers as you want. You could still learn K8S or whatever you want with less bother.
It’s amazing what you can get these days for a couple hundred dollars.