My FreeNAS Build: Supermicro X10SL7, Intel Haswell Xeon, ECC RAM

In my last post, I talked about my decision to build a FreeNAS box. Today I’m going to dive into the hardware I selected, starting with the motherboard, CPU, and memory. FreeNAS runs on any PC hardware, but building a reliable and scalable storage solution means picking higher-end components. I selected a Supermicro X10SL7 server-class motherboard with 14 (!) SAS/SATA ports paired with an Intel Xeon E3-1231v3 (Haswell) CPU and ECC memory from Crucial.

2017 Update: About 6 months after building this system, I bought the “other” 16 GB of RAM, not because I needed it now but because I wanted to get it while I could still buy identical sticks. At least now I have it. I’ve also upgraded the hard drives to 8 TB WD Red NAS drives, which are really sweet. The SuperMicro board and Xeon CPU are bulletproof, though I did swap in a CoolerMaster Hyper 212 EVO CPU fan to keep it cool. I’ve been disappointed by the SuperMicro IPMI, especially the useless remote console, but what can you do?

FreeNAS Hardware Decisions

I began my FreeNAS build by asking myself a few questions:

1. How much money do I really want to spend?
2. How much storage capacity do I need?
3. What physical form will my FreeNAS server take?
4. How important is the power/performance trade-off?
5. Do I want to re-use any existing hardware?

Rather than geeking out and going for the best hardware I could buy, I let the answers to these questions guide my quest. When presented by options, I leaned toward more power, capacity, and capability since I want my FreeNAS server to last many years. But I didn’t let my nerd-greed drive me to stupid spending.

I set a hard spending limit at $1,350 since that’s how much a FreeNAS Mini XL costs. That device offers just about everything I could want but it’s really quite a lot more than I wanted to spend on this project. But if I couldn’t beat it on price, why bother?

Ideally, I wanted to keep the price under $800 since that’s how much a nice Synology or QNAP box would cost. They don’t offer the capacity or performance my home-built FreeNAS will reach, but they’re integrated and supported and available.

I’m happy to report that I achieved this goal: Without drives, my FreeNAS build is easily under $800 including shipping and tax!

It’s All About the Storage!

Let’s start with capacity. I have a 500 GB iTunes library (music and movies), another 500 GB of photos, and about 400 GB of other archived application data and system backups. All this currently resides on my Drobo and is backed up onto a pair of identical external hard drives using rsync scripts. I also have a massive library (about 15 TB) of HD video from Tech Field Day which I want to protect. This is currently stored on another set of external hard drives, with the most-recent year or two on my two Drobos for easier access. I’d like to store this on reliable checksummed online storage for posterity. And all this is growing all the time. Since I got the Leica Q camera, I’m storing 40 MB DMG (raw) images by the dozens, and the music and movie library keeps growing. Then there’s the Tech Field Day archive, which adds 100-200 GB per event, or well over 2 TB per year.

I’ll want a system that supports 8 or more hard disk drives: I’ll eventually have five or six 5 TB hard disks the FreeNAS as an archive. Using RAID-Z2 for dual-disk parity, this will give 15-20 TB of archive capacity. I also have a pair of enterprise-grade SAS SSD’s given to me by SanDisk, so I’ll use those for active data. And I’ll start with a few spare 2 TB disks I have lying around.

This presented a problem. There are very few enclosures that can handle that many hard disk drives, and fewer still motherboards with that kind of connectivity. I had thought I would need to start with a PCIe HBA to get that many storage ports, but discovered two intriguing alternatives:

• ASRock Rack makes a pair of motherboards with 12 SATA ports based on Intel’s storage server “Avoton” Atom CPUs. The 8-core C2750D4I sells for about $450 while the 4-core C2550D4I is about $300. Both are Mini-ITX boards with very low power draw, but that’s quite a lot of money for an Atom!
• Supermicro’s X10 motherboards are very popular in the FreeNAS community, and one offering, the X10SL7, includes an integrated LSI SAS controller, boosting storage connectivity from 6 to a whopping 14 ports! But this is a Micro-ATX board, meaning it needs a larger enclosure.

Another option would be to buy a motherboard with less storage connectivity and add a PCIe storage HBA. Supermicro’s AOC-SAS2LP-MV8 8-lane SAS card is a popular option and sells for a bit over $100. Since SAS HBA’s can support SAS or SATA drives, this would allow just about any motherboard with a PCIe x8 slot to meet my needs.

What Form Factor and Case?

This left me with a related question: Which enclosure is right for my FreeNAS build? If I went with the Mini-ITX ASRock Rack motherboard I could use a mini case like the cool U-NAS NSC-800. But reviewers aren’t kind to that case in terms of workability and cooling. It’s hard to get 8 hard drives and a motherboard to work well in such close confines!

I turned instead to the idea of using hot-swap drive racks (like this Rosewill cage) in a conventional case. Every three 5.25 inch bays could securely house four 3.5 inch hard disk drives, so a case with six 5.25 bays would do. But these are few and far between, with most cases having just one or two full-size bays and up to six smaller ones.

I ultimately decided to look for a conventional tower case, meaning I could pick any ATX or smaller motherboard for my build. With Mini-ITX no longer a requirement, the ASRock Rack motherboard lost its allure. Why spend that much money on a tiny Atom board when I could have a real Xeon system?

Power and Performance

I love the idea of low-power systems, and this is even more important for a 24×7 system like a FreeNAS build. That’s one thing I liked about the Atom – it draws under 25 Watts! But hard disk drives need lots of power to spin up, and an active RAID system will have them powered on quite often. My NAS needs a big power supply anyway, so I might as well go with a more powerful CPU.

The X10SL7 board has an LGA 1150 socket, so I could pick any Haswell or Broadwell CPU. I also considered the newer X11 motherboards, which have LGA 1151 sockets for Skylake. I even considered repurposing my old Ivy Bridge Core i5 desktop CPU since LGA 1155 motherboards are still available.

I did some research on CPUs and ended up choosing a Haswell Xeon:

• Skylake Xeon CPUs and motherboards are still pretty expensive and don’t seem to offer much benefit over Broadwell or Haswell
• My old Core i5 would be good, but it’s pretty old tech and I couldn’t find a motherboard I liked
• Broadwell Xeons are surprisingly rare, so I was left with Haswell as the ideal compromise

I considered using a desktop chip, but the Xeon was much more compelling for a server build. A Xeon E3 costs about the same as a Core i5 but offers more. I wanted at least 4 cores and liked the idea of hyper-threading and Intel’s Turbo Boost 2.0 but didn’t want integrated graphics or other desktop features. At about $250 retail, the Xeon E3-1231 v3 is just about perfect, and easily $50 cheaper than a comparable Core i7.

Memory, Storage, Etc

Not using my old Core i5 meant my old RAM wouldn’t be useful either. The reason I retired that old system was a series of unexplained stability problems I blamed on either the motherboard or power supply, so none of that would make its way into my FreeNAS server.

It is critical to use the right memory with these motherboards, and the FreeNAS site strongly recommended a particular Crucial set which is identical to a Micron DIMM on the supported list. I decided to stick with 16 GB (two 8 GB DIMMs) at this point, but since the motherboard has four DIMM slots I can always double this in the future. FreeNAS does eat up RAM, though, especially if you turn on de-duplication!

Although SAS drives are over-kill for FreeNAS, I wanted to leverage the pair of SanDisk Lightning SSDs I had. As long as I had a SAS controller, they were in. I decided to use them for storage instead of as a large L2ARC cache, as I’ll discuss in the next post.

My old hard drives would make the jump as well. Being a storage nerd, I’ve got quite an assortment of 2.5 and 3.5 inch SATA drives. I decided to repurpose the 2 TB drives left over from my old Drobo along with a set of 5 TB drives I had in use for a while. I’ll probably add more over time. Since FreeNAS uses ZFS, it’s best to keep drives together in pools by size, so I’ll have my SSD pool, my 2 TB drive pool, and a 5 TB drive pool.

This left the case. I could have continued using the same 1990’s-era case but fell in love with the NZXT H440 while shopping for drive cages. I’ll write more about that later!

The Question of Money

With all this research done, I began considering how much all this would cost. Surprisingly, all this impressive high-end gear was much cheaper than a Synology, let alone a FreeNAS Mini!

Here’s the cost, approximated since prices vary all the time:

• Supermicro MBD-X10SL7-F-O motherboard: $200
• Intel Xeon E3-1231 v3 CPU: $250
• Crucial CT2KIT102472BD160B 2x 8 GB ECC memory: $90
• Corsair RM850x power supply: $110
• NZXT H440 tower case: $100

Coming in under $800, including miscellaneous cables, this is really an impressive system. It will absolutely scream as a FreeNAS server and can be pressed into other service in the future if needed. I doubt there’s much I couldn’t do with such a system!

Stephen’s Stance

My Supermicro/Xeon system is perfect for FreeNAS, with plenty of power, expansion possibilities, and a solid reputation. The fact that the whole bundle cost less than most integrated NAS offerings all while offering much, much more power is just icing on the cake! And I haven’t even mentioned Supermicro’s enterprise features like IPMI…

Here’s my entire FreeNAS series so far:

1. Hello FreeNAS! Goodbye Drobo and Iomega…
2. My FreeNAS Build: Supermicro X10SL7, Intel Haswell Xeon, ECC RAM
3. 14 Drives For 14 Ports: A Case For FreeNAS
4. It’s Fine To Mount Hard Drives On Their Side Or Even Upside-Down
5. Migrating Data With ZFS Send and Receive
6. FreeNAS First Impressions

Note: I’ve linked to Amazon for these products and used my Associates ID so I will make some money if you click through and buy something. I don’t really make money this way (I have a real job) but it’s nice to offset the cost of gear this way. I bought everything from NewEgg rather than Amazon, but their associates program isn’t as good.