Update: Check out the latest multi-vendor iSCSI post!
I usually don’t write about other peoples’ articles on this blog, preferring to stick to my own independent work. But this time I’m making an exception.
If you use or are interested in VMware ESX 3.x and iSCSI, you simply must go read Chad Sakac’s post on the topic. Co-written with just about everyone in the industry (including EMC, VMware, NetApp, Dell/EqualLogic, and HP/Lefthand), Chad has put together a “cheat sheet” on the ins and outs of iSCSI connectivity and performance in ESX 3.x.
Top takeaways (and I’ve been preaching about these for a while myself, too!)
- Ethernet link aggregation doesn’t buy you anything in iSCSI environments
- iSCSI HBA’s don’t buy you much other than boot-from-SAN in ESX, either
- The most common configuration (ESX software iSCSI) is limited to about 160 MB/s per iSCSI target over one-gigabit Ethernet, but that’s probably fine for most applications
- Adding multiple iSCSI targets adds performance across the board, but configurations vary by array
- Maximum per-target performance comes from guest-side software iSCSI, which can make use of multiple Ethernet links to push each array as fast as it can go
Thanks for putting together such a great article, guys!
This post can also be found on Gestalt IT: Essential Reading for VMware ESX iSCSI Users!
Thanks Steve – it was hard to make it multivendor, and involved some compromises – but nothing fatal, and I think the compromises were right to make it multi-vendor.
Two corrections (I’m being an nit/engineer about this 🙂
– the 4th bullet is the key one.
– on link aggr , I think you’ve distilled it a bit too far… it can help, just less than people would naturally expect…. You still need to use something to ensure outbound use of multiple links (something I don’t think we did well enough in the post – for a followup) – this is either via link aggregation (but you still need multiple iSCSI targets, and likely will also use MPIO config)
Can anyone verify this comment from Equallogic? I don't remember seeing anywhere that you'll get performance problems by having a stanby NIC… strange
Please Note: To avoid severe performance degradation, please use the IP or MAC hash option with Multiple Physical trunked/channel group (non-stacked) switches. Having any „stand-by‟ physical NICs in your vSwitch can also result in severe performance degradation.
Can anyone verify this comment from Equallogic? I don’t remember seeing anywhere that you’ll get performance problems by having a stanby NIC… strange
Please Note: To avoid severe performance degradation, please use the IP or MAC hash option with Multiple Physical trunked/channel group (non-stacked) switches. Having any „stand-by‟ physical NICs in your vSwitch can also result in severe performance degradation.
Has anything changed about this since vsphere was released or is this article still up to date?
Yes, the folks have put together an updated post! See it here:
http://blogs.netapp.com/virtualstorageguy/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html
Yes, the folks have put together an updated post! See it here:
http://blogs.netapp.com/virtualstorageguy/2009/…
Very good posting. I just love it.
Good luck man with your work.
http://webroyalty.com
A step-by-step guide for configuring HA iSCSI SAN with CDP for vmware ESX/ ESXi Server.
http://www.kernsafe.com/article_product.aspx?id=5&aid=49
“The most common configuration (ESX software iSCSI) is limited to about 160 MB/s per iSCSI target over one-gigabit Ethernet”
…What??? I think not. One Gbps equals 1,073,741,824 BITs, not BYTEs. MB/s is the abbreviation for MegaBYTEs per second. If you take 1,073,741,824/8 you get 134,217,728 Bytes, or other words 128MB/s. However, there is IP over head, iSCSI over head, etc. The best real world performance you might hope for is 90 MB/s, and that’s assuming everything is completely optimal. There is not a snow balls chance in hell you are ever going to see 160MB/s, or anything close, over a 1000 Base-T link, period.
Now if you want to say he meant 160Mb/s, I would hope that was wrong, as 160Mb/s is only 20MB/s, which is SUPER slow. Though that is certainly in the neighborhood of the performance I’m seeing, so that might be the case. But I would have to wonder why it would only be able to saturate 22% of the available bandwidth over a dedicated gigabit connection. (Running direct cable from ESXi to Solaris ZFS.)