Thin provisioning needs communication to function, and zero page reclaim is only the array side of the story. WRITE_SAME helps reduce I/O load, but the server needs to use it. Wouldn’t it be nice if the operating system, file system, or volume manager would use these commands to help recover capacity?

We need a bridge. We need to cross the gulf between the storage array (that expects all these zeros or some other way to recover this space), and the host (who has this information)

That’s the thing that gets me riled up. The application and the file system know that this data is no longer used. If there was only some way, some bridge, that allowed them to talk to the storage array and say, “Hey, you can get rid of that now. I don’t need it anymore.” See? We need a bridge.

Well, guess what? We have some bridges. One of my favorite developments in storage of the last year is this Thin Reclamation API that Symantec has. I really like what they’re doing.

Symantec (formerly Veritas) have the file system and the volume manager. If you’re an enterprise guy like me (I was using HP-UX in 1996) you have used the Veritas File System and Veritas Volume Manager before.

Symantec calls all this Veritas Storage Foundation now. It has the information thin provisioning storage arrays need, and they’ve come up with a way to communicate: Being a software company, Symantec implemented just about every communication method, because not everybody uses this WRITE_SAME, and not everybody uses zero page reclaim. Some of them have their own API for recovery. So Symantec tried to implement as many thin methods as they possibly could right in the file system and volume manager.

I’ve talked to a lot of people who are using it, and they say that it works. They say that it’s brilliant. Because you turn on this feature and, suddenly, the file system starts telling the array “I don’t need this data anymore.” Suddenly one of my major complaints about thin provisioning goes away.

But I have a question: Why doesn’t everybody do this? Why isn’t this in the Linux file systems and volume managers? Why isn’t Microsoft working on this? Why doesn’t everybody have this capability? It’s not rocket science. It just takes a little bit of development effort to try to come up with some kind of way to have the file systems talk.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Bridge: Veritas Thin (Provisioning) API
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Western Digital is first to bring "Advanced Format" 4k-sector drives to market, but buyers should beware of installing them in old systems!

Personal computer hard disk drive access methods have repeatedly been forced to adapt to ever-expanding capacity. Physical (CHS) addressing and 28-bit LBA have been abandoned, but the disk drives themselves retain the tiny 512-byte sectors they have had since IBM introduced fixed-block addressing in the 1970′s. But time marches on, and Western Digital is leading the change to larger 4 kilobyte hard disk blocks. Although this new “Advanced Format” includes mechanisms for backwards compatibility, buyers should be wary of these new drives for the time being.

A Little History

A long, long time ago, before there were personal computers, IBM set the standard for data storage with their CKD (“count key device”) format. Variable-sized records were stored on a raw disk with gaps between them. Each included a sequence number (“count”), an optional key, and the data itself.

This flexible format proved impractical as multi-process systems evolved in the 1970′s, so IBM introduced a new format: Fixed-block architecture (FBA). FBA disk drives were divided up into equal-sized blocks and addressed by their physical location on a specific cylinder accessed by a specific head. Each cylinder/head pairing had its own series of sectors, starting at the outer edge of the physical disk platter and moving inward.

This system of addressing hard disk capacity with a triple number identifying the cylinder, head, and sector (“CHS”) continued in the PC space in the 1980′s. Since early hard disk drives were tiny by modern standards, the PC industry chose the smallest sector size specified by IBM, 512 bytes. The earliest MS-DOS/BIOS PCs were limited to just 504 MB, thanks to incompatible defaults for the CHS address space. This forced a transition, initially raising the addressable capacity to 7.8 GB.

But the writing was on the wall: CHS just didn’t make sense. Storage protocols, including ATA, were being pressed into service for devices that didn’t have cylinders or heads, and the 128 GB limit of ATA-1 wouldn’t cut it forever. So the industry transitioned again, abandoning physical references in favor of a simple logical block addressing (LBA) scheme. By the time ATA-6 was introduced in 2002, all systems used LBA, and the new 48-bit address system allowed a maximum of 128 PB of capacity per device.

Though hard disk capacity had ballooned from the megabyte to the terabyte range (a million-times increase in 30 years), the 512-byte disk sector remained, each with its own error-correcting code (ECC) and gap.

Up To Date

Processors and file systems have long since moved on from 512-byte blocks of data, with 4 kilobytes being the most common size used by the x86 CPU and NTFS, ext3/4, and HFS+ filesystems in Windows, Linux, and Mac OS X. With just about every storage access triggering the reading or writing of eight disk sectors, an increase in hard disk sector size to 4 KB seems obvious. Stepping up to 4K enables better ECC checksums, making disks more reliable (in theory) and freeing up capacity.

Modern operating systems, including Windows Vista, Windows 2008, Windows 7, Mac OS X 10.4-10.6, VMware ESX 3.x-4.x, and most recent versions of Linux, will have no problem with 4 KB sectors in most cases. Installing a 4K drive, like Western Digital’s new “R” line, and partitioning and formatting it fresh should present no issues for most users. In fact, these operating systems will not even “know” they are talking to any special kind of drive, since Western Digital presents its new 4 KB sectors as plain old 512 byte sectors for compatibility reasons.

What’s Your Alignment?

Other systems present a challenge, however. MS-DOS traditionally started filesystems at sector number 63, and most other PC operating systems (including old versions of Linux and VMware) followed this convention. This presented no issue at all as long as disks used 512-byte sectors: Requesting 8 sectors starting at number 63 was the same as requesting eight starting at sector 64. But things get mucked up when the disk drive uses 4 KB sectors.

See the misalignment here? A starting offset of 63 moves the filesystem cluster across the 4K sector boundary. This misalignment means that every filesystem cluster access results in two disk sector accesses. This is a Very Bad Thing, especially for to write performance. Layers upon layers of convention, standards, and compatibility have left us with a problem.

Western Digital solves this problem in one of two ways:

1. Jumper pins 7 and 8 on the drive and it will internally offset all of its logical 512-byte sectors by one
2. Correctly align all partitions written to these new drives

The jumper method, though simple, is not a great idea. If the drive was ever moved or reformatted to a system that aligns its volumes differently (as do all modern operating systems), it would be puzzlingly slow. Instead, it’s best to simply realign any filesystems written to the drives, and Western Digital is providing a utility do just that.

USB, Drobo, and TiVo, Oh My!

Most external drive enclosures will simply pass read and write requests to these new drives as they get them. This means that an external USB enclosure with an Advanced Format drive will have exactly the same issues as an internal drive if it is partitioned under Windows XP or another old operating system. Not only that, but portable drives often get moved from system to system, so an XP-formatted USB disk will have this misalignment issue even when attached to a Windows 7 machine! As these 4K drives begin hitting the market, users should be careful to note if they are correctly aligned.

But what about other disk-using devices like Drobos and TiVos? The jury is still out here. TiVo appears to use misaligned partitions, so I don’t recommend installing 4K drives in a TiVo at this point. I contacted a number of other manufacturers, most of whom are “looking into” the matter.

Data Robotics did respond, and even posted a tech note about 4K drives on their support site. Their short answer is no, that 4K drives should not be used in Drobo storage devices. However, it is unclear if they are affected or just erring on the side of caution until they can test these new hard disks. They also promise an update allowing the use of 4K drives in the future.

I would be cautious about using 4K drives in anything other than a newish PC or Mac at this point. I will continue following the situation and will post updates on my blog as news comes out.

More Info

The following articles also offer great insight into these new 4K drives:

• Western Digital: Advanced Format Technology White Paper
• StorageMojo: Why we need 4k drives
• AnandTech: Western Digital’s Advanced Format: The 4K Sector Transition Begins

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Drobo, XP Users: Beware of 4K “Advanced Format” Drives!
This post was categorized as Apple, Computer History, Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

File Descriptors

Anyone familiar with its internals will tell you that everything is a file to a UNIX operating system. It’s the philosophy behind the system: Network connections, storage systems, system parameters, and processes all have file interfaces, and each of these pseudo-files needs a unique file descriptor.

What does this mean for lighttpd? Well, every time a visitor accesses a page, lighttpd uses three file descriptors: An IP socket to the client, a fastCGI process socket, and a filehandle for the document accessed. Lighttpd stops accepting new connections when 90% of the available sockets are in use, restarting again when usage has fallen to 80%. With the default setting of 1024 file descriptors, lighttpd can handle a maximum of 307 connections. This is a lot. But it is possible to exceed this number under times of high load.

To prevent this from happening, we can double the limit without any trouble. Simply set “server.max-fds” to 2048 in /etc/lighttpd/lighttpd.conf.

Contrary to much of the advice I found on the Internet, lighttpd spawned by root does not appear to use the “nofile” limits set in /etc/security/limits.conf, since these are for PAM and only apply to full interactive logins. There is a system-wide limit that can be set in /etc/sysctl.conf, however. Check your default with “cat /proc/sys/fs/file-max” and make sure it’s over 10,000. Mine was set to 12640 so I left that alone.

HTTP Keep-Alive

One reason that file descriptors get used up so quickly is HTTP keep-alive. To improve performance, modern web servers keep client connections alive to handle multiple requests instead of building up and tearing down connections for each item in a page. Keep-alive is tremendously beneficial to performance, but tends to keep unnecessary connections alive, too. By default, lighttpd allows 16 keep-alive requests per connection, allows idle sessions to remain alive for 5 seconds, and gives reads and writes 1 minute and 6 minutes to complete, respectively.

Although lighttpd has pretty aggressive defaults (especially compared to Apache), a period of heavy traffic and a few slow clients could see many unused connections sticking around. The server.max-keep-alive-idle setting default of 5 seconds can be reduced to as low as 2, if you assume your clients are reasonably quick about requesting data, but a value of 3 or 4 is probably realistic. You may want to increase the server.max-keep-alive-requests value from the default of 16, but you probably don’t need to. The server.max-read-idle and server.max-write-idle settings are tempting targets, but these situations are usually fairly rare so let’s not monkey with them.

Mechanics: Polling and Sending

The best bang for your lighttpd buck is to tune the server to use better kernel resources to check for file changes and write data to the network. There are three critical items here, each of which is set to a conservative universal setting by default.

One of the major areas of UNIX development over the last decade was how to handle the tens of thousands of connections experienced by Internet servers. This “C10K Problem” is documented in excruciating detail if you’re interested, but the net of it is that each version of UNIX has an advanced mechanism to handle I/O events. Since kernel version 2.6, Linux has sys_epoll, a so-called edge-triggered polling mechanism which scales linearly with the number of connections. But lighttpd runs on many different flavors of UNIX, so it has to default to the older and less-scalable “level-triggered” poll system. To remedy this, set “server.event-handler” to “linux-sysepoll”.

Another mechanism that varies widely across UNIX systems is how to actually read and write data from the disk to the network. All systems include basic read() and write() calls, which transfer data into and out of system memory. Lighttpd defaults to using these to move data around. But Linux includes a more advanced call, sendfile, which can move data around without copying it into memory. We can enable this by setting “server.network-backend” to “linux-sendfile”, which ought to improve performance for larger (multi-megabyte) files without impacting smaller ones.

Lighttpd attempts to improve performance further by caching the output of the UNIX stat() command. It includes a basic (“simple”) cache which keeps the result of file system calls in memory for one second. But many Linux distributions include more advanced accelerators: FAM was the original, and a lighter-weight workalike called Gamin is now included by default in Ubuntu’s lighttpd install. Therefore, we can improve stat calls simply by allowing lighttpd to use Gamin: Set “server.stat-cache-engine” to “fam” and you’re rolling!

One more useful tweak to consider, although it’s not included in the official lighttpd performance document, is not updating the “atime” parameter on served pages. This is a bit of a religious issue among some UNIX administrators, but I feel safe in saying that since my web server logs all accesses and I’m not using any kind of hierarchical storage system to store them, I don’t care when each php, html, and png file was last accessed. We can stop writing atime values by mounting the entire filesystem with “noatime”, but I like the more granular approach offered by lighttpd: Simply set “server.use-noatime” to “enable” and it won’t bother keeping this updated for the files it accesses. Everything else will continue as it always has but with reduced disk I/O.

Summary

Lighttpd has pretty good default settings, but a few might be tweaked if we need to respond to higher server loads. The more important area of tuning is simply enabling the advanced features of the 2.6.x Linux kernel and Ubuntu system we are using: Enable sys_epoll, sendfile, and Gamin and disable atime updates.

I’ll post more information as I stumble across it. I’m still learning, but my server performance as improved dramatically: Pingdom tools reports that it used to take upwards of half a minute to load my blog’s home page and it now loads in under seven seconds! That’s progress!

# Maximum number of file descriptors, default = 1024
server.max-fds = 2048
# Maximum number of request within a keep-alive session before the server terminates the connection, default = 16
server.max-keep-alive-requests = 16
# Maximum number of seconds until an idling keep-alive connection is dropped, default = 5
server.max-keep-alive-idle = 4
# Maximum number of seconds until a waiting, non keep-alive read times out and closes the connection, default = 60
server.max-read-idle = 60
# Maximum number of seconds until a waiting write call times out and closes the connection, default = 360
server.max-write-idle = 360
# Which event handler to use, default = poll
server.event-handler = "linux-sysepoll"
# How to handle network writes, default = writev
server.network-backend = "linux-sendfile"
# Requires FAM or Gamin to be installed, default = simple
server.stat-cache-engine = "fam"
# Whether to update the atime setting on file access, default = disable
server.use-noatime = "enable"

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Tuning Lighttpd For Linux
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I had switched from Dreamhost to Slicehost back in February to improve reliability and performance, but the meagre 256 MB of RAM in my virtual private server (VPS) “slice” proved insufficient. The time had come to completely redo my core hosting infrastructure. After some experimentation, I have settled on a simple two-server configuration based on Ubuntu Linux, MySQL, and lighttpd. I thought it would be a good idea to document this new configuration, as well as my previous experiments, for posterity.

If you’re interested in setting up a web site as outlined here, I recommend Slicehost. They’re not the cheapest, but their VPS servers are fast and reliable.

History

My web hosting environment has transitioned over the past six months. I had relied on shared hosting from Dreamhost for almost a decade, with my servers sharing infrastructure and management with thousands of others. This worked fine until I began to see significant traffic increases since creating this blog. Shared hosting just didn’t cut it once I had more than a few thousand pageviews per day.

I tried Dreamhost’s interesting and very flexible virtual private server capabilities, but could never get them working reliably. Plus, the core networking and storage performance of the Dreamhost infrastructure left something to be desired. After much research I switched to Slicehost, an all-VPS provider that has recently been acquired by Rackspace. Although they are not the cheapest or most flexible, Slicehost is a very professional service with good support and excellent infrastructure and connectivity.

I had been using a single 256 MB slice to host my entire site, and had managed to get everything well with lighttpd and MySQL, but this configuration ran into serious performance issues once traffic built again. Once I passed 10,000 pageviews per day, which happened quicker than I hoped, it was again time to upgrade.

Server Configuration

My core question was whether to go with a single 512MB or two 256 MB slices. Would resource contention in a single server be outweighed by the extra available RAM? After consulting with the experts, I decided that it was time to separate the database and web servers.

A multi-server setup delivers performance, reliability, and future capability.

As illustrated above, the database and web servers communicate via a high-speed private network, a standard Slicehost component. I created extremely restrictive iptables firewall policies to control access to both servers, disabling almost all communication. The database server in particular is inaccessible except from the web server, and even then only allows MySQL and public key ssh. Both servers are running bare-bones versions of Ubuntu 9.04.

MySQL Configuration

MySQL is much happier on its own dedicated server. It makes excellent use of its own query cache and operating system buffers and has very little disk access at all. Query performance immediately jumped, and site performance noticeably improved.

My only change to my.cnf was to enable the query cache. This works great on my 256 MB slice:

# Enable query cache
query_cache_limit       = 2M
query_cache_size        = 32M

I’m not a believer in security through obscurity. Sure, you can use a special high MySQL port, but by then you’ve probably lost anyway. Regardless, set up iptables to only allow access from your web server’s private interface using the -s parameter:

# Allow connections on our MySQL port
-A INPUT -p tcp --dport mysql -s 10.176.x.x -j ACCEPT

It’s a pretty simple configuration, but it works well. MySQL is humming along without much paging at all, using the entirety of the 256 MB available for caching. I haven’t had to resort to any tricks to keep the performance up.

One more configuration suggestion: Use a unique username and password for each database application and grant access only to that user on the private network interface. That way, your risk is segmented to a single database should an intruder use a SQL injection or something similar.

For example, say you were configuring the database “myblog_wp” for the user “me_myblog” with the password “123456abcdef”. You use the web server’s private interface address, 10.176.x.x as well so it is harder to get in even if your firewall is breached.

mysql -u root -p
Enter password:
mysql> grant all on myblog_wp.* to 'me_myblog'@'10.176.x.x' identified by '123456abcdef';

All set!

Basic Lighttpd Configuration

Back on the web server, we configure iptables to only allow connections to lighttpd on port 80 and ssh on whatever port you decide. Again, some suggest using a high port for ssh, but if you’ve configured sshd correctly then using an obscure port number is more likely to be a hassle for you than keep anyone out. Here are some good settings for /etc/ssh/sshd_config:

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
X11Forwarding no
UsePAM no
UseDNS no

This forces all logins to use established public/private key pairs rather than passwords. Protect your keys and you’re in good shape. Key passphrases are a very good idea here! You’re also disallowing root in any case, since you’re using sudo for all administrative tasks, right?

Next we need to set up lighttpd. I’m using the basic packaged versions for ease of maintenance rather than building my own. I also loaded the stock versions of php5 and mysql-client which integrate nicely. I used to use eaccelerator but didn’t love having to recompile it myself and had some mysterious lockups with it running. So I went with XCache, which is developed by the same fine folks who created lighttpd.

sudo aptitude install mysql-client lighttpd php5-xcache

Why not use Apache? I actually tested Apache with my site before deciding to use lighttpd. I was able to tweak it to run in 256 MB of RAM by limiting the number of worker processes created to 4, but Apache just couldn’t handle the site load. Connections were stacking up and pageviews dropped as people gave up. Lighttpd with XCache is blindingly fast and fits in my 256 MB RAM envelope nicely.

The only real issue I have with lighttpd is that configuration is entirely different from Apache and remains less well-supported. It doesn’t use with .htaccess files, for example, and rewrites use a unique syntax.

Lighttpd for WordPress

Here’s a basic virtual server for /etc/lighttpd/lighttpd.conf for a WordPress domain:

$HTTP["host"] =~ "^(blog\.)?yourdomain\.com$" {
  server.name = "yourdomain.com"
  server.document-root = basedir + server.name + "/blog"
  url.access-deny = ( "wp-config.php" )
  dir-listing.activate = "disable"
  url.rewrite-final = (
    # Exclude some directories from rewriting
    "^/(wp-admin|wp-includes|wp-content)/(.*)" => "$0",
    # Uncomment to exclude Google FriendConnect files
    # "^/(canvas.html|rpc_relay.html)" => "$0",
    # Exclude .php, robots.txt, favicon.*, and sitemap.xml files at root from rewriting
    "^/(.*.php|sitemap.xml|robots.txt|favicon.*)" => "$0",
    # Handle WordPress permalinks and feeds
    "^/(.*)$" => "/index.php/$1"
  )
  accesslog.filename = basedir + "/" + server.name + "/log/blog.access.log"
}

This configuration accomplishes many of the important tasks of WordPress configuration. Most importantly, it works!

1. Permalinks and feeds are correctly redirected so no ugly index.php is used
2. Critical directories like wp-admin and files like robots.txt still work
3. Your wp-config.php file is explicitly protected
4. Directory listings are disabled
5. A site-specific access log is used (like Apache)

I’d love feedback on this configuration! One weird thing I’ve not yet figured out is how to use a site-specific error log. I haven’t configured pretty MediaWiki links in lighttpd yet, but imagine a similar configuration would work there.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Setting Up a Multi-Server Web Hosting Environment
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

One of the great things about being a Microsoft MVP is the access I get to Microsoft software and staff. As I mentioned in my post about the 10 cool storage features from the 2009 Microsoft MVP Summit, I was able to preview a lot of what Microsoft is doing with their Server software and storage features. And the best part is that the Microsoft product teams are keenly interested in our feedback and suggestions. I’m told, for example, that the awesome iSCSI Quick Connect feature in the new Windows iSCSI initiator software was developed based on my feedback!

As I note on my Windows Storage Server 2008 preview on Gestalt IT, Microsoft has always kept WSS close to the vest. It’s only available to OEMs, not retail customers, and has never even been shared with TechNet or MSDN subscribers in the past. So I was really pleased when Microsoft gave the File System Storage MVPs access to a beta version of WSS 2008 so we could get a feel for all of the new features. I’ve also had some great conversations this week with the Microsoft product managers responsible for it.

What’s exciting about Windows Storage Server 2008?

1. It includes all of the storage enhancements in Windows Server 2008, including SMB 2.0 for much much faster file servicing over higher-latency links, SMfS, FSRM, enhanced VDS, and failover clustering.
2. WSS is the only way to get access to Microsoft’s iSCSI target software. It’s been improved in many ways from the prior releases, but its support for what Microsoft calls dual-active clustering is probably its most notable feature: You can’t share the same active LUN between cluster members, but each can have its own active LUNs and the can all fail over in the event that one member goes down.
3. The included single-instance storage (SIS) file-based deduplication has been much improved, scaling to 128 volumes per server and millions of files. It’s still not as effective capacity-wise as block-level deduplication (which I’d love to see, hint hint), but the performance is solid enough to use it for primary storage with production applications.
4. Probably the coolest feature exclusive to Windows Storage Server 2008 is its new browser-based remote administration capability. Just point your browser to the Storage Server machine (for example, “http://wss/desktop”) and you’ll get a full ActiveX version of RDP. Don’t use Internet Exploder? Firefox and Linux users will get a Java-based RDP instead! I will cover this feature more in the future, but let’s just say that every operating system should offer this!

Want to try Windows Storage Server for yourself? Breaking from the past, Microsoft will soon (like next week!) allow TechNet subscribers to download the full install. OEMs have a sekrit back-door site to try it out, too.

One more thing… This will be the last release of Windows Storage Server. There won’t even be a special Storage Server version of Server 2008 Service Pack 2! Starting now, Storage Server is just an optional feature of Windows Server. Purchasing and production use will still be limited to storage OEMs, but Microsoft has finally reconciled Storage Server with the rest of the Windows Server world. I imagine that most OEMs will release Service Pack 2 updates for their Storage Server customers shortly, and that future versions of the product will come closer to the base Server versions than WSS 2008. Although I can’t share what I know, I will say that Microsoft is continuing active development on their iSCSI target, single-instance storage, and other Storage Server features. I imagine that Windows Server 2008 R2 will support storage systems in the very near future!

On a personal note, reading A Brief History of Windows Storage Server Releases from the Microsoft Storage Server blog reminded me of the original unveiling of Windows Storage Server at Storage Decisions Chicago in June, 2003. The company loaded us all on one of those lake cruise boats with some celebrity impersonators, chocolate “Oscar”-style statues, an open bar, and a band. Good times ensued!

For more details, check out my Gestalt IT piece, Windows Storage Server-Based Systems Step Into 2008

Microsoft is detailing the new version of Windows Storage Server 2008 in a webcast Thursday at 8 AM Pacific. You should also check out the Microsoft Storage Server blog.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | I Can Finally Talk About Windows Storage Server 2008!
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Iomega has been a staple of the desktop computing environment for decades, but the company’s products have never been quite at home in even small corporate data centers. That changes today with the introduction of the iSCSI StorCenter Pro ix4-200r. As of now, EMC’s SOHO storage subsidiary is a serious challenger in the small business and entry-level VMware ESX storage market.

It might look like the existing NAS 200rL, but the ix4-200r sports upgraded hardware and a new rev of EMC’s LifeLine storage software. This unit packs a serious punch, boasting full iSCSI target support for servers running Windows or Linux (or anything else with an iSCSI initiator) in addition to NFS, SMB, media streaming, print services, and just about every other protocol.

Although both Iomega and VMware are under the EMC corporate umbrella, it was a surprise to find that the ix4-200r is certified compatible with ESX using both iSCSI and NFS right out of the gate. This is the only inexpensive storage system to wear a VMware badge, and this alone will likely make it a fixture in small offices and VMware labs. The desktop StorCenter ix4-100 and StorCenter ix2 are already widely used in these environments even without iSCSI, after all. The ix4-200r provides a complete SAN-in-a-box, supporting multiple NAS and iSCSI shares with dynamic allocation of the internal RAID-5 protected storage.

Although aimed at the office, the ix4-200r retains the vast set of LifeLine capability we’ve seen in Iomega’s other offerings. This includes media streaming for UPnP (Twonky) and iTunes (Firefly), remote access, Active Directory support, and print services. The new unit even packs the more unusual Axis video surveillance capture capability. It sports two USB ports on the back and one on the front for expansion, data import, backup, or printers as well. Probably the best software feature is EMC’s Retrospect backup client, which was recently updated on the Mac platform.

The ix4-200r starts at just $1,799 (list) for 2 TB, and I expect resellers to dip well below that number. For comparison, Amazon currently sells the smaller non-iSCSI desktop 2 TB ix4-100 for $675 and the 1 TB ix2 for $268 and I’ve seen each for much less. I expect a street price of $1600 for the 2 TB rackmount unit – competing products from Buffalo and Netgear are priced and marked down similarly. The 4 TB model is priced $1,000 higher, perhaps unrealistically high given that the only difference is the use of 1 TB hard drive units instead of the 2 TB’s 500 GB drives. For comparison, Drobo just introduced their limited single-server 8-bay iSCSI DroboPro at $1,750 configured with four 500 GB drives. But none of these alternatives boast a spot on the ESX compatibility list, and I suspect this may be a deciding factor for many. Note that you can’t buy less than four hard drives in an ix4-200r, though the drives are easy to replace.

Iomega was kind enough to give me a preview of the ix4-200r at their offices, and I came away impressed by the new array and the company in general. They have a solid vision of the needs of the small office and are hard at work on products to meet them. Although the iSCSI support is not coming to the company’s other LifeLine-powered systems (the ix2, ix4-100, and Home Media) at this point, I would not be at all surprised to see it become a staple in future networked storage systems. A large gap remains below the EMC CLARiiON range, so I suspect that larger Iomega systems are on the way as well. As a potential buyer, I’d like to see Windows logo qualification, and Hyper-V support would be super as well. And as a Mac user, I’d love to see Time Machine support and for Iomega follow Drobo by offering a free iSCSI initiator – a guy can dream, right?

Updates and clarifications:

• Iomega has added the StorCenter Pro ix4-200r to their web site alongside the non-LifeLine StorCenterPro 200rL
• The ix4-200r will not be released until April 22, 2009
• The new rackmount ix4-200r is listed at $1799.99 for 2 TB and $2799.99 for 4 TB. I don’t expect to see either sell for less than a few hundred off those list prices
• The ix4-200r has been listed in the VMware ESX compatibility guide for a few days now for both iSCSI and NFS connectivity – I’m surprised no one noticed!
• Although it’s not mentioned in the press release, Iomega tells me that the StorCenter Pro ix4-200r does still support the BlueTooth file exchange found on its little brothers

More coverage:

• EMC’s StorageZilla posted his impressions as well: Iomega adds iSCSI, threatens war on us all
• Carlo Costanzo is excited to use this in VMware environments: EMC’s Low Cost SAN Starter for VMware (Iomega)
• Chris Mellor gives it a UK spin in The Register: Iomega opens sub-£2k box of storage tricks
• Duncan Epping is also excited about Home Lab Storage

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Iomega Grows Up and Moves Out of the House
This post was categorized as Enterprise storage, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Homes now need data storage as well as closets...

This is part of an ongoing series of longer articles I will be posting every Sunday as part of an experiment in offering more in-depth content.

Along with my professional focus on enterprise storage systems, I’m enamored of home networking, and recently passed the three terabyte mark at home! This got me thinking about where home storage is heading.

As you can see in the photo, my office closet is overflowing with computer equipment (and one sweet guitar), but my data storage is much better organized. I have a hacked Linksys NSLU2 with 500 GB as a file server, a 500 GB PC backup disk, a 160 GB Time Machine disk, 1 TB of TiVo storage, and the rest. But wouldn’t it be nice if this could all be combined into some kind of super home server?

Past Failures: Home Servers

Home storage appliances and servers have come and gone over the year, with none seeming to make much of a mark. The market remains littered with UPNP media servers and home NAS boxes dashed on the shoals of an unappreciative public. Nearly every home network device company has produced one or two home storage servers, none of which have succeeded. Although I use a Linksys NSLU2 at home, I had to hack its Linux software and completely replace Linksys’ features to create a useful device! The un-hacked NAS devices of Buffalo, Western Digital, Netgear, and the rest have generally failed to find buyers as well. So far, consumers seem content with simple USB and FireWire external drives.

The most adventurous home storage servers came from Zetera and Ximeta, both of whom relied on proprietary IP SAN protocols. Note that these were SAN products, sharing block storage over Ethernet, rather than conventional NAS solutions. Both required drivers, limiting client support. The one Zetera buyer I know was pleased by the performance but never used the device as anything but a large hard drive for one PC.

Then there is Microsoft. Recall that the latest Windows Home Server is only their latest attempt to enter this market, and yet I know of no one who has adopted the device. The same can be said of the various media center servers from Microsoft and others. At this point, it seems likely that the future of home storage servers will not come from Microsoft, though their two XBox generations have great potential as clients.

Even EMC has entered the market with their nifty (but largely unnoticed) LifeLine product and Iomega acquisition. Supporting file services and backup for computers as well as audio and video for media players, EMC positions LifeLine much like their Retrospect backup product, but goes further in offering a complete software solution for hardware OEMs wanting to offer a non-Windows home server. Although an impressive offering, it is too early to tell if EMC will have much success with this product.

The Sleek, Shiny Elephant in the Living Room

Of course, there is one company that sells media players and servers by the bushel, complete with sleek, shiny interfaces. Apple’s tremendous success with the iPod has led to their iTunes software becoming the dominant media organization platform, complete with its own proprietary discovery and sharing protocol. Now, with the Apple TV and video iPods, the company is broadening into more media categories. Surely their dominance here puts them in a special position when it comes to setting the stage for a home server or storage revolution.

They also have a strong position in the world of dedicated home storage. Their Airport products are among the only routers to be widely implemented with shared storage. Although many other companies offer similar products, low customer understanding means that these functions are not widely used. And the new Time Capsule device is surely already the most widely-used home NAS product.

But Apple has not yet shown any home server strategy. Administering multiple iTunes servers can be frustrating for users, with no inter-iTunes synchronization or centralization capability. Although the Mac Mini, Apple TV, or Time Capsule could certainly be seen as a home server, the company does not position them as such in the market. Indeed, some iTunes users like myself rely on compatible third party media servers like Firefly and TwonkyVision rather than using iTunes itself. Still, rumors of an Apple home server persist.

One issue for Apple is their reliance on proprietary protocols. Although the Bonjour discovery protocol is certainly simpler than UPnP in practice, Apple stands alone in relying on it. They also steadfastly stick to AFP for NAS and DAAP for remote media streaming. This limits the number of third-party clients and servers that can be used with their hardware and software.

The Future is Friendly

Although Apple has not yet tipped a home storage strategy beyond Time Capsule and Airport Extreme, they are best positioned to deliver a real home storage solution. A simple step would be to create an iTunes media server integrated with Time Capsule and add client/server media synchronization. The company already has OS X backup and file services integrated, and this move would further centralize the digital home around Apple products. But the company’s reliance on closed protocols like DAAP is worrisome, since it locks consumers into nearly all-Apple solutions.

Microsoft’s Media Center and Home Server combination, based around UPnP, shows great promise, with many compatible third-party clients and servers already available. But my own experience with the solution has not been at all positive (I still can’t get my Roku SoundBridge, Vista Ultimate laptop, and Media Center PC to see each other!), leading me to question the viability of this option.

Although Apple or Microsoft could come to dominate, I suspect the future of home storage is out of both companies hands. A number of others are working on improved home server experiences, including EMC’s LifeLine and the expanding use of Debian Linux and open source tools. But all could be sidelined by improved Internet-based services. Google, Microsoft, and Apple continue to expand their online consumer suites with greater storage, synchronization, multimedia integration, and all have the potential to reduce or eliminate the need for in-home storage.

Although I cannot yet tell which service will win, one thing is certain: Consumers demand friendly, flexible solutions. They don’t want to fuss with their media, and they don’t want simple shared storage. They want integration with multiple devices and flexibility to access their content on any device. The first company to offer a simple, flexible storage server for the home will surely be on the right track!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | The Future of Home Storage
This post was categorized as Apple, Computer History, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

If you haven’t already tried VirtualBox, you ought to. It works very well, virtualizing Windows and Linux guests on Windows, Linux, Solaris, and Mac hosts with respectable performance. One thing I really love about it is that it doesn’t bog down my Vista system like VMware with installed drivers and services. It just installs and works and gets out of the way when you’re not using it.

VirtualBox supports shared folders, USB, and has guest additions for Windows and Linux to provide resolution independence and performance boosts. The new 2.0 version adds 64-bit guest OS support for 64-bit hosts along with performance and stability fixes. The Mac version now has a native look and feel and better networking, too.

One really interesting twist for folks interested in desktop virtualization (aka VDI on VMware) is integration with remote desktop protocol (RDP). VirtualBox guests can be configured to act as RDP servers, with thin(ish) clients accessing them over a network and even sharing their USB devices seamlessly. I’m getting pretty excited about the desktop virtualization concept – I’ll be keeping my eyes on companies like stealthy Old Road Computing Virtual Computer to see what they’re up to!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Sun’s Excellent VirtualBox Goes 2.0
This post was categorized as Apple, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

See, I’ve had my work ripped off by big content providers repeatedly over the years, but never could put into words why it bothered me so much.  I mean, it’s not like I was profiting from that snapshot or HOWTO document, so why should I care if someone else does?

But once you juxtapose the attitude of those same companies about my fair use rights of legally-purchased content, the grain that’s been chafing me becomes clear.  Media companies seem to think it’s ok for them to steal from the little guys (either by choice or through a lack of rigor, but they’ll come after me if I try to do anything at all with their content.  The article’s anecdote of a stolen dog photo used in a sports broadcast surrounded by ominous copyright warnings really sums up the whole situation!

The first time I experienced this was back in 1997 or so, when I received an email from a fellow Linux hippie alerting me that a recent Linux book contained a number of online texts, mine included.  The book, Linux File Systems, was “written” by Moshe Bar, better known for his work on OpenMosix and Xen, but also known as a writer for a number of other books and a columnist for latter-days Byte.  I confronted him about his wholesale copying of my LVM walkthrough in his book, and he apologized, claimed he’d run out of time (which is easy to believe, looking at the book), and blamed his editor.  The publisher, McGraw Hill, claimed the book wasn’t selling well anyway so they couldn’t offer me any compensation.  Being young(er) and foolish(er) at the time, I let it drop.

Over my later years of writing columns and articles for Storage Magazine, InfoStor, and others, I became aware of wholesale unauthorized translation and reprinting of English magazines in other languages.  A number of my articles were published in Russian magazines, for example.  Again, I did nothing but chuckle about seeing my name in Cyrilic.

So why the picture of the Saab 900 above?  Because just last month, I was notified that that exact photo was used by the German paper, Süddeutsche Zeitung, (Nov 23th 2007, page 11) without permission and in violation of the license.  This has happened to a number of my other Wikimedia Commons photos, with appearances in a number of papers and magazines that I know of, and probably more besides…

What’s to do?  I suppose I should have pushed harder when these uses were brought to my attention.  I suppose I could have banded together with others to protest.  But I did nothing.  What would you do?

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Living in a Copyrighted World
This post was categorized as Computer History, Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Notice that all of my PC’s run windows, while all of my servers run Linux! This got me wondering what role Linux plays in enterprise storage. Sure, Linux has a huge role to play on the computing side of the equation. But which enterprise storage devices are based on a Linux kernel?

Xiotech made a big splash a few years ago by announcing that they would switch from a proprietary operating system to Linux. I remember seeing Open-E’s Linux based iSCSI software somewhere, and hearing that Snap Appliance (now part Adaptec) of was using it as well. I consulted LinuxDevices.com and found out about Infrant (now part of NetGear), MaXXan (nee CipherMax), and Raidtec.

There have got to be more! So tell me, who is using Linux as their embedded kernel and why? Was it for convenience, hardware support, or perhaps a financial decision?

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Where is Linux in Storage?
This post was categorized as Computer History, Enterprise storage, Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.