Mac OS X 10.7 "Lion" is coming, and it's bringing a few storage features to the table

Apple’s not an enterprise company or a storage company. In fact, they’re rapidly jettisoning both – consider the sad fate of the Xserve RAID. But Apple does have enterprise features and storage features in their operating systems. And Mac OS X 10.7 “Lion” is a great case in point. From Versions to Time Machine Local Snapshots to AirDrop, Lion brings some storage love, and iCloud’s Storage API could be game-changing. Let’s look at what’s new and key in terms of storage in the latest version of Mac OS X.

New and Updated Storage Features in Lion

Lion is strong on consumer-oriented features, of course. And Apple is leading the industry in pushing user-friendly storage features for data protection and sharing. OS features like Core Data incremental storage enable Auto Save and Versions, Time Machine gets local snapshots, and FileVault is updated into a whole-disk encryption (WDE) tool.

Update: Probably the most important storage feature, CoreStorage, went un-covered! Read more at Mac OS X Lion Adds CoreStorage, a Volume Manager (Finally!)

Auto Save, Versions, and Resume

AutoSave and Versions leverage advances in HFS+ and a new Core API

Like iOS, Lion enhances the “back where you were” nature of computing with Auto Save, Versions, and system Resume. These completely change the end-user computing experience: Applications don’t have temporal “use once” interfaces but have lasting, historical state. And the ability to move through time (à la Time Machine)

Of course, lots of applications have had auto-save in the past. But Lion adds OS-level interfaces and APIs to enable applications to save data in a standard way. And these will be integrated with Resume (see below) for a very iOS-like experience.

Lion enables all this by enhancing the old, familiar HFS+ filesystem with Core Data incremental storage, a snapshot-like interface to save and recover multiple point-in-time instances of a single document. This is a delta differencing system, probably on a block level, in the filesystem.

Versions are accessed through the title bar document name, as well as a Time Machine-like interface in some applications. Applications can open up multiple versions of the same document at once, and you can cut and paste between them.

Resume is very cool. Applications using the new Lion APIs can save their state, even through reboots! This is what computers should have always done, but no one ever implemented it. Why should a reboot wipe out where you were last? Although not really a storage feature, resume relies on Auto Save and new APIs to store application state.

Time Machine Local Snapshots

Time Machine in Mac OS X "Lion" includes local snapshots as well as storage of backups on external disks

See also Local Snapshots in Mac OS X Lion Time Machine: Is It A Good Idea?

And speaking of Time Machine, it’s been enhanced with Local Snapshots, a mechanism for storing data on the local drive in addition to an external Time Machine drive. The Time Machine interface combines local snapshots and the backup drive or Time Capsule into a single timeline when browsing.

This appears to use Core Data incremental storage as well, though it’s not definite. And it’s unclear how local snapshots will interact with Versions.

AirDrop

See also Snooping on AirDrop in Apple’s Mac OS X 10.7 “Lion”

AirDrop is a new wireless file sharing protocol. Although it smacks of Wi-Fi Direct or Bluetooth, it’s neither. Instead, AirDrop is a newly-developed proprietary mechanism of sharing data between two Macs over a Wi-Fi link. It’ll be speedy and simple but incompatible with the vast majority of systems out there.

FileVault 2

Face it, FileVault was little-used and unfriendly. That’s all changed with FileVault 2 in Lion. It’s encryption done right and might just push average Apple users to protect their data. Kind of like what Time Machine did for backup. That’s a good thing.

What’s new in FileVault 2?

• Full-disk encryption, rather than an image-based hack.
• Encrypt in place, in the background, while you work. No more waiting and losing your computer for hours or days while encryption is installed. And the encryption process will resume even after a reboot (or two)!
• Encrypt external drives, too. Very sweet, especially for Time Machine backups!
• Remote wipe service, probably through iCloud
• XTS-AES 128 support

FileVault 2 is full-disk only, but supports multiple users (each with their own encrypted home area) by storing the full-disk key in each user’s keychain. They access it by using their login password, just like they always did. Apple will offer an option to store the encryption key in iCloud, but this does not appear to be the default condition.

Of course, users with poor passwords, or those vulnerable to social engineering, will still be vulnerable. But some encryption beats no encryption any day! Lion uses the login window for sleep and screen savers, so FileVault 2 protects in those states as well.

NFSv4

Lion adds support for NFS version 4, which is a welcome update. But it doesn’t appear to support version 4.1, or parallel NFS.

SMB with DFS

DFS is a technology in Microsoft Windows that virtualizes file server shares, improving flexibility. Lion’s SMB client supports DFS when connecting to Windows file servers.

Xsan Built In

Mac OS X Server will now be an optional extra to add to the standard Lion install, rather than a separate SKU. Part of this shift is the integration of Xsan, Apple’s OEM version of Quantum’s StorNext SAN file system. Xsan has been quite popular in the media space on client machines, and it appears that it does not require a Server license.

Xsan also gets case-insensitive volumes, which brings it more in line with the expectations of users used to HFS+. Finally, Lion gets ALUA-compatible multipathing, supporting a wide variety of storage arrays.

Probable: TRIM Support for SSDs

Apple enabled TRIM in certain versions of 10.6 “Snow Leopard”, but it only supported their OEM SSDs. Lion probably gets a fully-supported TRIM implementation that works with any SSD with TRIM, though this was not mentioned at WWDC or online. This is a welcome update, and one too-long in coming!

Stephen’s Stance

Mac OS X “Lion” doesn’t give storage folks much, but it could be a harbinger of massive changes. Although not part of Lion per se, the iCloud Storage API will likely see much use by application developers. But even Lion’s smaller storage feature list is welcome. Simple full-disk encryption in FileVault and Core Data incremental storage are promising. Let’s just hope TRIM comes along for the ride!

On the enterprise side of things, Lion is improving as a storage client with NFSv4, DFS, and Xsan built in. But there’s no mention of an iSCSI client, suggesting that project is dead. Thank goodness for Studio Network Solutions! And we’re still stuck with HFS+, suggesting the ZFS transition is off the table, too. Too bad – Versions and FileVault would have been much easier to implement on ZFS!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Key Storage Features in Mac OS X 10.7 “Lion”
This post was categorized as Apple, Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

W. Curtis Preston launched his own series of Backup Central Live! seminars for 2011

Last week, after the Exec Event in Palo Alto, I joined my friend W. Curtis Preston for his first Backup Central Live! event. Curtis has spent years educating IT pros about data protection, this was the first week of a new series of self-produced events. And let me tell you, although I’ve seen him present dozens of times, Curtis was really in his element here. He held the packed room enthralled, and the vendor sponsors I talked to were very pleased about the event!

Introducing Backup Central Live!

The Backup Central Live! series are day-long seminars across the USA in 2011. Each event includes over 3 hours of content from “Mr. Backup”, W. Curtis Preston, as well as presentations from Jacob Farmer and the sponsoring vendors. The seminars are free for qualified end-users, which includes most of the readers of this blog!

Curtis and company will cover the challenges of backing up and recovering data in a variety of settings:

• Virtualized servers (e.g. VMware, Hyper-V, Xen)
• Very large servers and data centers
• Remote offices and laptops
• Data retained for multiple years

The session also includes technical detail about key products and technologies:

• Cloud Backup Services
• Deduplication
• Continuous data protection (CDP) and near-CDP
• Archive software
• Tape and its proper role

Attendees even get free breakfast and lunch, which was of a good hotel caterer quality in my opinion.

Stephen’s Stance

The Backup Central Live! crew does a great job putting together a professional event

I knew Curtis could put together quality backup content, but the crew deserves credit for such a professional and successful event. They attracted some great sponsors, too, including AppAsure, Aptare, FalconStor, NEC, Quantum, Spectra Logic, and Cirtas. And Jacob Farmer’s involvement was a pleasant surprise, too: I’ve always enjoyed the deep technical conversations I’ve had with him!

If you enjoyed my own backup, archiving, and storage seminars in the past, I know you’ll love this event. The next Backup Central Live! cities are as follows. If you’ll be around, you really ought to attend!

• Orlando, FL Feb 1 Register here
• Houston, TX Feb 8 Register here
• Chicago, IL Feb 22 Register here

My only suggestion for the crew is that they get a bigger room next time!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | See W. Curtis Preston’s Backup Central Live!
This post was categorized as Enterprise storage, Gestalt IT, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Overland's SnapServer will soon scale out to hundreds of nodes, thanks to MaxiScale's technology

Overland Storage is showing intriguing signs of life. Once relegated to OEM tape library duty, Overland received an injection of cash and (more importantly) talent this year. Now the company is stepping up the technology behind their SnapServer NAS array by acquiring scale-out file storage company, MaxiScale. They intend to bring the scalable capacity and performance normally associated with enterprise and high-performance computing systems to the mass market.

Overland In A Snap

Snap Appliance and Overland Storage are familiar names in the enterprise storage space, but the two companies have had a convoluted history. Snap was born in 1998 as a product of Meridian Data and acquired by backup giant Quantum one year later for about $85 million. Although successful, the Snap NAS products were not at home among Quantum’s tape-heavy product lines. The division was re-purchased in 2002 as Snap Appliance by private investors, led by current CEO Eric Kelly, for just $11 million. Two years later Snap was again absorbed by a familiar storage name: This time $100 million brought it into Adaptec.

Quantum competitor Overland Storage re-purchased Snap from Adaptec in 2008 for just $3.6 million and brought Kelly back as CEO in 2009. Kelly brought back former Snap (and Data Robotics) marketing guru Jillian Mansolf this year, and they were soon joined by Dr. Geoff Barrall, founder of Data Robotics and BlueArc. Although a public company, Overland (NASDAQ:OVRL) also raised an additional $12 million in private equity financing, an unknown amount of which went to purchase MaxiScale.

You may also want to read The Rebirth of Overland Storage

This brings us to today, with Overland Storage on the rise, with strong OEM and channel sales in tape and disk storage devices. Overland’s product lines include the NEO tape libraries, REO virtual tape devices, SnapSAN block arrays and various SnapServer unified and NAS systems.

The highlight of Overland’s storage offerings is the SnapServer N2000, a unified iSCSI and NAS array solidly placed in the entry- to mid-market. Like many competitors, Overland relies on a single- or dual-controller storage “head” based on industry-standard CPU technology and scales with additional disk “shelves”. Unusually, Overland’s N2000 line currently relies on a single SAS connection daisy-chained to its E2000 expansion units. One hopes dual connections and faster connectivity are coming soon.

MaxiScale: Scale-Out NAS

Venture-funded startup MaxiScale developed a scale-out NAS capability, allowing multiple NAS servers to be consolidated into a unified namespace without the usual bottlenecks of clustered solutions. The company headed for the cloud storage market, boasting massive scalability and management benefits.

MaxiScale’s technology differentiator was in their rejection of conventional clustering and scale-out approaches. Many scale-out solutions rely on shared-nothing coordinated nodes, but it is often difficult to scale and manage these solutions. This has caused much effort to build cluster coordination through metadata, exemplified by SAN filesystems (including EMC’s MPFS and Quantum’s StorNext) as well as clustered NAS solutions like pNFS.

This MaxiScale illustration shows their distribution of metadata and content as well as their unified namespace

MaxiScale’s “FLEX” technology takes an entirely different approach. It distributes “peer sets” of data across multiple nodes like pNFS but also distributes metadata rather than relying on a single central node. Combining this with a unified namespace, MaxiScale claims to scale to thousands of nodes with linear performance. The company had targeted cloud service providers interested in massive scale using commodity hardware.

Overland + MaxiScale = SnapScale?

Overland appears to be bringing MaxiScale back down to earth, leveraging their technology to combine a set of in-datacenter SnapServers into a unified, high-performance cluster. Although a number of other companies claim scale-out capabilities (Isilon and HP X9000 spring to mind), this is a unique capability in the SMB and midrange NAS market, and Overland will surely make hay as a low-priced alternative.

The company could quickly implement MaxiScale’s software on their existing industry-standard hardware and push a line of “SnapScale” appliances. But things will get more interesting once they begin integrating the technology into their GuardianOS software. Imagine a scale-out solution that enables a company to grow from a single sub-$10,000 NAS to a vast cluster of thousands of nodes with linear performance gains. Even being able to add a half-dozen NAS heads would be a welcome alternative to Dell, HP, and NetApp offerings. In the longer term, Overland could use this technology to break from the head-and-expansion architecture they currently employ, shifting to a cloud-like distributed architecture.

Stephen’s Stance

Overland is making all the right moves. Eric Kelly should be commended for taking bold action to re-energize the company with an infusion of money, talent, and technology. The MaxiScale technology is impressive, and adding it to Snap for a mid-market offering is potentially game-changing.

There are lots of SMB and midrange NAS systems, and most are fairly similar in terms of technology. Overland fell in that group as well until now, but the addition of MaxiScale is a real differentiator for the company. Mansolf and Barrall have proven that they can take out-of-the-ordinary technology and market it to the masses, and this acquisition gives Overland another chance to execute. I will be watching their progress with great interest!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Overland Snaps Up MaxiScale to Scale Up Snap
This post was categorized as Computer History, Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Capacity increases exponentially based on the formula for the area of a disc: π times radius squared
• Sequential performance increases algebraically based on the formula for the circumference of a disc: π times diameter

Therefore, sequential performance grows smoothly with disk density, but capacity increases much faster. Double the density of disk media and you can read twice as many bits in the same amount of time, but the disk now contains four times as much data. Iterate this a dozen times, a miracle performed regularly by hard disk drive manufacturers, and you have a serious bottleneck to both performance and reliability.

Disk capacity has outpaced performance over the last decade

Back in 2004, I gave this metric a name: Flush time. It is a simple calculation to answer the question, how long would it take to read the entire content of a hard disk drive? Let’s look at some real-world examples:

• In 2000, a 45 GB Western Digital 450AA disk could stream data at 25.4 MB/s, requiring 30 minutes to flush every byte out its UDMA/66 interface. This was a massive and slow drive at the time – enterprise disks were much faster. A 2000 Quantum Atlas 10K II SCSI drive (36 GB and 31 MB/s) could flush in 19 minutes!
• A 2004-era Seagate Barracuda 7200.7 boasted 160 GB ad averaged 44.5 MB/s, requiring about an hour for a full flush.
• By 2007, high-performance drives like the Hitachi 15K450 had hit 450 GB and about 100 MB/s in sustained throughput, but flush times were well over an hour.
• Today’s enterprise drives can push 200 MB/s and average 160 MB/s across the entire 600 GB of capacity. But this is still about an hour for a flush. But large-capacity SATA drives are much more popular for bulk storage. The Samsung Spinpoint F2 EcoGreen drive I use in my Drobo only delivers about 110 MB/s, requiring almost four hours to flush at 1.5 TB of capacity! Think this is unusual? Check out Hitachi’s popular E7K1000, which needs 2.5 hours at 1 TB and 118 MB/s.

What will happen to flush time over the next half decade if density continues to increase? How about 16 TB drives, 400 MB/s, and RAID rebuilds that last more than half a day?

Since (traditional) RAID rebuilds are directly impacted by flush time, today’s massive disk drives are killing RAID. And flush time is only the minimum required time – most RAID rebuilds take much longer! Then there is the issue of media reliability!

Note: Yes, I know there are alternative RAID schemes that get around this problem. Far from ignoring that point, I’ll be promoting these in future posts! Stay tuned for more on these topics…

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Flush Time
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Frankenweenie saves young Victor in Tim Burton's macabre short film

When is a solution integrated and when is it a Frankenstein-like mashup of tangled tech? Apparently, that line is crossed when it’s your competitor’s offering…

In my time in the storage industry, I’ve seen enough franken-storage come and go to make me skeptical whenever a new “integrated” solution is announced. But a lot of this stuff works just fine, so I also know that integrated solutions aren’t always bad!

The latest industry blog flame war centers around NetApp’s recently-announced solid state storage solution, which pairs a V-Series NAS head and a Texas Memory Systems RamSan-500 flash storage system. Perhaps NetApp’s Val Bercovici did get a bit over-excited in his post on the topic, but he wasn’t just talking about the RamSan: He was laying out how NetApp’s WAFL technology can work in an SSD world, and using some recent performance test numbers on that solution as well as their PAM cache cards as an illustration of this.

The next thing you know, we have EMC’s Storagezilla and IBM’s Barry Whyte calling the company out for what they (and others. like Storagebod) see as an underwhelming product offering. That’s all well and good, and I’ll let the reader decide if NetApp’s moves warranted a press release, but now things have gotten uglier…

EMC’s Chuck Hollis called the whole RamSan idea to account, saying it was “Frankenstorage“, causing NetApp’s Alex MacDonald to engage in a little “I know you are but what am I” in reference to EMC’s CLARiiON/Celerra “unified storage” solutions.

It’s time to bring some sanity to this whole integrated solution concept. Every product in the storage world is an amalgamation of OEM parts to one extent or another, and there are always integration issues. Certainly many of EMC’s offerings could be the subject of name-calling: They use STEC SSD drives in the DMX, they use Quantum deduplication engines in their CDLs, and their Celerra NS platform does include a complete Fibre Channel SAN behind the curtain. But they’re not alone, and not even wrong in doing this: Every vendor relies on OEMs, and as a wise man said, “working with an OEM gives you the flexibility to pick best of breed technologies” and that’s exactly what customers want. Any objective person would welcome qualification and integration of TMS’ RamSan with a solid platform like the NetApp V-Series – it’s a certifiable win for the customer. Just like they would be happy to see EMC leveraging great technology from Quantum and STEC.

Chuck goes on to point out some downsides to these OEM combinations, and they’re certainly fair criticisms:

• When you’re buying this from this guy and that from that guy, it’s bound to cost more because everyone needs their cut.
• Since all attempts at unified heterogeneous device management have failed, a combo is certainly harder to manage than a single device.
• With multiple vendors in the mix, fingerpointing is common once support is needed.

But these criticisms can be mitigated by the vendors themselves. They can give up some margin in order to gain market share. They can create unified management interfaces for the combinations they sell and support. And they can really support what they sell, refusing to give in to the temptation to say “not my problem” when the going gets rough. And companies deal with these problems all the time! Frankenstorage doesn’t have to be so scary…

This post can also be found on Gestalt IT: The Difference Between “Integration” and “Frankenstein”

© sfoskett for Stephen Foskett, Pack Rat, 2009. | The Difference Between “Integration” and “Frankenstein”
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Apparently, some were annoyed, offended, or hurt by this post. For that, I sincerely apologize. My intent was to reach out to my friends at the company in this difficult time, but what I meant doesn’t help those who are directly affected.

I have only one question: What is the right thing to do when bad things happen? Should one offer help or remain silent? This is not a rhetorical question – I’d like some advice.

Although nothing official has yet been announced, enterprise storage giant EMC is apparently conducting a massive workforce reduction today. Update: EMC reported in January that they would take a restructuring charge in Q4 2008 and would reduce overall headcount by 2400. The job losses reported here were part of this restructuring. I hear that there is a voluntary early retirement program, too.  Multiple sources have confirmed to me that perhaps as much at 10% of the folks from Hopkinton will be let go. Update 2: The Boston Globe reports EMC’s claims that only “a small number of workers” has been let go. The Register reports that the numbers are “small and modest.” I personally know of a handful who were affected, and they claimed widely divergent losses in their departments, so I have no idea what the number was.

Update: I’m hearing conflicting reports from inside the company, with some absolutely confirming the RIF and others scratching their heads. Might be divisional rather than company-wide, but a good number of people both in Hopkinton and elsewhere are apparently losing their jobs. A number of others are confirmed to have been let go over the past three weeks.

This follows last week’s news that Quantum would also let 180 employees go, and Pillar and Overland also reduced their respective workforces this quarter.

There are also rumors that one of my favorite storage resources (which will remain nameless for now) will not last into 2009.

I’m really speechless about all of this. There are many, many fine people involved at these companies, and I wish all of them the best.

If you are affected, please drop me a line. I can’t offer you a job, but I promise to do everything I can to help you land on your feet.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | EMC Cuts Staff as Recession Continues
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Overall, I was impressed with many of these products this last year, but certainly CommVault, HDS, ProStor, Xsigo, and Akorri (winners all) stood out with important products. I too was surprised to see the absence of some heavyweights: Symantec’s strength in archiving and backup didn’t net them an award, and both Emulex and QLogic were skipped for the 8 Gb FC market. EMC was overlooked, too, but I’m certain that will be remedied for 2008 after the flurry of excellent new products announced these last few months.

Backup Software

• Winners: CommVault’s Simpana 7.0 – an impressive product that brings a very wide range of data protection features in an integrated package. They’d be in my top three! Next was the intriguing RecoverGuard from Continuity Software, a product which intrigues me. Finally, Yosemite’s amazingly priced FileKeeper Professional took bronze – I definitely would have put this up there, too!
• Strong contenders: EVault’s solution would have been my pick since it effectively tackles a real need – mobile and SMB data protection! Also, Symantec’s strong NetBackup 6.5 upgrade deserved more attention.

Backup Hardware

• Winners: NEC’s HydraStor HS8 isn’t a platform I’m familiar with – I’ll have to take the judges at their word! But I can’t argue with silver and bronze winners, Copan and ProStor’s cool RDX – both are truly revolutionary products and deserve the spotlight.
• Strong contenders: I would definitely have added Gresham’s Clareti VTL to the list – it’s much more than “just a VTL” and I hope it gets some more attention!

Disk and Disk Subsystems

• Winners: Seagate’s Momentus 5400 FDE.2 deserves considerable attention, bringing built-in hardware encryption to the mobile data market – every mobile disk should have this technology! NetApp’s FAS2000 (silver) is nice enough, but I was much more impressed with the bronze-winning USP V from HDS this year.
• Strong contenders: I would have given the USP V the top prize for 2007, but I can’t argue with the little Seagate disk. The AMCC 3ware 9600 RAID adapter deserves some attention, too.

Storage Management Software

• Winners: Akorri’s BalancePoint (gold) impresses me much like bronze-winning Onaro’s Application Insight does. Both make the big leap from “storage” to “application data”, and both are worthy winners. I’ve never used the silver-winning Finisar NetWisdom product, but it looks like others are impressed with it.
• Strong contenders: This was a crowded field, and Symantec, WysDM, Quantum StorNext, and the rest probably came close. I agree with the Akorri placing, but would have picked one of these instead of the potentially mis-categorized Finisar product.

Networking Equipment

• Winners: Xsigo took the top honors with their InfiniBand-based VP780 platform. This is truly a next-generation product, and it is getting serious attention and traction, and deserved a spot on the list! Riverbed’s excellent Optimization System (silver) also deserved its ranking, but I’m not familiar enough with the Storwize product to know if it’s truly bronze-worthy.
• Strong contenders: I would have tipped either QLogic or Emulex‘s 8 Gb offerings for a spot. Despite my jokes, 8 Gb FC is an important element of the modern SAN and both companies have carved out a compelling product, but apparently neither shipped in volume until this month…

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Reacting to TechTarget’s Storage Products of the Year
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.