Mac OS X 10.7 "Lion" is coming, and it's bringing a few storage features to the table

Apple’s not an enterprise company or a storage company. In fact, they’re rapidly jettisoning both – consider the sad fate of the Xserve RAID. But Apple does have enterprise features and storage features in their operating systems. And Mac OS X 10.7 “Lion” is a great case in point. From Versions to Time Machine Local Snapshots to AirDrop, Lion brings some storage love, and iCloud’s Storage API could be game-changing. Let’s look at what’s new and key in terms of storage in the latest version of Mac OS X.

New and Updated Storage Features in Lion

Lion is strong on consumer-oriented features, of course. And Apple is leading the industry in pushing user-friendly storage features for data protection and sharing. OS features like Core Data incremental storage enable Auto Save and Versions, Time Machine gets local snapshots, and FileVault is updated into a whole-disk encryption (WDE) tool.

Update: Probably the most important storage feature, CoreStorage, went un-covered! Read more at Mac OS X Lion Adds CoreStorage, a Volume Manager (Finally!)

Auto Save, Versions, and Resume

AutoSave and Versions leverage advances in HFS+ and a new Core API

Like iOS, Lion enhances the “back where you were” nature of computing with Auto Save, Versions, and system Resume. These completely change the end-user computing experience: Applications don’t have temporal “use once” interfaces but have lasting, historical state. And the ability to move through time (à la Time Machine)

Of course, lots of applications have had auto-save in the past. But Lion adds OS-level interfaces and APIs to enable applications to save data in a standard way. And these will be integrated with Resume (see below) for a very iOS-like experience.

Lion enables all this by enhancing the old, familiar HFS+ filesystem with Core Data incremental storage, a snapshot-like interface to save and recover multiple point-in-time instances of a single document. This is a delta differencing system, probably on a block level, in the filesystem.

Versions are accessed through the title bar document name, as well as a Time Machine-like interface in some applications. Applications can open up multiple versions of the same document at once, and you can cut and paste between them.

Resume is very cool. Applications using the new Lion APIs can save their state, even through reboots! This is what computers should have always done, but no one ever implemented it. Why should a reboot wipe out where you were last? Although not really a storage feature, resume relies on Auto Save and new APIs to store application state.

Time Machine Local Snapshots

Time Machine in Mac OS X "Lion" includes local snapshots as well as storage of backups on external disks

See also Local Snapshots in Mac OS X Lion Time Machine: Is It A Good Idea?

And speaking of Time Machine, it’s been enhanced with Local Snapshots, a mechanism for storing data on the local drive in addition to an external Time Machine drive. The Time Machine interface combines local snapshots and the backup drive or Time Capsule into a single timeline when browsing.

This appears to use Core Data incremental storage as well, though it’s not definite. And it’s unclear how local snapshots will interact with Versions.

AirDrop

See also Snooping on AirDrop in Apple’s Mac OS X 10.7 “Lion”

AirDrop is a new wireless file sharing protocol. Although it smacks of Wi-Fi Direct or Bluetooth, it’s neither. Instead, AirDrop is a newly-developed proprietary mechanism of sharing data between two Macs over a Wi-Fi link. It’ll be speedy and simple but incompatible with the vast majority of systems out there.

FileVault 2

Face it, FileVault was little-used and unfriendly. That’s all changed with FileVault 2 in Lion. It’s encryption done right and might just push average Apple users to protect their data. Kind of like what Time Machine did for backup. That’s a good thing.

What’s new in FileVault 2?

• Full-disk encryption, rather than an image-based hack.
• Encrypt in place, in the background, while you work. No more waiting and losing your computer for hours or days while encryption is installed. And the encryption process will resume even after a reboot (or two)!
• Encrypt external drives, too. Very sweet, especially for Time Machine backups!
• Remote wipe service, probably through iCloud
• XTS-AES 128 support

FileVault 2 is full-disk only, but supports multiple users (each with their own encrypted home area) by storing the full-disk key in each user’s keychain. They access it by using their login password, just like they always did. Apple will offer an option to store the encryption key in iCloud, but this does not appear to be the default condition.

Of course, users with poor passwords, or those vulnerable to social engineering, will still be vulnerable. But some encryption beats no encryption any day! Lion uses the login window for sleep and screen savers, so FileVault 2 protects in those states as well.

NFSv4

Lion adds support for NFS version 4, which is a welcome update. But it doesn’t appear to support version 4.1, or parallel NFS.

SMB with DFS

DFS is a technology in Microsoft Windows that virtualizes file server shares, improving flexibility. Lion’s SMB client supports DFS when connecting to Windows file servers.

Xsan Built In

Mac OS X Server will now be an optional extra to add to the standard Lion install, rather than a separate SKU. Part of this shift is the integration of Xsan, Apple’s OEM version of Quantum’s StorNext SAN file system. Xsan has been quite popular in the media space on client machines, and it appears that it does not require a Server license.

Xsan also gets case-insensitive volumes, which brings it more in line with the expectations of users used to HFS+. Finally, Lion gets ALUA-compatible multipathing, supporting a wide variety of storage arrays.

Probable: TRIM Support for SSDs

Apple enabled TRIM in certain versions of 10.6 “Snow Leopard”, but it only supported their OEM SSDs. Lion probably gets a fully-supported TRIM implementation that works with any SSD with TRIM, though this was not mentioned at WWDC or online. This is a welcome update, and one too-long in coming!

Stephen’s Stance

Mac OS X “Lion” doesn’t give storage folks much, but it could be a harbinger of massive changes. Although not part of Lion per se, the iCloud Storage API will likely see much use by application developers. But even Lion’s smaller storage feature list is welcome. Simple full-disk encryption in FileVault and Core Data incremental storage are promising. Let’s just hope TRIM comes along for the ride!

On the enterprise side of things, Lion is improving as a storage client with NFSv4, DFS, and Xsan built in. But there’s no mention of an iSCSI client, suggesting that project is dead. Thank goodness for Studio Network Solutions! And we’re still stuck with HFS+, suggesting the ZFS transition is off the table, too. Too bad – Versions and FileVault would have been much easier to implement on ZFS!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Key Storage Features in Mac OS X 10.7 “Lion”
This post was categorized as Apple, Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

How granular is your storage? This question is just as relevant today as when I first published this article in 2008!

Many storage challenges focus on correlating high-level uses of data (such as applications) with the nuts and bolts of storage infrastructure. These discussions often revolve around the conflict between data management, which demands an ever-smaller unit of management, and storage management, which benefits most from consolidation. Developing data management capability that is both granular enough for applications and scalable enough for storage is one key to the future of storage.

Storage Management: Scaling Up

As I discussed in a previous Sunday Series piece, Turning the Page on RAID, the data storage industry has traditionally focused on reducing granularity. Disk capacity has expanded, and RAID technology has multiplied this by combining multiple physical drive mechanisms into a single virtual one. Storage virtualization technologies, from the SAN to the server, have also often been touted primarily as a mechanism to reduce heterogeneity. From a technical perspective, therefore, granularity has been an obstacle to overcome.

The core organizational best practice for storage management is the reduction of complexity and the enforcement of standardization. Consolidation of storage arrays and file servers is a common goal, as IT seeks to benefit from economies of scale. The goal of both initiatives is the creation of a storage utility or managed storage service. This mirrors efforts on the server and network sides to consolidate and virtualize hardware.

Although both technological and organizational factors have traditionally driven granularity out of storage, this does not have to be the case. Virtual pools of storage are ideal for providing storage on demand, as disk-focused RAID groups give way to more flexible sub-disk storage arrangements. And an operational focus on standardized storage service offerings has the potential to enable scalable management of these smaller units.

Filing Service

File-based protocols would seem to have more potential for granular storage management, but they have been undermined by the hierarchical nature of modern file storage. Whether the connection to a file server uses NFS, CIFS, or AFP, the key unit of management is actually the shared directory, not the file. All files in the share \\firefly\backups would be located on the same server and would be managed as a unit.

NAS virtualization can change this somewhat, as can more specialized NAS servers. Although Microsoft DFS enables consolidation and virtualization of NAS shares, it does not allow subdivision of shares below the directory level – all files in a directory must be placed on the same server. Tricks like stubbing and links allow for some movement, but these do not solve the core issue. Specialized virtual NAS devices from F5 (the ARX, nee Acopia), NetApp, BlueArc, Symantec, and others have the ability to move files individually, providing as much a virtualized storage environment as any block-focused enterprise array. Avere is also beginning to talk about granular file management.

But even an ideal virtualized file server lacks the kind of granularity demanded by users. They care about data, not files, and most applications consolidate their data storage into a few files. Consider a database, for example, where users want each record treated uniquely but storage devices see just a few much larger files. We need a storage revolution, where someone creates an ideal storage platform in which each individual record or object includes custom metadata and is managed independently. This would truly be a massive change, however, and it is not clear that all applications will follow the object storage model of Google and Amazon.

Small is Beautiful

Barring a revolution in data management, our best hope is to allow greater granularity in storage management. As mentioned above, virtualization technology has the potential to enable management and protection of any unit of storage, right down to the individual block or record. But the reality of storage virtualization has not matched its promise.

What is needed is greater integration. Each layer of virtualization (file system, volume manager, hypervisor, network, array, and RAID) also hides necessary details from lower layers. Consider the case of a virtual server snapshot: The application and filesystem must be in a quiesced state to allow a snapshot to be taken at the storage level, but the storage array has no intrinsic information about how its capacity is used. A given LUN might contain dozens of servers on a shared VMFS volume, so all must be snapped together.

Integration can be enabled by sharing more information through APIs. VMware leverages Microsoft Volume Shadow Copy Service (VSS) integration for shared storage so a VMFS snapshot can call the operating system and even applications (Windows Server 2003 only, for now) to prepare the data. Similarly, VSS can communicate directly with supported iSCSI and Fibre Channel arrays, calling a snapshot at the right moment. And Microsoft is, no doubt, enhancing VSS as we speak.

As virtualization technology matures, expect this type of integration to improve. We hope to see more APIs exposed by VMware and Microsoft, allowing communication up and down the stack to break through the information barrier. Imagine a future where a standard API like VSS can pass a message through VMware, Xen, and Hyper-V to the underlying storage array to initiate a snap. I predict that this kind of integration-enabled granularity is not too far off.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Granularity: The Hidden Challenge of Storage Management
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

So let me sum up:

1. Server Message Block (SMB) 2.0 is probably the biggest news.  Microsoft re-worked this protocol for Vista and 2008 to reduce chattiness, combine multiple commands in a single packet, and allow larger packets.  This should improve performance for just about everyone, but is only in those two operating systems right now.
2. The excellent multi-path I/O (MPIO) driver from Microsoft is also included right out of the box, including device-specific modules for both iSCSI and Fibre Channel.  This is the first version of Windows to come with MPIO, though it’s available with the iSCSI initiator and OEM drivers for other versions.
3. As in Vista, Server 2008 allows you to shrink NTFS filesystems on the fly.  It’s not quite as flexible or forgiving as some third-party tools, but it’s certainly easy to use!  Just go into the disk administrator snap-in and try it for yourself.
4. Windows finally has the option to leave new LUNs alone instead of trying to mount them.  This was one of those things that gave Windows a rep as a bad neighbor in SAN environments.
5. NTBackup is gone, replaced by a new Server Backup MMC snap-in.  I haven’t tried it, but I hope it’s better than the one that comes in Vista!  I ended up keeping NTBackup around on my laptop…
6. A new MMC snap-in called Storage Explorer lets you manage WMI-compliant SAN devices.
7. The Volume Shadow Copy Service (VSS) and DFS interfaces are improved and are now scriptable.
8. Server 2008 now automatically aligns filesystem boundaries with storage, which was one of those dark and secret skills us storage guys used to share amongst ourselves.  This can increase performance in high-I/O environments.
9. NTFS (in both Server 2008 and Vista) now has symbolic link support, just like UNIX and Mac OS X.  Note that this got mangled in the TechTarget article.
10. NTFS was also tuned and tweaked a bit for better stability and crash recovery.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Windows Server 2008 Changes Storage
This post was categorized as Enterprise storage, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Dave called me for this piece, and I was pleased with the question.  Truth be told, there really are compelling benefits from virtualization, but most folks have been waiting for a real “must have” killer application for the technology.  In order for this tech to make the impact it should, we in the industry have to change some of our thinking:

• Storage virtualization means more than just Fibre Channel block aggregation.  There are great applications inside servers and arrays and in the NAS world, too.
• Speaking of NAS, Microsoft DFS is probably the most-implemented storage virtualization product, and just about every NAS array has cool aggregation and migration features.
• Virtualization is a feature, not a product.  HDS has seen the amazing potential for block virtualization in migration and storage flexibility, and this is just the tip of the iceberg.
• Storage and server virtualization go well together – so well, in fact, that ESG reports that 24% of those who have implemented the latter are also using the former!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Storage Virtualization: What Is It Good For?
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.