Deletion of data is not a high priority for most IT shops, but this ought to change

When was the last time you deleted data? Even at home, where we have autonomy and authority over our own data, many of us are digital pack rats. But at work? Never! No one ever deletes anything! Let’s talk about why this is.

Retention vs. Deletion

Just about everything we do in IT infrastructure is focused on retention. We back up our data and implement other data protection tools like snapshots and mirrors. We might also archive data so that the General Counsel can place legal hold on it, as well as perform data discovery during litigation. And then there’s the whole field of data security, focused on locking people out of data, keeping it intact and un-viewed.

But what about deletion? Almost no effort is put towards removing data, though the rapid growth of storage might lead one to think this is a key area for IT. We certainly could put some effort on revision control, and especially deleting drafts and outdated data. We could easily expire content that was no longer needed, if only we had some way to know that. And we’ve talked a lot about secure deletion, even though we hardly ever actually perform that task except when moving to new physical storage hardware.

The greatest challenge for deletion is a simple question: What should we delete and when?

IT can not answer these questions. They must be put to the business people who really own the data. Without permission and buy-in, IT is in serious legal peril when it comes to deleting data: Any deletion must be in accordance with policy and must be legal, that is there is no legal or regulatory hold on it. And there is no way most IT staff feel empowered to do that!

Some Data Should Be Deleted

Certainly, not all data should be saved. There is “low-hanging fruit” in every storage estate that can and should be deleted:

• Ephemeral copies – Drafts, temporary data, working copies
• Time-limited projects – Third-party or client data, test and development
• Expired data – Retention policies that are expired and no legal hold remains
• Legally required – Data that isn’t yours, or that legal demands deleted

Tackling these data sets is much easier to tackle than cleaning out primary data stores, since it doesn’t require as much sifting and sorting: These data sets can often be identified programmatically! If you have data sets like these, this is the ideal place to start a deletion effort.

Delete on Demand

Regardless of the type, however, IT should not delete data without direction. It is perilous in today’s legal environment to destroy data without a policy directing that action. So we should continue to focus on retention for most data, while we work with legal to determine which data can be deleted and come up with a process for approval.

But it’s important to start offer a deletion-friendly environment for certain data types. Such a storage system would reduce the difficulties associated with data deletion. Really, only an integrated solution can truly delete data:

• It must maintain custody of data from start to end and not allow it to leak all over the organization
• It must be accessible since any restrictions tempt users to create “working copies”, thus thwarting deletion
• It must be secure – Data must always be encrypted to avoid remnants on media
• It must be protected so data will not spread to external systems and sites

Stephen’s Stance

Data deletion is a real problem for most IT shops. I’m just getting my head around the ramifications, and continue to look for an ideal deletion-friendly storage solution.

If you’re interested in the topic of data deletion, I recommend joining me for a webinar on the topic on Wednesday, April 13. Sponsored by Nasuni, I will discuss the dilemma of deletion and CEO Andres Rodriguez will weigh in about the capabilities of his cloud storage solution. Register now!

Note: Nasuni is sponsoring this webinar, but the content was created by me. This blog post is intended to engage my audience in discussion of the subject, and is not a paid promotion or advertisement.

Image credit: “Delete” by blmurch

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Deletion Dilemma
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

First, consider whether to store records as atoms or bits. You can convert paper to data or vice versa, and there are pros and cons to both:

• Properly handled physical (paper or film) records should last for hundreds of years and can remain readable without software or devices. But they’re hard to search (you need an index), and paper is bulky, heavy, and difficult to work with.
• Digital records can either be stored offline or kept “alive,” but questions remain about their long-term reliability and readability. Living records can be easy to search and use, and digital storage can be very space-efficient, but data tends to pile up “out of sight.”

Long-term storage of records on physical media is proven – think about papyrus, tablets, gold or nickel discs, film, and paper. But will digital media fare as well? Data tapes and disks can degrade over time, and manufacturer reliability specs are based on accelerated testing, not actual experience. Regardless of media type, careful handling can extend media life.

But will you still be able to read it? Tapes and optical disks require additional hardware to read, while disk drives are paired with their read heads. Software applications are needed to read and interpret data (backup, archiving, compression, encryption, deduplication, database) as well. What about content format? Should you use ASCII, XML, PDF/A?

I’ll be presenting a webinar on this topic tomorrow, Wednesday, December 3, at 2:00 PM Eastern time. Register on-line at the AIIM web site and join me for the discussion!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Long-Term Versus Longer-Term Archiving
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Webcast: Automating Policy With Email Archiving Technology
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• How to improve the receptivity of the e-mail archieve message to management to overcome the biases you mentioned? Consider what management is interested in most: They don’t necessarily care about email archiving, but they will respond to well-presented solutions to business objectives like cost avoidance for litigation and e-discovery or future reduced spending on IT infrastructure (storage).
• How do you get things moving when Legal is sponsor and can demonstrate that one lawsuit discovery failure will be the same cost as the project? This would be an ideal scenereo. Legal probably already understands the high cost of e-discovery and potential risk of adverse judgement. Now demonstrate the ways that email archiving can help. But be realistic – an email archiving app will help with e-discovery, but is definitely not a complete solution. You will still need policy and process for hold and discovery, and may even need additional e-discovery tools.
• Have you seen standard or default retention periods for email, and how are they determined? I would not want to suggest standard or “rule of thumb” retention periods. In my experience with many organizations, I have seen such a wide variety of carefully considered retention schedules that I have come to believe that there is no such thing as a standard. However, I would like to suggest not making a too-short maximum (say, less than three months) or else you will drive end users to underground archiving to get around the policy. Such a short retention period can work for certain classes of messages (company picnic, lunch invites, mass mailings), but I can’t imagine a business where no one would need to keep messages longer. Try to be realistic in setting your retention, balancing productivity with compliance. Setting retention periods is an exercise in consensus building, with representatives from IT, Legal, HR, Compliance, and others each contributing their own needs. It’s doable, though – I do it for a living!
• How many years back do people hold on to mail typically? Most organizations lack standards or policies for retention, so their typical retention varies based on employee actions. When they do have retention policies for email, they tend to classify messages (by organization or user, keywords, or through user action) into a few “buckets” with appropriate retention for each. Most have short retention (3 to 9 months) for some content, and longer retention (on the order of a few years) for other types. Most also have a “keep forever” category for special record types.
• Do you have an opinion on whether journaling is better or not for meeting legal requirements? Most legal groups desire the most complete set of messages possible. Both journaling and log shipping should meet this need.
• Have you seen much use for email archiving for supporting Audit to prove that an event was reviewed and approved? Many organizations use email as part of their workflow. In these cases, maintaining a complete set of messages is critical for audit and compliance. Many certainly do use email archives for this purpose, while others prefer to route such approvals to other content management systems to maintain these records.
• Is there a legal argument to be made that getting rid of pst files and having a central archive will help with privacy law issues? Decentralized personal archives like PST files are a privacy nightmare waiting to happen – they are highly mobile, can be copied and viewed easily, and tend to “live” on theft-prone devices like laptops and portable drives. Replacing PST files with a centralized archive has many many benefits, and the reduced risk of privacy breaches is certainly among these. However, most mail clients still maintain offline caches of mail server (and archive) data, so simply replacing PST with an archive will not elimate the privacy risk.
• You might want to mention that accelerating volumes of stored email are not necessarily bad because it is electronic and therefore searchable. Offline stuff dumped into file servers with no meta data is actually more of a nightmare because no one knows who created it or why it is needed (or not). Adding structure to electronic stored information is always valuable, and email archives certainly do make that data much more organized and searchable, along with enabling retention policies to be applied (assuming they exist!) Unstructured data on file servers is indeed a nightmare, and can pose a serious risk. There are other tools to tackle this type of data, however, and they might do a better job than relying on an email archive as a primary repository of critical records.
• Can you elaborate on user training needs? I was under the impression that changes to the end user were minimal. The amount of impact an email archiving system has on end users can vary from minimal to significant, based mostly on how much user interaction is desired for classification. However, even the most unobtrusive system is likely to change certain aspects of user behavior, from PST files to searches to stubbing and deletion of messages. Therefore, training is probably needed with every email archiving implementation.
• Litigation holds can last for YEARS !! Tape is DEAD especially to try to rebuild mail boxes. Tape remains non-dead for certain applications like backup, despite years of prognostication. However, you are correct that e-discovery of data on tape is incredibly difficult and time-consuming for most systems. There are tools to help, but an online archive is vastly more flexible and speedy than recovering backup data from tape. I’ve seen backup-sourced e-discovery costs rise into the millions of dollars, too!
• How important is classification in an archive solution? The importance of classification depends on the business objectives that an archive is meant to serve. However, most implementations will require at least some message classification mechanism in order to determine how long different messages should be retained, since a one-size-fits-all retention schedule is unlikely to meet anyone’s needs. Therefore, it is important to begin developing an overall record retention policy for email when implementing an email archive, and to consider how messages will be classified. Classification of email messages is a complicated topic – worthy of its own webinar at least!
• We are a globally dispersed organization which increases the complexity and cost. Do you feel implementation is an all or nothing scenario or does it make sense to focus on domestic and large campuses and then in a second phase chase funding for remote and international sites? Dispersed organizations face many unique technical challenges, especially when implementing email archiving. Certainly it is better to have some archiving than none at all, but unless your retention requirements are also segmented by geography, focusing on only a single location can be risky. I’d definitely suggest starting with the biggest fish, but you will probably have to get to everyone fairly soon afterward.
• Are there solution that do both email and file data (shared folders) archiving? Yes, some email archiving products also do file archiving. However, some choose to implement these functions separately to keep one from standing in the way of the other – it’s more important to have some archiving than to wait and wait for a complete solution.
• Not to mention, extremely short retention periods for email messages will NOT allow an enterprise to comply with litigation hold needs… Once a litigation hold is ordered, all retention schedules for covered data must be suspended. Therefore, a short retention schedule does not necessarily preclude compliance with a legal hold order. However, unreasonably short retention will not look good in court!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Answering Your Email Archiving Questions
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.