Deletion of data is not a high priority for most IT shops, but this ought to change

When was the last time you deleted data? Even at home, where we have autonomy and authority over our own data, many of us are digital pack rats. But at work? Never! No one ever deletes anything! Let’s talk about why this is.

Retention vs. Deletion

Just about everything we do in IT infrastructure is focused on retention. We back up our data and implement other data protection tools like snapshots and mirrors. We might also archive data so that the General Counsel can place legal hold on it, as well as perform data discovery during litigation. And then there’s the whole field of data security, focused on locking people out of data, keeping it intact and un-viewed.

But what about deletion? Almost no effort is put towards removing data, though the rapid growth of storage might lead one to think this is a key area for IT. We certainly could put some effort on revision control, and especially deleting drafts and outdated data. We could easily expire content that was no longer needed, if only we had some way to know that. And we’ve talked a lot about secure deletion, even though we hardly ever actually perform that task except when moving to new physical storage hardware.

The greatest challenge for deletion is a simple question: What should we delete and when?

IT can not answer these questions. They must be put to the business people who really own the data. Without permission and buy-in, IT is in serious legal peril when it comes to deleting data: Any deletion must be in accordance with policy and must be legal, that is there is no legal or regulatory hold on it. And there is no way most IT staff feel empowered to do that!

Some Data Should Be Deleted

Certainly, not all data should be saved. There is “low-hanging fruit” in every storage estate that can and should be deleted:

• Ephemeral copies – Drafts, temporary data, working copies
• Time-limited projects – Third-party or client data, test and development
• Expired data – Retention policies that are expired and no legal hold remains
• Legally required – Data that isn’t yours, or that legal demands deleted

Tackling these data sets is much easier to tackle than cleaning out primary data stores, since it doesn’t require as much sifting and sorting: These data sets can often be identified programmatically! If you have data sets like these, this is the ideal place to start a deletion effort.

Delete on Demand

Regardless of the type, however, IT should not delete data without direction. It is perilous in today’s legal environment to destroy data without a policy directing that action. So we should continue to focus on retention for most data, while we work with legal to determine which data can be deleted and come up with a process for approval.

But it’s important to start offer a deletion-friendly environment for certain data types. Such a storage system would reduce the difficulties associated with data deletion. Really, only an integrated solution can truly delete data:

• It must maintain custody of data from start to end and not allow it to leak all over the organization
• It must be accessible since any restrictions tempt users to create “working copies”, thus thwarting deletion
• It must be secure – Data must always be encrypted to avoid remnants on media
• It must be protected so data will not spread to external systems and sites

Stephen’s Stance

Data deletion is a real problem for most IT shops. I’m just getting my head around the ramifications, and continue to look for an ideal deletion-friendly storage solution.

If you’re interested in the topic of data deletion, I recommend joining me for a webinar on the topic on Wednesday, April 13. Sponsored by Nasuni, I will discuss the dilemma of deletion and CEO Andres Rodriguez will weigh in about the capabilities of his cloud storage solution. Register now!

Note: Nasuni is sponsoring this webinar, but the content was created by me. This blog post is intended to engage my audience in discussion of the subject, and is not a paid promotion or advertisement.

Image credit: “Delete” by blmurch

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Deletion Dilemma
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• What: Is There a Real ROI for E-mail Archiving: Building a Business Case
• When: Thursday, March 26, 2009 at 1 PM Eastern/10 AM Pacific
• Sponsored By: Google Security Services Events & Webinars

In a difficult financial environment, organizations are scrutinizing IT expenditures. Only those projects with a proven return on investment (ROI) are receiving funding. Those looking at e-mail archiving need to ask if there is an ROI for their company, and how can these cost savings be justified. What really does and does not save companies money for e-mail archiving may surprise you. The webcast will cover:

• Four critical factors driving ROI – what to look at and what not
• Actual ROI case studies
• Strategies for building a business case

You’ll hear from Stephen Foskett, Director of the Data Practice at Contoural Inc., a leading independent provider of business and technology consulting. Drawing upon the experiences of hundreds of large and small companies, Stephen will focus on the nuts and bolts of building an ROI and business. Any organization considering an e-mail archiving system will need to face these issues.

Register and attend for free online!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Is There A Real ROI For Email Archiving?
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Register online at Contoural’s web site!

E-mail Archiving 101: A Non-Technical Person’s Overview of How E-mail Archiving Works and How to Pick the Right Tool
Thursday, January 8, 2009
10:00 AM Pacific Time (US & Canada)
Faculty: Independent expert Stephen Foskett, Principal Consultant, Contoural, Inc.

E-mail archiving systems are becoming an important component of many organization document retention and litigation preparedness strategy. Nevertheless, to the non-technical person, understanding what these systems do and how they are different can be bewildering or even downright confusing. A good idea sometimes gets lost in technical translation. In this webinar, industry expert and independent Contoural Principal Consultant Stephen Foskett will walk through a high-level overview of how these systems work in simple terms aimed at the non-technical person.

Topics include:

• What do e-mail archiving products do and how do they support litigation readiness?
• High-level description of how different products work and are used
• What are some of the real differences between systems?
• How to evaluate which features are best for your organization

This webinar will also provide time for questions from the audience.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Email Archiving 101 Webinar, January 8
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

1. What are vital records versus ordinary records? I like this suggestion from another attendee: “Vital Record” is, by definition, a record without which the organization could not continue to function.
2. Would you consider Accounts Payable and Accounts Receivable receipts and related backup documentations vital records? They may be vital to some businesses and not vital to others. I suspect that these would be much more vital in the short term and much less important after the year is closed out. But I can imagine scenarios where they would be required for decades.
3. It was mentioned that it was risky to scan paper into a electronic format without a written, compact retention policy. My question is this: If you are in the process of setting policy and adding to your policy and procedures manual is it still ok to scan? My boss has pushed for me to begin the conversion process even though I haven’t managed to complete the new policy and procedures. I think this is his way of showing his superiors that we are making progress on a project that they weren’t neccesarily behind in the first place. I would worry that you’ll waste effort scanning documents that shouldn’t be saved. But it’s better to duplicate and waste some effort than to not save at all. So scan away! But get that policy done too!
4. What about TIFF storage? Is that still viable? and For long term storage, why isn’t tiff G4 an option? TIFF is certainly a widely-used format for images, and G3 compressed TIFF will probably be readable a century from now (assuming we’re still here!) Plus, TIFF is (usually) lossless. However, it’s very flexible, and a TIFF file can contain lots of different data – you could easily create a TIFF that won’t be readable next week! Also, there have been some patent/royalty arguments about TIFF and related technologies. Finally, TIFF files tend to be huge (since they’re lossless) compared to other (lossy) formats. I like open standards like PNG better since they’re fully documented and portable, but TIFF is probably a decent choice.
5. When storing paper does the newer printing technology – inkjet, laser etc. match the archival quality of older paper and inks? and What do we need to know about the kind of paper that should be used for long term archiving? I’m concerned that newer technologies, including printer tech, won’t last like old ones. I have impact-printed pages from the 1980s that look like new, and laser-printed pages from the 1990s that are clear and clean. But my old inkjet pages are very faded. Now, this could have to do with the paper I chose, and it could be that old inkjet isn’t a predictor of new inkjet, but I’m skeptical. HP claims that their Vivera inkjet ink will last “108 years”, which seems oddly precise. I guess they took a Photosmart printer back to 1900 and printed out some pages? In all seriousness, be concerned about any claims like this that cannot possibly be tested. And consider handling as well – proper temperature, humidity, and storage will make documents last much, much longer!
6. Dry caskets of nuclear waste are dangerously radioactive for over 10,000 years. Consider the “pancake” drive: titanium disks written with an ion beam in both analog and digital. Even without ASCII, it can be read with an electron microscope. Now that is longterm storage. Do you really trust current formats for 50-100 years? Think of how much change there has been in the last 50 years. and If There was not a question of Blu-ray discs lasting for decades , what is your opinion on the technology as a viable archive solution? I’m skeptical of all unprovable longevity claims, since media has never lived up to them in the past! If I had to guess, I’d say that mechanical devices like disks and tapes will have more problems than plain discs, and that optical will last longer than magnetic. So I bet Blu Ray media will outlive most other current mainstream media.
7. If it isn’t vital you should not convert. On another project (our archives) I am trying to save and prevent any further damage to the records because they are currently stored in a basement with water leaks, rats, and no climate control. Is this instance how do you weigh the vital and non-vital? Specifically, I was asked to bring in outside companies to see what they had to say and get quotes after my warnings and price estimations were rejected. Any suggestions on how to get across the importance of not allowing these records to just rot? Rats don’t make great records managers! Seriously, though, I’d say that something is seriously out of whack when records are stored like this. Maybe you should come in to work one day and claim that all of the records are destroyed and see what reaction you get? But be sure to have another job lined up before you pull this stunt…
8. I just want to point out there is a whole profession that deals with this every day – the Archivist. Information can be found at www.archivists.org – they even have a group of Business Archivists that you can talk to. Thank you for that link – I’ll look into it!
9. Can you provide more information on what Open Standard Format is? I like standards that are well documented in public places – ASCII charts are freely available and widely distributed, for example. These are a message to future generations – when you encounter this file, decode it with this chart! But proprietary formats concern me, especially when their documentation isn’t freely available and open.
10. When will the presentation slides be available? The webinar and slides should be posted at AIIM’s site within two weeks.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Thoughts on Long-Term Archiving
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

First, consider whether to store records as atoms or bits. You can convert paper to data or vice versa, and there are pros and cons to both:

• Properly handled physical (paper or film) records should last for hundreds of years and can remain readable without software or devices. But they’re hard to search (you need an index), and paper is bulky, heavy, and difficult to work with.
• Digital records can either be stored offline or kept “alive,” but questions remain about their long-term reliability and readability. Living records can be easy to search and use, and digital storage can be very space-efficient, but data tends to pile up “out of sight.”

Long-term storage of records on physical media is proven – think about papyrus, tablets, gold or nickel discs, film, and paper. But will digital media fare as well? Data tapes and disks can degrade over time, and manufacturer reliability specs are based on accelerated testing, not actual experience. Regardless of media type, careful handling can extend media life.

But will you still be able to read it? Tapes and optical disks require additional hardware to read, while disk drives are paired with their read heads. Software applications are needed to read and interpret data (backup, archiving, compression, encryption, deduplication, database) as well. What about content format? Should you use ASCII, XML, PDF/A?

I’ll be presenting a webinar on this topic tomorrow, Wednesday, December 3, at 2:00 PM Eastern time. Register on-line at the AIIM web site and join me for the discussion!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Long-Term Versus Longer-Term Archiving
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The gist is that email archiving can be an incredibly useful tool to manage legal risks and enable more effective discovery of message content. Electronic discovery is growing at an amazing clip, and most cases now include email messages – in fact, email is estimated to be 60% to 70% of all legal discovery in the United States today!

At the same time, companies are challenged with ineffective or un-enforced retention and litigation hold policies and rampant “underground archiving” of messages in offline PST files. And to make matters worse, the 2006 revisions to the Federal Rules of Civil Procedure drastically shortened the amount of time companies have to describe and produce electronic records! It’s simply impossible to begin recovering messages from backup tapes in response to legal requests – the only way forward is a proactive strategy based on email archiving.

If this piques your interest, grab a copy of that whitepaper, join me on Wednesday for the webinar, and check out some of my other email-related content. I’ll also be speaking on the topic at Storage Decisions in San Francisco, and would be happy to respond to your emailed questions any time!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Six Critical Steps For Managing Email E-Discovery
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• How to improve the receptivity of the e-mail archieve message to management to overcome the biases you mentioned? Consider what management is interested in most: They don’t necessarily care about email archiving, but they will respond to well-presented solutions to business objectives like cost avoidance for litigation and e-discovery or future reduced spending on IT infrastructure (storage).
• How do you get things moving when Legal is sponsor and can demonstrate that one lawsuit discovery failure will be the same cost as the project? This would be an ideal scenereo. Legal probably already understands the high cost of e-discovery and potential risk of adverse judgement. Now demonstrate the ways that email archiving can help. But be realistic – an email archiving app will help with e-discovery, but is definitely not a complete solution. You will still need policy and process for hold and discovery, and may even need additional e-discovery tools.
• Have you seen standard or default retention periods for email, and how are they determined? I would not want to suggest standard or “rule of thumb” retention periods. In my experience with many organizations, I have seen such a wide variety of carefully considered retention schedules that I have come to believe that there is no such thing as a standard. However, I would like to suggest not making a too-short maximum (say, less than three months) or else you will drive end users to underground archiving to get around the policy. Such a short retention period can work for certain classes of messages (company picnic, lunch invites, mass mailings), but I can’t imagine a business where no one would need to keep messages longer. Try to be realistic in setting your retention, balancing productivity with compliance. Setting retention periods is an exercise in consensus building, with representatives from IT, Legal, HR, Compliance, and others each contributing their own needs. It’s doable, though – I do it for a living!
• How many years back do people hold on to mail typically? Most organizations lack standards or policies for retention, so their typical retention varies based on employee actions. When they do have retention policies for email, they tend to classify messages (by organization or user, keywords, or through user action) into a few “buckets” with appropriate retention for each. Most have short retention (3 to 9 months) for some content, and longer retention (on the order of a few years) for other types. Most also have a “keep forever” category for special record types.
• Do you have an opinion on whether journaling is better or not for meeting legal requirements? Most legal groups desire the most complete set of messages possible. Both journaling and log shipping should meet this need.
• Have you seen much use for email archiving for supporting Audit to prove that an event was reviewed and approved? Many organizations use email as part of their workflow. In these cases, maintaining a complete set of messages is critical for audit and compliance. Many certainly do use email archives for this purpose, while others prefer to route such approvals to other content management systems to maintain these records.
• Is there a legal argument to be made that getting rid of pst files and having a central archive will help with privacy law issues? Decentralized personal archives like PST files are a privacy nightmare waiting to happen – they are highly mobile, can be copied and viewed easily, and tend to “live” on theft-prone devices like laptops and portable drives. Replacing PST files with a centralized archive has many many benefits, and the reduced risk of privacy breaches is certainly among these. However, most mail clients still maintain offline caches of mail server (and archive) data, so simply replacing PST with an archive will not elimate the privacy risk.
• You might want to mention that accelerating volumes of stored email are not necessarily bad because it is electronic and therefore searchable. Offline stuff dumped into file servers with no meta data is actually more of a nightmare because no one knows who created it or why it is needed (or not). Adding structure to electronic stored information is always valuable, and email archives certainly do make that data much more organized and searchable, along with enabling retention policies to be applied (assuming they exist!) Unstructured data on file servers is indeed a nightmare, and can pose a serious risk. There are other tools to tackle this type of data, however, and they might do a better job than relying on an email archive as a primary repository of critical records.
• Can you elaborate on user training needs? I was under the impression that changes to the end user were minimal. The amount of impact an email archiving system has on end users can vary from minimal to significant, based mostly on how much user interaction is desired for classification. However, even the most unobtrusive system is likely to change certain aspects of user behavior, from PST files to searches to stubbing and deletion of messages. Therefore, training is probably needed with every email archiving implementation.
• Litigation holds can last for YEARS !! Tape is DEAD especially to try to rebuild mail boxes. Tape remains non-dead for certain applications like backup, despite years of prognostication. However, you are correct that e-discovery of data on tape is incredibly difficult and time-consuming for most systems. There are tools to help, but an online archive is vastly more flexible and speedy than recovering backup data from tape. I’ve seen backup-sourced e-discovery costs rise into the millions of dollars, too!
• How important is classification in an archive solution? The importance of classification depends on the business objectives that an archive is meant to serve. However, most implementations will require at least some message classification mechanism in order to determine how long different messages should be retained, since a one-size-fits-all retention schedule is unlikely to meet anyone’s needs. Therefore, it is important to begin developing an overall record retention policy for email when implementing an email archive, and to consider how messages will be classified. Classification of email messages is a complicated topic – worthy of its own webinar at least!
• We are a globally dispersed organization which increases the complexity and cost. Do you feel implementation is an all or nothing scenario or does it make sense to focus on domestic and large campuses and then in a second phase chase funding for remote and international sites? Dispersed organizations face many unique technical challenges, especially when implementing email archiving. Certainly it is better to have some archiving than none at all, but unless your retention requirements are also segmented by geography, focusing on only a single location can be risky. I’d definitely suggest starting with the biggest fish, but you will probably have to get to everyone fairly soon afterward.
• Are there solution that do both email and file data (shared folders) archiving? Yes, some email archiving products also do file archiving. However, some choose to implement these functions separately to keep one from standing in the way of the other – it’s more important to have some archiving than to wait and wait for a complete solution.
• Not to mention, extremely short retention periods for email messages will NOT allow an enterprise to comply with litigation hold needs… Once a litigation hold is ordered, all retention schedules for covered data must be suspended. Therefore, a short retention schedule does not necessarily preclude compliance with a legal hold order. However, unreasonably short retention will not look good in court!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Answering Your Email Archiving Questions
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The session is presented by Mimosa, but the content is independent, so you won’t hear me pitch log shipping or trashing other vendors. Instead, I’ll focus on the benefits and beneficiaries of email archiving and how to make this critical technology a priority for the business, not just IT.

If you would like to attend, please register for the session at On24.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Trying To Get An Email Archiving Project Approved?
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.