Comments on: Compression, Encryption, Deduplication, and Replication: Strange Bedfellows http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/ Understanding the accumulation of data Fri, 19 Mar 2010 00:14:36 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Jered Floyd http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13988 Jered Floyd Fri, 27 Feb 2009 09:28:41 +0000 http://blog.fosketts.net/?p=1396#comment-13988 As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). --rsyncable totally works, but it's a bit of a hack... it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure -- again, you ought to be doing your replication over an encrypted channel anyhow.<br><br>At Permabit (<a href="http://www.permabit.com" rel="nofollow">http://www.permabit.com</a>), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.<br><br>Regards,<br> Jered Floyd<br> CTO, Permabit As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). –rsyncable totally works, but it's a bit of a hack… it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure — again, you ought to be doing your replication over an encrypted channel anyhow.

At Permabit (http://www.permabit.com), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.

Regards,
Jered Floyd
CTO, Permabit

]]> By: Jered Floyd http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13184 Jered Floyd Fri, 27 Feb 2009 04:28:41 +0000 http://blog.fosketts.net/?p=1396#comment-13184 As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). --rsyncable totally works, but it's a bit of a hack... it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure -- again, you ought to be doing your replication over an encrypted channel anyhow.<br><br>At Permabit (<a href="http://www.permabit.com" rel="nofollow">http://www.permabit.com</a>), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.<br><br>Regards,<br> Jered Floyd<br> CTO, Permabit As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). –rsyncable totally works, but it's a bit of a hack… it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure — again, you ought to be doing your replication over an encrypted channel anyhow.

At Permabit (http://www.permabit.com), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.

Regards,
Jered Floyd
CTO, Permabit

]]> By: Pete Steege http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13136 Pete Steege Tue, 10 Feb 2009 15:45:19 +0000 http://blog.fosketts.net/?p=1396#comment-13136 Hi Stephen,<br>Encryption seems to be evolving as a multi-level requirement. Encryption of data at the end of the line with self-encrypting drives covers data at rest and avoids the dedupe issue. <br><br>It's trickier, as you say, for encrpting data on the move. When do you see a viable solution standardized? Hi Stephen,
Encryption seems to be evolving as a multi-level requirement. Encryption of data at the end of the line with self-encrypting drives covers data at rest and avoids the dedupe issue.

It's trickier, as you say, for encrpting data on the move. When do you see a viable solution standardized?

]]> By: sfoskett http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13111 sfoskett Fri, 06 Feb 2009 09:06:37 +0000 http://blog.fosketts.net/?p=1396#comment-13111 That's the traditional way of doing it. But I'm excited by the idea of doing deduplication AFTER compression and encryption using gzip-rsyncable and rsyncrypto! That's the traditional way of doing it. But I'm excited by the idea of doing deduplication AFTER compression and encryption using gzip-rsyncable and rsyncrypto!

]]> By: David Slik http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13110 David Slik Fri, 06 Feb 2009 09:03:29 +0000 http://blog.fosketts.net/?p=1396#comment-13110 Compression, encryption, de-duplication, and replication can all coexist, you just need to do it in the right order.<br><br>You first de-duplicate, then you compress, then you encrypt, and last of all, you replicate.<br><br>And, of course, if you really care about your data, you take a has of it at the beginning so you can verify that after all those machinations, you're still getting your original data back at the end of the day. Compression, encryption, de-duplication, and replication can all coexist, you just need to do it in the right order.

You first de-duplicate, then you compress, then you encrypt, and last of all, you replicate.

And, of course, if you really care about your data, you take a has of it at the beginning so you can verify that after all those machinations, you're still getting your original data back at the end of the day.

]]> By: sfoskett http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13105 sfoskett Fri, 06 Feb 2009 01:06:37 +0000 http://blog.fosketts.net/?p=1396#comment-13105 That's the traditional way of doing it. But I'm excited by the idea of doing deduplication AFTER compression and encryption using gzip-rsyncable and rsyncrypto! That's the traditional way of doing it. But I'm excited by the idea of doing deduplication AFTER compression and encryption using gzip-rsyncable and rsyncrypto!

]]> By: David Slik http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13104 David Slik Fri, 06 Feb 2009 01:03:29 +0000 http://blog.fosketts.net/?p=1396#comment-13104 Compression, encryption, de-duplication, and replication can all coexist, you just need to do it in the right order.<br><br>You first de-duplicate, then you compress, then you encrypt, and last of all, you replicate.<br><br>And, of course, if you really care about your data, you take a has of it at the beginning so you can verify that after all those machinations, you're still getting your original data back at the end of the day. Compression, encryption, de-duplication, and replication can all coexist, you just need to do it in the right order.

You first de-duplicate, then you compress, then you encrypt, and last of all, you replicate.

And, of course, if you really care about your data, you take a has of it at the beginning so you can verify that after all those machinations, you're still getting your original data back at the end of the day.

]]>