Comments on: Compression, Encryption, Deduplication, and Replication: Strange Bedfellows http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/ Understanding the accumulation of data Sat, 11 Feb 2012 20:58:00 -0500 hourly 1 http://wordpress.org/?v= By: Is Security in the the Cloud….Secure? « Feztech's Blog http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-15904 Is Security in the the Cloud….Secure? « Feztech's Blog Tue, 17 May 2011 00:53:10 +0000 http://blog.fosketts.net/?p=1396#comment-15904 [...] A great article that kinda sums it up  – http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/ [...] [...] A great article that kinda sums it up  – http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/ [...]

]]> By: CD Duplication http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-15640 CD Duplication Tue, 25 Jan 2011 18:40:00 +0000 http://blog.fosketts.net/?p=1396#comment-15640 Yeah I agree, It's bit trickier encrpting data on the move. Yeah I agree, It’s bit trickier encrpting data on the move.

]]> By: sfoskett http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-15361 sfoskett Mon, 13 Dec 2010 13:47:00 +0000 http://blog.fosketts.net/?p=1396#comment-15361 It's enlightening and positive comments like this that make me glad to write and share! It’s enlightening and positive comments like this that make me glad to write and share!

]]> By: Norealemaill http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-15358 Norealemaill Sun, 12 Dec 2010 19:57:00 +0000 http://blog.fosketts.net/?p=1396#comment-15358 what a stupid article. The fact is better compression allows for better encryption due to Unicity Distances considerations. Its common knowledge read about Claude Shannon. what a stupid article. The fact is better compression allows for better encryption due to Unicity Distances considerations. Its common knowledge read about Claude Shannon.

]]> By: DVD Duplicators http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-15230 DVD Duplicators Wed, 10 Nov 2010 08:33:00 +0000 http://blog.fosketts.net/?p=1396#comment-15230 I agree what you have said that David Slik's idea is the traditional way of doing it. Is the gzip-rsyncable and rsyncrypto idea possible? I agree what you have said that David Slik’s idea is the traditional way of doing it. Is the gzip-rsyncable and rsyncrypto idea possible?

]]> By: MarkDCampbell http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-14426 MarkDCampbell Fri, 23 Apr 2010 15:01:52 +0000 http://blog.fosketts.net/?p=1396#comment-14426 Stephen: Thanks for this post. A customer called this posting out to me; I reference it several times at my blog and call out the potential problems in ordering. Blog postings regarding this are over at <a href="http://www.unitrends.com/weblog/" rel="nofollow">http://www.unitrends.com/weblog/</a> - the most detailed one is at <a href="http://www.unitrends.com/weblog/index.php/2010/04/23/backup-compression-encryption-deduplication-and-replication-solution/" rel="nofollow">http://www.unitrends.com/weblog/index.php/2010/...</a> Stephen: Thanks for this post. A customer called this posting out to me; I reference it several times at my blog and call out the potential problems in ordering. Blog postings regarding this are over at http://www.unitrends.com/weblog/ – the most detailed one is at http://www.unitrends.com/weblog/index.php/2010/…

]]> By: Backup, Compression, Encryption, Deduplication, and Replication http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-14422 Backup, Compression, Encryption, Deduplication, and Replication Thu, 22 Apr 2010 05:55:40 +0000 http://blog.fosketts.net/?p=1396#comment-14422 [...] I was going to post another article in this series; but ran across  a great blog post from Stephen Fosket entitled Compression, Encryption, Deduplication, and Replication: Strange Bedfellows.  It goes into more detail concerning a specific challenge that the author faced using standard [...] [...] I was going to post another article in this series; but ran across  a great blog post from Stephen Fosket entitled Compression, Encryption, Deduplication, and Replication: Strange Bedfellows.  It goes into more detail concerning a specific challenge that the author faced using standard [...]

]]> By: Jered Floyd http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13988 Jered Floyd Fri, 27 Feb 2009 09:28:41 +0000 http://blog.fosketts.net/?p=1396#comment-13988 As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). --rsyncable totally works, but it's a bit of a hack... it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure -- again, you ought to be doing your replication over an encrypted channel anyhow.<br><br>At Permabit (<a href="http://www.permabit.com" rel="nofollow">http://www.permabit.com</a>), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.<br><br>Regards,<br> Jered Floyd<br> CTO, Permabit As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). –rsyncable totally works, but it's a bit of a hack… it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure — again, you ought to be doing your replication over an encrypted channel anyhow.

At Permabit (http://www.permabit.com), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.

Regards,
Jered Floyd
CTO, Permabit

]]> By: Jered Floyd http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13184 Jered Floyd Fri, 27 Feb 2009 04:28:41 +0000 http://blog.fosketts.net/?p=1396#comment-13184 As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). --rsyncable totally works, but it's a bit of a hack... it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure -- again, you ought to be doing your replication over an encrypted channel anyhow.<br><br>At Permabit (<a href="http://www.permabit.com" rel="nofollow">http://www.permabit.com</a>), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.<br><br>Regards,<br> Jered Floyd<br> CTO, Permabit As David says, the right way to do this is deduplicate (or, at least, segment for deduplication), compress, encrypt, replicate (,wash, rinse, repeat). –rsyncable totally works, but it's a bit of a hack… it's doing non-optimal segmentation for deduplication, and of course doesn't help unless you reset your encryption cipher on the same boundaries as well. As you say, this makes the encryption somewhat less secure — again, you ought to be doing your replication over an encrypted channel anyhow.

At Permabit (http://www.permabit.com), we incorporates all of these technologies in our Enterprise Archive product in this order for maximum benefit. As data is being written an in-line process breaks files up in to variable-sized segments for optimal deduplication. Then these segments are (optionally) compressed, (optionally) encrypted, deduplicated, and written to disk. These compressed, encrypted chunks can be replicated, which is also done over an encrypted channel to eliminate traffic analysis. This provides the best of all worlds.

Regards,
Jered Floyd
CTO, Permabit

]]> By: Pete Steege http://blog.fosketts.net/2009/02/05/compression-encryption-deduplication-replication/comment-page-1/#comment-13136 Pete Steege Tue, 10 Feb 2009 15:45:19 +0000 http://blog.fosketts.net/?p=1396#comment-13136 Hi Stephen,<br>Encryption seems to be evolving as a multi-level requirement. Encryption of data at the end of the line with self-encrypting drives covers data at rest and avoids the dedupe issue. <br><br>It's trickier, as you say, for encrpting data on the move. When do you see a viable solution standardized? Hi Stephen,
Encryption seems to be evolving as a multi-level requirement. Encryption of data at the end of the line with self-encrypting drives covers data at rest and avoids the dedupe issue.

It's trickier, as you say, for encrpting data on the move. When do you see a viable solution standardized?

]]>