NetApp Technology Network : Thread List - Deduplication

SnapVault deduplication

community@netapp.com — Mon, 26 Jan 2009 19:35:52 GMT

With DataOnTap 7.3 and above with the data remain deduped upon replicating to the SV secondary device. I had thought prior to 7.3 that the data would be expanded across the wire and on the secondary device.

Whether to split CIFS volume into multiple volumes of like-data for de-dupe purposes?

community@netapp.com — Sat, 31 Oct 2009 10:08:40 GMT

Hi,

I have a 600GB volume that conatins all CIFS data for an environment including home directories, roaming profiles, TS profiles and general business data. I'm planning to move all this data to an ASIS-enabled volume, but I am unsure whether there would be any benefit from a de-duplication point of view in splitting the volume into multiple (i.e. one for home directories, one for profiles, one for business data).

From my understanding, there is no need to do this (surely the data would de-duplicate just as well whichever volume it is in?), but a colleague has recommended that I do.

There are obviously other considerations such as snapshot frequency, future snapmirror options, flexclones etc ... but from a de-dupe point of view, is there any benfit to splitting the like-data into separate volumes?

Any suggestions welcome!

Jon

De-Dupe volume sizes

community@netapp.com — Fri, 19 Jun 2009 16:36:26 GMT

My understanding is that there are limits for the size of volumes that can be de-duped. Is that right? What are those limits and where do they come from?

Thanks!

Dave

Where does meta data for dedupe get stored in aggregates

matthewt@netapp.com — Fri, 23 Oct 2009 21:38:03 GMT

We are working to manage a large SnapVault secondary environment that is deduplicated and thinly provisioned.

We are trying to manage the space in our aggregates through the use of a "blocker volume" or fully guaranteed volume that will ensure that the aggregates maintain free space blocks for performance reasons, and to act as an emergency reserve.

The question is, where exactly does all the metadata for the deduplication get stored?

We could alter our layout to include larger aggregate snapshot reserve space, or no reserved volume space if needed, if it would help the appliance store and process this metadata more efficiently.

Deduplication of SQL Server Backup Files

community@netapp.com — Thu, 15 Oct 2009 15:52:06 GMT

Hi everyone,

Does anyone have any good experience of de-duplicating SQL Server backups? We have a volume that holds .BAK backup files for a number of databases, and around a week's worth of daily backup files for each database are held. Since the data churn is low, I thought I would get reasonable success de-duping this volume, but having turned it on and run through, I got less than 1%.

I don't know the mechanisms of SQL Server backups particular well - maybe the backup algorithms and data compression mean that similar blocks is actually quite unlikely ..

Just wondering if anyone has had similar experiences or successes where I have not?

Cheers,

Jon

Future of A-SIS de-dupe

community@netapp.com — Thu, 18 Jun 2009 16:26:23 GMT

Hi,

This is a non-SAN specific question, yet fully applicable to SAN environment as well:

Can we expect (when?) aggregate-level de-duplication?

Current volume-level de-dupe is great anyway, yet taking this one step further would be nice!

Regards,
Radek

Dedup problem

community@netapp.com — Tue, 06 Oct 2009 11:08:41 GMT

Hi,

We are currently running Netapp Release 7.2.6.1 and have turned on dedup on our cif's shares in an attempt to free up some disk space. Unfortunatly we have around 20 days worth of snapshots (to ease restoring of data for our helpdesk staff). I noticed that we have a 250gb snapshot with the name sis.41a470 etc. The volume itself is around 900gb with a snapshot reserve of 10% and i cant figure out what this snapshot actually is and if it is safe to get rid of it. We also run the scheduled sis dedup process once a night and that also seems to make no difference to the size of teh sis.41a snapshot.

Can anybody help?

Thanks

Why does df -s and sis stat give different % save?

community@netapp.com — Wed, 09 Sep 2009 18:31:06 GMT

Comparing the output of the df -s command and the (diag mode) sis stat command I get two different results for % saved.

df -g -s vmware01
Filesystem used saved %saved
/vol/vmware01/ 628GB 498GB 44%

sis stat /vol/vmware01
Path Allocated Saving %Saved
/vol/vmware01 302 GB 498 GB 62 %

Can someone comment on why this is the case (while avoiding any comments of why I shouldn't be using diag commands) ?

Deduplication Performance Impact?

community@netapp.com — Thu, 18 Jun 2009 17:57:15 GMT

What kind of performance impact does deduplication have on performance? (both during the scheduled deduplication process and also "business hours")

A-SIS SnapVault destination question

community@netapp.com — Thu, 24 Sep 2009 19:15:36 GMT

Real quick one,

Do you turn on dedup before you do the initial transfer or after?

Dedupe with 7.3.1 and Snapmirror - best practices

community@netapp.com — Wed, 09 Sep 2009 19:30:33 GMT

Are there any published best practices running dedupe with 7.3.1 and VSM?

With metadata now being stored in the aggregate but replication (at least in my case) occuring at the volume level, what are the recommendations for dedupe operations on the DR Filer if the need arises to break the mirror and make the DR writable?

For one I'm guessing there's no metadata present as its all hiding at the aggregate level on the production system so should you recreate the metadata with a 'sis start -s'?

Do the same processes need to considered when reversing replication to restore production?

Is this process more straight forward if leveraging Protection and Provisioning Manager 3.8?

Thanks

Richard