2,306 Views 28 Replies Last post: Nov 15, 2009 2:23 PM by Andrew Miller RSS
Guest
Currently Being Moderated

Jun 19, 2009 9:36 AM

De-Dupe volume sizes

My understanding is that there are limits for the size of volumes that can be de-duped. Is that right? What are those limits and where do they come from?

 

Thanks!

 

Dave

sajan NetApp Employee Enthusiast 7 posts since
Aug 11, 2008
Currently Being Moderated
Jun 19, 2009 11:14 AM in response to: Jeremy
Re: De-Dupe volume sizes

Information on volume sizes for all supported platforms with various Data ONTAP versions is available in Dedupe DIG: http://media.netapp.com/documents/tr-3505.pdf.

calvarez NetApp Employee Enthusiast 11 posts since
Aug 27, 2008
Currently Being Moderated
Jun 19, 2009 11:34 AM in response to: Jeremy
Re: De-Dupe volume sizes

NetApp deduplication is the #1 implementation of deduplication on primary storage, meaning it is being used on production systems to deduplicate active data.  With well over 30,000 licenses installed, it is a proven technology.

 

A key factor in the success of NetApp deduplication for primary storage is the fact that the system resources of each storage system model are being considered so that they are not oversubscribed, such as system memory.  NetApp deduplication for FAS uses different max volume sizes for different models to help ensure resource availability so that the performance of the primary storage system is maintained.

 

It is worth noting that this max volume size is a delimiter of the physical size of the volume only.  That is to say that even though a volume size may be limited to 3 TB in size, it is still capable of storing greater than 3TB of deduplicated data.  For example, you might see 5 TB of data being stored, but it would only be using 2TB of storage thanks to deduplication.

 

Below are the max vol sizes by model and version of Data ONTAP.

 

 

Data ONTAP 7.2.X (Starting with 7.2.5.1) and Data ONTAP 7.3.0

FAS2020

FAS3020 N5200 FAS2050

FAS3050 N5500

FAS3040 FAS3140 N5300

R200

FAS3070 N5600 FAS3160

FAS6030 FAS6040 N7600 FAS3170

FAS6070 FAS6080 N7800

0.5TB

1TB

2TB

3TB

4TB

6TB

10TB

16TB

Data ONTAP 7.3.1 or higher

FAS2020

FAS3020 N5200 FAS2050

FAS3050 N5500

FAS3040 FAS3140 N5300

R200

FAS3070 N5600 FAS3160

FAS6030 FAS6040 N7600 FAS3170

FAS6070 FAS6080 N7800

1TB

2TB

3TB

4TB

4TB

16TB

16TB

16TB

davelightman   1 posts since
Jun 19, 2009
Currently Being Moderated
Jun 20, 2009 7:27 AM in response to: calvarez
Re: De-Dupe volume sizes

Thanks, Carlos!

 

A couple of follow up questions:

 

Where do these limits come from and why do they vary from system to system? For example, why is the limit 4TB on the FAS 3140 and 16TB on the FAS 3170?

 

Is there any work-around for this? For example, if I have a bunch of VMware boot images that would de-dupe down to 5TB of real disk space is there any way to do that (on a FAS 3140)? If the data that's common between images is about 1TB each, it would be a shame to have to duplicate that data a bunch of times due to this limit.

jooss NetApp Employee Enthusiast 12 posts since
Mar 18, 2008
Currently Being Moderated
Jun 22, 2009 5:01 AM in response to: davelightman
Re: De-Dupe volume sizes

> Where do these limits come from and why do they vary from system to system? For example, why is the limit 4TB on the FAS 3140 and 16TB on the FAS 3170?

 

The limits come from the available resources on the systems (CPU, memory, etc...).

 

> Is there any work-around for this?

 

The limits are volume based so one can  break things into multiple volumes.  That obviously has some trade-offs in terms of deduplication as well as other areas but is certainly a possibility.

radek.kubka Virtuoso 389 posts since
Jul 31, 2008
Currently Being Moderated
Jun 22, 2009 1:45 AM in response to: calvarez
Re: De-Dupe volume sizes

Hi Carlos,

 

Two things:

NetApp deduplication for FAS uses different max volume sizes for different models to help ensure resource availability so that the performance of the primary storage system is maintained.

 

Does it relate to system resources during actual de-duplication or outside of this process? The former may or may not be a problem, as in a non-24/7 environment hammering the system for say 8 hours only to de-dupe the data could be 100% feasible. The latter is a subject to discussion in two separate threads here & I am yet to hear a firm answer to this.

For example, you might see 5 TB of data being stored, but it would only be using 2TB of storage thanks to deduplication.

 

Well, nice. The problem is A-SIS is a post-process deduplication, so if above scenario happens on one of the smaller filers, it may mean repetitive adding the un-de-duped data to the volume, running A-SIS against it, adding more data & so on, so forth. A bit tedious & not every admin will have enough time / patience to actually do this.

 

Do not get me wrong - I love A-SIS, but what I am saying is that capping volume sizes can make people's life harder, so the question is whether there is a good reason behind that.

 

Regards,
Radek

calvarez NetApp Employee Enthusiast 11 posts since
Aug 27, 2008
Currently Being Moderated
Jun 22, 2009 9:36 AM in response to: radek.kubka
Re: De-Dupe volume sizes

The volume size limits help ensure that the actual process of deduplication does not oversubscribe the system resources.

 

Working with a post-processing deduplication process means that you must take the initial size of the un-deduped data into consideration.

 

You will need to consider best practices for each specific scenario, as described in TR-3505, mentioned in a previous reply.

nigelg1965   1 posts since
Jan 27, 2009
Currently Being Moderated
Aug 19, 2009 7:27 AM in response to: calvarez
Re: De-Dupe volume sizes

Why then can't you run A-SIS on a volume that was ONCE over the limit. We migrated from 3020 with a main volume of c. 4.5Tb to a 3140 with new shelves, and in the process I split the data across two different volumes, using Snapmirror, so as to be able to use A-SIS, unfortunately as you can't retrofit Qtrees we had to Snapmirror an entire volume and then delete the unwanted parts. All resulting volumes are well below 3TB, but A-SIS will not work on the volume that was briefly over 4TB or indeed a completely fresh copy of it!

radek.kubka Virtuoso 389 posts since
Jul 31, 2008
Currently Being Moderated
Aug 19, 2009 7:53 AM in response to: nigelg1965
Re: De-Dupe volume sizes

Interesting stuff - we are entering a wild area!

 

Unfortunately I have no good answer to your question (hopefully someone else does). The only thing on my mind would be to come back to square one & use QSM instead of VSM, which will definitely drop behind all legacy characteristics of the original volume.

 

What particular error message are you getting when trying to run A-SIS on the new volume?

 

Regards,
Radek

eric.barlier Master 142 posts since
Jun 20, 2009
Currently Being Moderated
Aug 19, 2009 4:38 PM in response to: nigelg1965
Re: De-Dupe volume sizes

hi Nigel,

 

Did you consider using QSM to mirror qtrees into volumes that would maybe solve this? I know I have not got enough information about this to make

a qualified statement, and its probably too late as well. but for the future maybe QSM would be good for you?

 

Cheers,
Eric

Andrew Miller Featured Member Guru 662 posts since
Oct 7, 2008
Currently Being Moderated
Aug 19, 2009 8:02 PM in response to: nigelg1965
Re: De-Dupe volume sizes

Erm...what version of Data ONTap are you running? This restriction was specifically lifted in 7.3.1+ (related to moving the fingerprint hashes from the volume to the aggregate I believe).

Andrew Miller Featured Member Guru 662 posts since
Oct 7, 2008
Currently Being Moderated
Jun 26, 2009 2:46 PM in response to: calvarez
Re: De-Dupe volume sizes

Perfect graph....I've found this information in multiple places but that's the nicest representation so far.

 

And with 7.3.1+, it's ever so much less painful given being able to shrink a volume back down under the limit to turn on dedup.

christop NetApp Employee Enthusiast 25 posts since
Feb 9, 2009
Currently Being Moderated
Sep 4, 2009 8:56 AM in response to: Dave Lightman
Re: De-Dupe volume sizes

Hi:

 

What is the max flexvol size for deduplication for a GF960 system?

 

I do not see any guidelines published for this platform.

 

Thanks,

Chris

tekaravind Enthusiast 42 posts since
Mar 18, 2008
Currently Being Moderated
Sep 4, 2009 8:56 AM in response to: christop
Re: De-Dupe volume sizes

Hello,

I am on leave till Monday, 7th September 2009. Please contact support@tek-tools.com for any assistance in the meantime, and I will attempt to respond to your email as soon as possible upon my return.

Regards,

Aravind

chriskranz Virtuoso 278 posts since
Aug 26, 2008
Currently Being Moderated
Sep 4, 2009 1:32 PM in response to: christop
Re: De-Dupe volume sizes

I don't believe A-SIS is supported on the 900 series.

pascalduk Master 87 posts since
Mar 10, 2008
Currently Being Moderated
Sep 6, 2009 11:16 PM in response to: chriskranz
Re: De-Dupe volume sizes

chriskranz wrote:

 

I don't believe A-SIS is supported on the 900 series.

 

Correct, but the R200 is supported which basically is  900 series hardware. Could it perhaps be that ontap software can't handle the nearstore personality license, which is required for the a-sis license, on the 900 series?

tekaravind Enthusiast 42 posts since
Mar 18, 2008
Currently Being Moderated
Sep 6, 2009 11:17 PM in response to: pascalduk
Re: De-Dupe volume sizes

Hello,

I am on leave till Monday, 7th September 2009. Please contact support@tek-tools.com for any assistance in the meantime, and I will attempt to respond to your email as soon as possible upon my return.

Regards,

Aravind

aarondelp Enthusiast 27 posts since
Nov 25, 2008
Currently Being Moderated
Sep 8, 2009 12:46 PM in response to: pascalduk
Re: De-Dupe volume sizes

A couple of things.  Dedupe is not SUPPORTED on the 7.2 code line but it will run.  I have a 940 in my lab and it works fine.  As of the 7.3 code line the functionality was disabled.

 

As to why you can't dedupe a volume once it goes over the limit.  You can't (and the reason why it doesn't make any sense) is because it is a bug.  I think that the bug was fixed in 7.3 but I'm not sure.  I know that bug existed in 7.2.

 

Regards,

Aaron

Andrew Miller Featured Member Guru 662 posts since
Oct 7, 2008
Currently Being Moderated
Sep 14, 2009 8:31 PM in response to: aarondelp
Re: De-Dupe volume sizes

Hmm...I think I might be missing something here but dedup is definitely supported in the 7.2.x code line (7.2.5.1 minimum with 7.2.6.1 greatly preferred...maybe even just one of the P releases).

 

As to not being able to dedup a volume if it goes over the limit in 7.2, my understanding is that this wasn't as much a bug as an architectural limitation around how the dedupe metadata was stored in 7.2 (in the volume). Once the metadata was moved to being stored in the aggregate in 7.3, that wasn't an issue anymore (so could shrink the volume down under the limit and enable dedup).

aarondelp Enthusiast 27 posts since
Nov 25, 2008
Currently Being Moderated
Sep 15, 2009 5:30 AM in response to: Andrew Miller
Re: De-Dupe volume sizes

Sorry for the confusion Andrew - my comment was dedupe on the 9XX FAS series.  It is supported on everything else obviously! 

 

I could be wrong on that one but that is what I was told by somebody at NetApp the other day, I haven't ever had a need to verify if the support statement is true for the 900 series.  If it isn't, please let me know!

 

Thanks!

aarondelp Enthusiast 27 posts since
Nov 25, 2008
Currently Being Moderated
Sep 15, 2009 8:23 AM in response to: Andrew Miller
Re: De-Dupe volume sizes

Hey Andrew - About the de-dupe sizes.  It was originally a bug that was turned into a "feature" because they don't plan on fixing it.  You are spot on for the reason why.  The original bug was #254856.  Thanks!!

Andrew Miller Featured Member Guru 662 posts since
Oct 7, 2008
Currently Being Moderated
Sep 27, 2009 8:53 PM in response to: aarondelp
Re: De-Dupe volume sizes

So....I'm horrendously late replying here....but you're right -- I didn't catch the reference to the 9xx line.

 

So...I'll bow to your technical superiority. (this time at least )

p.liniger Novice 5 posts since
Sep 16, 2009
Currently Being Moderated
Nov 6, 2009 4:49 AM in response to: Dave Lightman
Re: De-Dupe volume sizes

Hello,

 

Unfortunately I don't see any Information about the FAS-2040 in the "tr-3505" Document.

Can somebody tell me the dedup limit for the FAS-2040?

 

Thanks,

Pascal

radek.kubka Virtuoso 389 posts since
Jul 31, 2008
Currently Being Moderated
Nov 6, 2009 2:31 AM in response to: p.liniger
Re: De-Dupe volume sizes
Can somebody tell me the dedup limit for the FAS-2040?

 

That's a very good question. My (informed) guess is 3TB.

 

Here is why I think so:

http://now.netapp.com/NOW/knowledge/docs/ontap/rel732_vs/html/ontap/smg/GUID-7ED7077A-26E6-425A-B00E-E072FD7547F0.html


The title says "Operational limits for FlexClone files and FlexClone LUNs", but if you read carefully there is also this:

"Note: The block-sharing mechanism used by FlexClone files and LUNs is also used by deduplication."

 

Regards,

Radek

aarondelp Enthusiast 27 posts since
Nov 25, 2008
Currently Being Moderated
Nov 6, 2009 4:24 AM in response to: radek.kubka
Re: De-Dupe volume sizes

You are correct.  I have an e-mail from NetApp stating the 2040 limit is 3TB.

p.liniger Novice 5 posts since
Sep 16, 2009
Currently Being Moderated
Nov 6, 2009 4:29 AM in response to: aarondelp
Re: De-Dupe volume sizes

Ok, great to know.

Thanks for your help.

 

Regards,

Pascal

Andrew Miller Featured Member Guru 662 posts since
Oct 7, 2008

More Like This

  • Retrieving data ...

Bookmarked By (0)