Hello Jan-Jacob,

That's an excellent question, hopefully I can shed some light on it here.

Being that you're using a 3020 and 300GB FC disks, in order to meet the maximum capacity of a single aggregate, your ideal RG size would be 15

With these particular disks (300GB FC disks) you can have a maximum of 59 disks allocated to the aggregate, leaving you with room to grow into the final drives to fill out this aggregate.     While a RG size of 16 would also work, you'll get the best performance and space utilization by establishing an RG of 15.

I hope this was able to address your question and help you complete your storage design.

Thanks,

Christopher

Dear Christopher,

Just curious to know how you came up with your answer... based on the 42 disks that are availabe to Jan-Jacob, wouldn't it be more advisable to choose RG size of 14 disks to have them even more equally divided?

Suppose 1 disk is used as hotspare, your offered solution (RG size of 15 disks) yields:

2x (13 spindels + 2 parity disks) +

1x (9 spindels + 2 parity disks) +

1 hotspare

A 14 disks RG size yields:

2x (12 spindels + 2 parity disks) +

1x (11 spindels + 2 parity disks) +

1 hotspare

Since performance is limited by slowest set, the equally devided solution would give a better performance.

Can you please provide the reasoning behind your answer, hopefully increasing my insight in the NetApp setup  :-)

Dear Christopher,

Thank you very much for your reply, explaining the reasoning behind your recommendation.

Based on the reply I start running into more fundamental questions of sizing myself - probably a reference to the capacity documents mentioned in your post could help me with that.

When trying to calculate the maximum aggregate size I am still having a hard time deriving the 59 disks maximum that you mentioned. Once I get that I can understand your recommended RGsize of 15 disks since that would optimize capacity/performance for 59 disks in total, effectively having 4 raidgroups of (almost) equal size.

Some of the questions I find myself asking now:

- the 16 TB limit for an aggregate seems to include all disks (raw capacity), including parity disks?

- when calculating optimized sizes what GB's should be used, 1000/1024 based?

We are currently in the process of expanding a NetApp storage system and are looking into optimized setup, both now and in the future - yes, I tend to agree that we will in the end grow to the maximum size as well  :-)

Would it be possible to provide some pointers in sizing/capacity documentation to get us started? Also, a brief explanation on how you derived the 59 disks maximum for a 300 GB disk size would be much appreciated.

Kind regards,

Mark.

This table is pretty confusing as it doesn't show the parity disks at all.

The SATA numbers also don't add up, and I've never understood fully the SATA aggregate limits...

68x 250 = 17000

53x 320 = 16960

33x 500 = 16500

22x 750 = 16500

15x 1000 = 15000

Obviously all except the TB disks are well over the aggregate limit.

With FC disks are worked out on RAW including the parity disks, but SATA it doesn't seem so...

Back to the topic a little more. Are there any stats to show the performance improvements by having similar RAID group sizes across an entire aggregate? It'd be useful to backup the reasoning for changing the RAID group defaults.

Kevin, you hit it right on the head - regarding 7.3

During the original post I had promised to try to provide as much citeable evidence as possible as to why certain RG's would be more ideal than another (outside of the default best practice) in order to achieve maximum space while providing for maximum performance.   Every bit of public collateral I can find (such as that 7.3 based table) I wanted to ensure was available and in a consumable format.

As for the performance of similar raid group sizes also, best practice dictates that RG's should have at a minimum half as many disks as the RG size (So for a 14 disk RG you're looking at 7 disks minimum preferable) in order to avoid running into any kind of hot-disk scenarios and an immediate need to reallocate

Ofcourse as always, I prefer citable evidence as to the impacts, so until that point I'll continue setting preference for my min-half..Full philosophy - until I hear otherwise. :)

Thanks for your input and feedback on this Kevin!

Christopher

Sorry, the table gets a bit messed up as a post, I'll attach the excel spreadsheet instead...

Performance.

If you have an aggregate with 32 disks (albeit in (2) 16 disk RAID groups), your writes (and possibly reads) will be spread across many more spindles.

Not using large aggregates actually defeats one of the primary NetApp benefits and gets you more back storage management a la EMC.

Having large pools of disk with good performance visibility (statit or Performance Advisor....goes down to the aggregate/volume/disk) as well as prioritization (FlexShare) is a good bit of the NetApp magic sauce.

If you just have a single write, sure (a single write will actually just go to a single disk not even a single RAID group). But....all writes get cached in NVRAM and then flushed to disk (i.e. as many writes as all the disks can handle). So....the more disks you have (even if in multiple RAID groups), the faster your writes are as they do get spread across all the disks.

Similarly, there can be more reads as well for the same reason essentially.

Depending on if the ONTAP version is 7.2.x or 7.3.x, here is what I have been doing for 1TB drives...based on the maximum drives supported per aggregate for each ontap version.

7.2.x     19 total, disks supported (15 data max per aggr)  [ (RG0: 8D+2P) +   (RG1: 7D+2P) ]

7.3.x     23 total disks supported (19 data max per aggr)   [ (RG0: 10D+2P) + (RG1: 9D+2P) ]'

Since we need two raid groups to max the aggregate size (4 parity disks needed to get to max size), it's best to keep the raid groups as even as possible.. for example, with 7.3 using the default raid group size, you could go with rg0: 14D+2P, then rg1: 5D+2P - but that wouldn't be as good as 10D+2P and 9D+2P over two raid groups...

The trick with ONTAP 7.2 to do this is... create a 10 drive aggregate.. change the raidsize option to 10, then add 9 more drives to max it out..changing to 10 will force a new raid group.  For example:

• aggr create aggrname 10
• aggr options aggrname raidsize 10
• aggr add aggrname 9

Similarly, you can do the same with ONTAP 7.3 by creating with 12 drives, then change the raidsize option to 12, then add 11 more drives to max it out..  Additionally, 7.3 added a new option to create a new raid group with "-g new"

• aggr create aggrname 12
• aggr options aggrname raidsize 12
• aggr add aggrname 11

OR

• aggr create aggrname 12
• aggr add -g new aggrname 11   #-g forces a new raid group with the next 11 drives... prior to 7.3 -g is supported but only for existing raid groups..explicity use "new" as the raid group name and it will create a new raid group

The more I read this thread, the more confusing it becomes.

Let me stick to PRE-7.3 for a second.

• The excel table I saw above says optimal RAID group size for a 1TB SATA Drive is 17. First off, for SATA, the max RAID group size allowed is 16. So 17 is not even a workable option.
  

• Aggregate Size. From my readings, work and understanding, the total size of an aggregate was computed by adding together the raw size of all disks in the aggregate, including parity disks, and regardless of the amount of disk space available to be used in each data disk. What I am now confused about now is the use of the term "raw size". I believe it is the "physical capacity" might be the correct term. More specifically, it is the output from the "sysconfig -r" command from under the "Phys (MB/blks)" column.
  
  
  For e.g. for a 300GB FC Disk Drive, the Physical Size is 274845 MB = 268.40 GB. So for an aggregate limit of 16TB, the total # of drives = 16 * 1024 / 268.40 = 61 drives (parity and data together). 
  
  Am I correct ?
  

Ah....right....I should have remembered the emphasis on data disks.

For anyone following along, in my example above, with 7.3 I could now have 45 disks total (i.e. 3 RAID groups with (15) 500 GB drives in each -- that would be 39 data disks and 6 parity/dual parity disks).

There aren't any aggregate size limitations on the 2050 (only on the 2020 pre-7.3.1). But.....the 2050 with internal disk is a bit of an odd beast....multiple factors to consider here.

-You can put FC & SAS disk into the same aggregate (and raid group even)....but if you do so, you can't put the FC tray onto another filer easily (i.e. future upgrade).

-Although FC & SAS can go into the same aggregate/raid group, the spares are separate.

-I generally like having 2 hot spares of each kind of disk type as that enables "Disk Maintenance Center".

So....what I'd probably do with a FAS2050A (i.e. active-active controllers) would be....

-1 2050 head controlling the external FC tray (i.e. 14 disks) via software disk ownership

-- 12 disk aggregate with one RAID group for 2.26 TB usable (no snapshot reserve, aggr snap reserve at 3%)

-- 2 spares

-- root volume as a 20 GB FlexVol

-- rest of space for whichever volumes you like

-1 2050 head controlling the internal SAS drives (i.e. 20 disks) via software ownership

-- 18 disk aggregate with one RAID group for 3.62 TB usable (no snapshot reserve, aggr snap reserve at 3%)

-- 2 spares

-- root volume as a 20 GB FlexVol

-- rest of space for whichever volumes you like

This gets you the ability to move the external FC to another filer head easily in the future as well as maximizes spaces/speed by keeping the # of aggregates to the minimum (more spindles = good).

Since you'll only have one external FC loop, you'll have a spare FC port on each 2050 (presuming no FC clients) so could even do MPHA as well. That clusterd.pdf is incredibly old....stops at the FAS9xx line which was dropped 3 years back IIRC.

And....general forum etiquette would be that sticking this in a new thread would be best. (You could put a short reply in this thread with a link asking people to go check out your new thread for instance.)

Message was edited by: Andrew Miller

Quite welcome....and precisely right (as you found). I just really like MPHA for anything past a very small configuration....doesn't cost all that much if the FC ports are available. That article also confirms the 2 disk spare thing as well.

Not really an exact science...just what I've picked up over the last 10 years or so online. It also makes it easier for people searching to find specific pieces when they're split out logically into separate threads (rather than a huge monster thread).

And....I don't know that my configuration is what everyone would do (although I like it ;). I'm curious to hear how others might set it up.

Quite intentional because....

-you have to have 2 RAID groups at least (since FC/SAS max raid group size is 28 disk). They could be in the same aggregate but....

-if you have a 2050A (i.e. active-active cluster), you need two aggregates (one for each head)

-if you just have a 2050 (single head), having 2 aggregates makes it easier to upgrade it to a 2050A later

-by keeping the aggregates separate between the SAS and the FC disk, it makes it easy in the future to move the FC-based aggregate to another storage head (since with NetApp people often keep disk for 6+ years the disk usually lives through a head upgrade or two).

If you just have a single 2050, you could do one large aggregate.....but you'd have to add disk if you made it a 2050A later (or destroy the aggregate to split it up)