We’ve come across a strange situation today, I was wondering if you could advise how best to deal with this.
We have a qtree which is very nearly full. It is the only qtree in the containing volume and there is no other data outside the qtree in that volume.
The containing volume has 197GB free:
Filesystem total used avail capacity Mounted on
/vol/xxxxxx_fsdata1/ 820GB 622GB 197GB 76% /vol/xxxxxx_fsdata1/
/vol/xxxxxx_fsdata1/.snapshot 502GB 355GB 147GB 71% /vol/xxxxxx_fsdata1/.snapshot
There is only one qtree in that volume (called Profiles). According to quota report there is 806GB used in this qtree:
vfilerxx@filerxx> quota report
Type ID Volume Tree Used Limit Used Limit Quota Specifier
----- -------- -------- -------- --------- --------- ------- ------- ---------------
tree * xxxxxx_fsdata1 - 0 - 0 - *
tree 1 xxxxxx_fsdata1 Profiles 845595168 - 12409822 - /vol/xxxxxx_fsdata1/Profiles
Note that there is no usage limits set on this qtree – it should be using free space in the volume. The /etc/quotas entry looks like this:
* tree@/vol/xxxxxx_fsdata1 - - - - -
The Profiles qtree is shared via CIFS. When connecting via a Windows client, it shows nearly full (eg 818GB used of 820GB):
I believe this difference is due to the dedupe savings. If I look at the volume usage including dedupe savings, it looks like this:
vfilerxx@filerxx> df -hs xxxxxx_fsdata1
Filesystem used saved %saved
/vol/xxxxxx_fsdata1/ 622GB 208GB 25%
…so I’m assuming the qtree used is accounting for effective used, rather than the actual used values.
The main area of concern here is that Operations Manager (version 5.0.1) has not generated any alerts for this condition. Looking at the Event History for the qtree I see no alerts at all. The quota nearly full and quota full alert global thresholds are set to defaults (ie 80% and 90%) and the quota object in Operations Manager does not have any custom alerts set. If I look at the quota summary in Operations Manager, it shows the Capacity Used as 99%, so why no alert?
Any help to understand this would be appreciated.
I think the problem is that you don't have a quota limit set on the qtree; since there is no quota limit, you can't be at 80 or 90 percent of it - though I'm curious what view is giving you the 99% figure. As I recall, dedupe savings are passed to the client, so the windows view should mirror what you see in the df (i.e., ~200G free) - not sure about that though.
I think the reason you aren't getting qtree full alerts is due to the fact that your quota entry has not limits. By default the values for qtree full and nearly full is 90 and 80 % respectively. Also qtree space utilization alerts are only triggered if quota are set on the qtree.
Though in your case quotas are since there are not limits applied in the quota file, the 80 and 90% couldn't be applied. I think that's the reason you don't get your alert.
Can you set a value and run dfm host discover to see if alerts are being generated for qtree ?
Also as you said, in case of qtree quota its the amount of space written and doesn't include dedupe space savings. So its the effective space used and not the actual( which includes dedupe space savings)
Thanks Bill and Adai,
Yesterday I enabled a tree quota on that qtree, and sure enough it generates alerts/alarms in DFM. All good. This makes sense. I'm now trying to figure out why the qtree was showing full from the client, however, and this I'm struggling with.
So, to recap, the volume has 197GB free, and only one qtree. The 'effective' used, accounting for dedupe savings meant the qtree was effectively full. That made sense in a way, until I decided to look as some other volumes which also have /etc/quota entries that apply no limits, (just there so I get capcity stats in DFM). Here's an example that appears to break the previous theory:
df -h xxxxxx_fsdep2
|Filesystem||total||used||avail capacity Mounted on|
df -hs xxxxxx_fsdep2
The effective used is 2931GB + 1112GB = 4043GB which is more than the total size of the volume, right?
This volume has 24 qtrees, which is the only significant difference I can see. /etc/quotas entry looks like the following, and I've confirmed quotas are on for this vol:
|*||tree@/vol/xxxxxx_fsdep2||- - - - -|
From a Windows CIFS client, I can mount one of the qtrees in this volume and Windows reports 318GB free of 3.17TB, which matches the df -h output, not the effective used accounting for dedupe.
Am I missing something?