2 Replies Latest reply: Aug 21, 2012 8:49 AM by casey.condo RSS

Is PAM II working optimally?

casey.condo
Currently Being Moderated

We have a 512GB PAM II card

 

and when I run a stats show -p flexscale-pcs it shows a high % (92%) usage, a low Hit % (9%) but it doesnt show a high evict or insert rate.

 

But when I run a sysstat -u 1 I get a cache hit rate (94%) but my disk utilization is high (91%)

 

We run VMs on this server.  My question: is my system cache maninly being used a very little making it to the PAM ?  It shows a lot of information in PAM but the hit rate is very low.

 

Or is this to be expected because the VMs are large files etc?

  • Re: Is PAM II working optimally?
    GARDINEC_EBRD
    Currently Being Moderated

    Hi Casey,

     

    Here's some info from memory...

     

    Firstly, the cache hit rate shown in sysstat output isn't showing the same thing as the flexscale stats.  Ignore the sysstat one for now...

     

    With a low hit rate you should look into what type of data you are missing, and what you are caching.  If you use stats show ext_cache_obj this will give you more info.  Better still run it over a few minutes using stats start -I cache ext_cache_obj  ...wait for a minute or two during a busy period... stats stop -I cache.

     

    This will give an output like this (notes added for some counters to check):

     

    StatisticsID: cache

    ext_cache_obj:ec0:type:IOMEM-FLASH

    ext_cache_obj:ec0:blocks:67108864

    ext_cache_obj:ec0:size:256  <------- size of cache in GB

    ext_cache_obj:ec0:usage:92%  <------- How full the cache is (should be >90% unless you've just booted)

    ext_cache_obj:ec0:accesses:32047/s  <----- number of times the cache was accessed per sec

    ext_cache_obj:ec0:disk_reads_replaced:3768/s   <---- how many disk reads were not replaced by the cache.

    ext_cache_obj:ec0:hit:17898/s  <---- ave number of hits per sec (add to the number of misses below to get total)

    ext_cache_obj:ec0:hit_normal_lev0:16517/s  <---- this is 'normal' data hits.  here you can see most of the hits are 'normal' data

    ext_cache_obj:ec0:hit_metadata_file:907/s  <--- this and the next few are the types of hit

    ext_cache_obj:ec0:hit_directory:42/s

    ext_cache_obj:ec0:hit_indirect:435/s

    ext_cache_obj:ec0:total_metadata_hits:1385/s

    ext_cache_obj:ec0:total_metadata_misses:434/s  <--- if this is high relative to the total operations, you should consider metadata cache mode

    ext_cache_obj:ec0:miss:14149/s  <----  number of misses per sec

    ext_cache_obj:ec0:miss_metadata_file:85/s  <---- this and the next few are the types of miss

    ext_cache_obj:ec0:miss_directory:1/s

    ext_cache_obj:ec0:miss_indirect:347/s

    ext_cache_obj:ec0:hit_percent:55%  <--- % hits (hits / miss)

    ext_cache_obj:ec0:inserts:426/s

    ext_cache_obj:ec0:inserts_normal_lev0:337/s

    ext_cache_obj:ec0:inserts_metadata_file:49/s

    ext_cache_obj:ec0:inserts_directory:6/s

    ext_cache_obj:ec0:inserts_indirect:39/s

    ext_cache_obj:ec0:evicts:11/s  <---- this indicates the number of blocks evicted (ie not accesses frequently)  If this is very high you *may* need more flash caches

    ext_cache_obj:ec0:evicts_ref:6/s

    ext_cache_obj:ec0:readio_solitary:1524/s

    ext_cache_obj:ec0:readio_chains:3768/s

    ext_cache_obj:ec0:readio_blocks:15788/s

    ext_cache_obj:ec0:readio_max_in_flight:511

    ext_cache_obj:ec0:readio_avg_chainlength:4.19

    ext_cache_obj:ec0:readio_avg_latency:0.56ms

    ext_cache_obj:ec0:writeio_solitary:0/s

    ext_cache_obj:ec0:writeio_chains:6/s

    ext_cache_obj:ec0:writeio_blocks:426/s

    ext_cache_obj:ec0:writeio_max_in_flight:182

    ext_cache_obj:ec0:writeio_avg_chainlength:64.00

    ext_cache_obj:ec0:writeio_avg_latency:2.14ms

    ext_cache_obj:ec0:invalidates:514/s  <--- data in the cache that's been overwritten on disk (and thus 'invalidated')

     

    Based on this you may need to adjust your cache settings (options flexscale) to cache different data. 

     

    You should also check you're VM alignment - misaligned blocks may affect caching efficiency.  If that looks OK, dedupe may also help.

     

    If you have Performance Advisor you can also collect these stats and review regularly if required.

     

    Hope that helps,

     

    Craig

More Like This

  • Retrieving data ...