21 Replies Latest reply: Mar 5, 2013 8:50 PM by adaikkap RSS

Oncommand deleted object still in the database

Francesco Duranti
Currently Being Moderated

Hi all, checking on DFM database I see that there are really old data into it that have the "deleted" flag setup but are still into the database. There are also really old objects that don't exist anymore and in my view can be safely deleted (old storage that don't exist anymore or old volumes).

I see that there are sometimes really many of those items.... doing a dfm volume list -a return 7700 objects (volumes) while doing a dfm volume list return only 700.

My question is: can I delete those objects or they're needed by DFM in some way? Should they be automatically cleaned by DFM after a time period or they'll remain in the database indefinitely?

 

This is an example of some aggregate objects:

 

1009 itromfs22:aggr1-2008-11-04 12:07:37.000-1 AggregateunknownYes  dfmmonitor
1010 itromfs22:aggr1-2008-11-04 12:07:37.000-1.snapshot AggregateunknownYes  dfmmonitor
8212 itromfs22:aggr1-2010-03-08 00:04:49.000-1 AggregateunknownYes  dfmmonitor
8213 itromfs22:aggr1-2010-03-08 00:04:49.000-1.snapshot AggregateunknownYes  dfmmonitor
22234 itromfs22:aggr1.snapshot             Aggregate32_bit No                

 

 

Regards

Francesco

  • Re: Oncommand deleted object still in the database
    arunchak
    Currently Being Moderated

    HI,

     

    They will remain in the database. For now we don’t have any mechanism where DFM deletes them after a certain period of time.

     

    Also see this post https://communities.netapp.com/message/69742#69742.

     

    Thanks,

      Arun

     

    From: c-xdl-communities

    Sent: Wednesday, January 18, 2012 11:49 PM

    To: Chakravarthy, Arun

    Subject: Re: Oncommand deleted object still in the database - Oncommand deleted object still in the database

     

    Re: Oncommand deleted object still in the database<https://communities.netapp.com/index.jspa>

     

     

    Oncommand deleted object still in the database

    created by Francesco Duranti<https://communities.netapp.com/people/f.duranti> in OnCommand Mgmt Software - View the full discussion<https://communities.netapp.com/message/72020#72020>

     

    Hi all, checking on DFM database I see that there are really old data into it that have the "deleted" flag setup but are still into the database. There are also really old objects that don't exist anymore and in my view can be safely deleted (old storage that don't exist anymore or old volumes).

     

    I see that there are sometimes really many of those items.... doing a dfm volume list -a return 7700 objects (volumes) while doing a dfm volume list return only 700.

     

    My question is: can I delete those objects or they're needed by DFM in some way? Should they be automatically cleaned by DFM after a time period or they'll remain in the database indefinitely?

     

     

     

    This is an example of some aggregate objects:

     

     

    1009 itromfs22:aggr1-2008-11-04 12:07:37.000-1 Aggregate

     

    unknown

     

    Yes

     

    dfmmonitor

     

    1010 itromfs22:aggr1-2008-11-04 12:07:37.000-1.snapshot Aggregate

     

    unknown

     

    Yes

     

    dfmmonitor

     

    8212 itromfs22:aggr1-2010-03-08 00:04:49.000-1 Aggregate

     

    unknown

     

    Yes

     

    dfmmonitor

     

    8213 itromfs22:aggr1-2010-03-08 00:04:49.000-1.snapshot Aggregate

     

    unknown

     

    Yes

     

    dfmmonitor

     

    22234 itromfs22:aggr1.snapshot

     

    Aggregate

     

    32_bit

     

    No

     

     

     

     

     

     

    Regards

     

    Francesco

     

    Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/72020#72020>

     

    Start a new discussion in OnCommand Mgmt Software by email<mailto:discussions-community-products_and_solutions-storage_management_software@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2026>

    • Re: Oncommand deleted object still in the database
      Gireesh Awasthi
      Currently Being Moderated

      Hi Francesco,

       

      When you delete any volume from dfm using "dfm volume delete <volume-name-or-id>" command then that volume doesn't delete permanently from dfm database, only dfm stop monitoring that volumes.

       

      So, the volumes those have deleted flag mark as "Yes", that means those volumes are deleted using "dfm volume delete <volume-name-or-id> command and dfm is not monitoring those volumes", so, dfm volume list -a command list those volumes because those volumes are not permanently deleted from dfm database.

       

      So, "dfm volume list -a"  command will list all the volumes present in dfm database while "dfm volume list" command will list only those volume for which Deleted flag marked as "No" and in other word, dfm volume list

      command will list only those volume which are monitor by dfm server.

       

      To delete the objects permanently from dfm database, please refer the post link that Arun mention in his post.

       

       

      Regards,

      Gireesh

    • Re: Oncommand deleted object still in the database
      Francesco Duranti
      Currently Being Moderated

      Just a question about it... it's safe to clear those "old" data related to volume/aggregate that don't exist anymore in the database using the "delete -f" or they are in some way used by DFM for some kind of historical report?

       

      arunchak ha scritto:

       

      HI,

       

      They will remain in the database. For now we don’t have any mechanism where DFM deletes them after a certain period of time.

       

      Also see this post https://communities.netapp.com/message/69742#69742.

       

      Thanks,

        Arun

       

      • Re: Oncommand deleted object still in the database
        arunchak
        Currently Being Moderated

        HI Fransisco,

         

        AFAIK they are not used by DFM. Deleted objects are not monitored so I don't think it has to do anything with reporting.

         

        I would like to tell you one scenario where I use the deleted object:

         

        Say, I have a LUN and I would want to stop monitoring it so would have deleted the LUN from dfm "dfm lun delete <id>". Now the LUN is marked deleted in DFM and gets listed under -a option.

         

        After a month, I want to monitor the LUN again and want to add the same object in DFM, then I do the following (use dfm lun add)

         

        [root@shoemaker-rhel6x64-01 ~]# dfm lun list -a | grep -i 1532

        1532 vserver01_arun:/lun_07_vol/lun07                                   Yes     (object deleted is YES)

         

        [root@shoemaker-rhel6x64-01 ~]# dfm lun add 1532 (Adding back the LUN to enable monitoring on it)

        Added LUN vserver01_arun:/lun_07_vol/lun07 (1532).

         

        [root@shoemaker-rhel6x64-01 ~]# dfm lun list  | grep -i 1532

        1532 vserver01_arun:/lun_07_vol/lun07                                   No (Added LUN is now showing as not deleted)

         

         

        [root@shoemaker-rhel6x64-01 ~]# dfm lun add help

         

         

        NAME

            add -- re-monitor deleted LUNs

         

         

        SYNOPSIS

            dfm lun add { all | <objects> ... }

         

         

        Hope this is useful.

         

        -Arun

      • Re: Oncommand deleted object still in the database
        Gireesh Awasthi
        Currently Being Moderated

        Hi Francesco,

         

        Once you delete any aggregate/Volume using -f option then all the history data for that volume or aggregate will get deleted and you can not retrieve that data  after this.

        So, this is not safe operation.

         

         

        Regards,

        Gireesh

      • Re: Oncommand deleted object still in the database
        kjag
        Currently Being Moderated

        The deleted objects are not at all used in any of the reports as its in deleted state. But you can't retrive the history data for the deleted objects anymore, once deleted using -f.

         

        If you really dont want those objects, you can delete them from DB.

         

        -KJag

  • Re: Oncommand deleted object still in the database
    kjag
    Currently Being Moderated

    Yes. You can delete the objects using "dfm volume delete -f <Volume Id>" command. You need to restart your dfm services after running this command.

     

    -KJag

  • Re: Oncommand deleted object still in the database
    Juerg Schneebeli
    Currently Being Moderated

    May this help

     

    Solution

    To avoid this behavior, when removing/deleting a volume from a dataset and DFM, remove it from the database as well. This is because DFM will assume that a volume deleted (in DFM) could be re-added later and could retain the historical data.
    Execute the following commands to delete the Protection Manager volume permanently:
    WARNING: Deleting Objects from DFM in this manner will remove them forever and will not retain historical data about the volume.
    1. dfm volume list -a (to get the IDs of the desired volumes)
    2. dfm service stop
    3. dfm service start sql
    4. dfm volume delete -f {ID of volume}
    5. dfm service start
  • Re: Oncommand deleted object still in the database
    adaikkap
    Currently Being Moderated

    Hi Francesco,

         What problem is it causing in your environment. This is internal to OnCommand, this should not affect any of the functionality pls let us know if its otherwise.

    Also as kjag said, its not used in any reports or in calculation.

     

    regards

    adai

    • Re: Oncommand deleted object still in the database
      Francesco Duranti
      Currently Being Moderated

      There's no real problem it's just that it seems that DFM server is a bit slow right now when I work with dataset and volumes and I was thinking it could be because of all the old objects not used anymore but that are in the database. The server is a vm with 8 gb of memory and 4 vcpu and sometimes it takes also a minutes to add volumes to a dataset. We have 8 storage controller defined and about 700 real volumes managed from them.

      • Re: Oncommand deleted object still in the database
        adaikkap
        Currently Being Moderated

        Is your DFM server also doing protection related activity ?

        what are the values of the following options ?

         

        [root@ ~]# dfbm options list statusUpdateInterval

        Option               Value

        -------------------- ------------------------------

        statusUpdateInterval 2 minutes

        [root@ ~]# dfbm options list purgeJobsOlderThan

        Option             Value

        ------------------ ------------------------------

        purgeJobsOlderThan 12.86 weeks

         

        Regards

        adai

        • Re: Oncommand deleted object still in the database
          Francesco Duranti
          Currently Being Moderated

          Yes, it's doing it. It seems I cannot find the first option ... probably i can lower the 12 weeks..

           

          [root@itromlx13 ~]# dfbm options list statusUpdateInterval

          Error: There is no statusUpdateInterval option.

          [root@itromlx13 ~]# dfbm options list purgeJobsOlderThan

          Option             Value                        

          ------------------ ------------------------------

          purgeJobsOlderThan 12.86 weeks

           

           

          [root@itromlx13 ~]# dfbm options list

          Option                       Value                        

          ---------------------------- ------------------------------

          backupDirMonInterval         8 hours

          backupScriptRunAs           

          discoverNdmp                 Disabled

          ndmpMonInterval              30 minutes

          purgeJobsOlderThan           12.86 weeks

          snapvaultLagErrorThreshold   1 day,  2:00

          snapvaultLagWarningThreshold 1 day,  1:00

          • Re: Oncommand deleted object still in the database
            adaikkap
            Currently Being Moderated

            Hi Francesco,

                 These two options will keep  the db from growing in size due to the job progress message and retention of old jobs.

            The first option needs to be set before able to be listed.

            dfbm options set statusUpdateInterval=5min

             

            This will populate the job progress message in protection jobs once every 5min instead of the default 30seconds.

             

            dfbm options list purgeJobsOlderThan

             

            This will purge all jobs older than the days specified. By this we will reduce the size of the db and you should get some better response times. Btw what version of dfm are you running ?

             

            Regards

            adai

            • Re: Oncommand deleted object still in the database
              Francesco Duranti
              Currently Being Moderated

              Thanks adai it's a good information. 5 minutes seems a good interval to put on instead of 30 seconds but does this have any impact? I've Snapcreator using pm to do remote backup and I've saw that it checks the status every minute, will it find that the mirror finished only 5 minutes after (in the worst case) or the job completion is reported directly?

              I'm running 5.0d1 and I've just now discovered that dfm database backup is failing from 2 month (never checked it unfortunately) because the validatedb after backup is going wrong and it seems something related ti Host Services.

              Replicating the dbvalid command that dfm use I got this massage (that don't appear in the dbbackup.log:

               

              VALIDATE TABLE "DFM"."hsNotifications"

              Primary key for "hsNotifications" has missing index entries

              Foreign key "hostServiceId" has missing index entries

              SQL error (-300) -- Run time SQL error -- Validation of table "hsNotifications" has failed

               

              Any ideas? I've opened a support call but if someone had the same problem it will be nice to solve it faster:)

              Francesco

              • Re: Oncommand deleted object still in the database
                adaikkap
                Currently Being Moderated

                Hi Francesco,

                                  The 5minutes is only for progress events, which gives information like snapmirror.progress and snapvault.progress and not for other messages of jobs. Basically this give how much data is transferred.

                 

                Can you get the output of dfm backup create -v ?

                 

                Regards

                adai

                • Re: Oncommand deleted object still in the database
                  Francesco Duranti
                  Currently Being Moderated

                  Those are the logs:

                   

                  [root@itromlx13 ~]# dfm backup create -v

                  Disabling Performance Advisor.

                  DestinationPath = /opt/dfm_backup/

                  TempPath        = /opt/dfm_backup/dfm_temp_dest_dir/

                  InstallPath     = /opt/NTAPdfm/

                  DataPath        = /opt/NTAPdfm/data/

                  LogPath         = /opt/NTAPdfm/data/

                  KeysPath        = /opt/NTAPdfm/conf/keys/

                  PerfPath        = /opt/NTAPdfm/perfdata/

                  ScriptPath      = /opt/NTAPdfm/script-plugins/

                  RptPath        = /opt/NTAPdfm/reports/repository/custom/

                  DBFileName      = monitordb.db

                  LogFileName     = monitordb.log

                  KeysFileName    = dfmenc.keys

                  ArchivePath     = /opt/dfm_backup/dfm_backup_2012-01-23_17-25-05.ndb

                  ArchiveTempName = tmp.backup.file.ndb

                   

                  strDBFilePath= /opt/dfm_backup/dfm_temp_dest_dir/monitordb.db

                   

                  strDBLogFilePath= /opt/dfm_backup/dfm_temp_dest_dir/monitordb.log

                   

                  strKeysFilePath= /opt/dfm_backup/dfm_temp_dest_dir/dfmenc.keys

                  Copying keystore for backup ...

                  SQL Anywhere Backup Utility Version 10.0.1.4213

                  (107134 of 107120 pages, 100% complete)

                  Database backup completed

                  Enabling Performance Advisor.

                  Error: Database validation returned errors.

                  Please see dfmbackup.log file for details.

                   

                  This is the dfmbackup.log:

                  Jan 23 17:25:32 [dfm: INFO]: [23221:0x2b8da1a67080]: dbeng9 process spawned with pid = 23377.

                  Jan 23 17:25:32 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23378.

                  I. 01/23 17:25:32. SQL Anywhere Personal Server Version 10.0.1.4213

                  I. 01/23 17:25:32.

                  I. 01/23 17:25:32. Copyright (c) 2011, iAnywhere Solutions, Inc.

                  I. 01/23 17:25:32. Portions copyright (c) 2011, Sybase, Inc.

                  I. 01/23 17:25:32. All rights reserved. All unpublished rights reserved.

                  I. 01/23 17:25:32. 

                  I. 01/23 17:25:32. This software contains confidential and trade secret information of

                  I. 01/23 17:25:32. iAnywhere Solutions, Inc.

                  I. 01/23 17:25:32. Use, duplication or disclosure of the software and documentation

                  I. 01/23 17:25:32. by the U.S. Government is subject to restrictions set forth in a license

                  I. 01/23 17:25:32. agreement between the Government and iAnywhere Solutions, Inc. or

                  I. 01/23 17:25:32. other written agreement specifying the Government's rights to use the

                  I. 01/23 17:25:32. software and any applicable FAR provisions, for example, FAR 52.227-19.

                  I. 01/23 17:25:32. 

                  I. 01/23 17:25:32. iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA

                  I. 01/23 17:25:32.

                  I. 01/23 17:25:33. 4 logical processor(s) on 2 physical processor(s) detected.

                  I. 01/23 17:25:33. This server is licensed to:

                  I. 01/23 17:25:33.     DFM User

                  I. 01/23 17:25:33.     NetApp

                  I. 01/23 17:25:33. Running Linux 2.6.18-274.7.1.el5 #1 SMP Mon Oct 17 11:57:14 EDT 2011 on X86_64

                  I. 01/23 17:25:33. Server built for X86_64 processor architecture

                  I. 01/23 17:25:33. Authenticated Server licensed for use with Authenticated Applications only

                  I. 01/23 17:25:33. 904832K of memory used for caching

                  I. 01/23 17:25:33. Minimum cache size: 8192K, maximum cache size: 4087256K

                  I. 01/23 17:25:33. Using a maximum page size of 8192 bytes

                  I. 01/23 17:25:33. Starting database "dfmBackupDbvalidDatabase" (/opt/dfm_backup/dfm_temp_dest_dir//monitordb.db) at Mon Jan 23 2012 17:25

                  I. 01/23 17:25:33. Database recovery in progress

                  I. 01/23 17:25:33.     Last checkpoint at Mon Jan 23 2012 17:25

                  I. 01/23 17:25:33.     Checkpoint log...

                  Jan 23 17:25:33 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23394.

                  Jan 23 17:25:34 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23397.

                  I. 01/23 17:25:35.     Transaction log: /opt/dfm_backup/dfm_temp_dest_dir//monitordb.log...

                  Jan 23 17:25:35 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23400.

                  Jan 23 17:25:36 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23407.

                  I. 01/23 17:25:37.     Rollback log...

                  I. 01/23 17:25:37.     Checkpointing...

                  I. 01/23 17:25:37. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:25

                  I. 01/23 17:25:37. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:25

                  I. 01/23 17:25:37. Recovery complete

                  I. 01/23 17:25:37. Database "dfmBackupDbvalidDatabase" (monitordb.db) started at Mon Jan 23 2012 17:25

                  I. 01/23 17:25:37. This database is licensed for use with:

                  I. 01/23 17:25:37.     Application: DataFabric Manager

                  I. 01/23 17:25:37.     Company: NetApp

                  I. 01/23 17:25:37. Database server started at Mon Jan 23 2012 17:25

                  I. 01/23 17:25:37. Trying to start SharedMemory link ...

                  I. 01/23 17:25:37.     SharedMemory link started successfully

                  I. 01/23 17:25:37. Trying to start TDS (TCPIP) link ...

                  I. 01/23 17:25:37. Unable to start on default port; starting on port 49152 instead

                  I. 01/23 17:25:37.     TDS (TCPIP) link started successfully

                  I. 01/23 17:25:37. Now accepting requests

                  Jan 23 17:25:38 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23433.

                  I. 01/23 17:25:38. Cache size adjusted to 849288K

                  I. 01/23 17:25:42. Cache size adjusted to 836256K

                  I. 01/23 17:25:42. Cache size adjusted to 833672K

                  I. 01/23 17:25:48. Cache size adjusted to 833064K

                  Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

                  SQL Anywhere Stop Server Utility Version 10.0.1.4213

                  I. 01/23 17:30:46. Database server shutdown requested by DBSTOP

                  I. 01/23 17:30:46. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:30

                  I. 01/23 17:30:46. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:30

                  Jan 23 17:30:46 [dfm: INFO]: [23221:0x2b8da1a67080]: dbstop process returned successfully.

                  Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

                  Please see dfmbackup.log file for details.

                  Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

                  Please see dfmbackup.log file for details.

                  I. 01/23 17:30:46. Database server stopped at Mon Jan 23 2012 17:30

                  Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database backup create failed

  • Re: Oncommand deleted object still in the database
    adaikkap
    Currently Being Moderated

    We have this purge tool today that take care of cleanup all this. Dfmpurge which will remove all these stale instances but requires down time. The utility has 2 modes and gives the estimation of downtime required as well. In most cases it shouldn't take more than 30mins to cleanup.

    Pls take a look at this video ( 3.43 Mins) and read the KB on how this tool works.

    Video Link: DFM Purge Tool: How to Video

    KB link:  https://kb.netapp.com/support/index?page=content&id=1014077

    Link to tool chest: http://support.netapp.com/NOW/download/tools/dfmpurge/

    Regards

    adai

     

More Like This

  • Retrieving data ...