22 Replies Latest reply: Mar 18, 2014 3:34 PM by pburcin@ibm RSS

Broken? snapmirrors - how to delete them ?

hardy.maxa
Currently Being Moderated

Hi all

 

On my filer, I was doing some backup testing, and now have these snapmirrors showing, which I can't delete

 

aushafs2b> snapmirror status

Snapmirror is off.

Source                 Destination                  State          Lag        Status

aushafs1b:admin        aushafs2b:admin              Snapmirrored   02:35:19   Idle

aushafs1b:catiax       aushafs2b:catiax             Snapmirrored   10:35:19   Idle

aushafs1b:pdm          aushafs2b:pdm                Snapmirrored   10:05:18   Idle

aushafs1b:zen          aushafs2b:zen                Snapmirrored   09:55:20   Idle

aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

 

The first 4 entries are fine - they are my mirrors between systems.

But I want to cleanup the last three.

 

As far as I can tell, there are no backup or smtape sessions still active.

Thanks,

Hardy

  • Re: Broken? snapmirrors - how to delete them ?
    aborzenkov
    Currently Being Moderated

    Use "snapmirror release" to remove old SnapMirror snapshots. You may also need to remove them from snapmirror.conf.

  • Re: Broken? snapmirrors - how to delete them ?
    YEKYAWTHARMYINT
    Currently Being Moderated

    Use snapmirror release srcpath dst_filer:dstpath command to release the snapmirror relationship from source filer.

  • Re: Broken? snapmirrors - how to delete them ?
    lhoffman1
    Currently Being Moderated

    Can you show us the output from  "snap list"  admin / images and domino2_vol  ?

     

    If you have no orphaned records within the  "snapmirror.conf" file,  you have some relations within old snapshots.

     

    btw.  I wonder,  your snapmirror is set to off,  although your idletime is not so old.

    • Re: Broken? snapmirrors - how to delete them ?
      hardy.maxa
      Currently Being Moderated

      Snap List output:

       

      Volume images
        %/used       %/total  date          name
      ----------  ----------  ------------  --------
        0% ( 0%)    0% ( 0%)  Feb 03 00:00  weekly.0
        0% ( 0%)    0% ( 0%)  Dec 09 12:03  aushafs2b(1785237204)_images.1109

       

      Volume admin
        %/used       %/total  date          name
      ----------  ----------  ------------  --------
        0% ( 0%)    0% ( 0%)  Feb 05 08:00  aushafs2b(1785237204)_admin.1792
        0% ( 0%)    0% ( 0%)  Feb 05 00:00  nightly.0
        0% ( 0%)    0% ( 0%)  Feb 05 00:00  aushafs2b(1785237204)_admin.1791
        0% ( 0%)    0% ( 0%)  Feb 04 19:00  hourly.0
        0% ( 0%)    0% ( 0%)  Feb 04 15:00  hourly.1
        0% ( 0%)    0% ( 0%)  Feb 04 13:00  hourly.2
        0% ( 0%)    0% ( 0%)  Feb 04 11:00  hourly.3
        0% ( 0%)    0% ( 0%)  Feb 04 09:00  hourly.4
        0% ( 0%)    0% ( 0%)  Feb 04 00:00  nightly.1
        0% ( 0%)    0% ( 0%)  Feb 03 19:00  hourly.5
        0% ( 0%)    0% ( 0%)  Feb 03 00:00  weekly.0
        0% ( 0%)    0% ( 0%)  Feb 02 00:00  nightly.2
        0% ( 0%)    0% ( 0%)  Feb 01 00:00  nightly.3
        0% ( 0%)    0% ( 0%)  Jan 31 00:00  nightly.4
        0% ( 0%)    0% ( 0%)  Jan 30 00:00  nightly.5
        0% ( 0%)    0% ( 0%)  Jan 29 00:00  nightly.6
        0% ( 0%)    0% ( 0%)  Jan 27 00:00  weekly.1
        0% ( 0%)    0% ( 0%)  Jan 20 00:00  weekly.2

       

      Volume domino2_vol
      working...

      No snapshots exist.

       

      I think snapmirror was set off when I noticed the issue.

      Now though, it is on.

      aushafs2b> snapmirror status

      Snapmirror is on.

      Source                 Destination                  State          Lag        Status

      aushafs1b:admin        aushafs2b:admin              Snapmirrored   00:19:40   Idle

      aushafs1b:catiax       aushafs2b:catiax             Snapmirrored   08:19:37   Idle

      aushafs1b:pdm          aushafs2b:pdm                Snapmirrored   07:49:37   Idle

      aushafs1b:zen          aushafs2b:zen                Snapmirrored   07:39:39   Idle

      aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

      aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

      aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

      aushafs2b>

       

      snapmirror.conf looks fine too

       

      aushafs2b> rdfile /etc/snapmirror.conf

      #Regenerated by registry Wed Jan 15 00:39:45 GMT 2014

      aushafs1b:catiax aushafs2b:catiax - 0 0,12 * *

      aushafs1b:admin aushafs2b:admin - 0 0,8,16 * *

      aushafs1b:pdm aushafs2b:pdm - 30 0,12 * *

      aushafs1b:zen aushafs2b:zen - 40 0,12 * *

       

      Any other ideas ?

  • Re: Broken? snapmirrors - how to delete them ?
    SHARMAMANISH
    Currently Being Moderated

    Can you send me the output of  options snapmirror.enable

    • Re: Broken? snapmirrors - how to delete them ?
      hardy.maxa
      Currently Being Moderated

      Here's all the snapmirror options

      aushafs2b> options snapmirror

      snapmirror.access            legacy

      snapmirror.checkip.enable    off

      snapmirror.delayed_acks.enable on

      snapmirror.enable            on

      snapmirror.log.enable        on

      snapmirror.vbn_log_enable    off        (value might be overwritten in takeover)

      snapmirror.volume.local_nwk_bypass.enable on

      aushafs2b>

      • Re: Broken? snapmirrors - how to delete them ?
        lhoffman1
        Currently Being Moderated

        again:   pls output the command:

         

        "snapmirror status -l aushafs2b:domino2_vol  aushafs2b:nrst1a"

          

        thx


        • Re: Broken? snapmirrors - how to delete them ?
          hardy.maxa
          Currently Being Moderated

          aushafs2b> snapmirror status -l aushafs2b:domino2_vol  aushafs2b:nrst1a
          Snapmirror is on.

          Source:                 aushafs2b:domino2_vol
          Destination:            aushafs2b:nrst1a
          Status:                 Idle
          Progress:               -
          State:                  Source
          Lag:                    -
          Mirror Timestamp:       -
          Base Snapshot:          -
          Current Transfer Type:  Store
          Current Transfer Error: -
          Contents:               -
          Last Transfer Type:     -
          Last Transfer Size:     -
          Last Transfer Duration: -
          Last Transfer From:     -
          aushafs2b>

           

          Any ideas ?

          Thx

          • Re: Broken? snapmirrors - how to delete them ?
            ANANDONTAP
            Currently Being Moderated

            hi all,

             

            go to designation end follow the below steps,

             

            snapmirror status-- check it is snapmirror on status (continue follow steps)

             

            which u want to remove entry in this output ensure.

             

            snapmirror break designation volume

             

            goto ---> rdfile /etc/snapmirror.conf --take this output in notepad

             

            in notepad output--> which line of snapmirror you want to remove ensure and put # comment symbol.

             

            wrfile /etc/snapmirror.conf--> paste the commented output and ctrl+c to save and exit

             

            now u can run snapmirror status all should be clear.

            • Re: Broken? snapmirrors - how to delete them ?
              hardy.maxa
              Currently Being Moderated

              Thanks, but this doesn't help either

               

              aushafs2b> snapmirror break aushafs2b:nrst1a

              snapmirror break:  nrst1a: destination is offline, is restricted, or does not exist

               

              aushafs2b> rdfile /etc/snapmirror.conf

              #Regenerated by registry Wed Jan 15 00:39:45 GMT 2014

              aushafs1b:catiax aushafs2b:catiax - 0 0,12 * *

              aushafs1b:admin aushafs2b:admin - 0 0,8,16 * *

              aushafs1b:pdm aushafs2b:pdm - 30 0,12 * *

              aushafs1b:zen aushafs2b:zen - 40 0,12 * *

              aushafs2b>

               

              The entries are not in the conf file, but still show in the status

               

              Source                 Destination                  State          Lag        Status

              aushafs1b:admin        aushafs2b:admin              Snapmirrored   05:36:54   Idle

              aushafs1b:catiax       aushafs2b:catiax             Snapmirrored   01:36:58   Idle

              aushafs1b:pdm          aushafs2b:pdm                Snapmirrored   01:06:59   Idle

              aushafs1b:zen          aushafs2b:zen                Snapmirrored   00:56:59   Idle

              aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

              aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

              aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

              aushafs2b>

               

              I know its not windows, but what about a reboot ?

              • Re: Broken? snapmirrors - how to delete them ?
                YEKYAWTHARMYINT
                Currently Being Moderated

                Hi Hardy,

                 

                Have you already tried snapmirror release from aushafs2b?

              • Re: Broken? snapmirrors - how to delete them ?
                lhoffman1
                Currently Being Moderated

                No,  a reboot doesn't help.   I think the problem is on Backup Server side,  not filer.

                 

                But..  you are set the mirror from aushafs1b  to aushafs2b  .

                 

                What about the snapmirror status from  aushafs1b  ?   and  the  "snap list admin"   by aushafs1b  ??


              • Re: Broken? snapmirrors - how to delete them ?
                ANANDONTAP
                Currently Being Moderated

                hi,

                 

                aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

                aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

                aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

                 

                this should be source. please follow my steps in designation end.

                 

                can please share the below output from dst end.

                 

                snapmirror status

                 

                df -Vh nrst1a

                 

                lun show -l nrst1a

                 

                rdfile /etc/snapmirror.conf

                 

                lun show unmapped

                 

                one more Question this replication within filer or across the filer?

                • Re: Broken? snapmirrors - how to delete them ?
                  pburcin@ibm
                  Currently Being Moderated

                  We are having the same issue as Max Hardy.  Our problem is related to the use of TSM to create NFS volume backups.  TSM is creating and starting the smtape jobs.  We currently have more than 220 orphaned smtape jobs on the controllers where the destination is an NDMP_local output like in Max Hardy's original post.  If there is no snapmirror destination to "snapmirror release", then it appears that there is no way to remove these orphans in the snapmirror status list.  They are not found in the snapmirror.conf file because they are supposed to be temporary associations and should go away when the TSM backup has completed.  We currently have a case open because one of our controller pairs is experiencing filer panics, crashing the controllers.  We have been told that root cause is smtape exhausting resources.  The crash and subsequent reboot did not remove the 220 orphaned entries from the snapmirror status list.   I am left believing that the orphaned NDMP_local snapmirrors that cannot be removed, released or otherwise deleted are the root cause of the filer crashes.  Perhaps we have all found a bug in the smtape engine.  We are still waiting for an answer from support on how to remove the orphans. 

                  • Re: Broken? snapmirrors - how to delete them ?
                    hardy.maxa
                    Currently Being Moderated

                    Hi pburcin

                     

                    I look forward to hearing Netapp's reply.  I havn't gone that far yet - I stopped using the smtape backups partly because of this issue.

                    • Re: Broken? snapmirrors - how to delete them ?
                      pburcin@ibm
                      Currently Being Moderated

                      The official response from support is that these "orphaned" smtape jobs that show up in the snapmirror status list can be removed by using "snapmirror release <volname>  <destination controller:NDMP_local_xxxxxx>".  You WILL receive an error that indicates there is "no release-able destination".  However, the orphaned job that you released DOES get removed from the snapmirror status list.  In my case, I have hundreds of these orphaned smtape jobs to remove across numerous controllers running Data ONTAP 8.1.2x.  So the manual removal process will be tedious.  Basically, that was my reply to support.  There's gotta be a better way to fix this permanently!!!  We should not have to watch this and remove the orphans manually on a regular basis.

                       

                      So this is not a fix.  It is the way to remove the orphaned smtape jobs from the snapmirror status list only.  I received no admission of a bug found or a fix in process.

More Like This

  • Retrieving data ...