24 Replies Latest reply: May 24, 2013 12:40 AM by milohuang RSS

NDMP Backup 2MB/min

oweinmann
Currently Being Moderated

Hi,

 

is there a limitation (file count) for NDMP? We see a very very poor Performance with NDMP Backups of a 2040c with BackupExec 2010r3. We have not evaluated a different backup software yet so we are not sure if this is the bottle neck.

 

The volume we are trying to backup consists of several qtree's and has a total around 16 Mio Files.

 

Regards,

Oliver

  • NDMP Backup 2MB/min
    andrc
    Currently Being Moderated

    I'm assuming the filer is directly connected to the tape device and data flow isn't passing through a media server?

    As a start you can issue `sysstat -u 1` from the CLI and check for any obvious bottlenecks i.e. CPU, disk utilisation plus you'll get a clear view of approximate KB/s the filer is writing to tape.

  • Re: NDMP Backup 2MB/min
    aborzenkov
    Currently Being Moderated

    The first step dump does is to build full file list; with large number of files it can be quite time consuming. You can test the theoretical dump speed by calling it directly and dumping to NULL:

     

    https://kb.netapp.com/support/index?page=content&id=1011894

     

    Watch how much time each of steps takes. You should also be able to see dump logs in /etc/log/backup.

    • NDMP Backup 2MB/min
      oweinmann
      Currently Being Moderated

      Hi and thanks for the quick responses:

       

      I'm assuming the filer is directly connected to the tape device and data flow isn't passing through a media server?

       

      - Yes it is directly attached to our FAS2040 via FC.

       

      I have now snapmirrored the largest qtree (1.8mio files) to a new volume and started backup exec. Let's see how long it takes. If this is already an issue we have to review the space usage.

       

      Regards,

  • NDMP Backup 2MB/min
    jwrobertson
    Currently Being Moderated

    We've had problems similar to this when we back up volumes that contain large numbers of files.  It has to crawl the file list before it actually starts backing up, and sometimes that can take hours before it actually starts moving any data.  Commvault supposedly now has a solution that allows it to skip that crawl by using snapshots and only looking at changed data.  Another option that NetApp presented to us is doing a volume based backup.  That avoids the file crawl, but doesn't allow individual file restores... you have to restore the entire volume.  Might be an option if you can use snapshots for individual restores and the volume backup for major disasters.

    • NDMP Backup 2MB/min
      jwrobertson
      Currently Being Moderated

      I should also add that this is not a problem that is unique to NetApp.  We had the same issues doing NDMP backups from a Celerra as well.

      • NDMP Backup 2MB/min
        oweinmann
        Currently Being Moderated

        Hi,

         

        thanks for the recommendations. Is there a trial version of commvault available? I had recommended commvault in the past to our boss but we had backup exec for several years now and it is quite ok. I will now evacuate all qtrees wth more than 600.000 files to a new volume and create a separate backup job for it. We have two LTO 5 FC Tape drives, so we can split it up.

  • NDMP Backup 2MB/min
    Currently Being Moderated

    hi,

     

    You said: "The volume we are trying to backup consists of several qtree's and has a total around 16 Mio Files."

     

    Thats your problem right there. NDMP is not good for lots of small files. Off the top of my head I think NDMP backups

    have to 5 phases (or was it 6) and the first 3 ones are just for scanning the file system. Imagine having to scan

    that many files on a very small controller. You havent got the CPU to deal with this I reckon, even if you did it would

    take ages to back this up.

     

    The way to work around this is to use snap products as backup methology. As mentioned commvault integrates nicely

    with netapp snaps.

     

    Cheers,

    Eric

    • NDMP Backup 2MB/min
      oweinmann
      Currently Being Moderated

      I will try to contact commvault if  they can supply us with a trial version to test the ndmp backup.

    • NDMP Backup 2MB/min
      glen.eustace
      Currently Being Moderated

      We are using EMC Networker at are seeing extremely slow thoughput as well.  We get 7Mb/s regardless of whether the volume has many files or one large file of a similar size.  A CIFS mount to the same volume and we get 70Mb/s.  There is something extremely sucky about NDMP.  To date we have had no suggestions from either EMC or IBM (N-Eries) that has made any difference at all.

      • Re: NDMP Backup 2MB/min
        aborzenkov
        Currently Being Moderated

        I had several installations with NetWorker and backup speed was OK. Restore was different matter though ☹

         

        Incidentally, all installations were using 3-way backup (via NetWorker DSA) and not direct backup to tape. I constantly observed full wire speed (120MB/s for 1Gb/s interface). But these all were database installations with relatively small number of large files, 10-20GB each.

        • NDMP Backup 2MB/min
          glen.eustace
          Currently Being Moderated

          This is our scenario as well.  We have a DLT4 jukebox on a storage node.  Throughput is abysmal when using NDMP.  As mentioned in my previous post, using CIFS as the transport, with all other hardware the same we can max out the network connection, so we have ruled out the tape and networker.  Even trying to multiplex a couple of NDMP backups the trhoughput is unacceptably slow it seems that the filer is simply incapable of pushing the data out any faster.

           

          This filer is at our DR site and has no customer access.  It had been our intent to offline the snap-mirror detination volumes using NDMP to tape.  We should be able to get the entire IO capacity of the filer.  We have almost given up and are currently trying to find alternatives

      • Re: NDMP Backup 2MB/min
        aborzenkov
        Currently Being Moderated

        Oh, and the first thing that comes to mind is to check fragmentation level ☺

  • Re: NDMP Backup 2MB/min
    gary.thomas
    Currently Being Moderated

    We had a similar issue and now use smtape.

     

    We get about 280-350mb/s on volumes that have a millions of small files, restore is significantly faster.

     

    We are doing it via ethernet and not fc, so its probibly even faster yet again.

     

    For us we just keep enough disks spare to do a out of store replace and just q a job overnight to restore if required.

     

    smtape has the advantage of having all the snapshots in the backup and we keep 6 months of snapvaults so while you can't do a single file restore without restoring the volume, once you have done the restore its very easy to work with the client to get the exact file/s they require as 9 times out of 10 they can't get a accurate date or filename for stuff thats older then a couple of weeks.

     

    Oh i forgot, usually takes a couple of minutes for smtape to start, but once it starts it flys.

    • Re: NDMP Backup 2MB/min
      oweinmann
      Currently Being Moderated

      Hi,

       

      sm2tape is not an option for us. I will now try to create smaller volumes like VolAC, VolDF and so on. Hopefully this makes backup faster. We have two tape drives so we can run the jobs parallel.

      • Re: NDMP Backup 2MB/min
        oweinmann
        Currently Being Moderated

        Hi,

         

        we finally have backup jobs running with backup exec at a very good average speed of 3.8GB/min. So we need 13 hours for around 3.1TB. We have two tape drives so we could even split the jobs.

         

        The key was to restructure the folder structure. Before we had just one large volume Projects with up to 14 million files. I have now started to break it down in ProjectsAC to ProjectsXZ and used snapmirror to move the qtrees. This had a signifficant impact on the backup speed.

         

        Problem solved!

  • Re: NDMP Backup 2MB/min
    milohuang
    Currently Being Moderated

    I use direct connect 1xLTO5 to filer, we got close to theoritical value of 500GB/Hr

    Using NDMP over TCP/IP where tape is connect to backup server, we get only 300GB/Hr

More Like This

  • Retrieving data ...