Having to reseed a copy of the database in a DAG configuration with Exchange Server 2010 is painful enough that I actually have had nightmares involving reseed. Even though streaming online backup is deprecated in Exchange Server 2010, that code is utilized to perform a database reseed. Reseeding is notoriously slow, and doesn’t scale very well. In a recent customer visit I could not achieve greater than 300MB/sec out of a source mailbox server, even with 40 concurrent database reseeds. This on a Friday was projected to take over 3 days and was painfully surprised to find on Monday that the jobs were done, but nothing had been copied. It turns out by default, after reseeding the database it will reseed the content index. The content index reseed failed, and rolled back the whole thing! If you use Exchange to reseed be sure to run with the database only switch and reseed the content index in a separate job. (update-mailboxdatabasecopy –databaseonly) I found much faster throughput using a flat file copy, even eseutil (/y) can be utilized for this at the expense of requiring a dismounted source. The takeaway is that database reseeding is painful.
If you utilize NetApp for your Exchange Server 2010 storage, a little disk space will save you from ever having to reseed a database. Most NetApp deployments of Exchange Server 2010 involve taking a full backup on one copy of the database utilizing SnapManager for Exchange and snapshots. Since NetApp doesn’t use a copy on write snapshot scheme, there is no performance penalty to keeping multiple (hundreds) snapshots. Space is only consumed as changes are made. If you update a block, it is not touched, rather a new block is written and the pointer to the live file system is updated. Typical deployments in the 1-2GB mailbox range can expect a 1-3% daily change rate. If you have a 2% daily change rate, storing 7 days of snapshots will consume space equal to 14% of the dataset size. This is one of the points of efficiency NetApp excels at, as competing VSS solutions must at the very least copy the entire dataset and then house the changed blocks for a backup capacity cost of greater than 100% of the dataset size.
Other copies of the database should run copy backup jobs that do not truncate logs. If space is at a premium, keep only enough space for one day’s worth of snapshots. If you ever reboot, blue screen, or otherwise find failed database copies, you can:
• Suspend the failed copy
• Remove it
• Add it back with seeding postponed
• Restore a recent snapshot
• Resume the copy
At this point it will perform an incremental reseed copying just the transaction log files that are required to get up to date, a few megabytes.
I'm running tests where you don't remove and then add the database copy back...you simply restore the LUN. However, depending on the timing this can throw up Exchange errors, which may not be acceptable depending on the monitoring strategy.
Please click permalink below, or the title of this blog post above to see the attached AVI for a Video demonstration of our rapid reseed. We demonstrated this live at TechEd 2011 in Atlanta in less than 2 minutes. Please keep in mind most of the elapsed time in the video are pauses in the script so that I have time to talk about what actions are being performed.
- NetApp-DAG-Reseed-on-NetApp-WM9.wmv (4.9 MB)