If you’re familiar with Open Systems Snapvault, then you know it’s a way for a non-NetApp system to ‘vault’ it’s data off onto NetApp storage. NetApp intends to provide a way to keep data backed up in the Vault… but what if you’re trying to migrate off that source system?
From an older post:
Don’t ‘xcopy’ all your data off an old Storevault like a chump. And don’t accept when someone says “unsupported”.
Still true. 🙂
In our environment, prior to adding a monstrous SATA shelf, we were tight on space. Our most important, revenue-generating application, one that has existed for nearly 2 decades, is attached at the hip to a large amount of historical data. This data used to live on our old, old NetApp SAN, prior to decommissioning. To complete our migration, we housed it on some cheaper Synology-backed NAS storage. To meet our encryption requirements on this data, the server leveraged TrueCrypt, and unlocked the encrypted disk at boot. We were also using Windows’ FCS to replicate the data onto another server we could use in case of failure. Due to all of the traffic to the primary server’s disk, memory utilization peaked and the disk fell over and detached itself every so often, during business hours, and that probably bothered some people. We called this, “an ugly science project”.
This having been my first time setting up any form of SnapVault, I wondered if it would be possible to leverage the Open Systems variety to get our data onto the SAN. We needed to minimize downtime as much as possible, and a straight NDMP copy would have taken dozens of hours to complete. I engaged NetApp Support, asked some questions, installed the OSSV utility on our secondary FCS server, and started poking around, working through some basic communications issues between SAN and server. Then Support dropped this on me:
“SnapVault is generally not intended to be used for data migration. When a SnapVault relationship is stopped, the configuration, as well as the destination qtree is deleted. There are ways to make a destination SnapVault qtree read/write but it requires additional steps and is generally not used with OSSV.”
So, let’s get down to business.
The Primary — Configuratoring the OSSV Configurator
First, you’ll grab the OSSV installer and license key from the NetApp Support site. I installed on a Windows 2012 (R1) server, the FCS secondary server. From now on, in Snapvault language, this server is considered the primary.
Once installed, you’ll want to find the tool called OSSV Configurator on your primary, and check out 3 things:
- General tab – Database, Trace, and Temporary Directories – I had to change the default location away from C: and onto another disk I could grow more easily, or one with plenty of space. As OSSV does its thing, the DB and temp will grow as it stores more and more inode information. For ~2TB of data, I ended up with 136GB of data on my scratch disk. Plan accordingly. If your disk fills up, you’ll see Snapvault updates not completing, and you’ll have to peek the logs to find out why.
- To change the location, STOP the service using the General tab, copy the folders from their original location onto the new disk you’re using, then modify the location using the OSSV Configurator, then restart the service on the General tab.
- Trace Level tab – Process Manager, Communication Manager, Snapvault Listener, NDMP Server, QSM Server – until you know everything works, set all the logging levels to VERBOSE. I needed to check logs several times during my project.
- SnapVault tab – a few important values here, regarding communication between the client and your SAN. At first, I’d specified my SAN’s HOSTNAME, and was getting some errors that they couldn’t connect. Ultimately I ended up with the following:
- Check QSM Access List – uncheck, at least until you get positive confirmation your OSSV syncs are working, then you can start locking it down.
- QSM Access List – USE YOUR SAN’s IP. This is what caused me to end up contacting Support for assistance. Name wasn’t working. You can put a comma-separated list in here, and if you check the box for “Check Access List”, it will whitelist only these IPs.
The Secondary — Creating a New Home for Your Data
Now that your client is all set, switch over to System Manager and set yourself up a destination volume on the SAN to which you’ll be replicating (the secondary, in SnapVault language).
I was pulling about 10 shares/10 parent directories off the primary, and since they’re all related to the same system, I’m tucking them all under ONE parent volume. Since I’ll be serving CIFS, I set security style on the volume to NTFS. I thin-provisioned, sized it larger than I thought I’d need (2.5TB to start), since I can shrink it later, and enabled Storage Efficiency, so we can see what sort of savings we can achieve later.
The first command you’ll run to initialize a new transfer will be looking for a qtree name. An early mistake I made was to create this qtree myself, which just caused my initialization to fail, because the qtree already existed. So, come up with your qtree naming convention, but don’t create them. Let SnapVault do it.
Now you have your primary and your secondary configured, and you should be ready to press onward. To initialize and update your SnapVault relationships, the commands are run from the secondary. The syntax is as follows:
snapvault start -S [primary_server_name]:C:\folder\subfolder /vol/your_vol_name/your_qtree_name
And here’s the command for my test exercise:
snapvault start -S A1-PRD-FCS-002:E:\PROD\Shares\matthew /vol/vol_mftest/qtree_mftest
Once you’ve run the start command for your folders, they should begin to sync. Go make coffee and take a walk. One thing to look at if you’re syncing LOTS of data is your inodes count on the destination SAN volume. One of our folders has over 1.5million files inside it, which means I ran out of inodes pretty quickly. You can find out how to increase your inodes count here:
Progress can be monitored by running snapvault status from either the primary or the secondary. [In this case, there is a snapvault.exe you can hit via CMD prompt on the primary, under the C:\Program Files\netapp\bin folder.]
Once complete, you’ll see the State reflected back as “Snapvaulted” and the Source Status as “Idle”. The “Lag” column will give you HH:MM:SS idea of how far behind the last sync was.
One cool thing is, at this point you can verify the integrity of your data, and point a CIFS share at the destination qtree, and — tada! — there’s your data. Just delete it before cutover, it will cause problems later. Notes below.
Where Things Get Tricky — Planning a Cutover
This took the greatest amount of care and consideration, and researching through NetApp documentation, to plan, test, and implement. The reason is, you can’t simply run snapvault release and suddenly all your data on the secondary becomes accessible with no strings attached. When you release a destination from an OSSV relationship, the destination qtree is deleted. I tested it — it’s true, it zaps your data. But with some careful trickery, you can work around this.
At first I tried using Snapmirror to pull data from the secondary volume onto a new isolated volume on the same controller. The problem was, SnapVault saw it, and took it in as another Snapvault destination, and wrote an entry for it into the Snapvault config. This meant anything I tried to do with my secondary, Snapvault was also trying to do on my “safe copy” — not so safe.
So, to protect your data during cutover, there’s a special sequence of events — and a hidden ‘diag’-level command — you can use. The process is decently documented in a Netapp Support KB article here, but I had to flesh out some finer details:
This process really is used for a DR scenario, where the primary becomes inaccessible, but to me, data migrations are another good use-case for the technology. The diag-level command, snapmirror convert, converts a snapvault qtree to a snapmirror qtree, which allows it to be broken and made R/W.
To complete your cutover, you’d follow these steps:
- Sync any changes from the primary (the Windows server) to the destination with ‘snapvault update -S’, example:
snapvault update -S A1-PRD-FCS-002:E:\PROD\Shares\matthew /vol/vol_mftest/qtree_mftest
- Offline or disable any shares accessing the source directories, to ensure no further data is written.
- Perform a final sync from primary (OSSV on Windows) to destination (SAN) using the ‘snapvault update‘ command again.
- Before performing the following, ensure that any CIFS shares pointed to the destination qtrees are stopped/deleted. Any CIFS shares will prevent the destination qtree from being properly cleaned up by OnTAP.
- Disable Snapmirror and Snapvault on the controller with the following commands:
options snapvault.enable off
options snapmirror.enable off
- Enter diagnostic mode — ‘priv set diag’
- Run the command to convert the OSSV destination from Snapvault qtree to Snapmirror qtree:
snapmirror convert /vol/vol_mftest/qtree_mftest
- Exit diagnostic mode — ‘priv set’
- Re-enable Snapmirror:
options snapmirror.enable on
- Identify whether a Snapmirror relationship shows up: snapmirror status
- Quiesce and break the new snapmirror:
snapmirror quiesce Walrus-SAN2:/vol/vol_mftest/qtree_mftest
snapmirror break Walrus-SAN2:/vol/vol_mftest/qtree_mftest
- Re-enable snapvault
options snapvault.enable on
- Identify base snapshot:
snapmirror status -l Walrus-SAN2:/vol/vol_mftest/qtree_mftest
Record base snapshot: Walrus-SAN2(1886024756)_vol_mftest_qtree_mftest-dst.0
- Delete the snapshot:
snap delete vol_mftest Walrus-SAN2(1886024756)_vol_mftest_qtree_mftest-dst.0
- Verify snapmirror relationship has been removed from controller:
- On Windows server, browse C:\Program Files\netapp\snapvault\bin, open CMD and run:
snapvault release E:\PROD\Shares\matthew Walrus-SAN2:/vol/vol_mftest/qtree_mftest
- On SAN controller, Release snapvault qtree:
snapvault release /vol/vol_mftest/qtree_mftest A1-PRD-FCS-002:E:\PROD\Shares\matthew (this might not work, complains about no releasable destinations, and to run 'snapvault destinations', which shows nothing. Your results may vary.)
- Verify snapvault relationship has been removed:
snapvault status (no snapvault entry should appear)
snapvault status -c (configuration entries; should be cleaned up)
qtree status (qtree status should be ‘Normal’)
When you’ve completed the above steps, now you can point a CIFS share at either the parent volume or the qtree. In our case we are going to CIFS alias the primary server name and put in a DNS alias to redirect to our SAN. Awesome!
One caveat I’m still working through is, you still cannot modify the Storage Efficiency settings on your destination volume; it still seems to think this is a SnapVault volume. The sis start command also does not work, however you can rescan the entire volume manually by using the command sis start -s <volume_name>. Good enough for now. If I can figure out how to change the identity of this volume, I’ll update.
I’m hoping to leverage more of SnapVault for cool things like VM backups in the future, but for a primer on SnapVault, this sure felt like jumping into the deep end of the pool – however, overall this effort culminated in another successful project and I was very pleased with the results.