Preparing for Disaster – Houston-style Part 1
Where do you start preparing for a disaster? I was invited last Thursday afternoon to attend and observe a server and SAN failover at Champion Technologies in Houston. I jumped at the idea.
After renting a car (gas and a weekend car rental are cheaper than driving your own car), I arrived at Champion at 6:30 in the evening Friday. System shutdown had started at 6:00 and was already proceeding.
It seems that Champion was notified by building management that power would be cut to the building a 6:30 AM on Saturday morning. This necessitated the company to do a controlled failover and shift its operations overnight to a disaster recovery site in Scottsdale, Ariz. It was also a chance for Champion to test their disaster recovery plan.
The Champion system consists of 11 Dell PowerEdge 6850 four-core servers running SQL Server, SAP ERP and Microsoft Exchange, connected to 15.5TB of Dot Hill storage with FalconStor Network Storage Server (NSS) appliances that virtualizes the storage and replicates the system to the disaster recovery site. The configuration is mirrored with an identical storage and server implementation in Scottsdale.
To facilitate its disaster recovery plan, which consists of 364 steps well-documented in Microsoft Project, Champion brought in about 15 IT staff, myself, a video crew and a technical support representative from FalconStor Software.
By 7:40 the SQL Servers were shut down. By 8:00, all servers had been shut down, a manual snapshot of the drives had been taken and replication had been forced. Roles were reversed, with Scottsdale taking over primary operations. These operations were followed in Houston by shutting down the virtual tape library, the two NSS servers, the Cisco Fibre Channel switches and the Dot Hill Storage.
Right before 8:10 PM, operations had shifted over to Scottsdale, where operations resumed. All except for a 897GB volume which had replicated but whose UMAP file differed from that of its mirrored volume in Scottsdale. They’d take care of that on Sunday morning. Everything was done at 11:00 PM that night, ready to reverse roles early Sunday morning when the power goes back on. Last night, the FalconStor software also rescanned the volume and rebuilt the UMAP file. Everyone was in bed by 2:30 AM.
Yesterday Sunday – at 11:30 AM, I was sitting in a cubicle on the 28th floor in Houston, waiting to regroup with the Champion and FalconStor crew. They were going to bring everything back on line in Houston. If everything goes right, they’ll be back in business and ready for the start of business today.
Leave a Reply


