Early Sunday morning, specialty chemical company Champion Technologies failed back their data center to their Houston headquarters.
Originally disabled by power being down in their headquarters building, Champion had to execute a disaster recovery drill and fail over their data center operations to a site in Scottsdale, Ariz. That failover occurred Friday evening. All operations, which started at 6:00 PM, were completed at 11:00 PM that night –five hours later.
When the power went back on Saturday night about 9:00 PM, Champion’s IT staff and FalconStor Software staff re-convened at the Houston offices, ready to fail back data center operations to Houston in time for the workweek on Monday.
After some preparation, failback started. The UPS, Cisco Catalyst 6509 switch, MPLS router, Dot Hill storage and Cisco Fibre Channel switches were powered on, according to a carefully-orchestrated plan, documented in Microsoft Project.
FalconStor Network Storage Server appliances, which handle the replication of data from Houston to Scottsdale and vice versa, were powered on to start the replication process back to Houston. Data base synchronization from Scottsdale to Houston was finished 6 hrs later. At 11:00 am Champion staff validated the replication, then proceeded to assign SAN resources to the servers, SAP was brought up and VMware servers were powered on and validated. Microsoft Exchange message stores were mounted.I missed all those operations, blissfully sleeping in a hotel bed two blocks away.
When I arrived at Champion’s Houston office at 11:30 Sunday morning, failback was again in full progress. Left to do was to replicate the deltas back to Houston. Because only a small amount of changes had taken place to the data, this operation took place quickly and without any problems.
After operations had resumed in Houston, users were locked out of the system in Scottsdale, the Scottsdale servers were shut down, SAP was downed and the Exchange message stores were unmounted from the Scottsdale servers.
Everything was ‘All systems go,’ as they say in Texas by 2:30 PM Sunday. Most of us – Stanley Qin, vice president of professional services, excluded – were home in time for dinner. Stanley was on a plane back to FalconStor Software in New York.
Where do you start preparing for a disaster? I was invited last Thursday afternoon to attend and observe a server and SAN failover at Champion Technologies in Houston. I jumped at the idea.
After renting a car (gas and a weekend car rental are cheaper than driving your own car), I arrived at Champion at 6:30 in the evening Friday. System shutdown had started at 6:00 and was already proceeding.
It seems that Champion was notified by building management that power would be cut to the building a 6:30 AM on Saturday morning. This necessitated the company to do a controlled failover and shift its operations overnight to a disaster recovery site in Scottsdale, Ariz. It was also a chance for Champion to test their disaster recovery plan.
The Champion system consists of 11 Dell PowerEdge 6850 four-core servers running SQL Server, SAP ERP and Microsoft Exchange, connected to 15.5TB of Dot Hill storage with FalconStor Network Storage Server (NSS) appliances that virtualizes the storage and replicates the system to the disaster recovery site. The configuration is mirrored with an identical storage and server implementation in Scottsdale.
To facilitate its disaster recovery plan, which consists of 364 steps well-documented in Microsoft Project, Champion brought in about 15 IT staff, myself, a video crew and a technical support representative from FalconStor Software.
By 7:40 the SQL Servers were shut down. By 8:00, all servers had been shut down, a manual snapshot of the drives had been taken and replication had been forced. Roles were reversed, with Scottsdale taking over primary operations. These operations were followed in Houston by shutting down the virtual tape library, the two NSS servers, the Cisco Fibre Channel switches and the Dot Hill Storage.
Right before 8:10 PM, operations had shifted over to Scottsdale, where operations resumed. All except for a 897GB volume which had replicated but whose UMAP file differed from that of its mirrored volume in Scottsdale. They’d take care of that on Sunday morning. Everything was done at 11:00 PM that night, ready to reverse roles early Sunday morning when the power goes back on. Last night, the FalconStor software also rescanned the volume and rebuilt the UMAP file. Everyone was in bed by 2:30 AM.
Yesterday Sunday – at 11:30 AM, I was sitting in a cubicle on the 28th floor in Houston, waiting to regroup with the Champion and FalconStor crew. They were going to bring everything back on line in Houston. If everything goes right, they’ll be back in business and ready for the start of business today.
Late yesterday Brocade Communications announced its intent to acquire switching and routing vendor Foundry Networks for $3 billion. The deal, which combines vendors from the storage and networking space, validates the whole idea of a converged network using the newly proposed Fibre Channel over Ethernet (FCoE), as well as a combined drive from two vendors to rival Cisco in the converged networking space.
Brocade, who recently said that FCoE would not replace Fibre Channel or Ethernet networks, but would serve as a routing technology to them while preserving the specialized services of each, is now in the enviable position to make FCoE happen. By acquiring Foundry, Brocade has gained the Ethernet and IP routing and switching technology that will let it develop FCoE products that can compete with Cisco on an even front. Early in 2007, Brocade gained some additional IP experience when it snapped up iSCSI vendor Silverback Systems.
We’ll need to watch from the sidelines to see whether this acquisition accelerates Brocade’s place in the FCoE market and whether it can beat Cisco at its own Ethernet-centric game.
In the you heard it here first category, comes Cemaphore Systems, a little known but very successful e-mail disaster recovery and business continuity software company for Microsoft Exchange environments.
Founded in 2002 with offices in Provo, Utah and San Mateo, Calif., Cemaphore was founded by Tyrone Pike, formerly of Ungermann-Bass, Citrix and LAN Systems.
Cemaphore, like we said before, makes software that ensures the availability of Microsoft Exchange servers. The software, which is installed on a dedicated Windows 2003 server, allows the recovery of individual e-mails and message boxes, as well as entire e-mail servers. Data is replicated continuously from the primary site to the secondary disaster recovery site and as it is being replicated, it is also being de-duplicated.
The company competes with products from Double-Take, the Neverfail Group, Teneros and CA’s XOSoft.
Stay tuned for more on this newcomer to the e-mail DR/BC space. You’ll be sure to hear from them more.
Cemaphore’s MailShadow is licensed on a per mailbox basis for $50.
EMC trumped the storage industry today with its announcement of flash-based solid state drives for its Symmetrix DMX-4 storage arrays.
The company, in partnership with STEC, introduced what it calls Enterprise Flash Drives. These drives, which deliver the same amount of IOPs as 30 15,000RPM 300GB drives, will be used by organizations for their transaction processing applications.
The EMC Enterprise Flash Drive is different than the multi-level cell flash technology used in consumer electronics. Instead it uses Single-level NAND flash-based storage that fits in a standard disk carrier and can be upgraded non-disruptively.
Additionally, for those energy-conscious users out there, the flash drives consume as much as 38% less power than traditional disk drives.
EMC has dubbed its Enterprise Flash Drives ‘tier zero,’ adding to the Tier 1 represented by Fibre Channel drives and the Tier 2 of Serial ATA drives.
Using Enterprise Flash Drives in a Symmetrix will come at about a 10% premium, a bargain for those financial institutions and other organizations that need the performance.
The flash drives are expected to be available in 73GB and 146GB capacities by the end of the first quarter of this year.
While no announcements have yet been made, you can expect IBM, Hitachi and others to follow suit.
The rumor mill is starting early this year — tomorrow, that is. If it is to be believed, IBM is set to acquire XIV, an Israeli company involved in making a grid-based storage array, for between $300 and $350 million.
XIV (14 — the atomic number for silicon) Information Systems was founded in 2002 by five graduates of the Israeli armies’ high-tech Talpiot program, including Moshe Yanai, the developer of EMC’s Symmetrix storage array.
XIV’s NEXTRA array is supposed to be less expensive that other enterprise storage from companies such as EMC, IBM and Hitachi Data Systems. According to a story in NetworkComputing , Ofir Zohar, CEO of XIV claims that NEXTRA will cost $5,000 per terabyte. In a NEXTRA array, data is divided into 1MB blocks and written across many drives. Zohar claims that with this approach, rebuilding a failed drive takes only 15 minutes.
If NEXTRA is all Zohar claims it to be, IBM could soon be adding a new weapon to its storage arsenal.
Symantec’s ‘green data center’ study points out how little IT is actually doing to make their data centers more energy efficient.
The company recently released a study focusing on ITs’ efforts to implement green data centers. The survey pointed out that although the majority of data center managers are starting to think about implementing green data centers, only one in seven global data center managers have implemented or started to implement energy-efficient savings in their data centers.
Symantec, who surveyed 250 Global 2000 data center managers, says that as data centers are running out of space and as energy costs are skyrocketing, it is forcing data center managers to look for ways to make their data centers more energy-efficient.
Many data center managers have consolidated the servers in their data centers by using virtualization software – as many as half of the respondents cited virtualization efforts – to save on energy costs. Data center managers also need to also look at implementing servers and storage technologies that use more efficient CPUs and power supplies and improving the airflow, cooling and heat removal capabilities in their data centers.
The survey highlighted that the companies surveyed spent an average of $1.4 million each year on electricity for their data centers. Over half – 57% — are starting to talk about green data centers and as many as 29% haven’t even started considering the benefits of going green.
For most of the respondents – 68% — going green by reducing energy consumption was a high or critical priority. Approximately 1/3 of the respondents said reducing their use of hazardous and toxic materials was a high priority.
One of the most interesting conclusions of the survey, was that energy-efficiency was cited as an important factor in selecting a hardware vendor. A number of storage vendors, including Copan Systems, 3PAR, Compellent and Plasmon have ‘at-the-ready’ energy efficient pitches made for their products.
Asking vendors to address how their gear is energy efficient is a first step in power-savings in the data center. According to a recent Environmental Protection Agency (EPA) report, computer gear accounts for as much as 50% of data center energy costs.
The EPA report estimates that by 2011 U.S. businesses could reduce their electricity costs by $4.1 billion annually just by following best practices detailed in the report. Since electricity costs in data centers in 2006 were $4.5 billion, that’s a lot of savings.
