Information Flows Freely at Contra Costa Water District Utilizing StoneFly’s Storage Concentrators
Leading California Water District Pumps up Storage and Disaster Recovery Capabilities with Affordable IP SAN.
As one of the largest urban water districts in California, Contra Costa Water District (CCWD) uses best-of-class technology and information systems to ensure the delivery of safe, clean water to 450,000 residents in central and eastern Contra Costa County. The district in Northern California also is one of the Golden State’s leaders in drinking-water treatment and source-water protection solutions. Formed in 1936, CCWD operates as an independent self-governed agency that focuses on providing the highest quality of services at the lowest costs possible.
The district relies heavily on leading-edge technology to improve overall efficiency and productivity as part of an ongoing goal to maintain operating costs and reduce rates. According to Jim Morton, CCWD’s IS Manager, one of the biggest challenges is staying on top of escalating technology demands while keeping expenditures in line with a 10-year capital improvement plan. “Our technology decisions are scrutinized carefully since every purchase must be cost justified in terms of how it provides rate relief and supports our capital re-investment strategies,” he says. Stringent selection criteria and rigorous testing as well as a competitive bid process help identify leading contenders and demonstrate maximum value to CCWD’s Board of Directors and constituents.
The IS team relies on a variety of applications to support its wide-ranging information and data collection requirements. More than 350 end users work on HP desktops and Dell laptops linked to 11 servers including Dell Powerdge Xeons and IBM X360 P4 Xeons running Windows 2000, IBM X360 Xeons with Red Hat Linux and Oracle 8i, plus HP Netservers running Windows 2000 and Microsoft Exchange. A Foundry Networks’ FastIron JetCore Gigabit Ethernet backbone serves as the foundation for the district’s network.
Traditionally, CCWD’s IS department used direct attached storage to meet fluctuating storage demands but often struggled with interoperability across RAID devices and host systems. Since CCWD’s technology solutions were purchased as part of a two-year budget cycle, the IS team frequently encountered differences in RAID cards, which then caused compatibility problems between newer disk drives and older servers of the same brand. Over time, it became increasingly difficult to maintain older equipment with newer parts. “We couldn’t buy new disk controllers for proprietary drive cages and the cost of legacy drives was ridiculous,” explained Morton. “This forced us to obsolete servers early, restricting our ability to attain a full ROI on our host systems.”
In addition, CCWD’s storage requirements often increased during seasonal construction throughout the district, causing peak volumes in data collection and associated demands for increased, short-term storage. Under its current approach, the district had no way to consolidate and share mass storage across various systems. As a result, the IS department had to find an economical yet state-of-the-art solution for its mounting storage dilemma.
Surfing the Storage Pipeline
In July 2001, CCWD’s IS team began looking at the general trend toward network-attached storage (NAS) and storage-area networking (SAN). The consensus was that the pricey NAS alternatives delivered insufficient management control to warrant more evaluation. The group also investigated a variety of Fibre Channel SAN products from XIOtech, EMC Corp. and HP but found the price tag a big stumbling block. “We couldn’t cost justify purchasing an entry-level FC SAN for $70,000 – the ROI just wasn’t there,” said Morton. “At that point, it made more sense to stockpile DASD so we could swap out devices and maintain compatibility with our host-based systems as long as possible.”
A year later, however, the team still needed to consolidate the district’s mass storage to take advantage of the resulting reservoir of resources that could be shared ultimately. In August 2002, a review of other industries with similar storage requirements led CCWD to the Discovery Channel, which was evaluating SANs based on the emerging iSCSI standard. In particular, CCWD was impressed by the flexible storage provisioning, which made it easy to handle the rise and fall of ever-changing digital video production storage. The ability to reclaim or recycle storage was exactly what CCWD needed. In addition, the opportunity to take advantage of the district’s existing Gigabit Ethernet backbone made an Internet Protocol (IP)-based solution very appealing.
A call to Integrated Archive Systems, a Palo Alto, Calif.-based IT infrastructure integrator that specializes in enterprise storage, linked CCWD with StoneFly Networks, an innovator in IP-based storage provisioning appliances. As a pioneer leveraging the iSCSI standard, StoneFly was an early market entrant, shipping the first model in its Storage Concentrator line in June 2002.
CCWD also looked at comparable alternatives from Adaptec and Cisco. Since no ship date was available for Adaptec’s iSCSI-based product and Cisco’s IP-based model was cost prohibitive, the alternatives were eliminated. The StoneFly solution, on the other hand, was affordable, reliable and backed by solid reference accounts. CCWD visited several StoneFly reference sites and conducted basic benchmark testing to confirm functionality.
As a result, CCWD built a strong comfort level with StoneFly’s iSCSI appliance. “We were pleased to learn that the Storage Concentrator was built on a Dell foundation,” said Morton. “We found the reliability and general security on our Dell systems superior to other platforms. In addition, we felt that the common service scenario across our storage solution, Dell servers and back-office Dell laptops would ease ongoing maintenance efforts.”
The team next calculated the cost of a StoneFly-enabled IP SAN, comprising Storage Concentrators and Nexsan ATAboy 2 Storage Arrays, at about 30 cents on the dollar in comparison to the Fibre Channel SAN equivalents. This meant that CCWD could implement an IP SAN for just a few thousand dollars more than its existing direct-attached storage. “Once we considered the additional management benefits, it was an easy decision to choose StoneFly,” Morton added. “Adding redundancy to the IP SAN was a fraction of the cost of buying redundant controllers, cabinets and disks on the DASD front.”
Flood of Benefits
Initial testing of the Storage Concentrator was completed in half the projected time, which enabled CCWD to streamline its IP SAN deployment. Data from five production servers were migrated onto the Storage Concentrator over a week-long cutover. “Configuring the Storage Concentrator was extremely straightforward, especially since we could use standard industry nomenclature for assigning storage IDs, partitions and logical unit numbers (LUNs),” said Morton. “No special expertise was needed, so our technicians mastered the system in less than one day.”
The team transferred 500 gigabytes of Windows- and Linux-based data from dedicated drives onto the IP SAN, moving over files from various user groups including digital photo archives, Oracle databases and the district’s email repository. Plans are underway to migrate Unix-based data to the IP SAN by the end of 2003. Almost immediately, the team achieved a 20 percent improvement in performance throughput with the IP SAN. Overall, the IS team will use the IP SAN to manage more than a terabyte of production storage with 2-1/2 terabytes of total storage, giving them a three-year growth path on the Storage Concentrator.
In the past, CCWD was unable to reclaim storage or resize volumes when storage surges occurred. The Storage Concentrator purges this problem by giving operating systems, applications and end users access to a reservoir of storage resources or volumes that appears identical in all ways to a locally attached hard drive. It doesn’t matter where the volume is located on the network or if the volume is spread across multiple physical storage devices. “Volume-to-volume transfers eliminate duplicate files easily, while letting us double and collapse storage without worrying about interoperability issues,” explained Morton.
Another benefit of the StoneFly IP SAN is streamlined disk-to-disk backup and restore procedures. “We no longer have to cut as much tape for any given workgroup, which has generated sizable savings on tapes alone,” Morton noted. Previously, the district averaged $7,200 a year on tapes; now the yearly expense is reduced to about $3,300 with the IP SAN. “In reducing CCWD’s tape back up needs, we were able to realize a payback on our IP SAN investment in less than six months,” he added.
Once disk backup is complete on the Storage Concentrator, CCWD can send an archival copy to tape in just three hours, down from prior archival times which exceeded 27 hours and often impacted network performance. In addition to reducing backup windows and lowering operating costs, the IP SAN enables the district to strengthen its business continuity strategies.
Disaster Recovery on Tap
A major priority for the IS team is ensuring information availability of vital information while reducing the time—and high costs—of recovering critical systems. “It takes three weeks of around-the-clock effort to recover the district’s mission-critical systems,” said Morton. “We need to cut that timeframe drastically because it costs a lot of money every hour our systems aren’t up and running.”
As part of the district’s growing business continuity strategies, the IS team is planning to deploy an alternate data center to protect mission-critical data in the event that a manmade or natural disaster strikes its primary site. Toward that end, the team will co-locate a redundant IP SAN at the alternate data center by the end of 2003, comprising a second StoneFly Storage Concentrator with FailOver capabilities to protect data from unforeseen power loss, port failure, loss of any storage connection or inability to serve any volume. “We’re projecting that our redundant IP SAN will enable us to lower our system recovery from three weeks to three days—which represents a huge time and cost savings for the district,” concluded Morton. “StoneFly’s IP SAN plays a pivotal role in protecting vital data and supporting our underlying information infrastructure.”
Contra Costa Water District