News Stay informed about the latest enterprise technology news and product updates.

Child welfare firm boosts its number-crunching with all-flash arrays

Child welfare agency NCCD couldn't munch through its client databases fast enough to run analytics until installing Violin Memory all-flash arrays.

How do you know if you need an all-flash storage array or if a hybrid array will do?

For Joel Ehrlich, director of information systems for the National Council on Crime and Delinquency (NCCD), his workload made the decision for him.

"We were not looking for mixed use; we were interested in a dedicated use optimized for Oracle databases," said Ehrlich, explaining why he acquired an all-flash Violin Memory 6212 Flash Memory Array to replace Oracle Pillar Axiom hard drive storage that had been tuned specifically for Oracle software.

The NCCD uses an Oracle RAC database to run sets of analytics for child welfare and juvenile justice agencies out of its office in Madison, Wis. The nonprofit group does analytics and reports for California's statewide child welfare system, and also crunches data for agencies in New Jersey, Pennsylvania and Virginia. NCCD must refresh its reports every night to make sure it has the most recent data, and that requires a great deal of I/O -- more than it was generating with hard disk SAN arrays.

"For analytics, we always struggled because we have a fairly narrow window," Ehrlich said. "We're munching through a lot of data. And we were not having success getting through that overnight window, getting data flushed and replaced and back up so people could see refreshed data the first thing in the morning.

"We were getting hangs for various reasons," he said. "One reason was that we could not push data through fast enough. Another one was that even though we were running Oracle RAC, because we were running a single SAN behind it, you had an I/O bottleneck if all the systems were asking to pull data at the same time. Sometimes the whole system would hang in the morning until you did a reset."

The problem got so bad that Ehrlich and his team discussed hiring somebody to work overnight to keep an eye on the system, or setting up an emergency monitoring system that would call one of his staff at home in the middle of the night when a problem occurred. Ultimately, he decided to treat it as an I/O problem.

"I said, 'I think fundamentally we have an I/O problem. We get incremental improvements every time we improve our I/O system. We just need to have something more robust and faster,'" he said.

He first saw Violin at a booth at a Dell World conference in Austin, Texas in December 2012. After talking to Violin and other flash vendors, he brought Violin in for testing in February 2013, and acquired the 6212 with 12 TB of flash capacity after a 60-day trial. He said the all-flash system worked as he hoped it would, and reports now run through the night and complete in plenty of time to give clients fresh data the next morning.

"People aren't happy if the data looks old, and we could lose contracts," he said. "I would say at least once a week we would have a problem with at least one jurisdiction not getting data refreshed. Now, it's been more than 40 days since we've had any problem."

When he brought the Violin 6212 in for testing, he ran the entire California database. At the time, it would sometimes take 27 hours to run on the Pillar system, but Violin reduced that to three hours. After tuning the array, it now runs the California database in about three hours and 15 minutes in conjunction with running all the other state databases. The other states take far less time to complete.

NCCD uses its Axiom to store log file writes on the Oracle database while sending all the transactional data to the Violin array.

Ehrlich said he considered a hybrid array to cut costs, but he wanted a system built for flash. "I didn't want to have anything designed to be a kludge of flash onto your existing hard drive system," he said. "If there are any mechanical latencies, you're passing through that front-end flash. Why not go straight to all-flash arrays for anything you want to read and write quickly?"

NCCD's first SAN was an EMC Clariion CX300 installed more than 10 years ago, but Ehrlich said the cache of that entry-level system was not enough to handle 10 million to 30 million record tables, "and the whole thing would come to a halt." He said an upgrade would cost too much, so NCCD switched to a three-node Pillar Axiom array. Pillar was founded by Oracle CEO Larry Ellison (Oracle acquired Pillar in 2011) and the company's arrays are tuned to work with Oracle databases.

"That was a big improvement, but short-lived," Ehrlich said. "As we grew, we overloaded the front cache and were back to dealing with slow disk I/O. I said it would be nice not dealing with cache at all, have really fast storage for our I/O and not worry about if we're overloading the cache. That's what Violin offered."

He said he could have added more Slammers (storage controllers) to the Pillar SAN for less money than the Violin array cost, but that would not have been a long-term solution.

Ehrlich said that as he researched flash and talked to other all-flash vendors, he realized Violin was right for him. "The competitors were saying, 'We have more general-purpose storage, Violin needs to be carefully tuned for Oracle, and if you run general-purpose loads, you're going to have a problem.' I never validated that to see if it was true, but we were going to dedicate the Violin to an Oracle database, so that fit our use case."

He said the price -- $225,000 for the array and Fibre Channel switches -- gave the nonprofit pause, but "it was cheaper and more bang for the buck than buying more Oracle nodes. Oracle licensing is expensive."

Dig Deeper on All-flash arrays

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.