By Christine Kane
How EMC's Critical Incident Response Center provides business operations protection for a $15 billion revenue machine
It's a typical weekday at EMC's Critical Incident Response Center (CIRC), located in eastern Massachusetts. With its two tiers of analyst workstations, facing a wall of 52-inch flat-panel displays, the center resembles a small-scale war room. But the atmosphere is relaxed as five members of the Critical Incident Response Team (CIRT) sip coffee and go about the job of protecting an infrastructure that spans 60 countries and more than 500 sites.
Fed by more than 1,300 security devices-which generate 15 to 20 million security events per hour-the screens function like a giant dashboard, displaying in near real time the security status of EMC's global information infrastructure. One screen shows top attackers bombarding EMC IP addresses that day. Other screens depict the top viruses detected, the most common attack signatures, alerting trends, behavioral reports, and attempts by EMC servers and desktops to access sites known to harbor spyware and malware.
"The CIRT is charged with security incident response for all of EMC," says CIRT Manager V. Jay LaRosa. "We manage virus outbreaks and denial of service attacks. We hunt out malware that may have compromised critical servers or systems, and we look for business-sensitive information that may be leaking outside the corporation. When we identify suspicious activity, our job is to investigate, contain, remediate, learn from, and move on to the next investigation."
Protecting EMC's revenue machine and its customers
It's all about protecting the company and its customers, says Roland Cloutier, vice president and chief security officer of EMC. "As part of EMC's converged security organization, the CIRC protects the company's business operations, which is a $15 billion revenue machine," says Cloutier. "Indirectly, we are also protecting thousands of our customers' critical infrastructures in industries that affect everyone's lives every day, such as government, healthcare, electrical, oil, and gas. Because our products are so integral to these organizations, we have to protect every aspect of how those products are designed, manufactured, delivered, and serviced."
Cloutier notes that customers are very interested in the CIRC. "We are providing a whole new level of service around business operations protection that wasn't available before, and our customers want that," he says. "Our approach is innovative, but it is built on proven technology that people have deployed throughout their data centers for many years. But instead of using 20 percent of the available capability for an IT or security function, we are stretching it to 80 percent."
Gaining a worldwide view of security
As recently as October 2008, responsibility for managing security incidents at EMC was split three ways, among a security operations center, a data loss prevention team, and an "eyes on glass team" of skilled security analysts-the type who watch packets at two in the morning to see what the bad guys are doing. "By breaking up these functions, we were driving inefficiency and higher cost into the system," says Cloutier. "So we took a step back and said, 'How can we integrate this team of disparate practitioners? How can we re-engineer our platform to handle multiple streams of log data from different technologies? How can we implement a single workflow for monitoring and enforcement out of one center?' The end result is the CIRT, which enables us to see what is happening across the environment, prioritize the most critical issues, and respond to them very quickly."
The industry's best dog food
Following the philosophy that a company should demonstrate confidence in its products by "eating its own dog food," the CIRC is built on technologies from RSA, The Security Division of EMC. Most notably, these include the RSA enVision platform for centralized security monitoring and the RSA Data Loss Prevention (DLP) Suite. In addition, security technology embedded in EMC's information infrastructure products generates much of the event data streaming into the CIRC. As LaRosa explains, "EMC storage infrastructure products collect data on encryption and authentication. Content management systems generate logs relating to digital rights management. It's all sent to the security monitoring platform, giving us a holistic view of our security posture across the enterprise."
The RSA enVision system forms the core of the CIRC, collecting event logs generated by security devices worldwide. Advanced analytical software turns this mass of raw data into structured and actionable information and displays it in a highly visual form. When enVision detects suspicious activity, it generates an alert and classifies it based on the likely severity of an incident and whether it requires a Level 1, 2, or 3 analyst to address the issue. By checking the Task Triage Queue, analysts can quickly identify the incidents requiring their immediate attention. To investigate an incident, they can interrogate the full volume of stored data-for example, corroborating an intrusion detection alert by looking at related antivirus or firewall data. "Having one place to go, one pane of glass for viewing everything is a huge timesaver for us," said LaRosa.
Calling offenders to account
If a specific user or group repeatedly violates security policy or takes a high-risk action, such as connecting their own wireless router to the network to temporarily bypass the IT-managed wireless network for reasons of convenience-thereby neutralizing multiple layers of network protection-the CIRC has a couple of options: It may engage the Global Investigations Security group to conduct an investigation or contact Human Resources to initiate communication with the offender. Analyst A. J. Muccio recalls occasions where local personnel have been dispatched to walk the halls of a facility with a device that allows them to sniff out such rogue access points, which are sometimes tucked under a desk or in a closet to avoid detection.
Detecting and preventing data leakage
The CIRC also deploys the RSA DLP Suite on nine Internet gateways worldwide to detect sensitive information, such as EMC intellectual property, customer records or employee Social Security numbers that are being transmitted out of the company through myriad methods. A large datacenter deployment of RSA DLP constantly scans all fileservers around the world and classifies any sensitive information found on them.
As one of his responsibilities, Level 3 Analyst Jeff Hale develops DLP "content blades" to detect new categories of information that need protection. "If we have a new product launch, we get that information ahead of time and program DLP to look for that information and be sure no one is talking about it or sending it out ahead of time," says V. Jay LaRosa "Similarly, before quarterly or year-end financial results are announced, we activate a content blade that looks for financially sensitive information to make sure no one is leaking it early."
Keeping operations separate from incident response
Analyst Dave Earle notes that customers are sometimes curious about why the CIRT and EMC's security operations group are distinct entities, though they work closely together. "Where the CIRT is charged with monitoring and remediating security events, operations manages our Level 1 adds, moves, and changes for security devices," says Earle, who was recently promoted to a senior security engineer within the GSO Security Operations team from the CIRT.
"We've found that when one group is responsible for both functions, the daily demands of operations tend to pull people away from incident response, except for events that are too big to ignore," he says. "One result is that staff members can't find the time to continually enhance the security monitoring environment, which is critical if you want to keep up with evolving threats. When we talk to customers about the separation of duties, customers immediately get it.'"
Expanding and enhancing CIRC capabilities
When the CIRC was launched, the center was only staffed during business hours on the U.S. East Coast. Since then, shifts have been expanded to 13 hours a day, and a second, identical facility was opened in Bangalore, India, allowing the center to offer follow-the-sun service. Key enhancements under development include improved workflow, which will greatly streamline investigations, and integration of the CIRC with RSA's Anti-Fraud Command Center, which helps customers prevent phishing, pharming, and Trojan attacks. The center also helps shut down fraudulent sites globally.
Security must be a business enabler
Cloutier is adamant that security must be deployed in the service of business goals, enabling the innovation and responsiveness that create competitive advantage. "As security practitioners, our aim is to create an environment for our executives, engineers, and sales folks to build, deliver, and service the absolute best technologies without any impedance or concern about security in our environment," he says. "We want them to understand that security is not a business inhibitor."
Prev