In the late 1990s, large companies began installing newly-developed numeric Secure Shell keys to guard access to the software running their servers. Banks, retailers, governmental departments and other large organizations were increasingly relying on these servers to store sensitive customer information, from bank account balances to health data to Social Security information. The idea was to ensure that only network administrators could access the increasingly automated software tasked with keeping the servers up to date.
The SSH protocol devised by my company, Helsinki-based SSH Communications Security, was intended to add security to that process. Technologically speaking, it has performed very well. The keys have proven difficult to break through brute-force hacking attacks.
The success of a technology, however, can never be assessed in isolation from human factors, and unfortunately, that’s what’s happened in this case. Some organizations have no process for controlling who has access to their keys. Others have never updated their keys. This creates the specter of a disgruntled or corrupt former employee disseminating access to the software underlying an organization’s most sensitive servers.
Simply put, poor key management has turned the SSH keys into vectors for injecting viruses into large businesses.
The problem of mismanaged SSH keys is a vast one. Many large organizations have hundreds of thousands to over a million keys installed on their servers, since they often do not have a way to find or remove keys once an employee transfers responsibilities or leaves. Many organizations have no process for approving and enforcing who can grant permanent access to the servers via these keys. One study we conducted in 2012 at the invitation of a large bank found that 10 percent of the bank’s one million keys granted unlimited administrative access to production servers running critical banking applications with access to sensitive customer data.
A virus can spread over the Internet in less than a minute. Poor SSH key management exposes private intranets to a similarly-speedy risk: a worm attack. Once inside an organization’s network, a worm can use improperly managed SSH keys to spread from server to server. With so many keys, odds are the worm will infect nearly all servers – including disaster recovery and backup systems that are typically also managed using such keys – in a manner of seconds to minutes.
Once inside a server, the worm may open a backdoor, steal or alter data, subvert encryption systems or databases, or outright destroy the server and any data (including backup media) accessible to the server.
Worm attacks so far have been limited in scope but they should serve as a warning sign. A May 2012 IBM X-Force study, “2012 Cyber Security Threat Landscape,” concluded that most attacks against Linux/Unix servers utilized SSH. In 1988, the Morris Worm brought down the Internet, in part using very similar attacks.
So far, we’ve been spared the worst-case scenario in which flawed SSH key management allows a worm to spred globally via the public Internet. That scenario is still possible, however, because large organizations rely on SSH public key encryption for linking their IT systems internally, as well as for external file transfers and remote system administration.
Imagine life suddenly without banks, retail chains, logistics, manufacturing, government or power for weeks. That is why the security industry, the government and our customers need to come together to fix the SSH key management vulnerability.
How did we end up here? All major security standards require controlling access to servers and proper termination of access. Key-based access, however, has slipped under the radar of IT departments because it is deeply technical and obscure. Mainstream security professionals and auditors – even chief information security officers – are scarcely aware of the problem. Most companies have no idea how many SSH keys they have, and have no concept of the scope or potential impact of the issue.
Remedying the problem involves discovering all SSH keys within the network, eliminating keys that are no longer in use, preventing unauthorized key-based access grants and establishing process to regularly change all keys so that copied keys cease to work.
This is an issue that any large organization trusted with sensitive data – banks, governments, retailers and others – must take steps to resolve. My company is working with organizations to audit and centralize their key management processes, but frankly put, the problem is too big for any one company to solve. We would welcome a vigorous competition for cost-effective, efficient SSH key management services.
Standards organizations such as NIST and the Internet Engineering Task Force can help mitigate the problem by specifically including SSH key management in published best practices. Furthermore, if the U.S. federal government eventually does move forward with comprehensive cybersecurity legislation, SSH key management and other automated access mechanisms (including Kerberos-based access) must be included.
We can do it, but we need to start now.