Fighting crime used to be, if not easy, then at least relatively straightforward. It was a policeman’s job to catch anyone who had committed any of a well-understood set of crimes.

“When I was recruited in the good old days, the criteria was that I be at least 177cm high, have good eyes, and that I could hammer people in the head and follow an order,” says Troels Oerting, the assistant director of the cyber crime division at European criminal intelligence agency Europol.

Today, it is becoming more complex, thanks to the rise of cyber crime. The rapid evolution of cyber criminal methods is challenging the ability of police forces to define and identify crimes, let alone catch the perpetrators.

“At no time in our history have we had to stretch the definition of what constitutes crime more than we do now,” says Michael Welch, the deputy assistant director of the FBI’s cyber division. The division is expected to double in size during the next 12 to 18 months, he says.

The cyber criminal underground is a textbook example of how collaboration drives innovation. Malware marketplaces, for example, allow users to give feedback on what works and what does not, helping virus writers to hone their products for specific uses and rework code for new targets.

Related: Fighting cyber crime needs security champions

“What’s happened is the development of an illicit economy, where bad guys are trading tools, products and services with each other,” says James Lyne, a technology researcher at security firm Sophos. “You even see Q&A platforms, where you can upload your virus to check it against 30-odd antivirus products. It’s an astonishingly mature business model.”

But the frictionless collaboration seen in this illicit economy has not been matched by the authorities attempting to crack down on it. Instead, they run up against barriers to collaboration at every turn: between nations, between businesses, and between the public and private sectors.

This is an issue that the government acknowledged in its new cyber security strategy, published in November 2011.

The strategy specifically calls for greater collaboration between private and public sectors in securing the country’s interests online.

The strategy outlines plans for a cyber security ‘hub’, allowing the private and public sectors to exchange information on existing security threats within specific business sectors.

GCHQ, the Ministry of Defence’s listening station, will play a pivotal role in the strategy, the government revealed. It is to receive around half of the government’s planned £650 million investment in cyber security, and will be offering its security technologies to private industry.

“There may well be things developed by GCHQ that could be used for commercial purposes,” a GCHQ spokesperson told The Guardian newspaper. “Up until now, some of the clever things that have been developed have just sat on a shelf. GCHQ may not know how to use it, but private companies may be able to.”

Beyond that, though, many questions remain about how businesses, public bodies and police forces might collaborate in the fight against cyber crime. There are a number of legal and cultural barriers to this collaboration, and the government has yet to specify how it plans to overcome these.

Handling sensitive data

One organisation wrestling with these issues is the Danish government’s computer emergency response team, or GovCERT, which last month was moved from the National IT and Telecom Agency to its Ministry of Defence.

GovCERT monitors the networks of government institutions and utility companies. It tracks the movement of data in and out of those networks, recording every IP address, reading every packet and measuring how long the connections last.

Given the sensitive nature of this data, keeping it secure is of the utmost importance, explains GovCERT chief Thomas Kristmar. The data is transmitted to GovCERT’s classified network, which is separated from the unclassified, web-connected network by a one-way ‘datagate’.

“Once the data is in our classified network it will stay there,” Kristmar says. “Even if we are hacked and are penetrated, we can guarantee that the data inside that classified network stays there.”

The data is kept for six days. During this time, GovCERT correlates the IP addresses of a list of known bad IPs and the traffic is analysed for abnormalities, such as large volumes of data leaving government networks at unsociable hours.

“If we have an attack, we have the ability to go back and investigate what happened,” Kristmar says. “We can do risk assessments because we have a unique picture of what is happening on the public sector networks.”

Using this data collaboratively with other organisations is extremely difficult, Kristmar reports. Legal constraints mean that the data can only be shared with the Danish military’s own CERT and, if it is relevant to an ongoing investigation, the police.

Cooperation with the private sector works in other ways. The majority of Internet service providers voluntarily block cyber attacks that GovCERT detects. “We have very close working relationships with the ISPs,” says Kristmar. “There are certain attacks that they are just as eager to block as we are.”

Although there are certain technical challenges, Kristmar is clear that developing cross-sector collaboration in the fight against cyber crime is primarily a political challenge. “This is a political process,” he says. “You will not have a national impact just because you are a good technician.”

According to Europol’s Oerting, sharing information to help solve crimes is contrary to the dominant culture in most police forces. “The culture is to protect your own know-how and knowledge – it’s my informant, my information and I will not share it, not even with a colleague in another district,” he says.

This reluctance to share is particularly damaging to the fight against cyber crime, which requires rapid collaboration to keep up with the latest threats, he adds. “In the online world, we need a whole new speed and a whole new level of reaction,” Oerting says.

He adds that collaboration is not helped by the mismatch between data retention laws and the processes that national crime-fighting agencies must go through to access one another’s data. “In Germany, the police can keep information from the Internet for seven days,” he says. “But if we need to access a German server to identify who has been dealing with, say, child pornography, I can guarantee you that it will take more than seven days.”

Private businesses and cybercrime

None of this is to say that international police forces are not already successfully collaborating to fight cyber crime. In September this year, for example, Scotland Yard’s e-Crime unit worked with the FBI to arrest four men in connection with the hacking exploits of LulzSec.

But there are not enough resources to combat the long tail of less noticeable cyber crimes, which cause the majority of the economic damage, with special operations such as this. Tackling everyday cyber crime therefore requires collaboration in the private sector.

IT security companies already share information with one another to help tackle malware threats, says Sophos’s James Lyne.

“It’s a little-known secret that the major antivirus companies share [malware] samples, and we have done for years,” he explains. “It was a gentleman’s agreement in the early days of malware, when we realised that if we didn’t share we’d all fail. It’s a race to write the fastest and best protection based on available data.”

Not all the data that the likes of Sophos collects on malware would be of any use to the authorities, however. “You wouldn’t be able to do anything useful with a lot of that data because it’s just a pile of code,” Lyne says.

More useful for cyber crime fighters is information about the attribution of a threat – in other words, data about where it came from.

Sophos analyses attribution by tying together malware that it finds in the wild, building up a picture of the author of a particular string of malicious code. “When you link [malware code] together into a campaign, and you can get your hands on information like servers and IP addresses, that becomes useful to intelligence agencies, because they can coordinate international police efforts,” Lyne says. “They can do things that we can’t.”

However, Lyne says it remains to be seen how the lines of communication between the IT security industry and the police might function.

Another rich source of relevant data in the private sector lies with organisations that have fallen victim to attacks. But these organisations are typically reluctant to go public with their misfortunes.

When Sony, for example, fell victim to a wave of hacking attempts during the first half of this year, took weeks to reveal that its online gaming networks had been breached and the personal data of millions of customers stolen. This reluctance to disclose the breach provoked negative press coverage, launching the company into the media spotlight and making it a ripe target for further hacking.

Draft amendments to the EU’s Data Protection Initiative include the proposal of data breach notification law, however, which may go some way to dissolve the culture of secrecy, and lead to more cooperation.

Behind many of the UK government’s recent policy announcements around cyber security and cyber crime lies the additional motive of economic development.

The Cyber Security Strategy was upfront about this. “In order to support the private sector in taking the opportunities that cyberspace offers, we will aim to foster a vibrant and innovative cyber security sector in the UK, with global reach,” it says.

That point was welcomed by Raj Samani, chief technical officer for EMEA at security vendor McAfee. “I was really happy with the recognition of the economic opportunities that the cyber industry can represent in the UK,” he says.

“Ed Miliband stood up at the Labour Party conference in Liverpool and talked about manufacturing being the answer to breaking out of this economic situation, but can we really compete there? Our biggest asset in the UK is our brain power.”

But if the country is going to exploit the advent of cyber crime as a source of business opportunity, there is a danger that it may make cross-sector collaboration even more complex. Might there be occasions when the interests of UK cyber security businesses clash with the public interest?

The rules of engagement between the public and private sector over cyber criminal activity need to be clearly defined, and subject to public scrutiny.