On 27th April, Sony sent me a fairly long, simplistic, patronising e-mail that completely failed to in any way apologise or offer recompense for their inability to secure my data. It was so bad in fact that Outlook tried to protect me from it, throwing into the Junk Mail folder.
I haven’t pored and pored over it. Fortunately, being of reasonable intelligence, I can read something once, and often retain much of that content in my head.
Some things have struck me in the intervening time:
Why didn’t Sony systems see the data leaving its network?
Sony has said that all PSN users’ accounts have been compromised. Accordingly, someone, presumably external to the PSN, managed to pull 77 million records from the database, without the data type being analysed by the firewalls through which it passed.
To reliably deliver large quantities of content over the internet, where some of the content itself is in large chunks (e.g. downloadable game patches, at between 20 and 600MB each), the network is split into sectors. The user interactions with the store front (PlayStation Store) are handled by one set of servers. The delivery of the game data (patches, downloadable content etc) is managed by a separate “content-delivery” network. The PlayStation Store instructs the CDN server to send the file to the customer, and in that way the customer interactions with the store (or the interactions of other customers) are not affected by the large download in progress.
Separation of PSN account services from content delivery services (thanks to Sykonist for the PS3 icon used under Creative Commons)
The assertion I’m making here is that the PSN should expect to be serving large files from CDN servers, but that the user data in question (the 77m records) would not be on those CDN servers – or at least should not be. This would make the delivery of those user records – which would be a substantial file transfer – an unusual event that intrusion prevention systems can be configured to look for in the traffic flow of standard operations. Was an intrusion prevention system (IPS) in place, operational, and properly configured? (Note: in the recent hack of Barracuda, the company ‘fessed up to the fact that their IPS was in place, and was configured, but had accidentally been left in “watch & learn” mode).
Were attackers hitting the content delivery servers, not the Store servers?
The content delivery servers shouldn’t really have any interaction with the Store servers, except to confirm that a request coming in from a user is to be accepted or declined. This can be secured quite simplistically, with a challenge/response request between the CDN server and the Store server. Firewalls between the two networks can be configured to allow a maximum conversation size, and cut anything longer than that.
This allows a lesser level of security to be put in place to defend the CDN servers, as they’re fairly dumb boxes configured to dump large quantities of data to the internet. And economising on security can – if done correctly – save real money.
How were the attackers able to get access to all the database records?
One of the requirements I have for securing web database systems is that the web servers are only allowed very limited access to the database. The web server acts as a proxy for the user (or attacker), passing the user’s request for information to the database server. By correctly configuring the database user account that the web server uses, security can be engineered into the data store.
PSN Dialogue: How not to get user information
Fundamentally, there is no requirement for the web servers’ user accounts to be able to retrieve all database records. The web servers need to be able to query the entire database, but at no point should the content of the database be returned to that server. It is enough to query for the existence of a record, and then return only that record, instead of grabbing all the records, then checking to see if the data grabbed contained the right information.
Let's talk about this guy
This is also a design function that should be being used for performance and scalability reasons, rather than as a pure-play security measure. Once again, it requires the security consultant to understand the size of a valid conversation between servers, and then to configure the security systems to block outsize traffic streams.
So, how was the security landscape at Sony?
Well, there was the revelation that one of the attacked servers was running software at least ten years old… This is unforgivable. It points to an underlying security attitude of “if it ain’t broke, don’t fix it”, which was the prevailing attitude to software versioning in around 1998. The critical flaw with this attitude is that it assumes that the threat landscape does not change with time. An assumption so ridiculous that even as I type it out I want to flame it until it’s dead.
That aside, drawing conclusions from such a limited information set is always risky. However, it seems certain that the PSN was not secured on a behavioural level.
The fact that hackers were able to extract large quantities of data from systems that should only be serving data fragments suggests that no systems were in place to check that servers were doing things they should not be. It seems that the network layers being secured were low-level, and that there was little – if indeed any at all – application intelligence being applied. Consequently, there is little to support the existence of Intrusion Prevention System (IPS) devices in the Sony network. Making accusations is easy, but really, it’s difficult to conside the omission of IPS as anything other that professional negligence either by a security architect not specifying this equipment or by a manager deleting it for budgetary reasons.
Sony has talked of “adding” firewalls, implying that they may have been scrambling to introduce additional network tiers that add abstraction between the frontline servers that PS3s talk to and the back-end database that powers the network intelligence. Perhaps they’ve been implementing these IPS devices, but “simplifying for the masses” the accounts of their actions by just calling them firewalls.
Sony must also be scrambling to bring up to date the web servers that were running the aged software. This isn’t something that should be rushed: it’s likely that many programming methodologies used in developing software circa 2000 are now prohibited, and will have to be recoded, retested, and, well, generally pretty much rewritten from scratch.
They should be shifting network security to be both at the firewall (request/accept/deny) level, and at the behavioural (inspect/review/allow/block) level. Database servers configured to only provide very limited data sets. No account information in the content delivery network. Different access levels for PS3/PSP devices from other devices. All things that should have been implemented before, and should be relatively simple to implement now.
In the meantime, the irony is that my 21st Century internetworked high definition console sits idle, while my need for a gaming fix is satiated by an organisation that is proudly “Steampowered“.