I don’t think this makes much sense but generalized apathy in operations environments or the honest mistake of buying and forgetting what it takes to get the job done has happened at places like Microsoft before. Abstraction of responsibility is an art form.

In fact, the way it is cited looks as if you were trying to intentionally misinform your own readers.

Your own analysis, that ‘Danger’s IT people must be pretty stupid to fall for [sabotage] of this magnitude’ is kind of silly just because the event is pretty clearly evidence of rampant incompetence no matter what actually happened.

Further, as the article pointed out, Danger’s IT staff was largely dispersed over the last (nearly) two years that Microsoft ran the data center. Based on multiple sources I’ve spoken with, there was clearly a very high level of disgruntlement, anger, and frustration among former Danger employees.

Enough to sabotage the operation? Perhaps not on a physical level, but there have been enough leaks at damaging times (including the scathing leak published here on this site earlier) to completely sabotage Microsoft’s reputation as a product manager, datacenter manager, and corporate partner.

Rule #2. Store some of them offsite.

And while it would not be easy to write a logic bomb, it might be possible. I really do not know about those systems.

While there is not enough information to claim interference, it is clearly an interesting direction. We might hear more about it at the next hacker conference.

Also, maybe this will teach some people to back up their data? Even though it is stored “in the cloud”.

“Never ascribe to malice that which can be explained by incompetence”

I think the sabotage theory is a bunch of bunk. I do believe that there are a fair number of disgruntled Danger (ex or otherwise) employees who are venting though.

Start with:

“Microsoft has a 30 year history of total contempt for its users”

…then go up, out from there.

( It’ll be a short trip )

All of these supposed “inside sources” could very well be disgruntled Danger employees with an ax to grind, but they are burying that axe by spreading FUD not executing elaborate sabotage plans.

For starters its already been repeatedly reported that it wasn’t Microsoft engineers that did the SAN work but outsourced Hitachi ones, who really should have had the skill set.

“To the engineers familiar with Microsoft’s internal operations who spoke with us, that suggests two possible scenarios. First, that Microsoft decided to suddenly replace Danger’s existing infrastructure with its own, and simply failed to carry this out. Danger’s existing system to support Sidekick users was built using an Oracle Real Application Cluster, storing its data in a SAN (storage area network) so that the information would be available to a cluster of high availability servers. This approach is expressly designed to be resilient to hardware failure.”

Using Oracle’s RAC, despite how good it is, is no guarantee of unbreakable. There is a lot of things that can go wrong at a SAN level. Moving to new firmware could present the SAN as the wrong type of storage to a server and result in severe corruption of all data. RAC can’t prevent that kind of occurance.

If all the RAC data was on a single SAN (Perfectly possible, we don’t know), with no backups and on no other controllers, even replacing a hard disk could be potentially lethal to the stored data. RAC doesn’t protect you from SAN failure, or some form of logical database corruption.

SANs can and will fail. It’s a simple fact. Any sysadmin worth half their beans makes sure to have multiple redundancies in place for everything.

From another blog:
‘I’ve got an IBM doc (sg246363) that says:

“Prior to physically installing new hardware, refer to the instructions in IBM TotalStorage DS4000 hard drive and Storage Expansion Enclosure Installation and Migration Guide, GC26-7849, available at: [...snip...] Failure to consult this documentation may result in data loss, corruption, or loss of availability to your storage.”

Does that imply that plugging the wrong thing into the wrong place at the wrong time can ‘f up an entire SAN? Yep. It does, and from what the service manager for one of our vendors told me, it did happen recently – to one of the local Fortune 500’s.’

Heck, without the proper checks and balances in place even a fat-fingered DBA or sysadmin could accidentally wipe all the data on a system, or even just that little bit that is critical. The SAN upgrade could be entirely co-incidental to the whole thing. Without a working backup they’re still screwed.

All speculation on the subject is entirely moot because of a few simple points:

1) We don’t know the full architecture of their system. – How many redundancies were there? How many SANs? What Single Points of Failure were there?

2) We don’t know how the data was actually lost – DBA fat-finger? Misconfigured SAN? Misunderstood communication between DBAs and SAN team? Unforeseen bugs in the controller firmware (happens more often than you’d expect)

“We collect a lot of personal data, but don’t worry, we’ll probably lose it all in a server crash.”

They screwed up.. just man up and admit it.

I think it all boils down to HORRIBLE and disastrous execution while the upgraded hardware went into place.