I can’t stand it, i know you planned it
Ima set it straight, this Watergate
I can’t stop textin’ when i’m in there
’cause your crystal ball ain’t so crystal clear
So, while you sit back and wonder why
I’m missing my pics when I slide my ‘Kick
Oh my god, it’s a mirage
I’m tellin’ y’all it’s sabotage
So,so,so, so listen up ’cause you can’t say nothin’
You shut me down with a push of your button
But AppleInsider knows why your data’s gone
I’ll tell you now I keep it on and on:
…someone with access to the servers at the datacenter must have inserted a time bomb to wipe out not just all of the data, but also all of the backup tapes, and finally, I suspect, reformatting the server hard drives so that the service itself could not be restarted with a simple reboot (and to erase any traces of the time bomb itself)… If this was an ordinary sort of failure, the service would have come back within a day, so once again, all signs point to sabotage.
’cause what you see you might not get
And we can bet, so don’t you get backed-up yet
Scheming on a thing that’s a mirage
I’m tryin’ to tell you now it’s sabotage
In all honesty, I have a problem with this theory. A three-pronged attack on servers like this is pretty difficult, especially from afar. While it’s obvious that a laid-off employee could have planted this during Danger’s move to Microsoft, Danger’s IT people must be pretty stupid to fall for something of this magnitude, y’all.
I must confess I didn’t have any interest in reading about the sidekick, I just came in to watch Sabotage. It’s awesome.
I have not actually seen them before. But they are fun.
I really doubt this theory. It seems to vindictive and I want to believe that even a laid-off Danger employee wouldn’t stoop this low.
I think it all boils down to HORRIBLE and disastrous execution while the upgraded hardware went into place.
Umm, http://twitter.com/youngandvacant/status/4834904652
The backup loss was due to a SAN upgrade failure (it’s nothing like a Windows/Linux server). There is no ‘Right Click – Format Drive’. In fact, most SANS are running embedded operating systems, something you couldn’t easily write a logic bomb (or time bomb as they say) for. Finally, you cannot simply reboot a SAN. When we needed to reboot our SAN (at work) it was a 3 hour call with HP. Debunked.
I wish it’d be just a normal shutdown, but I would not assume that. While it could be an employee, it could also be an outside hacker.
And while it would not be easy to write a logic bomb, it might be possible. I really do not know about those systems.
While there is not enough information to claim interference, it is clearly an interesting direction. We might hear more about it at the next hacker conference.
I’m a linux sys admin and this is generally the type of stories you tell management to coverup laziness.
They screwed up.. just man up and admit it.
This is T-Mobile’s Project Black
Seems to me that whenever MS is involved it’s virtually alwasy a MicroSoft screw-up — oh yeah, it’s a feature!!…
This is just Microsoft preparing for the release of its updated privacy policy:
“We collect a lot of personal data, but don’t worry, we’ll probably lose it all in a server crash.”
Sadly that article is so full of half-guesses based on little to no knowledge of how the infrastructure was designed.
For starters its already been repeatedly reported that it wasn’t Microsoft engineers that did the SAN work but outsourced Hitachi ones, who really should have had the skill set.
“To the engineers familiar with Microsoft’s internal operations who spoke with us, that suggests two possible scenarios. First, that Microsoft decided to suddenly replace Danger’s existing infrastructure with its own, and simply failed to carry this out. Danger’s existing system to support Sidekick users was built using an Oracle Real Application Cluster, storing its data in a SAN (storage area network) so that the information would be available to a cluster of high availability servers. This approach is expressly designed to be resilient to hardware failure.”
Using Oracle’s RAC, despite how good it is, is no guarantee of unbreakable. There is a lot of things that can go wrong at a SAN level. Moving to new firmware could present the SAN as the wrong type of storage to a server and result in severe corruption of all data. RAC can’t prevent that kind of occurance.
If all the RAC data was on a single SAN (Perfectly possible, we don’t know), with no backups and on no other controllers, even replacing a hard disk could be potentially lethal to the stored data. RAC doesn’t protect you from SAN failure, or some form of logical database corruption.
SANs can and will fail. It’s a simple fact. Any sysadmin worth half their beans makes sure to have multiple redundancies in place for everything.
From another blog:
‘I’ve got an IBM doc (sg246363) that says:
“Prior to physically installing new hardware, refer to the instructions in IBM TotalStorage DS4000 hard drive and Storage Expansion Enclosure Installation and Migration Guide, GC26-7849, available at: [...snip...] Failure to consult this documentation may result in data loss, corruption, or loss of availability to your storage.”
Does that imply that plugging the wrong thing into the wrong place at the wrong time can ‘f up an entire SAN? Yep. It does, and from what the service manager for one of our vendors told me, it did happen recently – to one of the local Fortune 500’s.’
Heck, without the proper checks and balances in place even a fat-fingered DBA or sysadmin could accidentally wipe all the data on a system, or even just that little bit that is critical. The SAN upgrade could be entirely co-incidental to the whole thing. Without a working backup they’re still screwed.
All speculation on the subject is entirely moot because of a few simple points:
1) We don’t know the full architecture of their system. – How many redundancies were there? How many SANs? What Single Points of Failure were there?
2) We don’t know how the data was actually lost – DBA fat-finger? Misconfigured SAN? Misunderstood communication between DBAs and SAN team? Unforeseen bugs in the controller firmware (happens more often than you’d expect)
+1 – Great comment!
Please! What a really sleazy thing to say. It was sabotage… whatever. They’re acting like children, saying “it wasn’t me!” pwn up to and and move on.
Well, that’s what you get for having named your company Danger. Karma catches up with you sooner or later… :P
It takes a lot to do this kind of damage. I think the most suprising part to everyone is no signs of a backup. Rouge employees at a big company happen sometimes, so having a backup is that much more important.
I think people underestimate the problems bad communication causes. Microsoft and Danger obviously didn’t get along well. Take bad communication, bad information as a result and combine it with an unfortunate IT upgrade plan and boom, colossal data loss.
All of these supposed “inside sources” could very well be disgruntled Danger employees with an ax to grind, but they are burying that axe by spreading FUD not executing elaborate sabotage plans.
Occam’s razor.
Start with:
“Microsoft has a 30 year history of total contempt for its users”
…then go up, out from there.
( It’ll be a short trip )
You have to lay off the idiot koolaid. Microsoft has long been the front runner in quality products. You are just parroting the words of your fellow morons.
I am reminded of a saying, I don’t remember where from:
“Never ascribe to malice that which can be explained by incompetence”
I think the sabotage theory is a bunch of bunk. I do believe that there are a fair number of disgruntled Danger (ex or otherwise) employees who are venting though.
Good riddance (from Sidekicks). Now it’s time for TMo to offer something actually good.
Also, maybe this will teach some people to back up their data? Even though it is stored “in the cloud”.
I believe there’s no way to backup Sidekick data locally, you can only backup to their servers.
Rule #1. Make redundant backups.
Rule #2. Store some of them offsite.
It is somewhat disingenuous to represent that the quote taken out of context from the article was something dreamed up by the writer at AI, rather than simply one of two possible scenarios presented by sources within Microsoft/Danger.
In fact, the way it is cited looks as if you were trying to intentionally misinform your own readers.
Your own analysis, that ‘Danger’s IT people must be pretty stupid to fall for [sabotage] of this magnitude’ is kind of silly just because the event is pretty clearly evidence of rampant incompetence no matter what actually happened.
Further, as the article pointed out, Danger’s IT staff was largely dispersed over the last (nearly) two years that Microsoft ran the data center. Based on multiple sources I’ve spoken with, there was clearly a very high level of disgruntlement, anger, and frustration among former Danger employees.
Enough to sabotage the operation? Perhaps not on a physical level, but there have been enough leaks at damaging times (including the scathing leak published here on this site earlier) to completely sabotage Microsoft’s reputation as a product manager, datacenter manager, and corporate partner.
Commented just for the video.
I don’t think this makes much sense but generalized apathy in operations environments or the honest mistake of buying and forgetting what it takes to get the job done has happened at places like Microsoft before. Abstraction of responsibility is an art form.
its Fun…
http://www.justcompareit.com?ag=3