The Vista problem: a candid internal view

Posted on by

One of the puzzles that really interests me is why delivering Vista — the new incarnation of Windows — has proved so traumatic for Microsoft. It’s interesting because it raises the question of whether these code-monsters have now grown so large and complex that they are beyond the capacity of any single organisation — even one as smart as Microsoft — to manage. Here’s an extensive excerpt from a fascinatingly candid Blog post by a Microsoft insider. Apologies for the length, but it has already been removed once after posting (though the author says he came under no company pressure)…

Vista. The term stirs the imagination to conceive of beautiful possibilities just around the corner. And “just around the corner” is what Windows Vista has been, and has remained, for the past two years. In this time, Vista has suffered a series of high-profile delays, including most recently the announcement that it would be delayed until 2007. The largest software project in mankind’s history now threatens to also be the longest.

[…]

Admittedly, this essay would be easier written for Slashdot, where taut lines divide the world crisply into black and white. “Vista is a bloated piece of crap,” my furry little penguin would opine, “written by the bumbling serfs of an evil capitalistic megalomaniac.” But that’d be dead wrong. The truth is far more nuanced than that. Deeper than that. More subtle than that.

I managed developer teams in Windows for five years, and have only begun to reflect on the experience now that I have recently switched teams. Through a series of conversations with other leaders that have similarly left The Collective, several root causes have emerged as lasting characterizations of what’s really wrong in The Empire.

[…]

The Usual Suspects

Ask any developer in Windows why Vista is plagued by delays, and they’ll say that the code is way too complicated, and that the pace of coding has been tremendously slowed down by overbearing process. These claims have already been covered in other popular literature. A quick recap for those of you just joining the broadcast:

Windows code is too complicated. It’s not the components themselves, it’s their interdependencies. An architectural diagram of Windows would suggest there are more than 50 dependency layers (never mind that there also exist circular dependencies). After working in Windows for five years, you understand only, say, two of them. Add to this the fact that building Windows on a dual-proc dev box takes nearly 24 hours, and you’ll be slow enough to drive Miss Daisy.

Windows process has gone thermonuclear. Imagine each little email you send asking someone else to fill out a spreadsheet, comment on a report, sign off on a decision — is a little neutron shooting about in space. Your innocent-seeming little neutron now causes your heretofore mostly-harmless neighbors to release neutrons of their own. Now imagine there are 9000 of you, all jammed into a tight little space called Redmond. It’s Windows Gone Thermonuclear, a phenomenon by which process engenders further process, eventually becoming a self-sustaining buzz of fervent destructive activity.

Let’s see if, quantitatively, there’s any truth to the perception that the code velocity (net lines shipped per developer-year) of Windows has slowed, or is slow relative to the industry. Vista is said to have over 50 million lines of code, whereas XP was said to have around 40 million. There are about two thousand software developers in Windows today. Assuming there are 5 years between when XP shipped and when Vista ships, those quick on the draw with calculators will discover that, on average, the typical Windows developer has produced one thousand new lines of shipped code per year during Vista. Only a thousand lines a year. (Yes, developers don’t just write new code, they also fix old code. Yes, some of those Windows developers were partly busy shipping 64-bit XP. Yes, many of them also worked on hotfixes. Work with me here.)

Lest those of you who wrote 5,000 lines of code last weekend pass a kidney stone at the thought of Windows developers writing only a thousand lines of code a year, realize that the average software developer in the US only produces around (brace yourself) 6200 lines a year. So Windows is in bad shape — but only by a constant, not by an order of magnitude. And if it makes you feel any better, realize that the average US developer has fallen in KLOC productivity since 1999, when they produced about 9000 lines a year. So Windows isn’t alone in this.

The oft-cited, oft-watercooler-discussed dual phenomenon of Windows code complexity and Windows process burden seem to have dramatically affected its overall code velocity. But code can be simplified and re-architected (and is indeed being done so by a collection of veteran architects in Windows, none of whom, incidentally, look anything like Colonel Sanders). Process can be streamlined where inefficient, eliminated where unnecessary.

But that’s not where it ends. There are deeper causes of Windows’ propensity to slippage.

Cultured to Slip

Deep in the bowels of Windows, there remains the whiff of a bygone culture of belittlement and aggression. Windows can be a scary place to tell the truth.

When a vice president in Windows asks you whether your team will ship on time, they might well have asked you whether they look fat in their new Armani suit. The answer to the question is deeply meaningful to them. It’s certainly true in some sense that they genuinely want to know. But in a very important other sense, in a sense that you’ll come to regret night after night if you get it wrong, there’s really only one answer you can give.

After months of hearing of how a certain influential team in Windows was going to cause the Vista release to slip, I, full of abstract self-righteous misgivings as a stockholder, had at last the chance to speak with two of the team’s key managers, asking them how they could be so, please-excuse-the-term, I-don’t-mean-its-value-laden-connotation, ignorant as to proper estimation of software schedules. Turns out they’re actually great project managers. They knew months in advance that the schedule would never work. So they told their VP. And he, possibly influenced by one too many instances where engineering re-routes power to the warp core, thus completing the heretofore impossible six-hour task in a mere three, summarily sent the managers back to “figure out how to make it work.” The managers re-estimated, nipped and tucked, liposuctioned, did everything short of a lobotomy — and still did not have a schedule that fit. The VP was not pleased. “You’re smart people. Find a way!” This went back and forth for weeks, whereupon the intrepid managers finally understood how to get past the dilemma. They simply stopped telling the truth. “Sure, everything fits. We cut and cut, and here we are. Vista by August or bust. You got it, boss.”

Every once in a while, Truth still pipes up in meetings. When this happens, more often than not, Truth is simply bent over an authoritative knee and soundly spanked into silence.

The Joy of Cooking

Bundled with a tendency towards truth-intolerance, Windows also sometimes struggles with poor organizational decision-making. Good news is that the senior leaders already know this and have been taking active steps to change the situation.

There are too many cooks in the kitchen. Too many vice presidents, in reporting structures too narrow. When I was in Windows, I reported to Alec, who reported to Peter, to Bill, Rick, Will, Jim, Steve, and Bill. Remember that there were two layers of people under me as well, making a total path depth of 11 people from Bill Gates down to any developer on my team.

This isn’t necessarily bad, except sometimes the cooks flash-mob one corner of the kitchen. I once sat in a schedule review meeting with at least six VPs and ten general managers. When that many people have a say, things get confusing. Not to mention, since so many bosses are in the room, there are often negotiations between project managers prior to such meetings to make sure that no one ends up looking bad. “Bob, I’m giving you a heads-up that I’m going to say that your team’s component, which we depend on, was late.” “That’s fine, Sandy, but please be clear that the unforeseen delays were caused by a third party, not my team.”

Micromanagement, though not pervasive, is nevertheless evident. Senior vice presidents sometimes review UI designs of individual features, a nod to Steve Jobs that would in better days have betokened a true honor but for its randomizing effects. Give me a cathedral, give me a bazaar — really, either would be great. Just not this middle world in which some decisions are made freely while others are made by edict, with no apparent logic separating each from the other but the seeming curiosity of someone in charge.

In general, Windows suffers from a proclivity for action control, not results control. Instead of clearly stating desired outcomes, there’s a penchant for telling people exactly what steps they must take. By doing so, we risk creating a generation of McDevs. (For more on action control vs. results control, read Kenneth Merchant’s seminal work on the subject — all $150 of it, apparently).

Uncontrolled? Or Uncontrollable?

We shouldn’t forget despite all this that Windows Vista remains the largest concerted software project in human history. The types of software management issues being dealt with by Windows leaders are hard problems, problems that no other company has solved successfully. The solutions to these challenges are certainly not trivial.

An interesting question, however, is whether or not Windows Vista ever had a chance to ship on time to begin with. Is Vista merely uncontrolled? Or is it fundamentally uncontrollable? There is a critical difference.

It’s rumored that VPs in Windows were offered big bonuses contingent on shipping Vista by the much-publicized August 2006 date. Chris Jones even declared in writing that he wouldn’t take a bonus if Vista slips past August. If this is true, if folks like Brian Valentine held division-wide meetings where August 2006 was declared as the drop-dead ship date, if general managers were consistently told of the fiscal importance of hitting August, if everyone down to individual developers was told to sign on the dotted line to commit to the date, and to speak up if they had any doubts of hitting it — mind you, every last one of those things happened — and yet, and yet, the August date was slipped, one has to wonder whether it was merely illusory, given the collective failure of such unified human will, that Vista was ever controllable in the first place.

Are Vista-scale software projects essentially uncontrollable by nature? Or has Microsoft been beset by one too many broken windows? Talk amongst yourselves…

The answer to that question — Are Vista-scale software projects essentially uncontrollable by nature? — may well be “yes — if they’re done within a single organisation”. That’s why Steve Weber may well be right about Open Source: that it’s a better way of making unimaginably complex products.

Gates to spend more time with his money

Posted on by

It’s all over the Net, but here’s the News.com version…

REDMOND, Wash.–Bill Gates, the man who started Microsoft and has been its public face throughout its three decades of existence, plans to step away from daily work at the company.

Gates announced on Thursday that he will gradually relinquish his current role, ceding the title of chief software architect immediately, while remaining a full-time employee for the next two years. In July 2008, he will remain as a part-time employee and chairman.

The announcement comes as his company battles pressures on all fronts: a sagging stock price, competition from Google and nagging delays in the Vista operating system…

Happy Bloomsday!

Posted on by

JJ looking pensive in old age. We will have Burgundy and gorgonzola sandwiches at half past noon in honour of Mr Leopold Bloom, late of No. 7 Eccles Street, Dublin.

Which reminds me of a nice Irish joke:

First man: “Do you like gorgonzola?”
Second man: “No, but I hear his brother Emile is a bloody fine writer.”

An Duce, RIP (contd)

Posted on by

A friend telephones to tell me about the front cover of the new issue of The Phoenix, the nearest thing Ireland has to a satirical magazine. The issue marks the passing of its old adversary, An Duce. I can’t locate the publication online, alas, but my informant describes it thus:

The cover shows (President) Mary McAleese looking grimly presidential, eyes half closed, looking into the middle distance. The photograph was taken at some State occasion or other, possibly the centenary of the Easter Rising. Behind her stands Bertie ‘Gurrier’ Ahern, complete with black overcoat and red nose. McAleese is saying, “He touched millions”. Bertie is muttering, “You’re telling me”. Or words to that effect.

I also hear that some Joyceans, who often renact Paddy Dignam’s funeral on Bloomsday, are planning to make a detour to take in the Haughey obsequies. The Latin Americans have nothing on us when it comes to Magic Realism.

Jumpcut

Posted on by

The user-generated content bandwagon rolls on. Jumpcut is a web service which enables you to edit small movies (add soundtracks, efffects, etc) in your browser. According to this New York Times report, there are lots of workalike services on the way:

Eyespot, Grouper and VideoEgg, have been introduced within the last year. This summer, they will be joined by another site, Motionbox, based in New York.

Their shared objective, the founders of the sites say, is to reduce the complexity of video editing and to reduce the cost to zero.

“We wanted to make video editing over the Internet faster than desktop editing,” said Jim Kaskade, co-founder and chief executive of Eyespot, based in San Diego. “We think it will broaden the base of people who are creative, but may not have thought they were, by creating this tool kit for them. Editing video is eventually going to be as simple as sending e-mail.”

Mr. Kaskade refers to the process as “mixing,” however, saying he believes that the term “editing” may sound labor-intensive to the amateur videographer. Previously, putting together a multishot video like Mr. Moore’s would have involved installing and learning to use a piece of software like iMovie from Apple, Adobe Premiere or Studio from Pinnacle Systems. Some of that software is packaged free with new computers or sold for about $100.

The analyst firm Parks Associates estimated last year that only about four million people regularly use such software for video editing — far fewer than the number who capture video using camcorders, Webcams, digital still cameras and cellphones.

But with more videos of soccer games, weddings and cruise vacations being posted online — and potentially being seen by people who have not been dragooned into the living room for a showing — editing gains in importance, Mr. Kaskade says, even if it involves trimming only the dizzying camera whirls at the beginning of a shot, or the inevitable question, “Are you taping right now?”

The bandwidth implications of this are interesting. All of the sites, except Grouper, require that video clips be uploaded to their servers before they can be manipulated. That can take a long time, and there are limits to the size of the files that can be sent. (For Jumpcut, the limit is 50 megabytes per clip.)

Grouper users have to download a free piece of Windows-only software that works in conjunction with the Web site. It permits users to trim and rearrange clips on their PC and upload only the finished product, in compressed form.

An Duce*, RIP

Posted on by

My countrymen are disgracing themselves again. Charlie Haughey, the former Fianna Fail leader and Taoiseach (Prime Minister) has died, in his bed, of cancer, at the ripe old age of 80. As one of the most corrupt politicians in the short history of the Irish state (which means he came top of a high-quality field), he ought to have died in gaol. But that’s not what rattles me; it’s the unctuous drivel that prominent Irishmen and women are spouting today. Listening to them, you’d think that it was some weird combination of Spinoza and Nelson Mandela who had passed away. Listen, for example, to what the country’s leading sky pilots have been saying:

The Primate of All-Ireland, Dr Seán Brady, said Mr Haughey was an able and talented politician who did much to promote the interests of Ireland and her people.

Dr Brady said Mr Haughey was a reforming politician who had considerable success in introducing measures to take care of the less well-off and disadvantaged in our society.

He said Mr Haughey will also be remembered for pioneering public utility allowance schemes and free transport for the elderly.

The Archbishop of Dublin, Dr Diarmuid Martin, said Mr Haughey was a man who engaged the people of Ireland over the last 40 years on the public stage.

Archbishop Martin said that these days following the death of the former Taoiseach were not ones for writing history books. He said a full and balanced analysis of Mr Haughey’s impact on Irish life would take time and careful consideration….

John Hume, the Nobel laureate, said:

Peace and justice in the North of Ireland was always at the top of the agenda for Charles Haughey and when I started to talk to Gerry Adams, he strongly supported me. He worked very closely with me in preparation for the whole movement to get lasting peace and an end to violence with the Downing Street Declaration and he fully briefed his successor Albert Reynolds.

The former Fine Gael leader and Taoiseach, Garrett Fitzgerald said that Haughey was:

a man of formidable political skills. Despite their public political differences, their relationship was always marked by courtesy and absence of personal antagonism.

Eh?

It gets worse. Haughey is to be given a State Funeral on Friday, and the current Taoiseach and Fianna Fail leader, Bertie Ahern, is to give the oration at the graveside. I look forward to the solemn tones of the RTE commentators as the cortege passes various landmarks in Haughey’s rapacious career. The local branch of Allied Irish Banks, for example, which tried to call in Haughey’s six-figure overdraft and were told to get stuffed. (Deciding that it rather hoped to do further business in Ireland, the bank wrote off the debt.) Will there be a respectful pause when the procession reaches a branch of Dunne’s Stores, one of whose family directors (Ben) handed over colossal sums of money to Haughey in brown envelopes? And what of the numerous housing estates built on green belt land mysteriously rezoned for development after being purchased by Haughey and his mates? And will the cortege stop briefly at the polling station where Haughey’s election agent was caught voting twice (and later prosecuted for that offence)?

Compassionate souls will say that one shouldn’t speak ill of the dead, especially on the day of decease, and in general I agree. In which case, the right thing to have done today would have been to note Haughey’s passing, express condolences to his family and leave it at that. But for the State to honour so conspiciously a man who so comprehensively polluted Irish political life beggars belief. And it leads one to wonder what’s really going on.

Part of the problem with Haughey is that everybody knew he was bent — but nobody ever dared to say anything. It was only when Ben Dunne spilt the beans after being arrested for possession of drugs while on a junket to Florida that the whole can of worms was levered open. I remember once being on holiday in Dingle many years ago. Haughey had bought Inishvickillane — a beautiful, uninhabited island in the Blaskets off the Kerry coast — and was building a house on it. The problem with Inishvickillane is that it is largely inaccessible from the sea, so most of the building materials were airlifted in by helicopter. As I watched the aerial comings and goings I started to estimate the costs of the operation. At that time helicopter charter costs were something like £200 an hour. I looked up Charlie’s ministerial salary — it was, I think, about £60,000 a year. Eventually I said to a local onlooker: “How can Charlie afford this?” He looked at me, smiled slyly, and said “Aw sure, you know Charlie”.

And that, of course, was part of the problem. Everyone knew what Charlie was like. There was widespread tacit acceptance that the planning system — largely controlled by Fianna Fail — was comprehensively corrupt. Worse than that — there was a kind of cynical admiration of the brazenness of the Haughey clique — as Conor Cruise O’Brien discovered to his cost when he ran for election in the late 1960s.

O’Brien had held the Schweitzer Chair in the Humanities in New York University during most of the 1960s and was at that time a classic liberal intellectual. (He had, for example, been arrested during protests against the Vietnam war.) But he eventually decided that his country needed him and returned home to run for the Dail (the Irish Parliament). He ran against Haughey as a Labour candidate in the latter’s North Dublin constituency. (Under Ireland’s proportional representation system, there are multi-member constituencies.) During the campaign, O’Brien discovered that some farmland that Haughey had purchased in the locality had, mysteriously, been re-zoned for housing development, increasing its value tenfold. O’Brien fulminated against this apparent abuse of power and obviously calculated that in so doing he would damage Haughey. But he was wrong. Haughey was returned with a considerably increased majority. It was if the electorate was saying “Sure, he’s corrupt, but good luck to him.”

So why the sudden attack of amnesia brough on by Haughey’s demise? Could it be that it’s just too embarrassing for the proud jockeys of the Celtic Tiger to admit that, in the not very distant past, their country was a rotten little borough off the mainland of Europe, run by a corrupt bunch of shysters who were the direct political ancestors of our own dear Euro-friendly Taoiseach, Bertie Ahern? Better to emphasise the positive aspects of Charlie — for example his ‘contribution’ to the peace process — than to dwell on these sordid realities. To me, it smacks of the famous attempt to find something good to say about Mussolini: that at least he made the trains run on time.

* Footnote: ‘An’ is the definite article in Irish.

Stolen personal data not significant enough to report?

Posted on by

From Good Morning Silicon Valley…

On Friday, officials from The National Nuclear Security Administration told a House oversight committee that a malicious hacker stole a computer file containing the names and Social Security numbers of 1,500 employees of the Energy Department’s nuclear weapons agency. The theft was detected last September, but no one bothered to report it to senior officials until late last week. NNSA Administrator Linton Brooks blamed the cockup on “bureaucratic confusion.” “It appears that each side of that organization assumed that the other side had made the appropriate notification,” Brooks told the House energy panel’s oversight and investigations subcommittee. “Just as the secretary just learned about this week, I learned this week that the secretary didn’t know. There are a number of us who in hindsight should have done things differently on informing.” That explanation didn’t fly with Rep. Joe Barton, the chairman of the Energy and Commerce Committee, though. “That’s hogwash,” Rep. Barton told Brooks. “You report directly to the secretary. You meet with him or the deputy every day…. You had a major breach of your own security and yet you didn’t inform the secretary,” adding “you should be removed from your office as expeditiously as possible. And I mean like 5 o’clock this afternoon.”

Fact 1: The NNSA is a semi-autonomous arm of the Energy Department and also guards some of the U.S. military’s nuclear secrets and responds to global nuclear and radiological emergencies.

Fact 2: Earlier this week the Pentagon revealed that personal information on about 2.2 million active-duty, National Guard and Reserve troops was stolen last month from a government employee’s house.

[Source]