The project failure & cloud connection

Around 2000, I worked on a project with a co-worker – he was maybe fifty and had worked in the IT industry – initially at IBM then contracting and consulting for nearly thirty years. At the time; he was heading up a web-development and creative team for a small consultancy firm which he saw as a new opportunity for him. He had a good reputation, people liked him and he had a vast amount of experience from being involved in literally hundreds of projects – developing, managing, business developing etc.

One day he slips into conversation – as an aside really; that he has never worked on a project that was delivered. Every project he worked on had been cancelled, merged into another programme, didn’t meet its financial goals (as with infrastructure consolidation) or his involvement ceased before it ever went live (if any of them ever did – he didn’t know). He said it in a resigned manner, almost as a badge of honour that he had got through all that with no job satisfaction in terms of delivery payoff whatsoever.

I have seen this scenario repeated many times in the decade since; although perhaps none quite so extreme. I have myself worked on hundreds of projects in many roles and can count the number of successful systems deliveries on one hand. People naturally want to associate themselves with successes rather than failures so it is perhaps understandable that this is not a common consultant topic.

A whitepaper released late last year attempts to put a figure on the worldwide cost of IT project failures. This turns out to be $6.2 Trillion and it doesn’t look like sensationalism. The US in particular is apparently losing almost as much money per year to IT failure as it did to the financial meltdown (with no end in sight). The paper makes an attempt to factor in what it calls indirect costs; basically a lost opportunity cost from the time wasted on failed/abandoned projects. It does not however take into consideration the wider indirect costs of people training for careers that are not actually delivering, IT staff disillusionment (turnover), operational failures for delivered IT systems (one in five businesses lose £10,000/hour through systems downtime) and associated security failures.

The paper has received some condemnation (due to its base assumption of a 65% IT project failure rate) but there is dearth of analysis in this area and quoting this figure is about as good as it gets right now. 50-80% figures have been quoted in one form or another for decades. CIO thinks rates are actually rising due to the recession. People have the choice of either working with a figure (challenging assumptions/statistics etc) or burying their head in the sand.

We will never know the exact figure for IT project failure. Similarly, we will never know whether efficiency/functional benefits of those systems that have worked have paid for the failures i.e. What has IT really given us? We are simply reliant upon these systems going forward. What can we do to reduce future project failure rates?

Although there are superficial similarities with scientific/experiment and IT/project communities respectively, the "accept defeat" approach of the former is routed in constant learning whereas IT is surely about delivering benefit now. Learning is only the priority for the largest, most stable, lowest turnover organisations; that is to say - next to none of them. The scientific regimen of independent assessment however is invaluable for IT projects. Tool-based PM consultants such as Bestoutcome are probably as good as the big management consultants for this purpose though. Techniques from the engineering community (when introduced to IT) have not had a huge effect.

The paper ends with a call to arms to simplify – IT/business communications, projects goals etc.

I agree in principle but this is oversimplification - when your realm of influence is the organisation you are in. I have seen many projects that although conceptually simple and with genuine IT/business agreement start to fail the moment integration with other organisations – vendors/hosting providers/recruiters/sub-contractors/data sources are required. Despite solutions being simple and manageable inside your organisation – just a few touch-points with others (basically anything worth doing) cause them to be complex and therefore unpredictable/at risk no matter what your collective capabilities. There is even a case for blinkered-simplification/procedures actually contributing to project failure: Complexity at least brings an element of flexibility, allowing you to react if the project starts going bad.

Better project managers, SOA, ramped-up co-worker involvement, Facebook-like "hackathons", daily IT/business meetings, PRINCE certifications, extranets, more rigorous cost control and mathematical complexity models within your organisation will have minimal effect on the success of your multi-touch-point projects – for they are already in the realm of chaos. Even improved PM collaboration through tools such as Asana will have a minimal effect on success rates. The role of the “good project manager” is perhaps the most scrutinized, personality-driven, divisive and misunderstood of all IT positions. Radical open enterprise models such as BetterMeans that effectively remove Project Managers in favour of automation and decentralized decision making will similarly be ineffective. Although in the case of the open enterprise model, I do agree that this will ultimately prevail (crowd-sourcing, creativity enabling and ultimately efficiency/cost) but not for decades (due to the need to directly attack the failure rate first).

Although there are certainly sizeable success increases to be made if you are experienced in the particular technology, the IT project failure rate will only consistently and materially fall once there are flexible cloud services that organisations can get 80% of all their needs by just subscribing to them.

Integration ceases to be the bottleneck. The other 20% being “secret sauce” value-add-ons developed in-house; probably mainly algorithms that, by definition, do not require integration with other organisations or heavy project governance. Other components of the 20% would be device specific exploitation code; essentially building the so-called App-Internet model (rather than full-Cloud). Organisations already recognise the economic justification for cloud computing so it is perhaps inevitable that project failure rates will eventually fall by default.

Cloud transition will take organisations years yet however. Both InfoWorld and Gartner have arrived at 2013 for broadly when the majority of organisations will run cloud. This may be optimistic. In the interim (three more years+ of high project failure rates?), delivery is simply better served by being built upon cloud solutions now; building partnerships with cloud providers if they can and leveraging their buying power if they cannot. Also, interim/limited functionality cloud solutions must be considered in preference to bespoke development/on premises deployment. In a real sense the project (that they would have otherwise completed themselves) becomes a creativity-driven; architectural investment and commercial partnership instead. Both Project Management and Enterprise Architect roles will need to shift accordingly.

It would be interesting to see any future IT project failure analysis split between organisations that have implemented virtualisation, those that have implemented private cloud and those that have implemented public cloud. The project failure and cloud connection is not well documented.

Starbucks, here’s how you get into the outsourcing business

Hello Starbucks. You have made enormous success of the past fifteen years and have become an integral part of 21st century global cultural fabric. You have a store on the Great Wall of China, introduce new words to us like Yirgacheffe and are a bit like Viagra and The Simpsons (you aren’t technically the best but you’re easier to find and we have so much fun with you - we don’t care). We’d come to you eight-days-a-week if we could. You are to be applauded.

You have reached somewhat of an impasse though. You aren’t growing much; many of your stores are busy only at lunchtime and your brand doesn’t make us think – “that’s progressive!” anymore. Free Wifi/Foursquare deals, exclusive album sales and instant coffee will not get you those 40,000 stores you wanted a few years back. Your mantra of “A Starbucks on every corner” remains a good one. We know it’s tough out there but you just need to stick with your plan; maybe be a little bolder. Here’s what you do:

1) Recognise that you have to diversify. Your huge rent bill will surely eventually cripple you. You either need to dramatically cut costs (how? [Given your locations]), increase demand for coffee (how? [Given everyone drinks it anyway) or expand into new markets.
a. In 2008, the FM market was approximately $846BN, with approximately half ($426BN) apportioned to internal services meaning that the outsourced FM market in 2008 was worth around $420BN. It is surprisingly difficult to obtain free global branded coffee shop market sizes but for the UK at least, this is a $2.5BN market (2009). Let’s assume the UK is 5% of the global market (pretty standard), leaving a total market of $50BN. This means the FM market is roughly ten-times the size of the branded coffee-shop market.
2) Build more stores. Very roughly and taking the UK as a case in point; you have 750 stores and there are 30M people employed in the UK. With a few (contentious) assumptions, if you increased the number of stores tenfold (7,500), each store would need to accommodate just 200 people (1.5M/7,500). Obviously, it would need to take place over some years. A burgeoning senior citizen population, increased contract working and home-working will reduce the market, making the figure more manageable longer term.
a. Assume half actually work in an office (the rest in retail stores, hospitals, lumberjacks, machinists in plants, plumbers, nurses etc.). This takes the potential market down to 15M.
b. Assume half work for big name organisations that will want to maintain their own premises (taking it down to 7M - basically the SME market).
c. Assume half of those actually work in an office at any given time (the rest visiting clients, training, sick/vacation, travelling, WFH etc.) taking it down to 3M).
d. Assume you lose half the remaining people to other coffee-shops as the market is quite fragmented at the lower-end (taking it down to 1.5M).
3) Use your Starbucks card. At the moment this is used as a store card (arguably faster than paying otherwise). Put an RFID chip in and use it to track peoples employing organisations, access times and automatically bill the organisations according. You can hugely undercut existing FM services if you open-up this new revenue stream. You also expand your coffee market.
4) Build meeting rooms. Organisations need secure ad-hoc meeting rooms (HR, competitive, strategic discussions etc.). Let’s assume all new stores have one. These would need to be empty by default i.e. not having coffee drinkers in them and controlled by an online booking system. Let’s also put sophisticated video conferencing facilities in each one. Of course meetings are going to overrun and the people outside waiting for the next slot are going to have to either play nice/assertively claim their room but this happens in offices already. You might want to partner with others for larger, scheduled meetings.
5) Deploy IT Infrastructure. Cyber-cafes may be on the wane as cheap mobile devices rise but you would need to pop Internet terminals in your stores to mop up those without laptops at any given point. The shift to cloud-based computing means organisations won’t need development/file/application servers because they won’t have IT departments. Each store is also going to need a couple of wireless printer/scanner/copiers.
6) Go stealth. To avoid monstrously over-selling your brand, you are going to need to expand your stealth experiments on a wider scale. Focus individual stores on the areas they are in (creative/business/education etc.). Maybe change the decor to fit-in with local murals on the walls. It may be healthy to engender some competition between them. There would clearly need to be more variety in (interior and exterior) store design.
7) Forget the Baristas. Everyone knows this isn’t a skilled job. Stop pretending it is. It’s not like they spend years learning the correct Frappuchino for the chocolate Starbucks coin customers eat. They’re a bit like your Starbucks cards – over-engineered. Do give them training but make this in basic IT services in addition to working the coffee machine. They should need to know how to reboot the router, connect to it and any of the various wireless devices you have in your store from most portable devices, reset passwords, create accounts and escalate issues – that sort of thing. Ultimately, they’ll thank you for it. Future employers will place much more emphasis on IT service skills.
8) Culture shift slightly.
a. “Third-place”. This internal marketing needs to go. Yes – there’s a place for a safe haven, a "third place", that place outside of work and home where you know that you will be greeted with a smile and some respect. This is more than a coffee shop though. It is now a hackneyed term anyway. It was used at the Playstation 2 launch and is employed by countless gyms over the world. Is it really harder to create another market than get a good chunk of one (or both) of the existing ones?
b. Seat-saving. This needs to go. Someone cannot come in, sit down on one of your sofas and then “save” seats around them; dissuading potential users as their “friends are coming”. This prevents people from using stores for more than a quick coffee i.e. to work. Hot-desks are essential. It has to be first-come-first-served. Subtle advertising cues should be able to make this culturally frowned upon so it ceases to be an inhibitor.
c. Table Service. Your service isn’t great at lunchtimes. Queues can be large. People on laptops are dissuaded from leaving their laptop but they still want a coffee. Your new Trenta sizes may address this issue (slightly) but your smaller competitors offer table service for the same price.
d. Enhance security. You cannot have hoodies/Hells Angels/gypsies/beggars etc. associating with Senior Executives (can you?). You are likely going to need a security guard in most stores to gently dissuade them. Can’t they all do double-duty as Baristas too though? Security Barista? IT Barista? Table-service Barista? They can be more Pokémon than Borg.
e. Get out of food. You are not known for your food. Stay with chilled things that go well with hot coffee e.g. muffins, cakes, chocolates, biscotti etc. The hot breakfast sandwiches, wraps and salads all need to go. They take too long, are odorous, other brands do them better, people don’t want them in their office and will also want a break from you (their workplace) to go get them anyway. Get a food partner if you must and link it to your Starbucks card. We can work it out.

You have the cultural and economic reach to become our workplace. This isn’t something you can do quickly. It’s a goal over the next fifteen years. You can choose to move up from being an escape to being a destination. That journey will mean you need to take a leap and recognise that you’re big enough right now and that you’ll have missed service elements along the way (but that others will fill-in and contribute to the new eco-system). It may also mean you concentrate on the back-office, lose a bit of your élan/put your brand on the back-burner and cancel that order for corporation T-shirts.

A little like those faceless East-India type holding companies that keep going for hundreds of years. That’s OK though. You have certainly let your face grow long of late but to paraphrase The Beatles further; you are the coffee man. They are the coffee men. You are the water-cooler.

Why you should outsource your office to Starbucks

Office rules have relaxed a lot over the last ten years. Many office workers are now working from home regularly and when we are in the office; we are all comfortable having water-cooler discussions in a coffee-shop and taking our laptop in to work. Even Government recognises the benefit. The days of rigorously putting in a nine to five every day in a shirt and tie are all but gone. Why not go further and – do most or all of our work there; dispensing with the need for our current physical offices? Yes - it’s a little out-there but idly run with it for a while.

Yes, we are talking facilities management (FM) but – perhaps a narrower definition of it with all value-add services being done by partners. Could it actually be done? There is a definite market for professional, ad-hoc and casual working environments e.g. The Hub.

What would the benefits be?

1) Cost savings. FM has become an important industry/profession; responsible for approximately 5% of GDP in the most developed countries. After HR, FM accounts for an organisation’s greatest expenditure with 20% of total organisational expenditure. Significantly reducing this figure would allow smaller organisations to compete and stimulate growth.
2) High street utilization. High streets (or Main streets in the US) have lots of empty shop fronts. They could be re-commissioned (as Starbucks’) bringing much welcome new life and trade opportunities. This would add to our existing spaces - our homes (city/suburban), our work (downtown/business park) and the mall; maintaining a space distinct from these – a common (high street/main street). It’s ultimately about variety, possibilities, culture and escape.
3) Cultural cross-pollination. Organisations generally benefit from finding out about other organisation’s ideas/challenges. Some will consider this a drawback since they fear dilution of organisational “special sauce”. But what is this really? - People, process and IP – assets that aren’t going anywhere. It’s just the physical environment shifting.
4) Flexibility. Working in the same office is dull. Work in whichever Starbucks-office you like.

What would drawbacks be?

1) Noise. In the long-term, once Starbucks is recognised as more than a coffee shop, people will act differently there and noise will become no more of an issue than it currently is in offices. Headsets will help in the short-term.
2) Loss of status and image. If you are Swiss Re and you have spent $1BN on your gherkin building, you care about prestige, internal branding and providing a great environment for your workers. If there is a great environment elsewhere though – for free - is prestige and internal branding worth it? It is for the for big name organisations, the multi-nationals. For everyone else - no.
3) No physical storage. People keep things (coats, umbrellas etc.) at their place of work; they will need to keep them elsewhere. A small amount of lockers could be made available. HR and accounting would need to digitize all physical files. Is this a real issue? It shouldn’t be. For every filing cabinet, there’s a good reason why it should be in the cloud.
4) Team accommodation. If you are working solo or there are just a few of you then you can usually find seats together. There would be a problem accommodating project based teams (3-10) people in this way. This on-demand physical availability of teams is the biggest drawback to office-Starbucks. There would need to be a responsive real-time system that is capable of identifying empty seats together and placing a reservation on them.

Next up there will be an open letter to Starbucks asking them to consider our audacious plan.

Beware the IDEs? Not so much

Whether to run an Integrated Development Environment (IDE) in a browser or not can be a surprisingly emotive subject. The majority of developers, if they do not already have it, want a well appointed workstation running Visual Studio (for .NET), Eclipse (for Java), Dreamweaver (for JavaScript/HTML/CSS) or similar. They baulk at the idea of running a browser-based IDE despite building browser-based applications and clear advantages to working in this way. It is worth logically and dispassionately looking at the situation.

Key advantages are:

1) Portability. Developers can work from anywhere with a web connection. Does this really happen? Offshore developers will typically have a work desktop and maybe a personal laptop. They may also come onshore for a short period and use a client desktop. If they work through an outsourcer/consultancy they may have another. This counts but it is a Dropbox like aspect of portability. Its main advantage is in supporting those lifestyle situations; when you were not scheduled to develop but – thinking about it - you can; either to get ahead or to react to real time issues – even when you are on vacation, travelling or visiting friends. There are also humanitarian reasons for being able to learn a trade and contribute without having to own even a $100 laptop but let us save that for a later post.
2) Collaboration. Developers can let others debug their code by sharing it via unique URL. Anyone navigating to that link will receive a separate, fully modifiable and executable version of the code. That means no API version inconsistencies come compile time. Real-time collaborative coding is also easier.
3) Efficiency. Hours and on larger projects – days are wasted in setting-up workstations and supporting environments e.g. source control/configuration management at the start of each project. Even if it has been done several times before, at-some-point it will fail because there are just too many variables on a desktop. This just goes away.
4) Cost. Older workstations can be used since compilation and anything else heavy is performed on the server. Cost also improved by collaboration and efficiency (2 and 3 above).

Key disadvantages are:

1) Usability. The browser is not perceived as being rich enough to accommodate a responsive editor, allow class/library management and debug. It is also impracticable to design and test a GUI due to the greater drag/drop precision required.
2) Connectivity. You need to be connected to the Internet in order to run your IDE and therefore develop. This limits your portability (1 above).

There are other points on both sides but above are the key ones. Let us hold those two disadvantages up to the tiniest bit of scrutiny:

1) Usability. Large text file editing in a responsive (next to no latency) way (the main criticism) can now be achieved using HTML5 Canvas/JavaScript. See Bespin and also Kodingen (extending Bespin and integrating with other services) for examples of this using Python/PHP/ROR. See CodeRun for an example of full-on code management using .NET/JavaScript. They are both free, quick and have clean efficient interfaces. Bespin is more of a work-in-progress and does not yet support all browsers though. GUI design is admittedly more of an issue right now but:
a. People already successfully use graphical editors in a browser e.g. Splashup, SUMO Paint and the recently Google acquired Picnick.
b. HTML5 adoption is affording more options here.
c. In both consumer and enterprise spaces, we are moving toward a widget-based UX making designing GUIs from scratch less common.
2) Connectivity. By this - offline development is meant i.e. enabling those circumstances where the developer is using a laptop (since if they were using a desktop surely it would be connected to a network?) and they are in an area of no Wi-Fi coverage (since otherwise they would have network access?). Granted this is a situation that occurs but consider further that this also means:
a. There are no collaboration or research possibilities available (no IM/no Google). If you get stuck when developing or need to clarify a technical point – you’re on your own.
b. You need a single professional and automated solution that synchronises all code, images, configuration files, media (that has previously been unit and integration tested and checked in) and also synchronises test data and potentially business rules (since it is good practice to keep these out of code). Either it will synchronise actual data/business rules in which case you need high grade encryption on your laptop (as you are likely using customer data) or your solution needs to de-sensitize the data/business rules somehow (and you need to have agreed this process with any customer). What kind of developers will be happy with these two restrictions? Only sole developers working on their own project.

Staying with the logical analysis of this, there are four decent reasons in favour of widespread use of browser-based IDEs and two against. There is also enough mitigation to mostly address the two negatives. There is clearly a significant net gain to be made. Side points around “developers won’t stand for it”, “Development is an art (it’s not!)” or “you just do not understand” etc. are emotional and really do not have a place in the decision. It is understandable (in a carpenter cherishing his chisel kind of way) but this noise is a real contributory factor as to why browser-based IDE have not made more of an impact to-date.

When their new OS comes out later this year, are Google really going to say – buy Chrome laptops, they can do everything your regular laptop can – unless you are developing? Given their long developer-friendliness; this would appear a peculiar move. Unless of course it is precisely because they are developer-friendly; that they will pander to populist developer belief and treat them as artisans needing powerful magical workstations? If they do this though – they risk confusing consumers and certainly the non-developing IT community as to their strategy at a time when they are already perplexed by what is happening with Chrome OS, Chrome and Android as a run-time environment.

Google have a new programming language – Go which currently needs OSX or Linux. Like the majority of languages today, it is C-based. It has been out for nearly a year but has not received a great deal of press. It will need a differentiator other than speed to compete (how many web applications really have a processor bottleneck these days?). Surely there is an opportunity to build a browser-based IDE for Go and enable a new generation of more casual (but also more open) developers around the world?

MOSS project failure reasons

Here is a list of the main reasons for MOSS project failure based upon experience (many reasons will also be applicable to non-MOSS projects). Other (occasional) reasons contributing to failure include inadequate business domain understanding, MOSS deployment design and MSFT licensing appreciation but for the sake of focus, only the main reasons are shown here. Reasons are shown in descending priority order:

1) Scope management. Initial requirements gathering insufficient. Incremental scope additions undocumented and outside change management. Little or no architectural analysis of scope shifts before they are implemented resulting in downstream performance, testing and hosting integration issues. High-Level Design (HLD) used as sole written basis for development. HLD is generally insufficient for developers to interpret and develop against. A solution can be to factor in element of client MOSS training during the early analysis stages. This will reduce incidence of client scope expectation being significantly more advanced than what is possible Out-Of-the-Box (OOB). Basically, clients tend to think MOSS can do more than it can - OOB.
2) Configuration management. Robust problem and change management processes missing; inadequately defined or not maintained. Doubly important in a MOSS environment where configuration, content and code (and also data and process) are entwined. Effect of this is that developers become confused as to what they are supposed to be working on and client loses detail visibility. Senior developers may attempt to establish these processes but will generally be ineffective due to inexperience and their relative capability for enforcement. Multiple versions of both requirement and design documents stored in multiple places and little attempt at sign-off coupled with poor client communication means that entrenched positions are swiftly made. Developers can struggle with the same issue for days. If daily updates are recorded in a problem management system, this will be obvious to the Project Management (PM) function through oversight.
3) Project communication. Poor overall project communication. With a focus on gaining hard MOSS technical skills, softer skills such as communication are often overlooked. Developers typically send conflicting messages to client. No team scrums, little articulation of cause and effect in terms of project plan. No “bottom up” captures or extrapolation. Culture of sharing information not engendered at senior developer level. PM reporting limited; when requested at high level and non-specific.
4) Skill levels. Due to initial shortages of MOSS skills, developers have historically been staffed to projects with the effective remit of learning on the job. This is not now the issue it used to be due to wide-scale MOSS training focussing on .NET Web part Framework. There remain key poorly resourced areas e.g. workflow and InfoPath development and engineering experience in MOSS “lockdown” mode.
5) Administrative duties. Not really focussed on operational logistics e.g. reporting status, ensuring adequate vacation cover, maintaining a service incident log, ensuring network access for resources and providing a view to scheduling on upcoming resource requirements. In a market where MOSS resources are relatively scarce, focus on effective resource management is critical.
6) Engineering resource. Insufficient engineering resource can be applied to MOSS projects. This manifests itself as generally assumed acceptance of the client infrastructure and hosting plans and means the team cannot plan and iterate for performance from design onwards.
7) Client relationship. Diminished client relationship. This can be left by default to be built by developers. Without a consistent face being presented to the client through reporting or solution walkthrough, the possibility for a relationship to grow is diminished and client finds it easy to take an aggressive stance to the engagement when required. This is particularly important where a RAD approach is more applicable e.g. MOSS.

The eyes of the developer

(Originally posted 22 April 2009).

Offshore development resources are great. They are invariably well educated, diligent and critically in these trying times – effectively priced. Even now though, a key reservation organisations have around using them is - visibility. They want to see them and talk to them; their requirements are so exacting that only by looking into the eyes of the developer can they be understood. Bringing offshore resources onshore for the initial stages of a project (and so that they can return offshore for knowledge) is a proven way of mitigating this concern.

Lead times involved in procuring offshore resources onshore (often three months) can be ineffective for many projects; especially ones founded on a business case of cost reduction/avoidance. This can be expedited to less than a month but generally only if the resources are undertaking “training” and not developing the solution. Developing the solution though is where they will truly learn and become vested in the success of the project. A seemingly attractive alternative for large organisations (and for the consultancies that service them) is to establish an onshore pool of offshore resources to service future onshore projects. Is this a cost effective solution though?

Five offshore resources brought onshore for three months will cost around $88K (accommodation/fly-backs/insurance/travel/visas etc.). Assuming they can be cross-charged (or sold) at say $877/day for 80% of the time they are onshore; this makes $210K in revenue.

This appears good (140% ROI) except for the fact that once this process is started, the resources have to be retained i.e. taken out of (or reserved from) the offshore pool until they arrive. This can easily take three months. Assuming a cost of $146/day/resource, this totals $44K, taking the endeavour down to $79K profit (60% ROI). This may be able to be offset by them doing other (short-term) work in the interim but it certainly should not be counted upon. Once resources are onshore, organisations should also account for increased team lead/managerial support for them (perhaps one day/week across all of them – totalling around $22K in opportunity cost), taking the endeavour down to around $57K profit (38% ROI).

This should also be considered a high risk endeavour due to the fact that resources are being recruited for a pool rather than a specific project (project may not happen) and manifest cultural differences. Organisations are therefore already borderline as to whether this is a good idea financially or not. The best approach has to be simply to keep a close eye on “hot” skills and ensure that offshore pool resources in these areas already have visas.