The banking & cloud connection

Some people are very worried about their data. They contend that data stored in the cloud is open to advert targeting, compromises legal ownership, open to theft and abuse (multi-tenancy arguably making this easier), legally accessible by many Governments and not exactly open (i.e. in silos) in terms of passing data between organisations. All of this is oppressive, reduces flexibility and is ultimately little different to proprietary software.

These feelings are attributed to practical experience, distrust of large organisation motivation, appreciation that data centres are inevitably farmed out to third-world countries and doubts over centralised security. On this latter point, certainly hackers could break your firewall accessing your personal data but the effort/risk simply isn’t worth it just for you. The same effort/risk for millions of people’s data is a different matter entirely (not exactly like locking your door when you’re not at home).

Cloud vendors including social networks aren’t (understandably) forthcoming about their security protocols. Key targets for security concerns are Google/Facebook since they have been most successful at getting our data to date.

People evidently believe they want control of their own data back. What options do they have? There are parallels here with the start of banking: 

1. Leave it where it is but demand more visibility/control. Cloud vendors can expose detailed controls to the user e.g. privacy/security/access/ownership etc. Some options free, maybe some on a payment scale e.g. reducing the level of advertising/data mining. This is complicated for consumers especially since they need to do this consistently across several services. They like things nice-and-easy.
2. Take it back. Putting your data back into your own network is impracticable. You need to remove duplicates, comply with legalities, elect who can access it, keep it current, ensure its connected to the newest services, access it remotely through your firewall, tag it, back it up, archive it, analyse it, share it and maybe sell it. You need USB keys to move data around and you’re at risk of direct physical loss/theft. You need to handle all this using common standards (so you know it can be accessed in future). Directly controlling your own data is a lot of work for all but the most paranoid/justifiably wearisome.
3. Leave it where it is but apply competitive pressure. Data portability allows you to pull your data out of one site and put it into another at will. This too is convoluted. It is also somewhat of a nuclear option in that consumers won’t actually do it unless cloud abuses are so flagrant (and widely reported) that they feel compelled to and competing sites exist that can import it using a common format and their friends do it too i.e. almost never. The threat of it is arguably enough to keep cloud vendors mostly honest. Despite making in-roads over the last year, Facebook comes in for most criticism here (since at time of writing, it doesn’t allow you to download your social graph or emails). Google with Chrome OS though will surely trump this (due to its sheer cloud nature) when released.
4. Leave it where it is but apply third-party pressure. It is still unclear how much pressure third-parties such as the Cloud Security Alliance, ENISA, general certification or indeed entire Governments can realistically muster against distributed clouds operating under multiple jurisdictions.
5. Give it to a specialist. A mostly utopian ideal is the concept of the personal data locker. This focuses on holding your identifying information, financial credentials and personal information e.g. allergies/airline seating preferences centrally online with a trusted dedicated organisation. With your permission, companies/services you subscribe to e.g. social networks pull data from your locker – each using it for their own value-add functions. It could also act as an agent – storing your purchase criteria - providing deals to you and perhaps even trading your data on a open market for a return.

Continuing our banking analogy, personal data becomes less about control and more about oversight, trust, commoditization, service differentiation, commission and regulation.

The obvious missing element with cloud/data is an open market for trading – much like banking/cash relies upon today. Google are fine getting into Enron-esque energy trading to moderate their data centre energy requirements. Why not building the foundation for a data trading market? They are ideally placed to analyse, quantify, monetize and then sell data. Will someone else steal the lead much like Facebook did with our social graph? Building a data trading market might actually allow them to build on Facebook’s social graph and make real money trading. It would create a commission-based eco-system where much needed data integration/consolidation/MDM could be funded. In this world, Facebook are relegated to merely banknote printer. 

Which consumer model above will prevail? Much like retail banking at least they all will. There’s no silver bullet. Some people prefer direct control/hoarding/easy access, others will trust specialists as they are too busy (and pay for the privilege), others will lobby Government (maybe they closely identify w/ a particular ideology).

Cloud has been around for ages (pre-1990 it was called - Terminal). It is only in the last decade though that there has been both a wealth of data and the widespread desire/capability to do something with it. An open data trading market is needed to consolidate and then drive forward personal data management.

The project failure & cloud connection

Around 2000, I worked on a project with a co-worker – he was maybe fifty and had worked in the IT industry – initially at IBM then contracting and consulting for nearly thirty years. At the time; he was heading up a web-development and creative team for a small consultancy firm which he saw as a new opportunity for him. He had a good reputation, people liked him and he had a vast amount of experience from being involved in literally hundreds of projects – developing, managing, business developing etc.

One day he slips into conversation – as an aside really; that he has never worked on a project that was delivered. Every project he worked on had been cancelled, merged into another programme, didn’t meet its financial goals (as with infrastructure consolidation) or his involvement ceased before it ever went live (if any of them ever did – he didn’t know). He said it in a resigned manner, almost as a badge of honour that he had got through all that with no job satisfaction in terms of delivery payoff whatsoever.

I have seen this scenario repeated many times in the decade since; although perhaps none quite so extreme. I have myself worked on hundreds of projects in many roles and can count the number of successful systems deliveries on one hand. People naturally want to associate themselves with successes rather than failures so it is perhaps understandable that this is not a common consultant topic.

A whitepaper released late last year attempts to put a figure on the worldwide cost of IT project failures. This turns out to be $6.2 Trillion and it doesn’t look like sensationalism. The US in particular is apparently losing almost as much money per year to IT failure as it did to the financial meltdown (with no end in sight). The paper makes an attempt to factor in what it calls indirect costs; basically a lost opportunity cost from the time wasted on failed/abandoned projects. It does not however take into consideration the wider indirect costs of people training for careers that are not actually delivering, IT staff disillusionment (turnover), operational failures for delivered IT systems (one in five businesses lose £10,000/hour through systems downtime) and associated security failures.

The paper has received some condemnation (due to its base assumption of a 65% IT project failure rate) but there is dearth of analysis in this area and quoting this figure is about as good as it gets right now. 50-80% figures have been quoted in one form or another for decades. CIO thinks rates are actually rising due to the recession. People have the choice of either working with a figure (challenging assumptions/statistics etc) or burying their head in the sand.

We will never know the exact figure for IT project failure. Similarly, we will never know whether efficiency/functional benefits of those systems that have worked have paid for the failures i.e. What has IT really given us? We are simply reliant upon these systems going forward. What can we do to reduce future project failure rates?

Although there are superficial similarities with scientific/experiment and IT/project communities respectively, the "accept defeat" approach of the former is routed in constant learning whereas IT is surely about delivering benefit now. Learning is only the priority for the largest, most stable, lowest turnover organisations; that is to say - next to none of them. The scientific regimen of independent assessment however is invaluable for IT projects. Tool-based PM consultants such as Bestoutcome are probably as good as the big management consultants for this purpose though. Techniques from the engineering community (when introduced to IT) have not had a huge effect.

The paper ends with a call to arms to simplify – IT/business communications, projects goals etc.

I agree in principle but this is oversimplification - when your realm of influence is the organisation you are in. I have seen many projects that although conceptually simple and with genuine IT/business agreement start to fail the moment integration with other organisations – vendors/hosting providers/recruiters/sub-contractors/data sources are required. Despite solutions being simple and manageable inside your organisation – just a few touch-points with others (basically anything worth doing) cause them to be complex and therefore unpredictable/at risk no matter what your collective capabilities. There is even a case for blinkered-simplification/procedures actually contributing to project failure: Complexity at least brings an element of flexibility, allowing you to react if the project starts going bad.

Better project managers, SOA, ramped-up co-worker involvement, Facebook-like "hackathons", daily IT/business meetings, PRINCE certifications, extranets, more rigorous cost control and mathematical complexity models within your organisation will have minimal effect on the success of your multi-touch-point projects – for they are already in the realm of chaos. Even improved PM collaboration through tools such as Asana will have a minimal effect on success rates. The role of the “good project manager” is perhaps the most scrutinized, personality-driven, divisive and misunderstood of all IT positions. Radical open enterprise models such as BetterMeans that effectively remove Project Managers in favour of automation and decentralized decision making will similarly be ineffective. Although in the case of the open enterprise model, I do agree that this will ultimately prevail (crowd-sourcing, creativity enabling and ultimately efficiency/cost) but not for decades (due to the need to directly attack the failure rate first).

Although there are certainly sizeable success increases to be made if you are experienced in the particular technology, the IT project failure rate will only consistently and materially fall once there are flexible cloud services that organisations can get 80% of all their needs by just subscribing to them.

Integration ceases to be the bottleneck. The other 20% being “secret sauce” value-add-ons developed in-house; probably mainly algorithms that, by definition, do not require integration with other organisations or heavy project governance. Other components of the 20% would be device specific exploitation code; essentially building the so-called App-Internet model (rather than full-Cloud). Organisations already recognise the economic justification for cloud computing so it is perhaps inevitable that project failure rates will eventually fall by default.

Cloud transition will take organisations years yet however. Both InfoWorld and Gartner have arrived at 2013 for broadly when the majority of organisations will run cloud. This may be optimistic. In the interim (three more years+ of high project failure rates?), delivery is simply better served by being built upon cloud solutions now; building partnerships with cloud providers if they can and leveraging their buying power if they cannot. Also, interim/limited functionality cloud solutions must be considered in preference to bespoke development/on premises deployment. In a real sense the project (that they would have otherwise completed themselves) becomes a creativity-driven; architectural investment and commercial partnership instead. Both Project Management and Enterprise Architect roles will need to shift accordingly.

It would be interesting to see any future IT project failure analysis split between organisations that have implemented virtualisation, those that have implemented private cloud and those that have implemented public cloud. The project failure and cloud connection is not well documented.

MOSS Internet Site - Security

Building a corporate Internet site using MOSS has real security implications. Access from the Internet (for the users) and from within the organisation (Intranet) must be considered in terms of both authentication and authorisation. Although a sound solution, MOSS is not well documented here and an implementation typically needs to be designed. Following is a suggested solution template for organisations to use when attempting this. It should resonate with the majority of MSFT-shop organisations moving to a MOSS Internet site solution.

The solution supports two basic user categories:

1) Internal users. Employees accessing via the Intranet. Typically content authors - in charge of maintaining solution content. It is recognized that a requirement for employees to maintain content via the Internet (from home?) may well exist. In this situation, they should connect to their Intranet using whatever connectivity solution is already in-situ (VPN?). Although it is certainly possible to allow employees to manage content directly over the Internet, this creates additional complexity, cost and security risk for perhaps the sole benefit of; being able to manage content from an “unofficial” desktop (without the connectivity software) e.g. in a cybercafé.
2) External users. Public users accessing via the Internet. Able to access content only. May be further categorized as either - Anonymous user (Read only/Able to access all public content) or Registered user (Read only/Able to access special content in addition to public content).

The solution supports two-way authentication; one to support internal users (NTLM) and the other to support external users via Forms Based Authentication (FBA). Internal users are authenticated using the SSO (Single Sign-On) feature provided by the LDAP directory services. An internal SSO process synchronises the Intranet AD with a De-Militarized Zone (DMZ) LDAP. In practice, this will mean that an internal user can log onto the Intranet and be pass-through authenticated to the DMZ i.e. they will not have to log in twice.

Synchronisation between the Intranet AD and DMZ LDAP is one-way. This means that the creation of new internal users will have to take place on the AD and not directly in production. Assuming your organisation does not need to use profile import for LDAP users (and also does not provide custom profile properties for its internal users), it will only need to cater for authentication of LDAP users. This is perhaps a minor risk, since it will only be possible to test this once deployed in Production.

Similar to traditional NTLM authentication mechanisms, the FBA application requires a centralized store for external user credentials (in this case - SQL database). Internal user credentials are stored in AD (on the Intranet) and in LDAP (within the DMZ). An internal SSO process synchronizes them. No authentication is required for anonymous external users as you would expect.

User information stored in the SQL database is created only by a single registration process. A registration page defines the user information and the related validation. Using FBA, users will remain authenticated until the user closes their browser or until authentication timeout occurs.
 
Moving onto authorisation; authorization policies are required for granting content access rights for differing user categories. As stated above two different “extended” web applications are defined for managing both internal and external users. Authorization policies are applied to those distinct “extended” applications addressing the user categories’ tasks.

In the “extended” web application defined for the NTLM authentication (used by internal users) two different user groups are defined in the site settings permission section - Site Administrator (users with full control access rights) and Content Authors (users with design control access rights). No Anonymous access is supported in this “extended” web applications instance. In the “extended” web application defined for the FBA authentication (used by external users) all users are defined as users with read access rights. Anonymous access is supported for a subset of content as appropriate.

A final note on domains. The AD within the DMZ is for service accounts only e.g. SQL or MOSS services. It is not possible to trust the DMZ AD and the organisation AD proper. The DMZ is also unconnected to any other organisational system. MOSS based content deployment into the DMZ is unsupported. These measures should address internal security concerns. There are plenty of other security hoops to jump through when creating a MOSS Internet site but the above should get you started.

Handling Targets That Do Not Roll-up

Comparing actual figures against targets are fundamental to any Performance Management (PM) system. PM solutions are typically built upon cubes and correspondingly, cubes are built upon data marts. In a data mart, measurements are generally stored in fact tables at their lowest level e.g. Cost per Organisational Unit (OU) per Day. These measurements are then aggregated along dimension levels e.g. Cost per Area per Week is the sum of the daily site figures for that week and area. This solution works well with actual figures as they are almost always additive. Targets however can sometimes be independent e.g. Target Cost per OU per Week is not necessarily the sum of daily target costs for that week.

There may be legitimate business reasons for doing so e.g. cost is heavily market/EOS driven and needs to be targeted manually or, for whatever reason, targets cannot be decomposed into fully additive components. This means that the PM solution has to either store the data for all levels in the fact tables or not include targets in the data mart at all. This latter option creates its own issues though as many front-end BI/PM tools can only connect to a single data source at a time.

Design alternatives on how to store data for all levels in the fact tables are not well documented in BI/PM design literature. Options here are:

1) Single Fact Table. Storing all aggregates in the same fact table simplifies the data model but has the disadvantages of values being duplicated e.g. the year value is stored in every day record; the fact table will contain a large number of fields and the ETL process is more complex (either using updates or reloading every level for each load).
2) Multiple Measures. To reduce the number of fields in the fact table, the tables could be split along time levels or measure. This option has the advantages of a reduction in the number of measures per metric and no duplication along the Time dimension. Its disadvantages are that there is likely duplication along the OU dimension and there are multiple fact tables.
3) Fact Table per Dimension Level. Defining a table for each of the dimension levels has the advantages of no field duplication and the least number of measures overall. Its disadvantage is again that there are multiple fact tables.

Option 3 is generally recommended as this eliminates data duplication and simplifies the ETL process. It is somewhat unusual as typically within a star schema; a fact table is surrounded by multiple dimension tables. It is however, a practical solution and has been recently implemented for a large resources client.

The fact tables are connected to the dimension tables on differing granularities as shown below:

The CostDay fact table is linked to the Time dimension table through its TimeId Foreign Key (FK). Fact tables with different granularities need to have additional field linking them to the Time dimension at the desired granularity. All attributes within the fact table therefore have to share the same granularity.


This (admittedly complex) model can be later simplified using whatever tools you use to maintain your cubes. SQL Server for example provides Perspectives, Measure Groups and Calculated Members that can be used to hide the complexity of the underlying data model from the user. Perspectives can be used to hide objects e.g. fact tables from the user. Finally Measure Groups can be used to create logical groupings of fact members.

BI Strategy Planning Tips – Part 1

Planning a BI strategy for your organisation can be challenging; you need decent industry/vendor awareness, an appreciation of organisational data and ideally; a handle on budgeting. In no particular order, here are some practicable tips to get started with yours. More will be forthcoming.

1) Start with your Supply Chain. Reduction of energy use in data centres is a key IT issue. This is generally driven by either a cost saving drive or a Green IT focus (shouldn’t they really be the same thing though?). However, most organizations budget between 2-5% of revenue for IT budgets, yet spend roughly 50% of revenue on all aspects of supply chain management. Basically, there are significantly more savings to be made in the supply chain than in data centres. Large volumes of raw data are generated and stored by each process of the supply chain (plan, source, make, deliver and return) by automated enterprise applications being used at most large, global manufacturers. BI can help determine what information is necessary to drive improvements and efficiencies at each process in the supply chain and turn the raw data into meaningful metrics and KPIs.
2) Forget the BI-Search “evolution”. Just the training costs for commercial BI systems are expensive. Organisations want single (easy and simple) interfaces wherever possible and a search-based interface appears to be the key to engaging the masses. There has been heavy speculation over the last couple years (ongoing) that BI and Search technologies will somehow merge. This approach however only really surfaces existing BI reports for more detailed interrogation e.g. “July Sales Peaks”. A search string is not a rich enough interface to support ad-hoc queries. Think about it? You need either a devoted language e.g. MDX or a rich data visualisation package to traverse dimensional data. A search box will never explore correlation between marketing budget and operating income.
3) Forget “BI for the masses” (for now). BI has been actively used in the enterprise since the early nineties. The expectation of “BI for the masses” (basically - the SMB market) hasn't exactly happened. Why is this? It’s because people want to collaborate and jointly come up with ideas, solutions, figures and approaches. They need this for personal, political and commercial reasons. It will change when enterprise SOA is in place and it might change when there is a change in the way consumer impacts enterprise networking. It will not change in the short-term.

Universal enterprise UX - Part 1 (Concept)

(Originally posted 6 November 2008).

From a UX perspective, portals are a great way of centralising; personalising and publishing the various functions that a user needs to undertake operationally. They are ideal for combining both structured and unstructured data and contextualising between the two. We’re all essentially information workers and users do similar information-centric operations during their day e.g. browsing, analysing, contextualising, starting new events (based upon old events) and data entry. There should be the same UX available for them to do these actions. Not just a similar portal but the same (configurable) web parts.

Where are these gadgets? SAP has a clear concept of UX reuse that plays partly in this space but where are they for other platforms? There are a finite dozen or so UX interactions (Find, Alert, Link, Flag, Copy, Browse, Cascade, CRUD, Audit, Confirm, Search and Analyse) that can be mapped to six or less web parts. Two of these web parts would account for most UX interactions i.e. Web part 1 (Find, Alert, Link, Flag and Copy) and Web part 2 (Browse and Cascade). Web part 1 would mainly deal with lists and Web part 2 would mainly deal with hierarchies.

These could be mix and matched with social networking and content management web parts. That’s just six or so interrelated web parts that could handle the majority of bespoke operational applications today. For example, an order processing system user would be able to search for a particular user, see all their past orders (and correspondence), analyse their propensity to cross/up-sell and take or query their order and flag particular customers for a follow-up call. When the IT department roll-on new functions e.g. recording user feedback at POS, users will know how they work as the web parts would behave the same as existing functions. For the back office, ongoing development, training and testing of incremental functionality would be reduced. MOSS is already treated seriously as an application development platform (http://www.andrewconnell.com/blog/archive/2007/09/24/6116.aspx) as it handles the plumbing every application needs. Some organisations have worked on solutions where they have deployed a similar concept for customers (reusing the same web parts each time) but this approach only really pays dividends with incremental development or new solutions that use the same UX web parts. Why is the universal enterprise UX not more prevalent (?).