uploaders

The banking & cloud connection

2011-02-01T17:13:00.001Z

Some people are very worried about their data. They contend that data stored in the cloud is open to advert targeting, compromises legal ownership, open to theft and abuse (multi-tenancy arguably making this easier), legally accessible by many Governments and not exactly open (i.e. in silos) in terms of passing data between organisations. All of this is oppressive, reduces flexibility and is ultimately little different to proprietary software.

These feelings are attributed to practical experience, distrust of large organisation motivation, appreciation that data centres are inevitably farmed out to third-world countries and doubts over centralised security. On this latter point, certainly hackers could break your firewall accessing your personal data but the effort/risk simply isn’t worth it just for you. The same effort/risk for millions of people’s data is a different matter entirely (not exactly like locking your door when you’re not at home).

Cloud vendors including social networks aren’t (understandably) forthcoming about their security protocols. Key targets for security concerns are Google/Facebook since they have been most successful at getting our data to date.

People evidently believe they want control of their own data back. What options do they have? There are parallels here with the start of banking:

Leave it where it is but demand more visibility/control. Cloud vendors can expose detailed controls to the user e.g. privacy/security/access/ownership etc. Some options free, maybe some on a payment scale e.g. reducing the level of advertising/data mining. This is complicated for consumers especially since they need to do this consistently across several services. They like things nice-and-easy.
Take it back. Putting your data back into your own network is impracticable. You need to remove duplicates, comply with legalities, elect who can access it, keep it current, ensure its connected to the newest services, access it remotely through your firewall, tag it, back it up, archive it, analyse it, share it and maybe sell it. You need USB keys to move data around and you’re at risk of direct physical loss/theft. You need to handle all this using common standards (so you know it can be accessed in future). Directly controlling your own data is a lot of work for all but the most paranoid/justifiably wearisome.
Leave it where it is but apply competitive pressure. Data portability allows you to pull your data out of one site and put it into another at will. This too is convoluted. It is also somewhat of a nuclear option in that consumers won’t actually do it unless cloud abuses are so flagrant (and widely reported) that they feel compelled to and competing sites exist that can import it using a common format and their friends do it too i.e. almost never. The threat of it is arguably enough to keep cloud vendors mostly honest. Despite making in-roads over the last year, Facebook comes in for most criticism here (since at time of writing, it doesn’t allow you to download your social graph or emails). Google with Chrome OS though will surely trump this (due to its sheer cloud nature) when released.
Leave it where it is but apply third-party pressure. It is still unclear how much pressure third-parties such as the Cloud Security Alliance, ENISA, general certification or indeed entire Governments can realistically muster against distributed clouds operating under multiple jurisdictions.
Give it to a specialist. A mostly utopian ideal is the concept of the personal data locker. This focuses on holding your identifying information, financial credentials and personal information e.g. allergies/airline seating preferences centrally online with a trusted dedicated organisation. With your permission, companies/services you subscribe to e.g. social networks pull data from your locker – each using it for their own value-add functions. It could also act as an agent – storing your purchase criteria - providing deals to you and perhaps even trading your data on a open market for a return.

Continuing our banking analogy, personal data becomes less about control and more about oversight, trust, commoditization, service differentiation, commission and regulation.

The obvious missing element with cloud/data is an open market for trading – much like banking/cash relies upon today. Google are fine getting into Enron-esque energy trading to moderate their data centre energy requirements. Why not building the foundation for a data trading market? They are ideally placed to analyse, quantify, monetize and then sell data. Will someone else steal the lead much like Facebook did with our social graph? Building a data trading market might actually allow them to build on Facebook’s social graph and make real money trading. It would create a commission-based eco-system where much needed data integration/consolidation/MDM could be funded. In this world, Facebook are relegated to merely banknote printer.

Which consumer model above will prevail? Much like retail banking at least they all will. There’s no silver bullet. Some people prefer direct control/hoarding/easy access, others will trust specialists as they are too busy (and pay for the privilege), others will lobby Government (maybe they closely identify w/ a particular ideology).

Cloud has been around for ages (pre-1990 it was called - Terminal). It is only in the last decade though that there has been both a wealth of data and the widespread desire/capability to do something with it. An open data trading market is needed to consolidate and then drive forward personal data management.

Money makes money

2011-01-25T14:08:00.003Z

Following on from last post, clearly one of the key benefits of digital cash is that it does not incur transaction fees. This is the main method of monetization for services currently providing cashless transactions. The digital cash concept is not exclusively people-focussed/altruistic; there are ways to make an on-going business out of it. Here are the obvious ones off the top-of-my-head. I'm sure there are lots more (?):

Affiliate fees. The validator site can also function as a free service, able to connect to your bank(s), retrieve your bank account details/transactions and provide value-add services with the data e.g. Mint. In particular, would allow you to see where your money is being spent and give you hints on how to save money. Money would be made through affiliate fees/recommendations.
Marketing. Digital cash contains with it, details of what was bought, when and for how much. Analytics analyse consumer transaction patterns, build spending usage pie charts and suggesting relevant ways to save or make more money via competitive offers. Marketing managers would purchase the analytics to analyse usage patterns, create marketing campaigns and target specific demographics and customer types e.g. through Google AdSense. All of this would be tied to the (anonymous) cash rather than the individual.
Purchase sharing. A modish site (Blippy) enables the controlled sharing of purchases to see what others are buying online and in real life. Blippy lets you share purchases by syncing already existing e-commerce accounts e.g. iTunes, Netflix, Woot, eBay. It is thought to monetize through turning links on Amazon purchases into referrals/featured vendors. Analysing purchase data is a gold-mine for analytics/consumer behaviour insight. Card companies cannot share this information but having consumers proactively share with others overcomes this obstacle (and when aggregated sold on). Facebook's Buy With Friends feature works in a similar way for virtual goods only. Purchase sharing in general is a great way to drive group buying behaviour e.g. Groupon.
Exchange service. Digital cash could be freely exchangeable into physical cash and vice versa. Existing currency exchange outlets would have the infrastructure for this and this would afford them a new revenue opportunity. Also a non-monetary currency exchange could be established. There is a trend toward metacurrency: treating movement of non-monetary flows of attention, participation and trust e.g. frequent flier miles, college degrees/grades/credits, five-star ratings, certifications, bus passes, votes, your eBay rating, scores, coupons much like physical cash. Transactional commission may be made on converting between these. Obviously you cannot buy votes or college degrees but it is certainly possible to buy coupons or frequent flier miles.
Banking. Digital cash being essentially being stored in a personal data locker - free secure cloud storage identified to you. As with a regular bank, it would store your digital cash for when you need it (download to your wallet) and (like banks) make money by market speculation.
Loyalty/payment card service. Existing proprietary loyalty/payment cards such as those operated by Tesco and Starbucks respectively could be outsourced to a new consolidated, cheaper service. Anonymity could be preserved with this new solution and subscriptions could be charged to the store.

How to get money

2011-01-19T23:42:00.001Z

Following-on from last post, there were some questions around how you might actually get digital cash, there are three basic methods of obtaining digital cash that I can immediately think of:

It could be given. Directly acquired by interaction with others (tap and pay). This is the simplest method. It incurs zero transactional charge, is quick and can be performed offline.
1. Conceivably, digital cash could be printed out using a home printer (essentially becoming physical cash). The value is in the number; the bits containing its identity (value/history/security key). The number could be rendered as an abstract pattern as a security measure. Mobile app vision solutions e.g. Google Goggles on Android and Word Lens on iPhone are highly sophisticated and could read details of printed digital cash, enabling transactions/validation to be conducted, allowing printed digital cash could be exchanged without a mobile. Fundamentally, it's no different to one-use vouchers validated by merchants. If the consumer copies a voucher and tries to double-spend, it would be caught at POS.
It could be earned. This can be physically earned (in a way identical to above) e.g. a paymaster gives you $100 for a day's work (tap and pay). Other consumers/online solutions can also give you digital cash (downloadable to your mobile) in exchange for some service/incentive; for example, with Twitter, you earn a credit when someone acknowledges your tweet e.g. one cent for a view, three cents for a click, five cents for retweets and eight cents for a favourite. In this way, it behaves in a similar way to virtual cash only becoming digital cash once downloaded.
1. Conceivably, digital cash could be earned by enabling the mobile app to use idyll CPU cycles to run a distributed version of the validator. In this way, a separate validator web site would be unnecessary (w/ associated costs). It would function in a similar manner to Bitcoin, an open source virtual cash solution for desktops. However, due to the complexity of decentralising the encryption algorithms involved, the need to maintain a "spent" database, the limited processing power of mobiles and the need for physical separation to prevent hackers back-engineering the validation process, this may not be entirely practicable. The situation is worth monitoring however since it makes restricting digital cash solutions e.g. by Government sanction very difficult.
It could be bought. When bought online, PayPal or similar could be used. It would be analogous to going to an ATM. PayPal last year announced PayPal for Digital Goods. Payment transactions cost 5% plus $0.05 per purchase under $12, lower than most previous micropayment transaction standards. Alternatively it could be physically bought by visiting a currency exchange with higher exchange rates.

Creating money is easy

2011-01-12T22:22:00.020Z

Physical cash drives 60% of all transactions. Everyone knows it has a few issues though: production/distribution costs, counterfeiting/terrorism use, loss/theft, coinage doesn't pay interest, you need a purse/wallet and a digital divide (cannot use a physical $1 note to pay for a digital newspaper) amongst others.

Why can't we directly replace physical cash with digital cash?

By digital cash here, I mean specifically anonymous digital cash: a $10 note simply being a string of digits - a number. This number would most usefully be stored on a mobile for payment purposes (or anything else digital). What stops people inventing their own numbers – making their own money? - Cryptography. Cryptography acts like the intricate banknote design/watermark – preventing counterfeits.

How does it work? At a high level, validators and consumers have public-key encryption keys. Public-key encryption keys come in pairs. A private key known only to the owner and a public key, made available to everyone. Whatever the private key encrypts, the public key can decrypt and vice verse.

Digital cash itself accumulates the complete path the digital cash made through the economy and therefore "grows" each time it is spent. The history of each transaction (minus the identities involved) is appended and travel with it as it moves from person to person, merchant to vender. When deposited (or validated), the validator checks its database to see if the piece of digital cash was double spent. If it was copied or spent more than once, it will appear twice in the "spent" database. The validator uses the transactional history to identify dates and potentially locations associated with the double-spend and reacts appropriately e.g. contacts authorities/blacklists etc. It has more tracking potential than physical cash (despite still being anonymous).

The validator can always reconstruct the path the digital cash took through the economy (except who owned it). The validator will know what was bought, where it was bought, when it was bought and how much was paid. A side benefit of anonymity i.e. not including identities in transaction history means that the war being fought over global Internet identity is conveniently side-stepped (Facebook, Google, Twitter, OpenID [even Angry Birds] etc.).

If your mobile (or any other device containing your digital cash) is stolen or lost, you could remotely deactivate the cash (and additionally claim it back). This process would not (by necessity) be fully anonymous but it could be just tied to your mobile number, making it semi-anonymous in much the same way that Standard Bank in SA have done with mimoney.

In respect of transactional proof, the basic rule is: everyone can prove that they took part in a transaction but no-one can prove that someone else took part in a transaction.

Why is digital cash a good idea for both consumers and merchants? Simply, cost, convenience and privacy in that order.

Cost
1. Credit/debit card based solutions are expensive. Merchants prefer cash since it is the cheapest to process. A British Retail Consortium (BRC) survey claims that credit cards account for 11% of transactions (but 49% of merchants' costs in accepting them). The biggest single cost of card payment collection is the bank merchant service charges. These cost the UK merchant 2p per transaction for cash, compared with nearly 8p for debit cards, 35p for credit cards and 53p for cheques. These costs are inevitably passed onto the consumer in the form of higher prices. Credit/debit cards companies simply cannot reduce this charge and are obliged to focus on target based rewards programs in order to compete. Credit/debit card solutions may not be available to all. Finally many consumers have bad credit ratings and find it difficult to obtain credit/debit cards. This also affects PayPal since it takes these cards into its wallet. In the US, this figure is reportedly 25%. Finally, merchants are never quite sure how much a credit card transaction will cost. MasterCard and Visa charge hundreds of different rates (interchange fees) for every type of card that runs through their network. MasterCard for example has 243 different fees.

Convenience

Card/PayPal/Carrier solutions are all centralized, you lose a bet and want to give your friend $10; one of you has to pay the cashless overhead? Who? You and three friends share a dinner at a restaurant and you pay the bill in full. Your three friends each then need to be able to transfer a quarter of the total amount to you – creating four transactions in total (and four times more transactional revenue for payment processors). You all also need to be online.
Card/PayPal/Carrier solutions just don't work offline. With digital cash, if you and a friend are both offline, as your friend is in your social graph, you can accept it money he gives you at face value (and then perhaps validate it later when you are online). A bit like a cheque. The concept of trustspace could also be used to grade and evaluate trust outside a user's social graph.
1. With trustspace, consumers would be rated by how many times their balance has reached zero (since here you have contributed exactly as much to society as society has contributed to you). To avoid rating manipulation (a sort of new credit rating), trustspace could be parameterized by personal turnover, a damping factor and a connectivity index derived from the number of other people you have dealt with. Your rating would diminish over time and so you would need to continue earning/purchasing with digital cash to stay active.

Privacy

Card/PayPal/Carrier solutions log all transactions made by the consumer. If you lose a bet and give your friend $10, that transaction is recorded. If you go into a bar and rack-up a large tab, that transaction is recorded. If you pay for prescription medicine, a DVD in a bar, "herbal" remedies, a massage or Playboy magazine; all –all those transactions are recorded and used, at the very least, to target marketing to you. There is a growing demand for data privacy and consumers want the option to remain invisible to a payment made on their behalf. Privacy is not so much a blanket consumer need to be unseen in terms of online/digital activity but a desire for easy control. The recent WikiLeaks scandal compelled some services to stop handling Wikileaks' business including payment services (inc. PayPal). Anonymous digital cash helps fund enterprises that, for whatever reason, others object to.

Additionally, there are wider economic benefits to digital cash:

Being a zero transaction cost solution enables consumers to sell online content and would be an alternative to advertising revenue meaning a reduction in distracting/invasive banner ads (micropayment). The fact that it is free means that it will stimulate a free market economy on the web where the best people, organisations and content will rise to the top because they can be directly paid. This, in turn, would make it easier to find things of most importance (currently we are spending 53% of our time searching for the right information).
Consumers should also see an improved web experience through a freer market economy.

A 10% shift in consumer spending, from chains/Internet to locally owned retail (currently being driven out of business due to Internet purchases), would create nearly 1,300 new jobs and over $190M in increased economic output for San Francisco alone. More jobs and more economic output in a specific geography where you own a house also means houses increase in value.

Twenty years ago, a digital cash solution was developed called ecash. It was sold by a company called DigiCash. A now defunct US bank and a handful of small European banks went live with it in the mid nineties but DigiCash went bankrupt later that decade and assets were acquired by InfoSpace in 2002. There was a similar story with CyberCash (over a similar timeframe). They went bankrupt in 2001 and assets were acquired by PayPal in 2005. No commercial organisations are now known to be operating these or similar systems.

Both organisations are considered to have failed due to security, implementation and administrative problems. They also made the validators – banks (when they really just needed to be web-services) which added unnecessary overheads/slowed down time-to-market. Countries historically used to back bank notes with gold but this is barely done any more. Canada for example has no gold backing for its currency.

Media interest in digital cash coincided with the dot-com bubble of 1995-2000. The vast majority of books on the subject date from that time. Online finance in general has stalled since then e.g. W3C abandoned attempts to incorporate micropayment into HTML. InfoSpace itself was a notorious dot-com casualty (in March 2000 stock reached $1,305/share but by April 2001 had crashed to $22/share).

Other than the fact that over two-thirds of the world still have no Internet access, there are several convergent trends right now that could build a landscape for a digital cash solution:

Smart-phones replacing traditional dial and text mobile phones.

Cheap/pervasive contactless NFC technology with an open API. Estimated 40-50M phones on market in 2011. Widespread NFC adoption of payments (149+ projects worldwide). High-profile assertions that mobile is the safest way to pay.

A thriving start-up culture (possibly due to the recession?).

Widespread and growing mobile/app culture. Mobile app market to be worth $25BN by 2015. App downloads to increase 605% by 2014. There is also more evidence that consumers are more likely to buy using a mobile app than regular web applications. Location services are on the rise.

General distrust of existing financial services (esp. banks) with consumers being open to alternatives. With both cash and clients in limited supply, barter networks e.g.
Dibspace , ITEX, BarterCard and IMS are becoming popular. P2P lending in particular e.g. Zopa, FriendsClear, LendingClub is becoming accepted.

Rising social graphing acceptance (bolstering trust issues).

Widespread acceptance of storing personal details in the cloud via trusted sites.

Digital signature/public key based cryptography acceptance.

25% of consumers have poor credit scores.

Distinct lack of retail banking innovation.

What about other forms of cashless payment?

Visa's payWave system was introduced last year as a digital wallet for the iPhone although this requires a separate case for the mobile (basically containing the same chip as new Visa cards). AT&T, Verizon and T-Mobile also announced as Isis, a joint venture to equip mobiles with NFC chips; linking them to credit/debit cards. Barclaycard has signed on to issue credit accounts through this system. Google have said that they will work with industrial partners for their digital wallet solution and with them pulling players together (highlighting better loss rates); previous collaboration issues should be resolved. Google clearly say though they want to replace your credit card not your cash.

Square allows anyone to accept physical credit card payments through a mobile or computer by plugging in a free sugar-cube-sized device so no expensive card reader is required. Of course this solution suffers from credit/debit card dependency.

Obopay lets consumers and businesses purchase, pay and transfer money through a mobile phone using a mobile application, text message, mobile Web or Obopay.com. It works on any mobile phone and any US carrier; again though it is tied to a card, in this case MasterCard debit.

Start-up Bling Nation went live in the US last year. They partner with both PayPal and banks, who then offer consumers a Bling Nation and "Bank" branded chip that can be stuck onto any mobile device. The chip allows any user to make a payment directly out of their checking account similar to a debit payment.

Rovio is taking a carrier billing approach with their payment system Bad Piggy Bank, intended for in-app payment of virtual or online goods. Zong have a similar approach. They also run a points system if you link your credit/debit card to your mobile and have reportedly processed transactions from 15M unique consumers.

All of these solutions are various themes on Card/PayPal/Carrier solutions and so also carry their failings. It could be argued that consumers and merchants are having installed on their behalf technological solutions that continue established order in preference to directly addressing their needs. In much the same way that PayPal principally extended the credit card model into the online world, current social/cashless payment solutions are seemingly doing so now.

Digital cash, in theory at least, straddles online and physical worlds better than any other solution. It empowers consumers and merchants through clear cost, convenience and privacy benefits over existing/planned solutions (physical cash or cashless payment solutions) and affords wider economic benefits to both.

The market has moved away in the last couple years from online/online solutions (payer/payee respectively) e.g. PayPal toward online/offline e.g. Groupon, Uber. On the topic of Groupon, receiving/generating a coupon then integrating this with digital cash (applying usage constraints/"increasing" your digital cash as appropriate) automatically within your digital wallet would be very powerful. It is quite possible the market will shift again to an offline/offline model in the near future. Is the best way to realise this - digital cash?

Obviously, big issues of system and national security, cryptography restrictions, economic stability and consumer acceptance would need to be overcome. As Minsky said "creating money is easy; the hard part is getting it accepted". However, alternative payment systems are already being adopted; being 20% of all online transactions. We are also seeing a surge in self-organized /managed citizen activism especially around finance and digital cash perhaps hooks into this trend too.

UPDATE: There are several ways to obtain digital cash.
UPDATE: Possible ways to monetize digital cash.

Time to stop checking-in & start checking-out

2011-01-11T22:01:00.001Z

Thinking about geo-location (GPS) and its mobile under-utilization, I considered what applications it actually is well suited to. Checking-in and earning badges to become major of somewhere just has to be a short-term fad even if they do begin to enable deals (?).Obviously there is finding local stores/services but that is pretty well covered by Google Maps and others. Crowd-sourcing e.g. seeing where contacts have been might work if your contacts left comments about stores/services (which they don't). Seeing your contacts are close-by so you can call them and meet-up doesn't appear popular (you kind of already know where your close friends are). The forecast for the geo-location market leader - foursquare is hazy.

So what mobile app could really make use of it? Unlikely as it may seem at first, I came up with dating; although as we'll see it may be unattainable for social reasons (much like even though we already have well implemented video chat, people reject it).

Why what's the problem with online dating?

As online dating becomes more main-stream/more commoditized – even reportedly now used by celebrities, it also suffers from lack of innovation/differentiation. Online dating came to prominence on sheer convenience; being able to trawl through hundreds of profiles w/out leaving the comfort of your home. But scanning through ever more prospects due to poor matching/sparse profiles and an increased social need to be mobile/online have eroded this convenience.

Social and mobile technologies should be able to add value but they seem only to be starting to. Looking at online dating as an application area (I had to research it) and comparing it with the speed-dating space (slightly more experience) there is perhaps a solution waiting to be found.

Some sites solve several issues but, in general, online daters need to:

Maintain a separate/proprietary profile.
1. Tedious to enter/maintain personal information already online. Few dating sites are really focused on matching anyway; too complex/expensive and not immediately appreciated by the dater. It's about minimizing acquisition cost/extending CLV. They don't necessarily want to get people off the site.
Periodically check to see if there are people to contact/someone has contacted them (pull).
1. Average visit/site time is 22 minutes. Finding time for dating is a real problem, necessitating some to even look for ghost-writing solutions.
Deal w/ communication awkwardness.
1. Communicating by sending messages to strangers and then being rejected more times than not is unnatural, demoralizing long term and doesn't fit w/ the personality of anyone but extroverts.
Determine the best way to leave a date if they don't click w/ the other person.
Pay on a subscription basis.
1. Market leaders for online dating – Match and eHarmony monetize on subscription. Plenty of Fish is the only free site in top-10. Sites charge $35/month on average.
  1. Free sites have their own issues though. As there is no barrier to entry, profile quality is poor as people put less work in. Many actually prefer paid sites for this reason.
Deal w/ incomplete information.
1. Feedback that could be useful to prospective daters is lost.
  1. People typically don't reply if they feel there wasn't a connection. Consequently, the rejected person is unsure whether to approach the person again.
  2. Future prospective daters of the rejected person cannot take previous criteria (from others) into account before meeting.
Handle remaining social stigma among peers (despite relaxing recently).
Infer second degree social graphing.
1. Having 175 friends (pretty typical for those dating) means people can have 60,000 friends of friends (Friend of a Friend - FOAF). It is likely that many of these people are dating. Identifying these people is useful since their immediate friend can be contacted as they will provide a source of additional information (about their friend). Determining these connections becomes part of the date.
Cancel their account after a few successful dates.
1. After a few dates if the daters vaguely consider themselves "in a relationship", then it is expected that they cancel their accounts and live happily ever after.
Work w/ poor, non-existent or additionally paid for mobile versions of sites.

How could these problems be solved?

A free mobile app, connected to an existing/widespread profile that, once you register (one time/through Facebook) pushes meetings to you i.e. the solution opportunistically knows that you are available (from looking at respective schedules) and are physically w/in a five minute journey (from GPS) of someone that matches you/you match (based on filter criteria you have previously entered) and so arranges it automatically (pushes it). You get the other persons picture, a profile summary and a named coffee shop (in-between you both) together with a time. You can accept/reject there and then. If you reject, the meeting is simply off (could be because the solution itself decided there simply wasn't enough time for it so fewer social rejection issues). If you accept, you meet for a time-boxed duration at a coffee shop w/ the app managing the duration and other details (like a speed date assistant).

It prioritizes similar people (of the correct orientation). Everyone needs a facial photo on their profile so that they can be recognised. Basically, you don't need to log into the site again (unless you want to) as the mobile app will push everything to you.

No-one meets up anywhere other than a coffee-shop halfway between the two people and no-one knows where the other person is other than at the coffee shop. Coffee shops are necessary since they are an easily found public space. As such the solution would work better in cities. The solution is more general than dating i.e. intended to be used by everyone. For example, you could express an interest in whether to buy the new iPhone and then meet someone who has one. Online dating would certainly be the beach-head application though.

Post meeting, you "thumb-up" or "thumb-down" the other person. If you both mutually thumb-up each other, the solution informs you and gives you the ability to exchange 1:1 contact details. This information can be condensed into a simple personal reputation score which can be used as selection criteria e.g. you can elect to only meet people w/ a high reputation.

Since the solution is not specifically dating related, reputation scores can be made available (via API) to other sites. Existing reputation solutions are principally based upon Twitter followers/retweet activity e.g. Klout, Peerindex or Tweetreach. This allows those that don't use Twitter to compete online. Since it is more personal than tweeting, it is also potentially a more accurate reputational assessment for other sites to use e.g. when offering deals. Wider reputation usage helps drive traffic and because it is solution specific – makes replication by competitors difficult.

The solution described above addresses the online dating problems in the following ways:

Maintain a separate/proprietary profile.
1. Solution uses Facebook Connect to enable one-click login. Truncated versions of people's Facebook profiles also become their dating profiles. It is not a Facebook app and no-one can see whether you use it or not. People are already using Facebook to flirt and this solution extends that but removes the bad etiquette associated w/ trawling for dates.
Periodically check to see if there are people to contact/someone has contacted them.
1. Solution saves time. Trawling through online dating sites to find (pull [rather than push]) someone interesting to you is time-consuming. Push model enables solution to work around your schedule. Increased efficiency e.g. people can email each other for weeks but only really know whether they connect after a few minutes conversation. Makes the whole experience low expectation (good for first dates).
Deal w/ communication awkwardness.
1. Relying on the mutual knowledge aspect of mutual thumb-ups e.g. I know that you don't know that I thumb-upped you unless you thumb-up me - would remove much communication awkwardness.
Determine the best way to leave if they don't click w/ the other person.
1. Meetings are for five minutes only. After five-minutes, an alarm would sound through both phones. Timing starts once both people's phones are in immediate proximity to each other (to handle one/both people being late to the meeting).
Pay on a subscription basis.
1. Solution is w/out cost (clearly a winner). Barrier to entry/profile quality issues are partially addressed by the reputation score.
Deal w/ incomplete information.
1. Through the reputation score.
Handle remaining social stigma among peers (despite relaxing recently).
1. Solution is not specifically a dating solution. Even people that don't want to date may be interested as it enables them to opportunistically and simply meet new and interesting people or discuss specific topics w/out planning in advance.
  1. People's lives are increasingly scheduled and many are drawn to the idea of occasional "curve balls".
  2. Can offer beforehand to buy someone a coffee if they are able to discuss a particular issue from a position of experience e.g. rearing a puppy, fixing a computer, getting into a particular industry or continuing online questions from Q&A sites e.g. Quora. This could also work for sales e.g. you get a free coffee if you sit through a five minute pitch.
    1. Similar to Facebook Like button, a "Meet" button could be federated to other sites to indicate a willingness to discuss a particular topic.
2. Since it is based around your schedule i.e. whether you are physically close enough to meet there is less dating investment enabling it to be treated casually.
3. The time-box nature makes it more akin to speed dating which inherently carries less social stigma than online dating.
Infer second degree social graphing.
1. Through Facebook social graph. This has a freely available API.
Cancel their account after a few successful dates.
1. The solution moves on from scheduling that initial coffee shop date to bars for successive dates to theatre/film scheduling for later dates. Even years after two people have got together through the solution, they could still use the solution as a proximity meeting facilitator.
Work w/ poor, non-existent or additionally paid for mobile versions of sites.
1. Solution uses location awareness on exception basis (push)/much more simplified than current mobile dating sites (basically simplified versions of web sites). Conversation topics based on mutual interests could also be suggested seconds before you meet. It is the only access point to the service and free.

The solution would certainly be open to abuse. People are travelling five minutes for a five minute meeting. They can set filters to ensure the solution just pushes meets that they want and if it's apparent that the person is disingenuous - they leave after one minute (total six minutes wasted) then leave poor feedback or report/block the person - similar to email spam. It should actually be easier to control than email spam since you can leave poor feedback/report the person. It's true you have potentially wasted 5-10 minutes of your life though (w/ email spam it will be seconds to read the email). Then again consider risk/reward: W/ email you risk minutes/day w/ spam against the reward of instant written communication anywhere in the world. W/ mobile dating (or meeting), you risk 5-10 minutes against the reward of meeting someone cool/learning something new. People are already risking 22 minutes every time they visit an on-line dating site to meet someone cool (and unable to multi-task), so it could be posited as a time saving.

Following on from our high-level design, some detail on how to potentially realise it. Starting with - how might it make money? It would always be free to use but development cost should be able to be offset in short-term:

Short-term. Affiliate marketing would be impacted due to the "proximity/push" model i.e. people are not regularly hitting a specific site w/ which to see ads.
1. Small, targeted ads would need to be embedded and cycled w/in the mobile app and shown when date messages are pushed. The option for a paid version of the mobile app could remove those ads e.g. the Angry Birds model.
2. There is a trend for free online dating sites to be launched to great press (focussing on some minor innovative angle) and then closing down several months later e.g. Thread (FOAF), CrazyBlindDate (randomness). Likealittle (flirty commenting) has only recently been launched and received 1M uniques in first month (reportedly 20M page views). Conservatively taking uniques and using the Amazon Affiliate program, this would have generated $10,000 affiliate revenue (assume 1% click-through rate on average price $10). On page views this would have been $200,000.
3. Online donations e.g. through Flattr can generate a surprising amount of income e.g. Flattr reportedly kept WikiLeaks afloat for a period.
Medium-term. Assuming the solution successfully transcends short-term "viral effect", the amount of active registered users dictates revenue.
1. Plenty Of Fish for example, reportedly makes $30/registration and £10M annually through affiliate advertising. This is on 250,000 uniques each day.
2. Once the concept is proven, it could pivot to become a back-end service (accessible via API) for other dating sites i.e. they provide their own user experience and value adds e.g. specialist matching algorithms, niche markets. The solution would handle Facebook integration, tracking, messaging, rating/feedback and meeting handling. The solution becomes a natural monopoly (due to network effects) that future dating sites need as daters expect it. It could conservatively be licensed to the top-50 dating sites for $500/month ($300,000/year).
3. Equivalent services/outlets available around the interim point e.g. coffee shops/bars can compete for business w/ the solution paying the solution for "referrals".
Longer-term. A working solution would be attractive to acquisition.
1. Facebook may see it as a way to extend its brand - moving out of the digital world into the physical world/gaining deeper social data.
2. Coffee shops e.g. Starbucks may simply see the solution as a means to drive sales (interim coffee shop meeting points). Starbucks are keen to develop their loyalty programme e.g. they recently partnered w/ foursquare to unlock a "Barista badge".
3. Location-aware apps e.g. foursquare may view it as a way to extend the short-term conceit of checking-in/obtaining badges etc.
4. Other dating solutions may find it attractive to compete/diversify (much like Match tried to compete at the free end w/ Plenty of Fish by releasing Down To Earth.
5. Personal assistants e.g. Siri may see it as complementary.

What competition is there?

There is a much competition. Some solutions solve some of the problems above but none solve all of them and none have the proximity/push cornerstone approach supported by a speed dating assistant:

Classic online dating sites e.g. Match, eHarmony and Plenty of fish.
1. Market is fragmented/specialized e.g. 50+, Jewish, European (Meetic) etc. at the lower/start-up end.
Local speed dating outfits that also have an online presence e.g. SpeedDater.
Dating Facebook applications e.g. Zoosk and Flirtable.
Dating sites that use Facebook social context e.g. AreYouInterested.com, DateBuzz, SpeedDate.com and Chemistry.com.
1. DateBuzz has gained press for its innovative use of letting daters rate other dater's profiles. Profiles provide a fraction of the detail gained from even a five minute meeting though and are open to gaming/deception. Also, there is a case to be made for saying that daters don't actually know what they want until they experience it.
Friend locators e.g. Facebook Places, Google Latitude and face2face.
1. Facebook Places is geared toward friends/checking-in much like Gowalla or foursquare. Latitude is more organic in that it simply tracks you and allows you to see other friends. Both Facebook and Google are unlikely to provide direct online dating support themselves in order to protect their brand but it is conceivable.
Mobile dating apps that use location awareness e.g. Grindr, Skout, Flirtomatic and MeetMoi.
1. Grindr is very niche. MeetMoi is more general. Neither have speed date assistant capabilities e.g. interim coffee-shop scheduling, strict date timing and reputation scoring. Both also require proprietary profiles.
2. Skout and Flirtomatic use Facebook profiles and are probably closest to the solution however they hamstrings themselves by focussing on chatting w/ others in your location rather than proactively scheduling a meet with them while it is possible (they may only be there for a few minutes). You can chat anywhere. Geo-location only becomes useful to online dating once the opportunistic meeting element is introduced.
3. This approach is widespread in China although this is done on older technology e.g. many Chinese wireless service providers offer dating services based upon signal triangulation to drive text message usage.
Sites that use scientific matching foundations e.g. ScientificMatch and GenePartner.
1. These two actually use DNA comparison. While this may be more accurate than proprietary matching algorithms, it is still reliant on matching rather than chemistry and chance. Cost too is prohibitive for now.

To sum-up, a mobile dating solution (essentially: Social + Mobile + Speed dating) bought to market now could earn distinct competitive advantage if the social adoption question could be resolved:

Market. Online dating in general is a $1BN global market (some say $4BN), has yearly 10% growth and is known to weather economic storms. Market for free online dating solutions is fragmented. Plenty of Fish is the only one that has a real market (followed by OKCupid) yet the paid market is market is shrinking. This solution is wider than online dating; effectively creating a new market – proximity meeting enablement.
Innovation. Online dating solutions have yet to capitalize on mobile and (to a lesser extent) social. W/ smart-phones set to be ubiquitous, this is a clear opportunity for innovation. The infrastructure/audience for geo-location in particular is there but wasted on games e.g. earning badges/becoming mayor etc. Mobile usage also enables new approaches to online dating. In particular, the opportunistic proximity/push feature is a clear differentiator. Ultimately though, the innovation here is - to handle the ten problems w/ current online dating slightly better than everyone else. It is this that gives the solution a real chance of progressing past being a short-term trend to become a brand.
Risk/Reward. Roughly six weeks to build onshore on Android (assuming design completed) and cost around $35,000 to get a first release out from good developer. Refine app (assume 80 hours costing additional $8,000). Releasing a beta then would cost roughly $51,600 ($35,000 + $8,000 + 20% contingency). Plenty of Fish is reportedly mostly a one-man band. Development cost could be offset by combination of affiliate advertising/ad-free payments and donation w/in 3-6 months. A separate upper limit estimate for total cost (complex Android app) is $80,166. If the solution took just 10% of Plenty of fish business, this is £1M/year revenue on low overheads. Finally, there are clear approaches to extend solution functionality and acquisition from several varied sources would be possible.

Ultimately, this solution could potentially create an important new communication channel while addressing the online dating problem. By far, the biggest question is simply whether people (particularly women) would adopt the concept of having meetings (dates) proactively scheduled for them on-the-fly as they go about their lives. Conversely, there is a sort of chance/synchronicity element that they might consider romantic while men might be drawn more to the time saving/efficiency angle. As no-one has really done anything like this before, no-one can say. What can be said though is that it isn't a huge investment to find out and like many of the dates that would be pushed to your mobile; it is probably worth checking-out.

The project failure & cloud connection

2010-10-31T17:30:00.042Z

Around 2000, I worked on a project with a co-worker – he was maybe fifty and had worked in the IT industry – initially at IBM then contracting and consulting for nearly thirty years. At the time; he was heading up a web-development and creative team for a small consultancy firm which he saw as a new opportunity for him. He had a good reputation, people liked him and he had a vast amount of experience from being involved in literally hundreds of projects – developing, managing, business developing etc.

One day he slips into conversation – as an aside really; that he has never worked on a project that was delivered. Every project he worked on had been cancelled, merged into another programme, didn’t meet its financial goals (as with infrastructure consolidation) or his involvement ceased before it ever went live (if any of them ever did – he didn’t know). He said it in a resigned manner, almost as a badge of honour that he had got through all that with no job satisfaction in terms of delivery payoff whatsoever.

I have seen this scenario repeated many times in the decade since; although perhaps none quite so extreme. I have myself worked on hundreds of projects in many roles and can count the number of successful systems deliveries on one hand. People naturally want to associate themselves with successes rather than failures so it is perhaps understandable that this is not a common consultant topic.

A whitepaper released late last year attempts to put a figure on the worldwide cost of IT project failures. This turns out to be $6.2 Trillion and it doesn’t look like sensationalism. The US in particular is apparently losing almost as much money per year to IT failure as it did to the financial meltdown (with no end in sight). The paper makes an attempt to factor in what it calls indirect costs; basically a lost opportunity cost from the time wasted on failed/abandoned projects. It does not however take into consideration the wider indirect costs of people training for careers that are not actually delivering, IT staff disillusionment (turnover), operational failures for delivered IT systems (one in five businesses lose £10,000/hour through systems downtime) and associated security failures.

The paper has received some condemnation (due to its base assumption of a 65% IT project failure rate) but there is dearth of analysis in this area and quoting this figure is about as good as it gets right now. 50-80% figures have been quoted in one form or another for decades. CIO thinks rates are actually rising due to the recession. People have the choice of either working with a figure (challenging assumptions/statistics etc) or burying their head in the sand.

We will never know the exact figure for IT project failure. Similarly, we will never know whether efficiency/functional benefits of those systems that have worked have paid for the failures i.e. What has IT really given us? We are simply reliant upon these systems going forward. What can we do to reduce future project failure rates?

Although there are superficial similarities with scientific/experiment and IT/project communities respectively, the "accept defeat" approach of the former is routed in constant learning whereas IT is surely about delivering benefit now. Learning is only the priority for the largest, most stable, lowest turnover organisations; that is to say - next to none of them. The scientific regimen of independent assessment however is invaluable for IT projects. Tool-based PM consultants such as Bestoutcome are probably as good as the big management consultants for this purpose though. Techniques from the engineering community (when introduced to IT) have not had a huge effect.

The paper ends with a call to arms to simplify – IT/business communications, projects goals etc.

I agree in principle but this is oversimplification - when your realm of influence is the organisation you are in. I have seen many projects that although conceptually simple and with genuine IT/business agreement start to fail the moment integration with other organisations – vendors/hosting providers/recruiters/sub-contractors/data sources are required. Despite solutions being simple and manageable inside your organisation – just a few touch-points with others (basically anything worth doing) cause them to be complex and therefore unpredictable/at risk no matter what your collective capabilities. There is even a case for blinkered-simplification/procedures actually contributing to project failure: Complexity at least brings an element of flexibility, allowing you to react if the project starts going bad.

Better project managers, SOA, ramped-up co-worker involvement, Facebook-like "hackathons", daily IT/business meetings, PRINCE certifications, extranets, more rigorous cost control and mathematical complexity models within your organisation will have minimal effect on the success of your multi-touch-point projects – for they are already in the realm of chaos. Even improved PM collaboration through tools such as Asana will have a minimal effect on success rates. The role of the “good project manager” is perhaps the most scrutinized, personality-driven, divisive and misunderstood of all IT positions. Radical open enterprise models such as BetterMeans that effectively remove Project Managers in favour of automation and decentralized decision making will similarly be ineffective. Although in the case of the open enterprise model, I do agree that this will ultimately prevail (crowd-sourcing, creativity enabling and ultimately efficiency/cost) but not for decades (due to the need to directly attack the failure rate first).

Although there are certainly sizeable success increases to be made if you are experienced in the particular technology, the IT project failure rate will only consistently and materially fall once there are flexible cloud services that organisations can get 80% of all their needs by just subscribing to them.

Integration ceases to be the bottleneck. The other 20% being “secret sauce” value-add-ons developed in-house; probably mainly algorithms that, by definition, do not require integration with other organisations or heavy project governance. Other components of the 20% would be device specific exploitation code; essentially building the so-called App-Internet model (rather than full-Cloud). Organisations already recognise the economic justification for cloud computing so it is perhaps inevitable that project failure rates will eventually fall by default.

Cloud transition will take organisations years yet however. Both InfoWorld and Gartner have arrived at 2013 for broadly when the majority of organisations will run cloud. This may be optimistic. In the interim (three more years+ of high project failure rates?), delivery is simply better served by being built upon cloud solutions now; building partnerships with cloud providers if they can and leveraging their buying power if they cannot. Also, interim/limited functionality cloud solutions must be considered in preference to bespoke development/on premises deployment. In a real sense the project (that they would have otherwise completed themselves) becomes a creativity-driven; architectural investment and commercial partnership instead. Both Project Management and Enterprise Architect roles will need to shift accordingly.

It would be interesting to see any future IT project failure analysis split between organisations that have implemented virtualisation, those that have implemented private cloud and those that have implemented public cloud. The project failure and cloud connection is not well documented.

Why are we still talking about search?

2010-10-27T18:01:00.072+01:00

Google is obviously pretty successful due in large part to its search service. Despite them not having the best social networking track record, it’s an activity that maps well to the social world. We search for things – our keys, a present, a holiday, our next partner, a restaurant to take a friend to - our next role - ourselves. Google doesn’t give us exactly what we want but that’s OK because we don’t know exactly what we want either. It’s useful to be given a few social curve-balls.

Despite us all mainly using Google or Bing (Yahoo, Ask etc. all gone now), new search engines e.g. Blekko still regularly surface with varying differentiations and significant funding.

What if, rather than vague searches (inevitably) generating prioritized yet potentially vague results (M:M) [in terms of question:answer]; we could ask a specific question that provided a generally accepted, precise result (or if that question had previously recently been addressed, a link to same) that we (and others) could refer to (to later justify our logic). What if we wanted - an actual answer (1:1)?

Consider - “Which is the largest State in the US?” The question is vague (by land or population?) also Google is mainly matching keywords rather than actually understanding the question (and of course we get many results so the full exchange is M:M). Actually, Google have been delivering factual answers to some queries for five years. It only works for some though (it doesn’t work in this case). Regardless, Google’s first result correctly tells us Alaska and additionally gives us comparison with other States and square mileage too. We aren’t guaranteed getting the same result when we query in future (so accountability goes out the window) but the answer is OK for most purposes. If we make the question “Which is the largest Republican State in the US?” we still get an answer but takes us a few minutes to manually cross-reference the links. If we ask “Which is the largest, happiest Republican State in the US?” we simply don’t get a usable answer.

Someone living in Alaska is worried about perceived rising Russian immigration and they want to know if it has risen in their adult lifetime. There are no obvious figures on the Web. They could tweet/email the Governor and (if enough others have asked broadly the same question) then he might post a response on his blog (if he has one) once his research team have arrived at some defensible statistics for him to refer to (assuming they can find some).

This simple/current scenario has challenges and inconsistencies on both question and answer sides. Here are a few:

1) People have to feel so strongly about the issue that they have to find time to interact – search for any similar questions/answers, write their email, monitor the Governor’s twitter-stream/blog for an answer. They feel as though they don’t want to put themselves forward unless others feel the same. Conversely, enough people have to use similar phrasing of the question in order to kick-start the answer process.
2) The answer (if it is given) is not explicitly tied to the question. It could be made on his blog, a successive TV appearance, in a press release or personal speech.
3) There is no disambiguation or prioritization of questions – meaning that there is just so much noise that the Governor or his team simply don’t have the time to answer.
4) The answer could be presented in different ways – perhaps to achieve political ends or simply because they are underused by Government e.g. taking Russian immigration over a twenty year time-frame may show no year-on-year increase but taken over a two-year time-frame it may show a very different story.
5) There is little opportunity to to-and-fro with the question and answer thread – blending anecdotal evidence, redefining terms.
6) It’s a naive voter with a question - asking a team of well researched people. Odds are there will be a soft soap answer as independent experts are not involved in the process to moderate.
7) Statistics could come from Federal or State government, commercial, charity or other bodies. Also there is unlikely to be any checks and balances e.g. does the US figure for people incoming match with the Russian figure for people outgoing?
8) Any presented figures will be difficult to understand (unless an intuitive and consistent graphical approach is used).
9) Lawyers may need to be consulted to define the term “immigrant”.
10) There is no accepted “end” to either the question or answer.

In business, we need - to commit resources to a business plan, to indemnify ourselves against litigious customers, to make our point in meetings and to provide defensible input to decision making. Consumers largely search for things (M:M) while companies have to query things (1:1), although as with social networking, there’s some overlap (M:1) in the middle. Companies lose out as they maintain a blinkered-view missing critical information not captured through their own processes. Consumers lose out since they spend so much time searching for things (and the Internet is so rich) that, like travelling, the journey becomes the destination.

Google Search Appliance has done very well in the enterprise. There is a need for a one-ring-to-rule-them-all "get things" function. Also, our work and home lives are merging and when the answer you present is (close enough to be) unanimously accepted and is able to be referenced in any future analysis of your decision - then it will always stand. Once you don’t have to think about the logistics of getting answers, you can concentrate on actually doing things with them (for both consumers and the enterprise). To do this, quite simply - we need a single answer, a 1:1 model that works for everyone - like search – a Q&A service.

We have “I’m Feeling Lucky” of course (M:1) but this is an indulgent lucky dip that actually loses Google revenue. The other way around (1:M) doesn’t work since it entails definitively stating the question e.g. using something like QBE or NLQ (both of which are hard to do right) but still getting a range of results. What other services are currently available? Facebook released Facebook Questions earlier this year. This is slated to be integrated into Community Pages (already includes Wikipedia content). It is developing but is still very social/free-text. Wikipedia is an encyclopaedia rather than a Q&A service. Yahoo Answers has more of a wiki/free-text approach. It probably is the widest used but is also known for being random and open to abuse. PeerPong is expert-focussed. Off-topic a little, for product reviews, there are Hunch, DooYoo, Epinions, Mahalo (their Answers service is a bit wider focussed than just products though but still very social). There are mobile Q&A services such as Mosio and ChaCha. Quora (founded by ex-Facebook CTO Adam D’Angelo) appears to stack up against Facebook Questions and again is very social. Social Q&A in general is big. StackExchange is an open-source platform for developing your own social Q&A service. In the enterprise, Opzi is attempting to be a corporate Quora. MSFT shut down their offering last year. Qhub runs a niche hosted Q&A service. Some social ones such as Hunch have repositioned themselves as recommendation services. Many just cater to niches e.g. StackOverflow/development. OSQA is a free open-source one. Several dot-com bubble casualties have refocused on discussion based Q&A e.g. Ask has refocused on a mobile Q&A service similarly Startups.com.There are other start-ups.

All of these services are in some way niche, most require an account (which will put off many) and most provide multiple answers (1:M), although collectively they will probably end reliance on reference libraries (and on the product side – magazines like “Which?”). They helpfully hand you a piece of the jigsaw rather than solving the jigsaw for you. As has previously been said "None of these sites are Google-killers. In fact they make Google stronger because the questions and answers often will be indexed - extending Google's reach into the tail". Quora in particular has been recently heralded as a killer Q&A service but asking it "What are the most effective ways to engage news audiences?" receives a bunch of people's opinions; whereas we could have a single list in order of effectiveness that has been produced through actual operational data and curated by either a Journalist or Statistician. If there were any dissention around the term "effectiveness" then that could be hammered out through social interaction. There wouldn't be a need for a - discussion. As with all Quora questions, you need to read all responses to get a full answer but even then its just a subset of people that responded. Another example - RockmetIt better than Flock?

If the industry is framing Web 2.0 is the Social web and Web 3.0 as the Semantic web – it seems churlish not to leverage the powers of both in the Q&A service. The benefits of great data integration using common terms (Semantic) and crowd-sourcing (Social) in a Q&A service are obvious. By the same token, some answers will always require a (semi)/professional element to them (Expert) e.g. Someone asking a question about Alaska’s closeness to Russia (surprisingly - 2.5 miles) might get a quick answer but it will take an expert to plan how to get from one to the other through that particular route. Expert knowledge also comes into play around the presentation side – knowing which facts to present to most effectively answer questions e.g. the largest State question above – given to another search engine also gives Alaska but the square mileage is significantly less. A Geographer knows this is due to the second result not including water. There are other components but, in the main, both questions and answers have - social, semantic and expert elements.

Back to the Alaska question (“Which is the largest, happiest Republican State in the US?”). This requires both a semantic element (to determine the largest State) and a social element (who is happy?). If we make this “Which is the largest, happiest Republican State in the US liable to switch to Democratic?” then there is an expert element too.

Onto the proposed Q&A service - it needs to check a few boxes right off the bat:

1) Social. It would have to be based upon a popular, real-time social system. Despite perceived issues investing in their ecosystem, Twitter is arguably better than Facebook (or any other) for this purpose but it also needs to amplify and expand the reach of questions: present persistent questions that have undergone a process of disambiguation (both automated using semantic algorithms and manual through a public process of “backing” existing questions). Questions also need to be prioritized based on number of backers. Something like Kommons, Replyz or maybe Formspring or Open Media could work here. Amplify is maybe too discussion focussed. The focus here is on directly (directly to the tweeter) quantifying the importance of a question and the validity of an answer. Social media monitoring like Crimson Hexagon could offer supporting services but they typically rely upon sentiment analysis whereas we really need direct engagement through voting or other mechanisms.
2) Semantic. We all need to be sure at the very least that we are talking about the same data. In short it will need an index. Google have Rich Snippet functionality but this is not a common approach adopted by most site owners. Google also acquired Metaweb earlier for this year for this purpose but something more open like Sindice would really work providing organisations have an incentive to cooperate (integration/ontologies/micro-formats etc.). Providing and articulating the incentive is the toughest piece of this by far. Then – we want an interface on the results (produced by the index) that allows us to de-select terms that we are not interested in so that the system knows for next time but also to publish the results of our tinkering and link it to the threads in the question or answer. Something like Sig.ma would do this.
3) Expert. We need data sets that are both selected and curated by subject matter experts. Curated answers can often be the most useful. Google acquired Aardvark earlier this year for this purpose but something like Wolfram Alpha or Qwiki would work. Involving Wolfram Alpha would add other benefits too – computational power (through Mathematica) to aggregate on-the-fly, leading NLQ support to translate questions, updatable widgets to support the presentation side and the semi-celebrity name of Stephen Wolfram attracting academic curators. All Wolfram Alpha content would need to be indexed and we would probably need to extend the publication side to be able to deselect returned facts as required (doesn’t appear to currently be available).

We are talking about potentially lots of social and semantic data needing to be agreed and presented. We’ll need simple data visualization with a basic level of interactivity so, with the example above; the user can slide the time scale to represent his adult lifetime. Something like Many Eyes, Hohli, StatPlanet or some improved Google Charts should do here. Also, we will want to store all our questions and answers forever so that we can reference them. Partnership with a very large EMC-like network storage partner will be required.

The resulting composite solution would be the most advanced Q&A service in the world. Q&A is so important and multi-faceted that it needs multiple solutions to work together. Its influence, if accepted, would be profound.

The service would drive out real answers (reference and topical) through a combination of social, expert and semantic approaches and actually referentially improve itself - as bringing social attention to the data would force organisations to improve data. Usability-wise - it wouldn’t hamper ad-hoc users by having a scary QBE type interface or by blinding users with spreadsheets of data. Conversely, it wouldn’t be so simple that it misses the point of what people want. It would be a natural extension to tweet activity (retweet, favourite, reply etc.). Opportunities for monetization would be rife: not least advertising with the solution’s wider collaborative user base, increased opportunities for interactivity and greater relevancy. New forms of credibility and relevancy would also become available. Journalism would shift to sourcing stories from datasets.

The traditional search space e.g. "Alaska Jobs" is already being eroded by location-based services. Google is certainly well placed to build a killer Q&A service as is Facebook with its obvious social advantages and semantic/graph plans. Google is confusing though - on the one hand, they have the right investments and infrastructure but on the other they still appear search/data/click driven - even imperiously so ("...designers...need to learn how to adapt their intuition"). They also have issues with innovation.

The big search players generally recognise current search solutions do not meet the needs of the user but they don't have obvious solutions. There is an opportunity for innovation and for weaning people of this style of information seeking. There are smaller players that, if they cooperated now, could produce a more potent, compelling, flexible and lasting solution than any. Their biggest problem would be being creative about incentives for organisational involvement on the semantic side. Let’s hurry though - its hard going searching for things. We've got work to do now. We have questions.

Statistical thinking & the ability to read and write

2010-08-18T18:26:00.010+01:00

Statistics are hugely underused. Even when they are used, they are improperly used. Much of the bolstering of weak arguments, miscommunication and ad-hoc ideology creation performed by individuals and organisations from the industrial revolution onwards are down to poor statistical use.

Entire industries are devoted to generating statistics; duplicating monstrous amounts of effort and when they are referred to e.g. in sales pitches, organisational reports, building/product tolerances, war crimes tribunals, political and environmental manifestos, news articles and business plans – they are so heavily caveated as to make them insensible. This is by contrast to the relatively deterministic, transparent and auditable approach organisations use to produce BI to support their own business decisions. Coupling this BI with governmental statistics is often necessary for the best decision making support, so commercial BI is itself hamstrung by statistics.

There are two key reasons for this:

1) Statistics are hard to find. If you are looking for, for example, the number of people that work in London currently (a simple enough request that many service organisations would need to be aware of), you will find this close to impossible. The UK Office for National Statistics (ONS) does not have this on their main site. Nor does the Greater London Authority or the newly released UK government linked data site. After you have wasted maybe twenty minutes of your time, you will be reduced to searching for “how many people work in London” then trawling through answers others have given when that same question has been asked. You will receive answers but many will be by small organisations or individuals that do not quote their sources. In the worst case, you may not even find these – instead using an unofficial figure for the whole of the UK which you have had to factor down to make sense just for London. If you search hard enough you will find what you are looking for at an ONS micro-site (completely different URL to ONS) but this data is over five years old.

2) Statistics have a poor image. The blame for this, in part, may be attributed to the famous Disraeli quote – “Lies, damn lies and statistics” in which he set generations of professionals into thinking they were akin to a practicable yet modish Victorian politician by disregarding statistics and cocking-a-snoop at the establishment in favour of their own experience. Showing you are practicable with maverick tendencies while (in overtly disregarding information that may cast doubt on your decision-making) shoring yourself up from failure - are powerful incentives. By contrast, other famous statistical quotes have been forgotten (“Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write” [H.G Wells]). Short-sighted governmental data integration, hugely delayed/over-budget government data-centric projects such as the UK National Health Service’s Records System, confusion over keys statistics e.g. number of asylum seekers and high-profile data losses haven’t helped matters since.

There is also a bit of a myth that statistical interpretation is an art and that the general public can be confused – even sent entirely the wrong message by engaging in statistical understanding. Only statisticians can work with this data. Certainly there is this side to statistical analysis (basically anything involving probability, subsets, distributions and meta-statistics) but for the most part, both the general public and organisations are crying out for basic (the answer to one question without qualifiers e.g. where/if etc.) statistical information that is quite simply - on a web-site (we can all just about manage now thanks), produced by or sponsored by the Government (we need to have a basic trust level) with a creation date (we need to know if its old). If statistics are estimates, we need to know that and any proportions need to indicate the sample size. We need this since we are now sophisticated enough to know “8/10 owners said their cats preferred it” has less impact than if we are talking about a dataset of ten cats rather than 10K cats (we don’t really need this though since we’re capable of working out proportions ourselves).

We don’t want graphs since the scale can be manipulated. We don’t want averages since it is similarly open to abuse (mean, median or mode?). If we make a mistake and relate subsets incorrectly then the people that we are communicating to may identify this and that in itself becomes part of the informational mix (perhaps we were ill-prepared and they should treat everything else we say with care). We actually don’t need sophisticated Natural Language Processing (NLP), BI or Semantic Web techniques to do this. It would be nice if it were linked data but concentrate on sourcing it first. We really are not that bothered about accuracy either (since its unlikely we are budgeting or running up accounts on governmental statistics).

Mostly we are making decisions on this information and we are happy rounding to the nearest ten percent. Are we against further immigration? Is there enough footfall traffic to open a flower shop? Do renters prefer furnished or unfurnished properties in London? Which party has the record for the least taxation? What are the major industries for a given area? We just need all the governmental data to be gathered and kept current (on at least a yearly basis) on one site with a moderately well thought-out Query By Example (QBE)-based interface. That’s it.

Reading and writing have been fundamental human rights in developed countries for decades. Broadband Internet access is fast becoming one too. Surely we need to see access to consistent, underwritten government statistics in this vein too. Where other political parties dispute the figures, they should be able to launch an inquiry into them. Too many inquiries will themselves become a statistic – open to interpretation. It is absolutely in the interest of organisations and current affair-aware individuals.

Say hello, wave goodbye

2010-08-06T10:15:00.049+01:00

Google Wave is/was an interesting product. It is nothing less than an attempt to oust email as our primary communications medium and therein lays its story. It is basically Instant Messaging with two additional functions - the ability to automate a response so the user doesn't really see a difference between a human and a program (or Robot). It also supports mini-applications (or Gadgets) in a similar way to iGoogle and Facebook, blurring the lines between conversations and documents. Each individual exchange (or Wave) is logged and can be added to at anytime. It is better than email because it is real-time and richer exchanges can be made. It is worse than email because using this functionality is confusing for all but the tech savvy (having multiple Robots, Gadgets and humans all involved in the same Wave is rife with issues of ownership, progression and timing). Also the success of email is due to its ubiquity - everyone can use it. Hardly anyone knows what a - Wave is or what to do with it.

It was un-ceremoniously released to the general public a couple months ago (although some developers have been using it by invitation since its tech-celebrated debut last Summer) and pronounced dead by Google this week.

In Google's own words - "Wave has not seen the user adoption we would have liked". After less than fifty working days? For such a new and cart-upsetting product? With hardly any permeation of Wave concepts to the general public, next to no marketing and no specific commercial targeting? Of course! It was inevitable that this would be the outcome if an organisation were to review a key product launch after so short a time frame with so little support. This has to fall in the space of Google testing out new concepts/obtaining user feedback, with little fanfare (and so little possibility of failure) with a view to releasing the (inevitable) email replacement again at some point in the future.

Rich real-time communication supported by an easy/interactive interface to a computer (rather than a simple search dialogue) are most definitely the way forward. At the very least, collaborative programming environments benefit from this type of interface, as does trading and social networking proper. Google Wave uses the Extensible Messaging and Presence Protocol (XMPP) protocol which enables the most efficient peer-to-peer communication currently available. These needs are not going away and neither will the constituent parts of Google Wave (in some form).

Wave Robots are perhaps the most interesting component right now. In operation, they are a bit like a Turing Test or a Twitter-bot in that they facilitate a conversational and collaborative approach to establishing (and getting) what you want. Moving past the simple search/response model we have now, it is inevitable that there will need to be some interaction, some toing-and-froing and narrowing-down for something to understand (disambiguously) what it is that you actually want. It happens in real-life and so probably needs to be modelled in virtual-life.

Wave Robots are coded relatively straight-forwardly in Java using a Wave API. This element of Wave remains significantly undersold. As an example, I developed one to allow collaborative SPARQL queries to be made against any open linked data. A few working SPARQL queries have been uploaded to give you an idea.

Queries can be collaboratively changed in real-time and results can be sent out to a named Google Docs account as a spreadsheet. Once in Google Docs, there are several charting options available to make the data more accessible. Its like a hugely more powerful Google search (for those of a technical persuasion). You can add other Robots to the Wave to allow syntax highlighting of the code e.g. Kasyntaxy (although at the moment, this doesn't seem to specifically support SPARQL). As its all Robot based, its server-side and so you can use it on your mobile device - whatever that may be. Just add querytheweb@appspot.com to your Wave contacts to use. Type "cycle" to cycle between one of two endpoints. An endpoint is basically a SPARQL query engine. The two used are both generic meaning that they are not tied to a particular data set. This means your queries will need to use the FROM clause to identify which data you are querying (by URI). Type "help" to get a list of other options.

This Wave Robot took just a day to develop in Java and was deployed to Google App Engine. It is basic but even as it stands, it is still probably the best way to access and present open linked data currently available.

However long Google maintains/runs/supports Wave, its constituent parts will, at some point, be mainstream. The demise of Google Wave is not the demise of the email replacement concept. Get ahead of the game and develop some Wave Robots now. Get used to the concepts and the working environment Google has provided until the end of this year. Your work will be able to re-surface in some new product next year.

UPDATE: Interesting Scoble commentary on Google Wave ending here.

Starbucks, here’s how you get into the outsourcing business

2010-08-03T15:28:00.005+01:00

Hello Starbucks. You have made enormous success of the past fifteen years and have become an integral part of 21st century global cultural fabric. You have a store on the Great Wall of China, introduce new words to us like Yirgacheffe and are a bit like Viagra and The Simpsons (you aren’t technically the best but you’re easier to find and we have so much fun with you - we don’t care). We’d come to you eight-days-a-week if we could. You are to be applauded.

You have reached somewhat of an impasse though. You aren’t growing much; many of your stores are busy only at lunchtime and your brand doesn’t make us think – “that’s progressive!” anymore. Free Wifi/Foursquare deals, exclusive album sales and instant coffee will not get you those 40,000 stores you wanted a few years back. Your mantra of “A Starbucks on every corner” remains a good one. We know it’s tough out there but you just need to stick with your plan; maybe be a little bolder. Here’s what you do:

1) Recognise that you have to diversify. Your huge rent bill will surely eventually cripple you. You either need to dramatically cut costs (how? [Given your locations]), increase demand for coffee (how? [Given everyone drinks it anyway) or expand into new markets.
a. In 2008, the FM market was approximately $846BN, with approximately half ($426BN) apportioned to internal services meaning that the outsourced FM market in 2008 was worth around $420BN. It is surprisingly difficult to obtain free global branded coffee shop market sizes but for the UK at least, this is a $2.5BN market (2009). Let’s assume the UK is 5% of the global market (pretty standard), leaving a total market of $50BN. This means the FM market is roughly ten-times the size of the branded coffee-shop market.
2) Build more stores. Very roughly and taking the UK as a case in point; you have 750 stores and there are 30M people employed in the UK. With a few (contentious) assumptions, if you increased the number of stores tenfold (7,500), each store would need to accommodate just 200 people (1.5M/7,500). Obviously, it would need to take place over some years. A burgeoning senior citizen population, increased contract working and home-working will reduce the market, making the figure more manageable longer term.
a. Assume half actually work in an office (the rest in retail stores, hospitals, lumberjacks, machinists in plants, plumbers, nurses etc.). This takes the potential market down to 15M.
b. Assume half work for big name organisations that will want to maintain their own premises (taking it down to 7M - basically the SME market).
c. Assume half of those actually work in an office at any given time (the rest visiting clients, training, sick/vacation, travelling, WFH etc.) taking it down to 3M).
d. Assume you lose half the remaining people to other coffee-shops as the market is quite fragmented at the lower-end (taking it down to 1.5M).
3) Use your Starbucks card. At the moment this is used as a store card (arguably faster than paying otherwise). Put an RFID chip in and use it to track peoples employing organisations, access times and automatically bill the organisations according. You can hugely undercut existing FM services if you open-up this new revenue stream. You also expand your coffee market.
4) Build meeting rooms. Organisations need secure ad-hoc meeting rooms (HR, competitive, strategic discussions etc.). Let’s assume all new stores have one. These would need to be empty by default i.e. not having coffee drinkers in them and controlled by an online booking system. Let’s also put sophisticated video conferencing facilities in each one. Of course meetings are going to overrun and the people outside waiting for the next slot are going to have to either play nice/assertively claim their room but this happens in offices already. You might want to partner with others for larger, scheduled meetings.
5) Deploy IT Infrastructure. Cyber-cafes may be on the wane as cheap mobile devices rise but you would need to pop Internet terminals in your stores to mop up those without laptops at any given point. The shift to cloud-based computing means organisations won’t need development/file/application servers because they won’t have IT departments. Each store is also going to need a couple of wireless printer/scanner/copiers.
6) Go stealth. To avoid monstrously over-selling your brand, you are going to need to expand your stealth experiments on a wider scale. Focus individual stores on the areas they are in (creative/business/education etc.). Maybe change the decor to fit-in with local murals on the walls. It may be healthy to engender some competition between them. There would clearly need to be more variety in (interior and exterior) store design.
7) Forget the Baristas. Everyone knows this isn’t a skilled job. Stop pretending it is. It’s not like they spend years learning the correct Frappuchino for the chocolate Starbucks coin customers eat. They’re a bit like your Starbucks cards – over-engineered. Do give them training but make this in basic IT services in addition to working the coffee machine. They should need to know how to reboot the router, connect to it and any of the various wireless devices you have in your store from most portable devices, reset passwords, create accounts and escalate issues – that sort of thing. Ultimately, they’ll thank you for it. Future employers will place much more emphasis on IT service skills.
8) Culture shift slightly.
a. “Third-place”. This internal marketing needs to go. Yes – there’s a place for a safe haven, a "third place", that place outside of work and home where you know that you will be greeted with a smile and some respect. This is more than a coffee shop though. It is now a hackneyed term anyway. It was used at the Playstation 2 launch and is employed by countless gyms over the world. Is it really harder to create another market than get a good chunk of one (or both) of the existing ones?
b. Seat-saving. This needs to go. Someone cannot come in, sit down on one of your sofas and then “save” seats around them; dissuading potential users as their “friends are coming”. This prevents people from using stores for more than a quick coffee i.e. to work. Hot-desks are essential. It has to be first-come-first-served. Subtle advertising cues should be able to make this culturally frowned upon so it ceases to be an inhibitor.
c. Table Service. Your service isn’t great at lunchtimes. Queues can be large. People on laptops are dissuaded from leaving their laptop but they still want a coffee. Your new Trenta sizes may address this issue (slightly) but your smaller competitors offer table service for the same price.
d. Enhance security. You cannot have hoodies/Hells Angels/gypsies/beggars etc. associating with Senior Executives (can you?). You are likely going to need a security guard in most stores to gently dissuade them. Can’t they all do double-duty as Baristas too though? Security Barista? IT Barista? Table-service Barista? They can be more Pokémon than Borg.
e. Get out of food. You are not known for your food. Stay with chilled things that go well with hot coffee e.g. muffins, cakes, chocolates, biscotti etc. The hot breakfast sandwiches, wraps and salads all need to go. They take too long, are odorous, other brands do them better, people don’t want them in their office and will also want a break from you (their workplace) to go get them anyway. Get a food partner if you must and link it to your Starbucks card. We can work it out.

You have the cultural and economic reach to become our workplace. This isn’t something you can do quickly. It’s a goal over the next fifteen years. You can choose to move up from being an escape to being a destination. That journey will mean you need to take a leap and recognise that you’re big enough right now and that you’ll have missed service elements along the way (but that others will fill-in and contribute to the new eco-system). It may also mean you concentrate on the back-office, lose a bit of your élan/put your brand on the back-burner and cancel that order for corporation T-shirts.

A little like those faceless East-India type holding companies that keep going for hundreds of years. That’s OK though. You have certainly let your face grow long of late but to paraphrase The Beatles further; you are the coffee man. They are the coffee men. You are the water-cooler.

Why you should outsource your office to Starbucks

2010-08-03T15:25:00.004+01:00

Office rules have relaxed a lot over the last ten years. Many office workers are now working from home regularly and when we are in the office; we are all comfortable having water-cooler discussions in a coffee-shop and taking our laptop in to work. Even Government recognises the benefit. The days of rigorously putting in a nine to five every day in a shirt and tie are all but gone. Why not go further and – do most or all of our work there; dispensing with the need for our current physical offices? Yes - it’s a little out-there but idly run with it for a while.

Yes, we are talking facilities management (FM) but – perhaps a narrower definition of it with all value-add services being done by partners. Could it actually be done? There is a definite market for professional, ad-hoc and casual working environments e.g. The Hub.

What would the benefits be?

1) Cost savings. FM has become an important industry/profession; responsible for approximately 5% of GDP in the most developed countries. After HR, FM accounts for an organisation’s greatest expenditure with 20% of total organisational expenditure. Significantly reducing this figure would allow smaller organisations to compete and stimulate growth.
2) High street utilization. High streets (or Main streets in the US) have lots of empty shop fronts. They could be re-commissioned (as Starbucks’) bringing much welcome new life and trade opportunities. This would add to our existing spaces - our homes (city/suburban), our work (downtown/business park) and the mall; maintaining a space distinct from these – a common (high street/main street). It’s ultimately about variety, possibilities, culture and escape.
3) Cultural cross-pollination. Organisations generally benefit from finding out about other organisation’s ideas/challenges. Some will consider this a drawback since they fear dilution of organisational “special sauce”. But what is this really? - People, process and IP – assets that aren’t going anywhere. It’s just the physical environment shifting.
4) Flexibility. Working in the same office is dull. Work in whichever Starbucks-office you like.

What would drawbacks be?

1) Noise. In the long-term, once Starbucks is recognised as more than a coffee shop, people will act differently there and noise will become no more of an issue than it currently is in offices. Headsets will help in the short-term.
2) Loss of status and image. If you are Swiss Re and you have spent $1BN on your gherkin building, you care about prestige, internal branding and providing a great environment for your workers. If there is a great environment elsewhere though – for free - is prestige and internal branding worth it? It is for the for big name organisations, the multi-nationals. For everyone else - no.
3) No physical storage. People keep things (coats, umbrellas etc.) at their place of work; they will need to keep them elsewhere. A small amount of lockers could be made available. HR and accounting would need to digitize all physical files. Is this a real issue? It shouldn’t be. For every filing cabinet, there’s a good reason why it should be in the cloud.
4) Team accommodation. If you are working solo or there are just a few of you then you can usually find seats together. There would be a problem accommodating project based teams (3-10) people in this way. This on-demand physical availability of teams is the biggest drawback to office-Starbucks. There would need to be a responsive real-time system that is capable of identifying empty seats together and placing a reservation on them.

Next up there will be an open letter to Starbucks asking them to consider our audacious plan.

Semantic Web - Part 2 (Where is it?)

2010-06-13T23:28:00.045+01:00

The enterprise in general has barely given these technologies a second thought to-date and the consumer has little idea about them (short of a vague idea that Web 3.0 will make the Web more intelligent). The concept of storing more data in a graph/RDF format remains disputed. MSFT, for example (as a populist bridge between the two), could be said to have a less-than-enthusiastic approach (all the main APIs are in Java with .NET versions managed only by enthusiasts). Few MSFT products use RDF internally (Media Management). None (including SQL) use OWL/SPARQL. Google have their recent Rich Snippets initiative but their Chart API currently only works with spreadsheets (rather than RDF). Facebook are actively pursuing the graph format as its on a secret logo inside Mark Zuckerberg's hoodie. Twitter have recently announced annotations - a way to add meta-data to tweets (which could be used semantically in future). Some breaking sites use RDF e.g. Glue, Drupal and Tripit but there are no killer apps yet.

The world awaits an application that inherently makes a tool of the Semantic Web. This will likely be focussed around disambiguation since wholesale data integration is a tougher nut to crack.

Reasons follow (descending order of importance to-date):

1) Openness. There are few clear reasons for the enterprise (the people who manage the vast majority of data) to be more open with data. Especially their raw data (as opposed to their massaged/reporting data). Government has a remit of transparency so they have more data in this format.
2) Federation. Of business processes. There are a host of facts (and rule-based logic linking them together) required to make the above scenario (and anything like it) function; all working over several different organisations with different remits. Each taking revenue share. Building federated applications; using other peoples data are also rife with issues of SLA and legalities.
3) Performance. Storing data as facts (triple or graph) results, in most cases, in a poor performing back-end. Relational databases are widely recognised as most efficient for most scenarios and therefore what most organisations use. Tests indicate triple queries are on average 2-20 times slower than an equivalent relational query. This alone instantly negates OWL/RDF and SPARQL for widespread use in the enterprise. There are also huge uncertainties over how distributed SPARQL queries and reasoners will work in practice - at Internet scale.
4) Ambiguity. Many believe that the real world is just too complex to be mapped:
a. It is just not able to be boxed and named in terms of categorisation and relationship (or ontology as it is called). This is essentially the same as the schema or Entity Relationship Diagram (ERD) for relational databases.
b. Linking ontologies together accurately (differing owners, world-views, drivers) is impracticable at Internet scale. Related to this is the almost philosophic issue around the use of the URI to identify a resource. It is hard to give a crisp definition of what 'representation' (of a resource) means in order to justify an assertion e.g. an image of Texas does not 'represent' Texas.
c. The recombination of facts necessary for inference are too simplistic e.g. the addition of - drunks (subject) frequent (predicate) bars (object) to our scenario might allow our agent to infer that the CEO is therefore a drunk. This may or may not be true but given known facts, it is a rash statement to make (especially if you are considering networking with him). You might not even know that the agent considers the CEO to be a drunk (it will just be one of the many factors that it uses to provide actions for you). This makes the situation much worse since bad decisions are then difficult to debug/improve.
5) Validation. Maintaining graph data is less structured than a relational format. It makes validating data integrity challenging. Data quality is a huge issue for the enterprise. Many CIOs will look to the ceiling when you ask them about their data quality and meta-data (needed by the Semantic Web to function) takes a back step in priority. Existing relational databases have much better tools and processes.
6) Semantics. The Semantic Web community have not helped themselves by casually using unclear terms, bolting on reasoning/inference to the Semantic Web definition and generally setting-up camps about well - semantics. Running parallel to Semantic Web development has been Natural Language Processing (NLP) development which, by contrast, has a clearer mission statement, can achieve some of the same goals and is actually more about human language semantics than the Semantic Web.

The first two above are related. There is simply an absence of reason for the enterprise to be more open with its data and link its transactions and business processes with other enterprises right now. However, it is fair to say there is a small revolution going on in respect of consumer personal openness or privacy. Consumers are seriously considering the benefits of publishing their purchases and sharing their screens for mass consumption. At the same time, Facebook is criticized for being too open and free-wheeling with personal data. What this tells us is that – the consumer wants to be in control of their data. This desire to control their data, all of it, all the time is essentially a form of self-actualisation; managing their on-line presence – facts, hopes, desires, history and image. Self-actualisation as popularised by Maslow is an established end-game for individuals and society in general. This will happen initially due to consumer demand. The enterprise will then be dragged into this process by more powerful forces of expectation and socialization (Web 2.0) – market forces. They will have little choice but to serve up their data and integrate with others. This could happen surprisingly quickly – within a year: Six-months of observed consumer demand, three-months to get an enterprise pilot going and another three for everyone to assess their new situation (slightly faster than Web 2.0 enterprise adoption) and bang – Web 3.0 is a competitive differentiator.

It is tempting to suggest that performance will become a non-issue due to better future technology (and network is more of a bottleneck than database for many applications) but the reality is the information explosion is so potent that it will broadly compete with Moore’s law to keep performance in play. Poor performance simply blocks enterprise adoption right now. There are creative solutions mooted e.g. swarm based reasoners. Similar high concept solutions are likely necessary since the performance gap is huge.

Ambiguity removal is essentially what the Semantic Web is all about. Objections around whether the world can be generally mapped or not are valid concerns; they can certainly be well illustrated by examples showing literally stupid inferences that can be made. Such examples are on miniscule datasets though. With the real live Internet (as with all complex systems) - outliers even-out at scale.

It is easy to imagine a massively connected system refining ontologies in real time based on user input (although certainly people would need some carrot to do this). It is less easy to imagine this happening for reasoning but a great algorithm could well emerge that will simply not be questioned (or even understood by the majority of the population) as it just works most of the time and anyway; when it doesn’t, you can’t really tell e.g. PageRank.

Tool interest will be stimulated once the inhibitors above start to be addressed. Tangential to this, a general consumer focus on their personal data will mean organisations are compelled to improve their customer data in order to meet new consumer driven activity (personal profile, reputation management).

The semantics point is ultimately an artefact of Semantic Web technologies not being commercialized. Once the enterprise gets involved, efficiency will drive out any hint of academic ownership. NLP is actually complementary to the Semantic Web since it is a front-end to allow access to the semantic data.

None of the inhibitors above are outright Semantic Web deal breakers. It is not conceptually flawed. If RDF/OWL or SPARQL have implementation requirements (Grouping, BI/aggregation, lineage etc.), they can change; they are still evolving. Collectively though, the inhibitors are assuredly deal breakers for widespread adoption in the five to ten year range. Before that time, as it seeps gradually into collective consciousness, performance and reasoning visibility will likely become the main inhibitors. It is not an all-or-nothing idea though. A little semantics goes a long way as they say. Next post will explore how elements of the Semantic Web can be utilised now.

Semantic Web - Part 1 (What is it?)

2010-06-12T18:37:00.051+01:00

The science fiction future that futurologists love predicting is some way away because meaningful data is currently not well integrated. You cannot have anything like the rampant “Imagine...” scenarios (typically using “Agents” and RFIDs) that futurologists speak to on a wide scale until (in terms of data) a Single Version of Truth (SVOT) is agreed upon for the data we are looking at (or disambiguation as it is called) and it is integrated wholesale.

Many of these scenarios will happen since they portrait a more efficient, opportunity filled or simply fun lifestyle; one enabled by information. Someone in the future will find a way to monetize it (maybe in a Minority Report/advertising kind of way) because market forces always apply.

Let us try one out in the next paragraph to illustrate:

Imagine you are in a bar. A distributed agent *knows* your location through your mobile. It also *knows* that the CEO of a large company in your industry is in the same bar and that that company is hiring for a role one level above your current one. Of course, it *knows* that you have been in your current role for a while. It does not *know* whether the CEO has any people with him at the moment (or is waiting for them) as he has restricted this information by his personal privacy settings. The agent suggests though (by texting or calling you) that it is worth you going up to the CEO and introducing yourself but not before it has already emailed him your resume and informed you that his drink of choice is a dry Martini.

This scenario is by turns – fantastically cool, moderately disturbing, highly efficient, opportunistically enabling and culturally changing. Some version of it will likely happen. What it is not is - artificially intelligent. Aside from the technology existing to do it all right now (mine GPS location data in real-time to determine matches against pre-set scenarios e.g. connecting people for jobs then checking social encounter rules and privacy settings) and the rule-based logic involved is straightforward (if people in same location and same industry and networking opportunity exists then...) the fact that all data required to fulfil this scenario is in different formats and in any case – secured (since there is little reason for the owners to share) will prevent our vista from happening. A lesser inhibitor is the rule-based logic - straightforward certainly; but the types of scenarios we are talking about require a lot of rules and it is unclear who will maintain them.

Future agent does not *know* anything, it has simply traversed the Internet (using look-up tables or schemas) to find an SVOT (because data is well integrated your location is stored disambiguously) and acted upon them as directed by predefined rule-based logic. Basically it has acted like any program around today (but on better data).

To fully integrate data you need senders and receivers to agree on a standard for both storage and communication (storing data in one format and communicating it in another defeats the purpose of data integration). This standard needs to be simple (since we also want to exchange data with mobile and embedded devices and generally want rapid and broad diffusion of the format) and not restrict others from building niche and more complex standards on top. The simplest standard is - a fact (Sales, personal etc.). Facts – of course, are condensed down to, something that the fact is about (the subject), something about the subject (the predicate) and something that is related to the subject through the predicate (the object). Examples are:

You (subject) located in (object) bar (predicate)
bar (subject) place of (object) socializing (predicate)

You cannot decompose facts any more than this otherwise; they would not tell us anything. It is conceptually akin to storing data at a physical level as 0s and 1s. Any type of information in the world can ultimately be stored as a list of linked facts in this way.

I (subject) purchase (predicate) Jack Daniels (object)

What is missing here is (you might ask) the timestamp and location; don’t we have to add them as columns four and five? Surely our future scenario needs that information? No – the idea is that we stick with the simple triple representation and it becomes:

I (subject) purchase (predicate) Jack Daniels (object)
My purchase (subject) was timed at (predicate) 1430HRS (object)
My purchase (subject) was located at (predicate) O'Malley's bar

While it is certainly true that much rule-based logic will always be required to fulfil the type of scenarios above, the amount of it is significantly reduced by the ability to make inferences using facts. Consider our facts:

You (subject) located in (object) bar (predicate)

This fact is automatically generated by your phone broadcasting its GPS location and it being matched to its location as commercial premises and finally cross-referenced against its business type.

bar (subject) place of (object) socializing (predicate)

This is a core fact that never changes and was created by some internationally accepted community effort. Because we have a like-term (bar), we can now infer that:

You (subject) are currently (object) socializing (predicate).

You have not specifically told anyone that you are socializing. It has not been encoded. Indeed it may be the middle of the afternoon so, in lieu of further information anyone may have otherwise assumed you were at work. We could have built-in rule-based logic to achieve the same result (if you are in a bar then you are socialising) but we have been saved the trouble by inference. Performance has been maintained as inference was in memory. This type of logic – syllogism has been around since at least Ancient Greeks. The implicit knowledge that both you and the CEO are physically in the same informal situation at the same time allows an opportunistic suggestion to be made; opening and closing a loop in real-time without having written a rule for it.

If everyone used the fact format (and its inferencing - managed by reasoners) for data storage and communication then we should all be able to resign from our jobs and hang-out in bars; secure in the knowledge that we have reached a technological plateau and an agent will at some point fix us up with a new role. Imagine.

The existing Internet is still very page focussed. You go to a job search site and have to trawl through pages of job descriptions, applying your own experience to decide which ones are interesting e.g. is a “Sales Executive” the same as a “Business Development Executive”? Does that term differ by industry category? If so, should I follow these links instead? You have to do a lot of work to find the things you want. So much so that you either give-up or end up with things that you don’t want. Using the fact format at Internet scale with disambiguation removes the necessity for humans to contextualise the data and so enables machines to better process it; which in-turn leads to more pervasive automation and those Imagine scenarios. This is what is meant by the Semantic Web (Web 3.0/SemWeb).

Is the Semantic Web inevitable? Opinion is divided to say the least. The Imagine scenarios (or variations of them) are inevitable. They absolutely require disambiguation and wholesale data integration. This in-turn has to necessitate a standard for storage and communication of facts. The Semantic Web is an attempt (the only one in town) to deliver that fact standard. Inferencing must be considered an optional component of the Semantic Web. It may uncover previously unknown information or simply be required to make it work at scale. It may require too much effort to be practicable for many due to its current reliance on an Open World Assumption (OWA).

The core promise of the Semantic Web is - disambiguation and wholesale data integration (Linked Data). It is the primary enabler for data-mashups. There are certain parallels with the early days of the Object Orientated Programming (OOP) movement. The Semantic Web is inevitable but it won't be called such. It will still be the Web.

There is an established fact format right now – RDF (Resource Descriptor Framework). Much of the supporting systems are also in place e.g. query and descriptor languages (SPARQL and OWL respectively). They have been quietly developed over the last eight years or so and all focus around the core premise of the simple fact (or triple as it is known [subject/predicate/object]). Next post will explore why we have yet to see widespread adoption of these technologies.

UPDATE: Great source of further reading links here.

Beware the IDEs? Not so much

2010-06-05T23:05:00.011+01:00

Whether to run an Integrated Development Environment (IDE) in a browser or not can be a surprisingly emotive subject. The majority of developers, if they do not already have it, want a well appointed workstation running Visual Studio (for .NET), Eclipse (for Java), Dreamweaver (for JavaScript/HTML/CSS) or similar. They baulk at the idea of running a browser-based IDE despite building browser-based applications and clear advantages to working in this way. It is worth logically and dispassionately looking at the situation.

Key advantages are:

1) Portability. Developers can work from anywhere with a web connection. Does this really happen? Offshore developers will typically have a work desktop and maybe a personal laptop. They may also come onshore for a short period and use a client desktop. If they work through an outsourcer/consultancy they may have another. This counts but it is a Dropbox like aspect of portability. Its main advantage is in supporting those lifestyle situations; when you were not scheduled to develop but – thinking about it - you can; either to get ahead or to react to real time issues – even when you are on vacation, travelling or visiting friends. There are also humanitarian reasons for being able to learn a trade and contribute without having to own even a $100 laptop but let us save that for a later post.
2) Collaboration. Developers can let others debug their code by sharing it via unique URL. Anyone navigating to that link will receive a separate, fully modifiable and executable version of the code. That means no API version inconsistencies come compile time. Real-time collaborative coding is also easier.
3) Efficiency. Hours and on larger projects – days are wasted in setting-up workstations and supporting environments e.g. source control/configuration management at the start of each project. Even if it has been done several times before, at-some-point it will fail because there are just too many variables on a desktop. This just goes away.
4) Cost. Older workstations can be used since compilation and anything else heavy is performed on the server. Cost also improved by collaboration and efficiency (2 and 3 above).

Key disadvantages are:

1) Usability. The browser is not perceived as being rich enough to accommodate a responsive editor, allow class/library management and debug. It is also impracticable to design and test a GUI due to the greater drag/drop precision required.
2) Connectivity. You need to be connected to the Internet in order to run your IDE and therefore develop. This limits your portability (1 above).

There are other points on both sides but above are the key ones. Let us hold those two disadvantages up to the tiniest bit of scrutiny:

1) Usability. Large text file editing in a responsive (next to no latency) way (the main criticism) can now be achieved using HTML5 Canvas/JavaScript. See Bespin and also Kodingen (extending Bespin and integrating with other services) for examples of this using Python/PHP/ROR. See CodeRun for an example of full-on code management using .NET/JavaScript. They are both free, quick and have clean efficient interfaces. Bespin is more of a work-in-progress and does not yet support all browsers though. GUI design is admittedly more of an issue right now but:
a. People already successfully use graphical editors in a browser e.g. Splashup, SUMO Paint and the recently Google acquired Picnick.
b. HTML5 adoption is affording more options here.
c. In both consumer and enterprise spaces, we are moving toward a widget-based UX making designing GUIs from scratch less common.
2) Connectivity. By this - offline development is meant i.e. enabling those circumstances where the developer is using a laptop (since if they were using a desktop surely it would be connected to a network?) and they are in an area of no Wi-Fi coverage (since otherwise they would have network access?). Granted this is a situation that occurs but consider further that this also means:
a. There are no collaboration or research possibilities available (no IM/no Google). If you get stuck when developing or need to clarify a technical point – you’re on your own.
b. You need a single professional and automated solution that synchronises all code, images, configuration files, media (that has previously been unit and integration tested and checked in) and also synchronises test data and potentially business rules (since it is good practice to keep these out of code). Either it will synchronise actual data/business rules in which case you need high grade encryption on your laptop (as you are likely using customer data) or your solution needs to de-sensitize the data/business rules somehow (and you need to have agreed this process with any customer). What kind of developers will be happy with these two restrictions? Only sole developers working on their own project.

Staying with the logical analysis of this, there are four decent reasons in favour of widespread use of browser-based IDEs and two against. There is also enough mitigation to mostly address the two negatives. There is clearly a significant net gain to be made. Side points around “developers won’t stand for it”, “Development is an art (it’s not!)” or “you just do not understand” etc. are emotional and really do not have a place in the decision. It is understandable (in a carpenter cherishing his chisel kind of way) but this noise is a real contributory factor as to why browser-based IDE have not made more of an impact to-date.

When their new OS comes out later this year, are Google really going to say – buy Chrome laptops, they can do everything your regular laptop can – unless you are developing? Given their long developer-friendliness; this would appear a peculiar move. Unless of course it is precisely because they are developer-friendly; that they will pander to populist developer belief and treat them as artisans needing powerful magical workstations? If they do this though – they risk confusing consumers and certainly the non-developing IT community as to their strategy at a time when they are already perplexed by what is happening with Chrome OS, Chrome and Android as a run-time environment.

Google have a new programming language – Go which currently needs OSX or Linux. Like the majority of languages today, it is C-based. It has been out for nearly a year but has not received a great deal of press. It will need a differentiator other than speed to compete (how many web applications really have a processor bottleneck these days?). Surely there is an opportunity to build a browser-based IDE for Go and enable a new generation of more casual (but also more open) developers around the world?

Where did those mash-up tools go?

2010-06-04T22:15:00.018+01:00

Just eighteen months ago, mash-up tools were big. The promise of building applications quickly, with minimal development and with context directly reflected within the application (as they are made by SMEs rather than IT resources) remains appealing. They looked to be the perfect tool for civic activists and knowledge workers alike. They were high in Garner’s top ten technologies to watch. The enterprise was starting to take them seriously as a mechanism to reduce crippling data integration challenges and consumers - bored on a diet of pushing links - thought they would be fun and/or a showcase for themselves in much the same way as blogs have been.

Now, the wind has shifted and MSFT’s Popfly and Google’s Mash-up Editor are both gone. Other niche vendors e.g. Sprout Builder similarly disappeared. Of the big players, Intel Mash-up Maker and Yahoo Pipes continue. All of them attempt/have attempted to straddle the void between consumer and enterprise spaces. This is an important distinction since – mash-ups even within the enterprise rely upon an ad-hoc passionate approach rather than formal development. They are typically built at home by passionate non-programmers who want to invest time in a single non-niche tool so whatever they learn is portable (work and social/other organisations etc). Even more so than blogging (which also uses one tool across both domains) a single tool is required - as more learning investment is required. With the exception of SAP’s Visual Composer (if you are an SAP shop), none of the tools mentioned have been particularly successful in either space let alone both. Why is this?

1) No UX standards. For both enterprise and consumer, there are no standard for widgets (or gadgets or web-parts or whatever else you call discrete self-contained UX functions). There are standards for business cards (vCard) – why not widgets?
2) Slow linked open data adoption. More of a consumer inhibitor at the moment. Linked Data is a core component of the Semantic Web vision that uses a specific set of current technologies. It provides a way to readily mix-and-match data in a meaningful way and so is a key enabling technology for mash-ups. Sig.ma is a simple mash-up tool for RDF data. Unlike other data-based mashups which tend to be query-based, Sig.ma is search-based. You enter a search term, the search engine gets your data, you remove the bits that are not relevant and (if you like) re-publish the data again as RDF (or other formats). This is perhaps too simplistic for users right now but it is evolving and could become a potent research tool. Any mash-ups that rely upon open linked data (ideally the best data of all) suffer from a lack of it; although this is changing as Government initiatives in particular publish RDF data for transparency reasons.
3) Insular data integration. Although the various flavours (SOA/ETL/EII/EAI) have been core CIO agenda topics for over five years, they have been mainly confined to the particular enterprise itself; especially in the narrow form of web services and have been of limited success even there. Extranet take-up, where common data is shared between parties in the supply chain, has been leisurely and this is precisely where mash-ups are needed. Very few organisations treat meta-data with the same focus as data. This means, mash-ups have trouble vouching for data currency and lineage which detracts from user take-on. It is possible that the solution to data integration will be the Semantic Web and a greater openness of organisations to share data. If so, we will be waiting some years yet.
4) Industry standards. Have been slow to be adopted. A notable exception here is XBRL for common reporting.
5) SSO. This is not exactly an issue for consumer mash-ups; assuming you are using open linked data but it is still a huge inhibitor to data integration for many organisations.
6) Blogging comparison. Although superficially similar to blogging - mashing (if you are going to do it properly) requires a thorough data understanding and a lot more effort than committing stream-of-conscious thoughts before they float away into the ether (or linking to other peoples work). Blogging is simply an easier way to achieve microcelebrity and also - because the majority of posts are written in the first-person (I think...) they can be defended (if need be) by the simple statement - "These are my opinions". This is a segueway into a whole minefield of philosophy, politics and culture that is best left alone. People mainly do. Only a small proportion will directly challenge someone's written thoughts. Publishing a mashup however - where you are vouching for the legitimacy of the data opens you up to direct challenge (people may have provably better data) and so people resist it. Only when the number of single versions of truth in the world become smaller and more consolidated will this situation change.

Popfly showed early promise as a learning tool but never really got past being a Silverlight showcase. Its focus was on looking good (geo mash-ups and slick drag/drop) rather than data integration. It did not use RDF at all. MSFT have been slow to utilise semantic web approaches in general. Some of their media management technologies use RDF in the background but their main focus has been around semantic search through the Semantic Engine. This initiative uses recently acquired Powerset technologies and will be released through SQL Server. PowerPivot has been significantly downsized from its original Project Gemini remit which would have provided not just a potent reporting mash-up environment but the management and support processes and infrastructure to QA and promote the mash-ups throughout the enterprise. This latter point is a key inhibitor to mash-up growth in general.

There are still signs of life in the mash-up tool space. Dapper is advertising focused. NetVibes is portal focussed. Alchemy API takes a content management/annotation approach (similar to Intel Mash-up Maker). Birst takes an analytic portal approach. Jackbe looks interesting; it appears to take a sales analytics approach. Snaplogic is not exactly a mash-up tool but it certainly takes a non-technical approach to data integration. None of them really play in that sweet-spot between enterprise and consumer though.

The parallel economic downturn has influenced mash-up take-on. The enterprise essentially stopped unproven development and consumers have yet to be sold on the concept but let’s face it – the focus on one or two drop-downs for configuration and Google Maps didn’t help either. The future of mash-ups is secure because it is the future of building useful applications quickly by SMEs and that will always be desirable. Five back-end data sources (database, RSS etc.) linked to five middle-ware components (aggregation, integration etc.) and five front-end components (analytics, data entry etc.) generates 125 possible combinations of application straight off the bat. Adding tailoring through filtering, personalisation and general configuration takes in into the thousands. This simple logic guarantees a future at least in the enterprise.

Whether the name “mash-up” has been tainted by its recent hiatus and – like its raison d’etre will need to resurface as part of something new remains to be seen. We can be sure though that the next generation of mash-up activity needs to be three things in order to stick around; interactive, data focused and usable in both enterprise and consumer spaces.

We're all journalists. Can't we be designers too?

2010-05-09T02:17:00.005+01:00

Why do web sites still look so dull? Ten years ago when broadband adoption and the content management tools we take for granted today were evolving this was understandable. Now though, the opportunity for anyone, anywhere to knock together a free site with federated content, streaming media, real-time integration, lots of storage, payment options, embedded BI, bolt-on UX functions (gadgets/web-parts), graphically engaging templates and critically - keep it up-to-date all without writing any code (maybe CSS if you really get into it) does not afford the same excuses.

So why are - organisational sites and blogs are just as dull looking now as they were a decade ago? We have real-time integration and video but; from a graphical design perspective, they haven’t really moved on. Of course the focus always needs to be on content; form follows function after all but can we not just have a little graphical integrity on top?

Here are the main reasons why sites are dull:

1) Trend. There is still a Googly/Web 2.0 trend for basic, almost amateurish looking text with minimal graphics and an informal almost chummy way of addressing the consumer even for large organisations. This is partially a marketing approach to engendering trust in the consumer (you are dealing with a friendly colleague-kid/folksy dad/generally laid-back individual who does not just want to take your money - he has a cause). This trend extends to the name (something snappy/abstract; typically with an ‘r’ at the end) and certain marketing approaches. As with all trends, people will tire of them. Given that this one has been running for a good decade, we are due for a new one. There is an embryonic trend among career designers for overlaying text on web sites. Previously this was considered anathema due to usability reasons. It has been used for ages in print media but all you need to do there is read. When things become interactive, use of this technique needs to be handled with care. It can make sites more organic and stimulating though.

2) Tool. Existing free tools do not go far enough in supporting design. The Blogger Template Designer released earlier this year does a great job of allowing the user to customise fonts, backgrounds (and has a good quality variety of templates to start with). It is highly functional. It stops though (as with most tools in this space) at providing support for the creative process itself. In a sense it provides too much freedom and not enough creative support. It is not a stretch to imagine core graphic design principals – proximity (are you sure you want to put that there?), contrast (your pink on cyan scheme?), alignment and repetition being supported through a tool. Also, Google has a ready-to-go tool with “Find similar images” (potentially driven by Image Swirl that could help in building composite background images. The underutilized grid system could be implemented within a tool. At the very least, let’s have a way to get text anti-aliased.

3) Fear. Most people are not designers. Everyone is a critic on the Internet. People are worried that their design will be wrong, weird or somehow not good enough. These are the same worries people in the mid-nineties went through went they started blogging. Blogs have become complementary to the established newspaper industry. No-one (blogging in their first language at least) now feels threatened by comments criticising their use of semantics, grammar, structure etc. by journalists. Their enthusiasm more than makes up for any lack of storytelling narrative.

4) Advertising. It is straightforward for organisations and bloggers in particular to advertise through their sites. Over half of all bloggers do it. Typically, this entails embedding banners on a page in corporate colours/graphics/typeface (and so unable to be changed) which can break a design. There needs to be less restriction on what can be done to these banners (Photoshop etc.) so that they can be adapted to a particular design. This means more designers will eventually incorporate them. Bloggers should also consider whether the pennies they receive in advertising revenue is worth corrupting a design. If a blog starts getting serious hits (around 10K unique visitors/month) then by all means advertise but advertising is generally not compatible with graphical design right now.

This post has been about why sites are dull. They are by no means insurmountable reasons and a movement to amateur web design is to be welcomed. Why? – Because it will complement and ultimately improve the graphical design integrity of the Internet. Is this (non-tangible) result worth the effort? That is too large a question for the tail end of this post. This audience will be likely (fairly equally) split into those that maintain content is so much of the WWW-equation that it is barely even worth discussing presentation. Others will argue philosophically that life without art is impossible. Let us just leave with the knowledge that graphic design services are a $12BN/year industry in the US alone and attractive sites (like attractive people) will bring others back to them.

Music by a stream with clouds

2010-04-28T00:01:00.013+01:00

Consumers have an ample choice of online music services right now. The market is fragmented; each provides a slightly different set of offerings and outlays. Napster is the most established. A decade ago, it was innovative, subversive and popular. It then went dark for a couple years (while presumably it went through legal dialogue) and emerged as a competitive and commercial music rental/purchase service. It had cool features such as an embedded monthly music “magazine”, Windows XP Media Centre integration and a virtual monopoly for around five years in offering unlimited music rental for around $10/month. It should have been a second coming but an apparent inability to read the market caused it to slide from public consciousness; the magazine stopped after a few months, marketing was outmoded, DRM implementation was unwieldy, Media Centre integration disappeared for both Vista and Windows 7, barely anything was free and mobile, web-client and social networking integration were too little too late.

This opened up the market as smaller players gained footholds. Some concentrated on “radio” e.g. Pandora and Slacker. Several went for the rental market e.g. Spotify and Rhapsody. Others stayed with the download model e.g. iTunes and Amazon MP3. Across all of these (and others) there are varying combinations of payment/access methods/DRM/integration etc.

It is clear that streaming and downloading will both remain as different media access methods. Apple hardware owners will always use iTunes (and latterly Lala [US only at moment]) and the remainder of the download music service market looks to be being mopped up well by Amazon MP3.

The first service that can make a sustainable business out of free (and DRM-free) global audio streaming access in a browser (not just US like Pandora and Slacker or Europe like Spotify [plus Spotify does not currently provide web access]) with service continuity and payment options that include - Media Centre integration, mobile and offline support will have a relatively clear-run for the next five years or so. Aside from essential playlist functionality, adverts (or the lack of them), HD/loss-less quality, social networking features, upload capabilities and music discovery/search features are not mainstream deciding factors. Online music is mostly commoditized in terms of need. Consumers want a service that works, is accessible anywhere and is (at an entry level) free. Of available services, Spotify appears to have the best service continuity - being the fastest to start tracks (faster even than Windows Media Player) and never dropping out.

Of those available - the closest to this ideal is Grooveshark. Although to ensure its future, it still needs to improve service continuity, produce a Media Centre client and ideally - deliver the mobile experience without requiring an app (and a bit faster). Also, it is worth noting that the perceived legality of Grooveshark is topical; dissuading some from investing time in it. It simply seems too good to be true that right now; anyone - anywhere, on any PC or console (including PS3) browser can play pretty much anything (without audio adverts) - all day, without even creating an account. Further, if you do elect to create an account (requiring the barest minimum of details), the site will help you send links to people to share tracks or embed a play button in your blog (all without them creating an account). You can even post any streaming track to Facebook. Why isn't everyone using it? It’s great now. It will be fantastic if they can make their business model work long-term.

Given Google’s general cloud/consumer centric approach and the fact that they have recently released a video streaming solution, it appears incongruous that they do not possess an audio streaming solution. Yes - they came in late last year with search augmentation where samples are streamed and then directed to purchases from partners but Google does not own these organisations which means they will not be well integrated with the rest of the range (tags, social, semantic web etc.). They also suffer from comparison with the ideal above.

People can go to a friend’s house/cyber-cafe, use their computer to watch a film through YouTube (US only at moment) then access all of their pictures and documents through Picasa and Google Docs respectively; but they cannot play their music (using a Google product) while they are doing it? Shouldn’t Google buy Grooveshark and mix it up with everyone else?

Seraphis to Spreadsheets

2010-04-25T14:19:00.002+01:00

Offshore development is a trade-off of cost and skills against communication. It will always be this way until, of course, demand exceeds supply and the only reason organisations consider offshore is for premium skills. Communication will likely always be an issue; as it is when broaching any two cultures, time-zones and work ethics. It is however unhelpful to blanket term – communication.

The issue with offshore communication is communicating project goals (and also progress against them as this is the only real way to confirm understanding). Operational (day-to-day/immediate communication) is generally OK (and much of the reason why non-French speakers can order food in Paris and be understood). Even complex operational communication can be understood offshore. The problem is in appreciating (and buying into) the end-game or goal and that means keeping it simple.

Ancient Greeks understood this when they simply bolted on Greek Gods to Egyptian religion (creating The Cult of Seraphis) as a goal-bridge to help them manage Egypt.

The simplest method of maintaining project tasks is to place them into a single spreadsheet (easy comparison between developers) and to establish a daily process for recording developer feedback e.g. proportion of tasks they have worked on the previous day/changes to effort estimates based on greater exposure. We will certainly want a way of “parking” tasks for a while (if one of their pre-requisites is not ready). Next, we will want a system generated “estimate” for when the task will be completed (based upon daily progress). This will be necessarily different to both the original estimate and the ongoing developer estimate. Finally, we want web-enablement. That is it; the minimum required to capture/communicate developer progress against common goals/estimate when code will be complete.

The process of being code-complete further refines the goal (coding is usually not the bulk of a project). Developers can intersect actual communication (bugs, “please make it do this...” etc.) with code to obtain an accurate representation of what is really required. Further communications – testing, QA/UAT, environment support etc. is easier.

A reusable example of the spreadsheet we have just described is here. Cell comments aid interpretation. Just apportion 1.0 of a day over the tasks for each developer and do it every day. Simple spreadsheets like this have genuinely helped manage complex offshore development projects. Project servers, operational BI, timesheet integration etc. add complexity and detract from the “what/who/when” simplicity required for communicating goals.

MOSS Internet Site - Security

2010-04-19T18:32:00.008+01:00

Building a corporate Internet site using MOSS has real security implications. Access from the Internet (for the users) and from within the organisation (Intranet) must be considered in terms of both authentication and authorisation. Although a sound solution, MOSS is not well documented here and an implementation typically needs to be designed. Following is a suggested solution template for organisations to use when attempting this. It should resonate with the majority of MSFT-shop organisations moving to a MOSS Internet site solution.

The solution supports two basic user categories:

1) Internal users. Employees accessing via the Intranet. Typically content authors - in charge of maintaining solution content. It is recognized that a requirement for employees to maintain content via the Internet (from home?) may well exist. In this situation, they should connect to their Intranet using whatever connectivity solution is already in-situ (VPN?). Although it is certainly possible to allow employees to manage content directly over the Internet, this creates additional complexity, cost and security risk for perhaps the sole benefit of; being able to manage content from an “unofficial” desktop (without the connectivity software) e.g. in a cybercafé.
2) External users. Public users accessing via the Internet. Able to access content only. May be further categorized as either - Anonymous user (Read only/Able to access all public content) or Registered user (Read only/Able to access special content in addition to public content).

The solution supports two-way authentication; one to support internal users (NTLM) and the other to support external users via Forms Based Authentication (FBA). Internal users are authenticated using the SSO (Single Sign-On) feature provided by the LDAP directory services. An internal SSO process synchronises the Intranet AD with a De-Militarized Zone (DMZ) LDAP. In practice, this will mean that an internal user can log onto the Intranet and be pass-through authenticated to the DMZ i.e. they will not have to log in twice.

Synchronisation between the Intranet AD and DMZ LDAP is one-way. This means that the creation of new internal users will have to take place on the AD and not directly in production. Assuming your organisation does not need to use profile import for LDAP users (and also does not provide custom profile properties for its internal users), it will only need to cater for authentication of LDAP users. This is perhaps a minor risk, since it will only be possible to test this once deployed in Production.

Similar to traditional NTLM authentication mechanisms, the FBA application requires a centralized store for external user credentials (in this case - SQL database). Internal user credentials are stored in AD (on the Intranet) and in LDAP (within the DMZ). An internal SSO process synchronizes them. No authentication is required for anonymous external users as you would expect.

User information stored in the SQL database is created only by a single registration process. A registration page defines the user information and the related validation. Using FBA, users will remain authenticated until the user closes their browser or until authentication timeout occurs.

Moving onto authorisation; authorization policies are required for granting content access rights for differing user categories. As stated above two different “extended” web applications are defined for managing both internal and external users. Authorization policies are applied to those distinct “extended” applications addressing the user categories’ tasks.

In the “extended” web application defined for the NTLM authentication (used by internal users) two different user groups are defined in the site settings permission section - Site Administrator (users with full control access rights) and Content Authors (users with design control access rights). No Anonymous access is supported in this “extended” web applications instance. In the “extended” web application defined for the FBA authentication (used by external users) all users are defined as users with read access rights. Anonymous access is supported for a subset of content as appropriate.

A final note on domains. The AD within the DMZ is for service accounts only e.g. SQL or MOSS services. It is not possible to trust the DMZ AD and the organisation AD proper. The DMZ is also unconnected to any other organisational system. MOSS based content deployment into the DMZ is unsupported. These measures should address internal security concerns. There are plenty of other security hoops to jump through when creating a MOSS Internet site but the above should get you started.

Playing at Art

2010-04-18T23:44:00.015+01:00

With the flame-wars of whether video games are really art or not; over for several years (hopefully), we can perhaps move-on and look at methods to inject more artistic merit into videogames.

For those still unconvinced of its art status – the fact that videogames turnover as much money as film (appealing to your commercial side), an idle contemplation that society considers ceramics as art (appealing to your political side) and a casual play of Half Life, Psychonauts, Silent Hill, Bioshock, Grim Fandango, Myst, Okami, Impossible Creatures, Electroplankton, Heavy Rain and/or their sequels (appealing to your –well - artistic side) may cause you to reconsider.

Is art in video games desirable though? Well aside from the altruistic vision of it helping shift millions of young, single men out of darkened bedrooms and engaging in senses, emotions and the range of human experience (maybe even with girls); there is a commercial imperative. Many popular video games are simply too task-orientated (even Stakovian) and realistic to have much of a chance to hold artistic integrity e.g. Call of Duty, Splinter Cell, Total War series etc. Realism is still impressive in a video game but barely. In less than ten years – photo-realism will surely make real scenes/characters broadly indistinguishable from gaming ones. There will need to be new qualifiers of what makes a game interesting and different to others (in addition to playability of course).

Old games were weirder and more abstract due to technical necessity but the whole market was also less commercial. Individuals could straddle both coding and creative chasms and their vision alone (much akin to a film director) could carry a game along most of the development life cycle; creating crazy, organic, beautiful, engaging worlds that stay with you. As a result, old games were more – artistic. Indie gaming still largely works like this but the big names will need to blend production values with indie approaches to compete; that or at least - chillax and get their freak on.

If an indie gamer can produce a free online video game that has a goal, consistent art direction, makes a clear existential point and puts a smile on your face - in less than a week, what can a Duke Nukem Forever–esque development life-cycle and budget do?

Here are some practicable suggestions for making mainstream video games more artistic right now:

1) Reuse existing world concepts. There is very little truly new in art. Almost every piece of art could be argued to be (at least in part) a rehash of a previous piece. This is fine. One of the key reasons why people visit art galleries is because they also like history; they want to associate the lineage of ideas over time. They want to learn something new as well as simply see something new. Homage is rife in cinema and is completely relevant. This relevancy, together with fusion, synergy, consistency and challenging people with the vernacular of the time is more important than a (probably futile) attempt to come-up with something that no-one in history has ever considered before. Portal looks to be influenced by 2001, the indie game above looks to be influenced by French New Wave (which in turn was influenced by Film Noir); there are other examples. Instead of another dystopian sci-fi setting why not (re)create ancient Alexandria in 3D; including the Pharos Lighthouse and library then overlay it with modern graffiti/technology to create a hybrid/alternate world that – although new will also appeal to historians/art-lovers and gamers alike. There are plenty of street maps for ancient Alexandria on the web and no-one has ever created an FPS virtual world of it before. The whole thing could be rendered in a Banksy/Frank Auerbach style (linked through the natural monochrome ethic of the artists). There are a host of mathematical/astronomical and philosophical precedents in that location to hook an emergent storyline into. It would be new. It would be art.

2) Be creative around the medium. There is much more scope in terms of the medium for creativity than there is in building your gaming world. This is because a good deal of the infrastructure surrounding gaming hardware e.g. webcams, email, broadband, downloadable content, mobile, 3D, touch-screen, GPS, micro-payment, IM, photo-realistic graphics, accelerometers, tweets has only been in widespread use for a dozen or so years. Some games have attempted to capitalize here e.g. Little Big Planet uses the PS3 Eye camera for level design and Grand Turismo will purportedly use it for head-tracking. There was a point-and-click game from a few years back that sent actual emails from game characters to allow the plot to enfold in real-time. These games are a fraction of the market though. Nintendo have led the way here for the last few years but there is so much more creative scope; there is barely any integration between Wii and DS consoles for example.
3) Decorate the corridor at least. There is a staple in FPS video games levels of - the corridor. There is no real issue is using this device but surely they were born to show-off graffiti, framed prints/paintings or imaginative entrances/exits etc? Graffiti in particular is easy to incorporate. There is a certain motivation associated with it that perhaps many designers will not have also they are maybe weary of reproducing existing graffiti due to perceived copyright reasons but – take it. If later some street artist complains that you are infringing their copyright – pay them. The designer will always have the fact that graffiti in public spaces is illegal on their side (unless the Government has set-up a “graffiti wall”). Anyway, it is a better way of commercializing and improving graffiti quality (Biasquiat-izing it!) than the naive suggestion of sticking QR codes (linking to a site to allow prints to be purchased) onto graffiti.

4) Increase cooperative options. With the notable exception of the impressive Left4Dead series, cooperative games have declined in popularity. Gauntlet was once the most popular game in the arcades. Games designers have chiefly concerned themselves with death matches and single player experiences for the last twenty years. Yes - MMORPGs and social gaming e.g. WOW/Everquest/Mafia Wars and others have become popular over the same time but they are very - trading, chatting, building and/or fighting focused. Also the esteem these games afford Goblins and suchlike is arguably derivative and therefore not creative/art. It is tough to find cooperative PS3, Xbox or PC games. Cooperation, in theory, creates participation, unscripted interaction and a more open/fun approach. Removing (or sharing) competitive imperative should allow more focus on the environment and contemplating why we are there; mainstream artistic drivers.
5) Kill the main character. The concept of video game character was invented effectively as a vehicle to get around technical limitations e.g. “you are a bat trying to stop a ball (due to monochrome/block video output) in Pong”, “you are a Brooklyn-based plumber (due to sprite colour/pixel restrictions) in Super Mario Bros”, “you are a space soldier (due to all the corridors we plan to make you run around) in Doom”. Take this away and you remove the back story (an element of the art component certainly) but gaming back stories are invariably dull, childish or an afterthought anyway. You have the best back story – your own life to date. Video games that play off this, allowing you as much free-range as possible will create more possibilities for expression and that means more art.
6) Design for glitches. There is a sizeable sub-set of the video gaming community that seeks out glitches (Google “video game glitches” and get 11M+ hits). Glitches are well publicized and repeated when they are found. Ones that create hidden or bonus worlds (perhaps because you have been able to jump beyond a wall that the designer did not intend) are particularly popular. The downsides - screen freezes, save game corruption, lengthy waits should be minimized by creating a wrapper that the game runs in and (in real-time) elegantly returns the user to the game world. Glitches (basically software bugs) are always going to happen but unlike commercial software development, they create something new, something imperfect, something challenging/exciting, something able to be contemplated, explored and potentially even resolved; something very much like – art. The glitch as art is not limited to video games and is even becoming a form in its own right.
7) Allow objects to be uploaded (or at least changed) and persisted. This all adds to creating a richer world which others can incrementally build upon (see 1 above). Over time, we will then have something recognisably – newish affecting the game itself (new adventures, cultural touch-points or just new graphics). Farmville allows for simple pointillist graphics to be persisted by planting crops and Second Life supports complex scripting to allow anything to be built as a 3D model/photoshopped and uploaded. Some median between the two of these is probably appropriate for mainstream video games.

Universal enterprise UX – Part 7 (Browsing, finding & copying)

2010-04-12T22:48:00.002+01:00

Unless we implement a duplicate create button in the Type/Filter web-part, we would have to first select an existing Enquiry (or any other business data) in order to create a new one. Each form will have its own individual buttons but they should all contain the basic functionality indicated above ideally with the buttons placed in the same positions on the form.

When we see information on a form, it is likely that the user will want to browse (or drill-down) into the detail of that particular business entity e.g. an Enquiry. There are several options available to us to handle browse.

1) Consolidated View button. We would require this button in order to see more detail surrounding a particular piece of business data in a form. For example, if a particular note were displayed in a list-box of all notes for a particular Enquiry, clicking “View” would open up the Wizard Pane to give more details regarding that particular note without losing context with the Enquiry proper.

2) Multiple View buttons. This is as per the above except that each control capable of being drilled-down into has its own View button. The user would only be able to examine in detail the particulars of one business datum at a time on a form, so it makes sense to have just one (consolidated) view button rather than cluttering-up the form with one View button per-datum. The disadvantage of both of these approaches are that they disrupt standard hyperlink flow, for example; if the user sees an Order Part in a list of linked Order Parts, he has to select it and then select “View” in order to see it in detail. The advantage is that the linking process is simplified as the user doesn’t need to differentiate between the links i.e. the row containing the Order Part and the control i.e. the list-box containing all Order Parts.

3) Separate the hyperlink from the control. This would be required in order to both allow regular hyperlink operation but differentiate between this and the user merely wanting to change the value. If the user wanted to change the value in a control he would click the area of the control not covered by the hyperlink. If he wanted to instead drill-down into the value in the control, he would click the hyperlink instead. The main disadvantage is that the user needs to be aware of the difference and this is not typical UX behaviour. It also means that the user can drill-down into several values without losing focus upon a particular control. The main advantage is that all hyperlinks work in the regular manner.

Option 3 (Separate the hyperlink from the control) option is selected for our model; principally because this way means that we do not have to compromise on standard hyperlink operation.

In the case of the Enquiry example described above, the Wizard Pane did not have to be used to provide the detail view; although in the case of viewing the details of notes, which do not have a lot of information around them; this is recommended. An alternative approach would be to (upon View click) re-purpose the entire Browse/Next web-part to display the detailed information and provide a “Back” button towards the bottom of the form. In this way, much more screen real-estate can be used to display the detailed information; although this is at the expense of context i.e. Enquiry details will be temporarily hidden to accommodate the new details. This type of behaviour would be recommended for viewing the details of the principal business objects e.g. Enquiry, Task, Order, Order Part only.

The model uses existing Find and Browse UX components in order to find entities in forms, for example; finding a particular Contact from the Asset Filter (a Type/Filter UI component) to add to the Task Details form. This allows us to: Reduce development time; as additional find and browse capability is not required; Reduce testing efforts because the principal controls need only be tested once; Reduce training efforts as the user knows to always go to the Asset Browser to find assets (Customers, Classifications, Products and whatever else is deemed an asset in future) and finally - obtain data entry speed benefits as the user’s flow of operation is consistent. It also lends itself more readily to macro-automation.

Several options around implementing find functionality are available:

1) Accept loss of Type/Filter status. When highlighting a control on the form i.e. Contact details. We just go to the Asset Filter and choose the correct Contact Id to add to the form, clicking “Link” to add it to the requisite area on the form highlighted. Benefit: Don’t need a new button. Benefit: Can use existing one-way line of web-part communication i.e. from Type/Filter to Display/Update web-part. Drawback: Lose focus upon whatever was previously highlighted in the Asset Filter (Type/Filter) e.g. Particular Products of interest may have been in the Asset Filter; these may have had lengthy filter criteria which, if the user wants the list back, will need to be re-entered.
2) Add “Find” button to form. This would temporarily re-purpose the Asset Filter in order to find the Contact that is needed to add to the form. Clicking “Link” would in addition to putting the selected Contact or Contacts in the form, place the Asset Browser back into the state it was previously. Benefit: Don’t lose focus upon whatever was previously highlighted in the Asset Filter. Benefit: The “Type” and “Context” fields in the Asset Filter could be temporarily propagated to show correct context e.g. Type of “Customer” and Context of “Contact”. Drawback: Means that we have to build an additional line of web-part communication i.e. From Display/Update web-part to Type/Filter. This could be challenging if the Display/Update web-part is rendered in an iFrame.
3) Add “Save” button to Type/Filter. This would save the state of the Asset Filter (Type/Filter) before operating as for 1 above. Once at state is saved (only one state allowed to be saved), the button would turn into a “Restore” button, allowing the original state to be returned i.e. after we have located and added the correct Customer(s). In this way, it would operate much like the memory function on a regular calculator. Benefit: Adheres to the existing channels of web-part communication i.e. from Type/Filter to Display/Update web-part. No new channels required/means no additional testing required. Benefit: Provides additional functionality as a consequence of finding things e.g. a user could find some interesting Products and save them before looking at some Customers and then toggling between the two without even involving the Display/Update web-part and form filling. Drawback: It is an additional control to place upon a Type/Filter and so may add to general application complexity.

Option 3 (Add “Save” button to Type/Filter) is selected for our model due to this solution giving the user more power without adding behind-the-scenes complexity and risk.

The Type/Filter should also be used to copy things from one UX component to another. In this way, it is an extension of the method used to find things. For example, if a requirement exists to copy one or more Order Parts from one Order to another, the user would select the Order that he/she wants to copy to (bringing it up in the Display/Update web-part) and then switch to the To-Do List, selecting type of “Order” with a context of “Order Part”, clicking “Filter” and then selecting the checkboxes of the existing Order Parts to copy to the Order under focus and in the Display/Update web-part. It may be seen that the user can only copy things that are selectable in a linked Type/Filter through the standard filter options. If business elements outside of this selection are required, filter options should be changed to encompass the required selection rather than implementing some new copy/paste option. Filter options are easily changed as they are defined through exposed web-part properties.

Universal enterprise UX – Part 6 (Form Development)

2010-04-12T22:41:00.003+01:00

Form development is the bulk of programming activity using our model. The Type/Filter and Browse/Next web-parts should (in theory) be able to be configured (XML/web-part properties) rather than developed. Forms can be developed using whatever technology the organisation are familiar with e.g. Web/InfoPath/X.

The Browse/Next web-part is the wrapper that all forms reside within (when they are on screen). Many forms will be able to be rendered within it in their entirety. Some forms will require contextual information to be retained in the form for reference, while the user is engaged in an “offshoot” task; for example a form focused upon generating a new user might want to retain the core user details e.g. Name, ID on the Browse/Next, while also taking the user through a process to add that user to selected user groups. This sort of behaviour should make use of the Wizard Pane concept. This supplementary web-part permanently exists on the screen wherever a Display/Update web-part is placed and (like personalisation behaviour) expands whenever required.

Its use is not limited to wizard-like behaviour and it may be used to render discrete pieces of information, tables or other complex controls that require the original context (in the Browse/Next web-part) to be displayed at the same time that the user is interacting with the new process. Another example of use of the Wizard Pane would be to show details of Notes, where upon selecting the note in the list box (perhaps attached to an Order); the details of that note e.g. Name and Description, Raise Date, Raiser and Type are detailed in the Wizard Pane.

Form developers should be fully aware of the Wizard Pane and make appropriate use of it where necessary. Judicious use of the Wizard Pane could dramatically improve the UX. A good form developer will make appropriate use of it wherever possible.

When the Wizard Pane is expanded on a page, it will “compete” for space in the same way as for the other web-parts. Within a form, collapsible sections should be used wherever a logical grouping of form details exists, for example; “Personal details”. This allows the user to personalize the form experience by collapsing sections he is not interested in at a particular time. The combination of both the Wizard pane and collapsible sections should mean that use of tabbed dialogues and in particular pop-up dialogues should not be necessary within any application built using this UX model.

Universal enterprise UX – Part 5 (Personalisation)

2010-04-12T22:35:00.002+01:00

All commercial portal platforms allow for a level of personalisation; for example Windows SharePoint Services (WSS) allows the user to change their personal view of a WSS site e.g. resizes of web-parts, addition of new part-parts, minimising of web-parts.

Our model needs to extend this however. For example; when a web-part is minimised, the remaining web-parts should automatically re-size to consume as much of the screen real-estate as possible. This auto-expansion activity must only be constrained by underlying portal platform e.g. web-part zones for the page in question using WSS/MOSS. Unless all web-parts are specifically minimised, whatever web-parts are not minimised on the page, they will always consume exactly the same screen real-estate.

Examples of auto-expansion behaviour follow. These are not exhaustive and additional combinations may be allowed for example a page where the Task List is auto-expanded to consume the whole page (similarly to the Asset Filter whole page expansion below) would be possible.

Darkt-gray boxes above indicate that a web-part has been minimized by the user. Light-gray boxes indicate that a web-part has automatically re-sized itself to consume the available space within its web-part zone. Note that in the case of the page described in the top-right above, the Asset Browser has not horizontally expanded to fill the page, as this would exceed its web-part zone. Note also the lines between the pages as they indicate how a page may be successively personalized to the user’s taste.

Universal enterprise UX – Part 4 (Bringing it to life)

2010-04-12T22:15:00.007+01:00

Fleshing-out our order processing scenario a little; Orders consists of Order parts (Line items) and these are for various Products. Products grouped into Categories. Enquiries can be progressed into Orders and Tasks are tracked by the application for various reasons e.g. Tracking cold calls, customer complaints etc. All key entities are allocated a Classification e.g. Geography. Customers are front-ended using the order processing system but are likely managed using something else. Similarly Tasks are front-ended but this will likely be managed by some workflow solution. That is it - a standard order processing solution.

Several instances call for a non-linear browsing approach, for example, the selection of Categories and Products where related items (not specifically searched upon by the user) may be seen and the selection may be seen in context. In many instances, the user will also want to search upon these objects (Categories and Products) directly i.e. without browsing.

The arrows show UX contextual flow only. For example, user interaction in the Browse/Next web-part will not automatically affect the Order browser however; the user can select a business operation such as “Add to order” to insert a Destination (or any other allowed object) into the Order browser (and the corresponding Order [Part] highlighted).

Each Type/Filter type of web-part will operate as a three-fold UX process. The user will choose the type of data to work with (Step 1) then apply a series of general filters to that selection (Step 2) and then browse the result set (list box) by use of the vertical scroll-bars and column sort functionality (Step 3).

Initial rendering of the Type/Filter web-part (Step 1) should either default to the filter view (Step 2) or the list view (Step 3). As such this web-part has just two modes – filter mode (Step 2) or list mode (Step 3). Three steps are shown above in order to articulate the UX process involved i.e. the Type has to be chosen first.

In this (modal) way, the model can affect a search by the following means – filtering, sorting and then browsing (from switching from the Type/Filter web-part to the connected Browse/Next web-part). For both UX and performance reasons, a more unstructured search function where, for example, all matches for a particular string are returned across all tables, columns and rows will not be selected. This type of unstructured search is more appropriate to content searching.

A Task is selected in the Task list and the matching Task is automatically selected in the Browse/Next web-part (Order web-part) below it (with the hierarchy also automatically expanded for browsing). The Next functionality will be grayed-out in this particular case, as there will only be one matching Task amongst the Orders.

We would mainly use the Type/Filter web-part as an entry-point to the browsing (what users generally really want to do). For example, if we know which State we want, we will use the Type/Filter web-part to select a Level of “State” then switch to the list view to find our particular state e.g. California. Once we have selected this, in the same manner as with the Task description above, we would switch to the Browse/Next web-part to drill-down into California to see the regions within that State and the Cities within the Regions (or in fact any hierarchical description we define below California; political/ethnic etc.).

Find functionality to the Type/Filter may be extended by the provision of two Filter/Value pairs, the first of which has previously enumerated values i.e. in a drop-down, the second of which displays different Filters (fields) to the first and allows a “wildcard” search via a text-box.

If the user wished to search for all Marketing Tasks for the New York office created for customer “Smith”, they would enter the following; Type = “Task”, Context = “Marketing”, Level = “”, Filter = “Office”, Value = “New York”, Filter = “Customer”, Value = “Smith”. There is a purposeful and direct parallel with the underlying table structure e.g. Type = Table, Context = Table Type, Level = Table Relationship, Filter = Field, Value = Value of Field.

No additional placement of Filter/Value pairs is possible with the model above. Also note that no Boolean operators are operative between the Filter/Value pairs. Of course more pairs/Boolean operators would extend functionality but would your users really use them?

Universal enterprise UX – Part 3 (Some rules)

2010-04-12T21:13:00.003+01:00

For purposes of both user acceptance/reduction of technical complexity, rules regarding the use of web-parts on a page should now be enforced for our model:

1) No more than six web-parts on a page by default. The user may well be able to add more web-parts from a gallery but this should be at his/her discretion. By default, any more than six on a page risks disorientation.
2) Web-part communication only goes one way. Although certainly technically possible to have web-parts communicate both ways, i.e. both to and from each other. This creates unnecessary complexity i.e. users have to think modally also it increases the number of paths through the system which then have to be tested.
3) One web-part should communicate with a maximum of two other web-parts. Although generally desirable to have web-parts communicating with each other, any more than two other web-parts being “automatically” changed when a user selects an item in a third web-part is making the application too linear i.e. it prevents other activity from happening in parallel because UX components have just been re-purposed for the current task under consideration. Allowing for a level of UX multi-tasking is desirable.
4) No more than one Display/Update web-part per page. Any more than this would entail either a significant amount of scrolling or would be confusing to the user as two portions of the screen would then be devoted to displayed information. Which Display/Update web-part should they look at, for example, when the user selects a contextual data item from a Type/Filter?

Another rule is that the look and feel of web-parts and the way that they interact should be consistent. This benefits not only rapid development, in that UX code may be developed once and re-used for varying functions but also significantly reduces testing and training time.

Universal enterprise UX – Part 2 (Core Web-parts)

2010-04-12T20:44:00.010+01:00

The first post on this topic generated several requests for a UX design of the concept and some suggestions that it simply would not be implementable. Following posts on this topic will attempt to deliver at least a high level design so this question can be debated. We will use specific terms e.g. web-part but the design will be generic and not tied to any particular portal technology. We will also use the scenario of a standard order processing system to bring it to life but it could just as well be any application.

Each portal page will comprise a maximum of one large area (web-part) for the display and update of data (the Display/Update web-part). This will present data/allow updates (if the user has the requisite authority etc.). A page could be comprised of several other web-parts (without a Display/Update web-part) but it will never have more than one Display/Update web-part as the scope of the page would then be too confusing for the user. Optionally, this web-part will invoke actions; for example, “New customer entry” or “Add contact to enquiry”. As such it may be described as a “big, dumb” area of the page in that it has no filter, selection (of types) or browsing capability. It exists for displaying and updating data – CRUD operations.

The Display/Update web-part must then be controlled (and contextualized) through other web-parts e.g. filtering, selection (of types) and browsing. These functions may be combined in several different ways to create the web-parts described in the earlier post.

Data with a rich-hierarchy such as Products lend themselves to a browse metaphor and data which is mainly sequential such as Tasks lend themselves to a list metaphor. Our model will implement these as such. There are also a couple other ancillary functions; namely Next (for moving through peers in a hierarchy) and Filtering (for selecting what is in the list) functionality respectively.

Now that we have defined the core functions, several options for combining them into web-parts are possible:

The “Next” functionality simply finds the first match and allows the user to “hop” through matches to whatever filters are set in the Type/Filter web-part.

1) Multiple Filter web-parts. Easier for multiple developers to code, for example; they can each be given a web-part describing a “type” of information e.g. Developer X is responsible for creating a web-part to select “Geography” and Developer Y is responsible for building a web-part to identify a “Category” for successive editing. Issues with code-reuse, consistency and the amount of screen real-estate required.
2) Combined Type and Filter web-parts. Here, the same web-part is used to display all types of data, for example; a drop-down will allow the selection of “Classification” as a type and the user would use filter functionality to select whether “Geography” or “Category” would be displayed in the same web-part as a list. The user would then choose one at a time to display or update in the Display/Update web-part. The user will use the filter capability in the controlling Type/Filter web-part to select the first in the list of matches and then use the Next button to move through matches.
3) Separate Type and Filter web-parts. As above, but the type functionality in the drop-down is taken out of the controlling web-part and placed into a dedicated Type web-part. The issue here is that it may be confusing for the user in that they have to know which web-part is actually controlling the Display/Update web-part i.e. Type or Filter.
4) Combined Type/Filter and Browse/Next web-parts. As with Option 2 but the additional Browse web-part functionality is also included. This option has the advantage of being very flexible in terms of user interaction. It suffers however from a compromise around consistency i.e. The Type/Filter/Browse web-part is used for all controlling functionality (even for those areas that do not need it or it could be potentially confusing e.g. Task list) or we selectively use it for areas that may benefit from browsing either by a list or a hierarchy e.g. Classification.

Option 2 is selected for our model as it is a combination of usability and judicious screen utilization and affords a consistent UX. It also does not result in extraneous functionality being deployed e.g. around the Task example described above.

Our model now has three core web-parts; Display/Update, Browse/Next and Type/Filter. Together, these will enable us to build most applications. They are effectively UX classes; used to build the physical web-parts of the application e.g. both a Product browser and an Order browser can be made from (differently configured) Browse/Next web-parts.

On Microblogging & Microcelebrity

2010-04-08T22:53:00.008+01:00

Why Tweet? Many tweeters are motivated by learning. A clinical psychologist suggests it “...stems from a lack of identity...”. Arguably the most widespread theory (other than simply - "its cool") is a narcissistic/observer cycle with celebrities and civilians respectively. It is a viable way for many people to feel like they are “hobnobbing” with celebrities. Fleetingly and tangentially (as their exchange is typically over a specific topic/shared link) they too are famous. The real reason, as with many truths, is likely some combination of these. Whatever the motive, its - popular, free, requires no training and is on-the-surface at least - an effective communications channel. Is it of any organisational value though?

There are a host of organisational microblogging tools. At enterprise scale, Office Communications Server (OCS) 2007 R2 supports persistent group chat (through its acquisition/integration of Parlano’s MindAlign) as does Lotus Sametime. MSFT are additionally in development with Office Talk, a Yammer-esque enterprise microblogging service. This is a research project though and may not see the-light-of-day or be subsumed into MOSS/OCS. The space is well served.

Are organisations really using microblogging tools productively in a business sense though? They are not fundamentally different to persistent group chat; functionality that has been in heavy use in trading (to collaboratively form trading strategies in real-time) and automated application feeds for years (particularly within the Finance and Resources sectors). An immediate, informal, concise and one-to-many communication channel makes perfect sense in these scenarios.

Organisations that already use it will carry-on – happy that what they have been doing for years is now modish. Organisations that are rolling out microblogging now though – because they feel it is expected by their workforce, will cause an injection of productivity or (more likely) there is a single vocal micro blogging champion that treats it as a personal cause will experience challenges. As with many consumer/social tools adopted by the enterprise, the process of obtaining similar enterprise versions (security, bulk updates, internal hosting options, directory integration, archiving and moderation) will lose much of the usability and fun element of the originals. For the employee, having to keep two copies of your Facebook, Twitter, Blogger, Delicious, Foursquare and (in the not very distant future) - Quora and Plancast profiles (and settings and friends) will be both tedious and impracticable. Also, many people enjoy the disposable identity present in consumer social networking; especially the young; where they can try on different pseudonyms and personas. This is directly at odds with organisational goals. Organisations could federate employee personal networks with the work networks but how many are going to attempt this given the reasons for enterprise versions in the first place? Basically, how many business decisions are really going to be influenced by microblogging if you are not a trader to justify the cost?

The most important element for organisations to take from the micro-blogging phenomenon right now is not the tool; it is recognition of the potency of microcelebrity:

1) Build it into business processes. This can be harnessed without even having tools at all. For example, a key strategic requirement for many organisations is to empower employees to create, publish and maintain ad-hoc reports in response to the needs of the moment (versus some lead time – typically in weeks); it is cheaper and employees understand their own data best. Driving this behavioural change requires more than training. Showing the “Top-10” most popular reports, who created them and broadcasting them as web parts/gadgets on home pages affords both a sense of microcelebrity (in terms of the report writer) and positively contributes to the willingness of other employees to create these reports (or hobnob with them).

2) Build it into employee recruitment/retention. Personal branding awareness is on the rise. Employees are able to articulate their ideas, successes and worth using the Internet as a market place. They can operate at an influential global level if they want to despite their actual role not quite matching up. For those that do, they can be frustrated with HR policies that do not recognise this. Micro-celebrity recognition should be supplementary to standard HR/line manager recognition, bottom-up and borne from audited and metricized operational usage. It should be inarguable and fundamentally honest come employee review time. This is due to any peaks/troughs being levelled out by sourcing the data from many people rather than HR, the line manager and occasional others that are bought in to provide their views.

No-one is listening though.

Handling Targets That Do Not Roll-up

2010-04-07T19:40:00.006+01:00

Comparing actual figures against targets are fundamental to any Performance Management (PM) system. PM solutions are typically built upon cubes and correspondingly, cubes are built upon data marts. In a data mart, measurements are generally stored in fact tables at their lowest level e.g. Cost per Organisational Unit (OU) per Day. These measurements are then aggregated along dimension levels e.g. Cost per Area per Week is the sum of the daily site figures for that week and area. This solution works well with actual figures as they are almost always additive. Targets however can sometimes be independent e.g. Target Cost per OU per Week is not necessarily the sum of daily target costs for that week.

There may be legitimate business reasons for doing so e.g. cost is heavily market/EOS driven and needs to be targeted manually or, for whatever reason, targets cannot be decomposed into fully additive components. This means that the PM solution has to either store the data for all levels in the fact tables or not include targets in the data mart at all. This latter option creates its own issues though as many front-end BI/PM tools can only connect to a single data source at a time.

Design alternatives on how to store data for all levels in the fact tables are not well documented in BI/PM design literature. Options here are:

1) Single Fact Table. Storing all aggregates in the same fact table simplifies the data model but has the disadvantages of values being duplicated e.g. the year value is stored in every day record; the fact table will contain a large number of fields and the ETL process is more complex (either using updates or reloading every level for each load).
2) Multiple Measures. To reduce the number of fields in the fact table, the tables could be split along time levels or measure. This option has the advantages of a reduction in the number of measures per metric and no duplication along the Time dimension. Its disadvantages are that there is likely duplication along the OU dimension and there are multiple fact tables.
3) Fact Table per Dimension Level. Defining a table for each of the dimension levels has the advantages of no field duplication and the least number of measures overall. Its disadvantage is again that there are multiple fact tables.

Option 3 is generally recommended as this eliminates data duplication and simplifies the ETL process. It is somewhat unusual as typically within a star schema; a fact table is surrounded by multiple dimension tables. It is however, a practical solution and has been recently implemented for a large resources client.

The fact tables are connected to the dimension tables on differing granularities as shown below:

The CostDay fact table is linked to the Time dimension table through its TimeId Foreign Key (FK). Fact tables with different granularities need to have additional field linking them to the Time dimension at the desired granularity. All attributes within the fact table therefore have to share the same granularity.

This (admittedly complex) model can be later simplified using whatever tools you use to maintain your cubes. SQL Server for example provides Perspectives, Measure Groups and Calculated Members that can be used to hide the complexity of the underlying data model from the user. Perspectives can be used to hide objects e.g. fact tables from the user. Finally Measure Groups can be used to create logical groupings of fact members.

BI Strategy Planning Tips - Part 2

2010-04-05T22:01:00.005+01:00

4) Directly address credit crunch sensibilities. BI endeavours can be expensive and challenging to articulate in terms of benefit. You may need to plan a BI strategy; but right now, projects need to have a critical-mass of internal support and short-term benefits in order to obtain funding and/or avoid being postponed. Key areas are:
a. Cash flow management. Cash is King. Cash flow metrics e.g. Price to Cash Flow/Free Cash Flow enable managers and potential buyers to see basically how much cash an organisation can generate. Data mining can predict cash flow problems e.g. bad debt to allow recovery and/or credit arrangements to be made in a timely (and typically cheaper) manner.
b. Business planning capabilities. While information systems are in place for many organisations to generate representative management information, the level of manual intervention needed to deliver essential reporting can result in unacceptable delay and therefore data latency and inconsistency. This directly impacts the organisations ability to predict and respond to threats to its operations (many - in an adverse climate). Outcomes can be financial penalties, operational inefficiencies and lower than desired customer satisfaction. Building habitual Performance Management (PM) cycles of Monitoring (What happened? /what is happening?), Analysing (Why?) and Planning (What will happen? /what do I want to happen?) - places the focus back on results as well as affording a host of additional benefits.
5) Release something early. Unlike a couple years ago, you likely cannot now go through a six-month analysis and following data cleansing and integration phase. This work does not deliver tangible business benefits in the short-term. Instead, look at getting basic BI capabilities out within weeks. This will allow you to incrementally build upon your successes, gain business/operations experience (through monitoring usage) and build user alliances gradually. How much you can do here depends on your chosen BI platform but building your BI/PM on-top of existing reports (data pre-sourced/cleaned), selectively using in-memory analytics tools (no need for ETL) and SaaS BI; if your immediate needs are relatively modest, all should be seriously considered.
6) Avoid the metadata conundrum. Metadata is undoubtedly important. It is well known to assure adoption; convincing those making decisions (from the system) that they are using the best data available BUT it is a complex problem and intersects other disciplines e.g. data integration, information management and search. Most data objects, whether Word files, Excel files, blog postings, tweets, XML, relational databases, text files, HTML files, registry files, LDAPs, Outlook etc. can be expressed relationally i.e. they make at least some sense in a tabular format. They also span the spectrum of metadata complexity. The end-game is to build on all the ideas of ODBC and JDBC to provide the same logical interface to all of them. A DBMS or file system can then treat them all the same logical way, as linked databases and extract the metadata, create the entities and relationships in the same way and use the same syntax to interrogate, create, read, write and update them. Tools/theories are evolving but this is rarely achievable in practice. If you can satisfy regulatory requirements with the bare minimum – do it. Just concentrate on data lineage if you cannot; building as much of a story around the ETL process as possible. Less than 15% of BI users use metadata extensively anyway.

MOSS project failure reasons

2010-03-31T19:17:00.007+01:00

Here is a list of the main reasons for MOSS project failure based upon experience (many reasons will also be applicable to non-MOSS projects). Other (occasional) reasons contributing to failure include inadequate business domain understanding, MOSS deployment design and MSFT licensing appreciation but for the sake of focus, only the main reasons are shown here. Reasons are shown in descending priority order:

1) Scope management. Initial requirements gathering insufficient. Incremental scope additions undocumented and outside change management. Little or no architectural analysis of scope shifts before they are implemented resulting in downstream performance, testing and hosting integration issues. High-Level Design (HLD) used as sole written basis for development. HLD is generally insufficient for developers to interpret and develop against. A solution can be to factor in element of client MOSS training during the early analysis stages. This will reduce incidence of client scope expectation being significantly more advanced than what is possible Out-Of-the-Box (OOB). Basically, clients tend to think MOSS can do more than it can - OOB.
2) Configuration management. Robust problem and change management processes missing; inadequately defined or not maintained. Doubly important in a MOSS environment where configuration, content and code (and also data and process) are entwined. Effect of this is that developers become confused as to what they are supposed to be working on and client loses detail visibility. Senior developers may attempt to establish these processes but will generally be ineffective due to inexperience and their relative capability for enforcement. Multiple versions of both requirement and design documents stored in multiple places and little attempt at sign-off coupled with poor client communication means that entrenched positions are swiftly made. Developers can struggle with the same issue for days. If daily updates are recorded in a problem management system, this will be obvious to the Project Management (PM) function through oversight.
3) Project communication. Poor overall project communication. With a focus on gaining hard MOSS technical skills, softer skills such as communication are often overlooked. Developers typically send conflicting messages to client. No team scrums, little articulation of cause and effect in terms of project plan. No “bottom up” captures or extrapolation. Culture of sharing information not engendered at senior developer level. PM reporting limited; when requested at high level and non-specific.
4) Skill levels. Due to initial shortages of MOSS skills, developers have historically been staffed to projects with the effective remit of learning on the job. This is not now the issue it used to be due to wide-scale MOSS training focussing on .NET Web part Framework. There remain key poorly resourced areas e.g. workflow and InfoPath development and engineering experience in MOSS “lockdown” mode.
5) Administrative duties. Not really focussed on operational logistics e.g. reporting status, ensuring adequate vacation cover, maintaining a service incident log, ensuring network access for resources and providing a view to scheduling on upcoming resource requirements. In a market where MOSS resources are relatively scarce, focus on effective resource management is critical.
6) Engineering resource. Insufficient engineering resource can be applied to MOSS projects. This manifests itself as generally assumed acceptance of the client infrastructure and hosting plans and means the team cannot plan and iterate for performance from design onwards.
7) Client relationship. Diminished client relationship. This can be left by default to be built by developers. Without a consistent face being presented to the client through reporting or solution walkthrough, the possibility for a relationship to grow is diminished and client finds it easy to take an aggressive stance to the engagement when required. This is particularly important where a RAD approach is more applicable e.g. MOSS.

BI Strategy Planning Tips – Part 1

2010-03-21T15:48:00.011Z

Planning a BI strategy for your organisation can be challenging; you need decent industry/vendor awareness, an appreciation of organisational data and ideally; a handle on budgeting. In no particular order, here are some practicable tips to get started with yours. More will be forthcoming.

1) Start with your Supply Chain. Reduction of energy use in data centres is a key IT issue. This is generally driven by either a cost saving drive or a Green IT focus (shouldn’t they really be the same thing though?). However, most organizations budget between 2-5% of revenue for IT budgets, yet spend roughly 50% of revenue on all aspects of supply chain management. Basically, there are significantly more savings to be made in the supply chain than in data centres. Large volumes of raw data are generated and stored by each process of the supply chain (plan, source, make, deliver and return) by automated enterprise applications being used at most large, global manufacturers. BI can help determine what information is necessary to drive improvements and efficiencies at each process in the supply chain and turn the raw data into meaningful metrics and KPIs.
2) Forget the BI-Search “evolution”. Just the training costs for commercial BI systems are expensive. Organisations want single (easy and simple) interfaces wherever possible and a search-based interface appears to be the key to engaging the masses. There has been heavy speculation over the last couple years (ongoing) that BI and Search technologies will somehow merge. This approach however only really surfaces existing BI reports for more detailed interrogation e.g. “July Sales Peaks”. A search string is not a rich enough interface to support ad-hoc queries. Think about it? You need either a devoted language e.g. MDX or a rich data visualisation package to traverse dimensional data. A search box will never explore correlation between marketing budget and operating income.
3) Forget “BI for the masses” (for now). BI has been actively used in the enterprise since the early nineties. The expectation of “BI for the masses” (basically - the SMB market) hasn't exactly happened. Why is this? It’s because people want to collaborate and jointly come up with ideas, solutions, figures and approaches. They need this for personal, political and commercial reasons. It will change when enterprise SOA is in place and it might change when there is a change in the way consumer impacts enterprise networking. It will not change in the short-term.

You can tag. A lot.

2010-03-18T15:32:00.007Z

(Originally posted 1 May 2009).

MSFT Tag is a Windows Mobile application that uses the device’s camera to recognize a graphical image or tag for the purposes of obtaining information. You could print one out as a sticker and place it on a landmark to get a link to its site or Wikipedia definition for example. Each tag consists of a 5X10 grid of triangles (although these can also be smaller circles; overlaid over a background image to create a custom tag), each of which can be in one of four colours. This allows a tag to hold thirteen bytes of data. Information received is of four types – URL, Telephone number, Vcard and free text.

Telephone numbers, text and URLs (with a bit of compression) could maybe resolve from the device i.e. the tag itself stores the URL and the device either directly calls the telephone number or jumps to the URL/displays the text. QR codes work in this way. This is disproved however when a device is switched to “Airplane Mode” and tag recognition just does not work. This means that MSFT Tag on the device jumps to a proxy server that resolves the reference (GUID) extracted from the tag into the information already uploaded to the MSFT Tag site. The proxy servers must be pretty impressive as the whole thing works very smoothly.

Thirteen bytes is enough to store a number of 2.02E+31; each one referencing a separate piece of information. This is broadly comparable to the number of grains of sand on all of the beaches of the Earth so MSFT must be expecting a decent take-up of this.

Tags are really useful for URL links that contain filters tied to a geography in the link e.g. you are at a bus stop and there is a tag that shows bus schedules; whether they’re late or not – for that particular stop. Vcards, Telephone numbers and free text are cool rather than essential e.g. you might use a tag embedded into a colleagues’ email sign-off to get their details into Outlook but it’s never going to be a killer application.

The eyes of the developer

2010-03-18T15:30:00.006Z

(Originally posted 22 April 2009).

Offshore development resources are great. They are invariably well educated, diligent and critically in these trying times – effectively priced. Even now though, a key reservation organisations have around using them is - visibility. They want to see them and talk to them; their requirements are so exacting that only by looking into the eyes of the developer can they be understood. Bringing offshore resources onshore for the initial stages of a project (and so that they can return offshore for knowledge) is a proven way of mitigating this concern.

Lead times involved in procuring offshore resources onshore (often three months) can be ineffective for many projects; especially ones founded on a business case of cost reduction/avoidance. This can be expedited to less than a month but generally only if the resources are undertaking “training” and not developing the solution. Developing the solution though is where they will truly learn and become vested in the success of the project. A seemingly attractive alternative for large organisations (and for the consultancies that service them) is to establish an onshore pool of offshore resources to service future onshore projects. Is this a cost effective solution though?

Five offshore resources brought onshore for three months will cost around $88K (accommodation/fly-backs/insurance/travel/visas etc.). Assuming they can be cross-charged (or sold) at say $877/day for 80% of the time they are onshore; this makes $210K in revenue.

This appears good (140% ROI) except for the fact that once this process is started, the resources have to be retained i.e. taken out of (or reserved from) the offshore pool until they arrive. This can easily take three months. Assuming a cost of $146/day/resource, this totals $44K, taking the endeavour down to $79K profit (60% ROI). This may be able to be offset by them doing other (short-term) work in the interim but it certainly should not be counted upon. Once resources are onshore, organisations should also account for increased team lead/managerial support for them (perhaps one day/week across all of them – totalling around $22K in opportunity cost), taking the endeavour down to around $57K profit (38% ROI).

This should also be considered a high risk endeavour due to the fact that resources are being recruited for a pool rather than a specific project (project may not happen) and manifest cultural differences. Organisations are therefore already borderline as to whether this is a good idea financially or not. The best approach has to be simply to keep a close eye on “hot” skills and ensure that offshore pool resources in these areas already have visas.

Floating up to the Spatial Web

2010-03-18T15:28:00.006Z

(Originally posted 20 April 2009).

Google Street View is an option in Google Earth that shows sequential 2D images down the world’s main streets. This tool provides for a simple 3D effect if you traverse up or down a particular street. Photosynth is MSFT Live Labs software that creates a 3D effect from multiple 2D images of the same scene taken from different angles. You can zoom in/move around the scene if there are enough 2D images of it available. It is more sophisticated than Street View since it is not tied to a particular vector i.e. a street and also because it performs image extrapolation to complete partial views.

Moving through alternate realities like this holds a natural attraction for people; training, POS, medical diagnosis, national security, gaming, film-making, virtual tourism and urban planning would all immediately benefit. Photographs of existing places and objects are increasing at a huge rate through (camera phone) image uploads to social networking sites and the thousands of commercial and Government camera installations throughout the world. There will surely eventually be blanket photograph coverage of pretty much everything everywhere (and once that is achieved - at every time). Geo-tagging will help connect these pictures together.

These technologies basically link photographs of existing scenes up. Your brain mainly adds missing spatial orientation. There is no 3D model of structures behind them unlike say as with Google Sketch-up, AutoCAD or the various First Person Shoot-em-up (FPS) games e.g. Call of Duty etc.

Does there need to be a 3D model behind them? You could ignore surfaces/obstacles, simply walking through them as a disembodied ghost. This might be acceptable for many applications but for gaming it will not. 3D models are time consuming to plot/maintain with changing actual landscapes. The reason why models are created now is primarily for edge detection e.g. you see a wall and because the 3D model has been defined at design-time to identify it as a wall, you are prevented from moving through it. It is much better to do edge detection at run-time using some algorithm to identify the wall as an edge (or barrier), preventing it from being moved through. Gaming will likely be the driver for this new standard. This will be accessible through some cloud-based service and other applications will simply adopt it.

Development of the Internet has shown us that competing infrastructure technologies will co-exist for a while but they will eventually “float-up” to the highest common denominator e.g. the most extensible, cost-effective, egalitarian and “fit-for-purpose” technology. Applications built on selected-out infrastructure will eventually switch to the new standard. It is easy to see a gaming driven cloud-based service, regularly updated with near-real time images as being adopted by all other applications requiring spatial understanding.

In this way, we will see a single alternate reality develop where synergies and opportunities are created by markets colliding (as with actual reality) and the “Spatial Web” described last year by MSFT’s Craig Mundie will become normal (if not “real”!).

UPDATE: Swiss Computer Scientists use Flickr to do this now http://www.openculture.com/2010/12/3d_rome_was_built_in_a_day.html.

Virtual Earth & SSRS Integration

2010-03-18T15:25:00.005Z

(Originally posted 27 March 2009).

Virtual Earth integration with SSRS can be a challenge due to uncertainty around approach and a general lack of resources on the topic. Developers can use three Virtual Earth APIs: the MapPoint Web Services, the Virtual Earth Map Control and the Virtual Earth Web Services. SSRS cannot currently use the Virtual Earth Map Control since this allows users to make requests via JavaScript to an AJAX map object and SSRS will not allow JavaScript to run (as otherwise it will not be able to export to PDF amongst other things). This would have been preferable since it affords a level of interactivity e.g. zooming-in on tiles. This leaves either the MapPoint Web Services or the Virtual Earth Web Services. Both methods effectively work the same in that points to plot are passed to the web service, rendered on the map server-side and passed back to the browser as an image. MSFT no longer develops MapPoint Web Services (although existing applications built using MapPoint Web Services will remain functional) so if you want Virtual Earth integration with SSRS, your only real choice is Virtual Earth Web Services. Here is a snippet on how to pass coordinates (Longitude/Latitude) to Virtual Earth. This should display a pushpin on the map (The Imagery Service is used).

You will need a Virtual Earth Platform Developer account and this itself will need a Windows Live ID. Once you have this, the SSRS report should call a function in a C# custom assembly, passing in an MDX query string to retrieve co-ordinates. The SSRS report only calls the custom assembly. The output of the function will be a URL for the map image to display on the report. The custom assembly queries the database/cube to retrieve co-ordinates for the points to plot. It should then call Virtual Earth Web Services passing through the co-ordinates to be plotted. Finally, the GetBestMapView method should be used within the assembly to scale the image appropriately e.g. if all points are plotted in the South East of England, the image needs to zoom in to the South East of England and not the whole of the UK.

Text breeds data. Data breeds information.

2010-03-18T15:24:00.004Z

(Originally posted 7 March 2009).

Received wisdom tells us that unstructured information is 80% of the data in an organisation. Reporting, BI and PM systems are still tied, in the main, to structured information in transactional systems (obtained through ETL, staging, dimensional modelling, what-if modelling and data mining). An opportunity has existed for some time to incorporate unstructured data. The inhibitor is technology. Where product sales figures can be extracted over years and then extrapolated to determine likely sales next period; how does a BI solution use the dozens of sales reports, emails, blogs, unstructured data embedded within database fields, call centre logs, reviews and correspondence describing the product as outmoded, expensive or unsafe? If they do not, they may lose out since this information (the 80%) can affect the decision of how much of the product to produce.

Most organisations currently handle the general need to unlock unstructured data by market sampling through techniques of interviewing, questionnaires and group discussion. They attempt to apply structure to the data by categorizing it e.g. “On a scale of one to ten – how satisfied are you with this product?” They either ignore information already there or manually transpose it; typically by outsourcing. A minority use the only technology that can truly unlock unstructured data within the enterprise right now – text mining. Note that this is different to both Sentiment Analysis (too interpretive right now) and the Semantic Web (too much data integration required right now) .

Many of these organisations however believe text mining simply makes information easier to find. This is a function of currently available products. The principal MSFT text mining capability is in its high-end search platform Fast. Such products use text mining techniques to cluster related unstructured content. It is not enough to loosely link data however, they need to be linked at an entity (ERM) level so they are subject to identical policies of governance, accountability and crucially; the same decision making criteria. It is worth stating the pedestrian; text mining is like data mining (except it’s for text!); establishing relationships between content and linking this information together to produce new content; in this case new data (whereas data mining produces new information).

Consider a scenario where an order management application retrieves all orders for a customer. At the same time, search technologies return all policy documents relating to that customer segment together with scanned correspondence stating that dozens of the orders were returned due to defects. The user now has to perform a series of manual steps; read and understand the policy documents, determine which returned order fields identify policy adherence, check the returned orders against these fields, read and understand the correspondence, make a list of all orders that were returned (perhaps by logging them in a spreadsheet), calculate the client value by subtracting their value from the value displayed in the application and finally modify their behaviour to the customer based upon their value to the organisation. Human error at any step can adversely affect customer value and experience. Much better to build and propagate an ERM on-the-fly (by establishing “Policy” as a new entity with a relationship to “Customer Segment”), grouping the orders as they are displayed while at the same time removing returned orders. I’m not aware of any organisations currently working in this way. This is the technology inhibitor and where text mining needs to go next.

MOSS development tips

2010-03-18T15:21:00.003Z

(Originally posted 19 February 2009).

Other than the classic (and largely non-technical) project concerns of scope management, configuration management etc. there are key challenges to a MOSS deployment. The process for moving the MOSS solution between environments, for example, is more complex than other .NET projects. This is due to the way MOSS separates configuration, content and code. The process is also heavily dependent upon the environments being used. Implementing changes to Production post go-live are particularly challenging as of course you need to retain content. Generally, the process is routinely underspecified. A document explains the steps involved here .

A general lack of adherence to a MOSS centric methodology other than general MSFT best practices causes developer confusion. The key MSFT document here is – “SharePoint Products and Technologies Customization Policy 2007”. This is intended to be a framework for organisations providing MOSS platforms. Organisations should also use this list as a starting point to verify the quality of solutions that are submitted for deployment. Along with providing a check after a solution has been developed, the code acceptance checklist can make a good training tool. The steps required to plan, design and deploy MOSS projects are defined here. A governance plan is recommended as a means of gaining early support from key stakeholders across organizations. It forces consensus and designates ownership for numerous key deployment considerations for governance and for information architecture. Tracing and error logging support are required reading for MOSS developers.

Reporting is generally required from MOSS applications. The method used to write them is typically C#.NET against the MOSS object model. IT departments are generally aware of SSRS and like its OOB functional abilities e.g. export to PDF/Excel, publish to MOSS, tie to workflow and non-specialist skills required. They find it difficult to understand why MOSS reporting is not so straight-forward.

MSFT wording here has been relaxed around use of SSRS against MOSS and now it may be prudent to copy relevant MOSS tables or database to a reporting database periodically and query directly against that. The decision to do this requires explicit KT and consultation with the IT department e.g. MSFT are more likely to change the DB schema than the object model therefore any reports written against the DB schema could be rendered unusable with future SPs or releases. It will generally be less problematic however than the alternative.

Dumbing down of learning

2010-03-18T15:19:00.009Z

(Originally posted 18 February 2009).

Learning within the enterprise has always been a fragmented affair. There are dozens of specialised methodologies, packages and even languages devoted to it. It has had its own standards organisations e.g. SCORM and it seems to take several months to deploy a new training programme by which time, the processes/tools or methods it was describing are likely outdated. Actually attending training courses is also tedious; comprised of combinations of self and group study exercises and instructor directed sessions punctuated by role-play sessions or videos to add variety. Occasionally there are also group discussions which are intended to share the group’s experiences but are in practice a mechanism to update course content. E-Learning or Computer based Training (CBT) or whatever you term it, is over-intellectualised.

This is to say nothing of the huge cost (and risk) of fielding several resources (often from the same department) for two to five day training courses every couple of quarters (many times with additional hotel costs and associated travel/carbon footprint costs). This expenditure is routinely tolerated because organisations recognise investment in people is associated with lower employee turnover, which is associated with higher customer satisfaction, which in turn is a driver of profitability. On the employee side, attendance is justified because everyone gives broadly positive feedback at the end of the course because they don't want to rock the boat; they enjoy their periodic change-of-pace after all. Both parties are complicit in the charade.

Web 2.0 tenets of collaboration, user-generated content, crowd-sourcing and social networking have made it into the enterprise on, by comparison, shaky justification; efficiency savings per head (useless when the person does nothing business-like with the time saved), back-door routes where enterprising individuals have brought the tools in unmanaged (helped because many of them are open-source and free) or loose assumptions that Generation Y are fundamentally different to Generation X and expect these tools to be in their workplace or they will leave. These technologies are all about real-time sharing of information, commenting on each other’s content and utilising the inherent wisdom of crowds; in other words, the precise foundation for effective learning.

Rather than just knowing how to do something because they have been previously trained in it (perhaps some years ago), people will instead look to current solutions; contacting someone else who has done it before or failing that - picking through search results. They'll have a low chance of finding the support they need and either learn through their mistakes or give-up and escalate it to someone else. Both alternatives carry enterprise cost.

An alternative is redeploy efforts spent on training to build-in learning opportunities (based on Web 2.0) into operational solutions that people use each day. In this way, they don't have to search for knowledge (it is already there and contextualised for them). It will also more likely be what they need because it will have been created from others that do the same thing each day. Finally, because it is all in one place (the line-of-business systems), it will, (paradoxically for Web 2.0) be easier to control centrally (compliance, consistency etc.). Learning is not about remembering things from years past. It is about being able to find, validate, synthesize, leverage, communicate, collaborate, and problem solve with the facts, ideas, and concepts. In real-time.

This creates new enterprise challenges such as providing usable tools for the production of user generated content and providing an effective incentive mechanism (for them to produce the content). These are being addressed though. MSFT has recently announced Semblio - a product/SDK using .NET/WPF to develop collaborative learning material. The SDK is available now and the product part (for content creation by non-technical users) will come with Office 14. Other mash-up tools e.g. Popfly can also be appropriate for creating learning content and let's not underestimate the power of video podcasting. 5MP+ cameras (required for workable video quality) are almost main-stream. BT have taken a lead on pervasive, on-demand, pod-cast based learning through its Dare2Share programme.

In terms of incentivizing, management consultants are learning to culture the concept of microcelebrity within the organisation. In many cases, this can be met by technology; implementing (and publishing on the Intranet) a ranking mechanism for training content. In other cases, it will involve some BPR. Whichever route is taken, we'll all be happier even if we lose our little changes-of-pace.

UPDATE: Mahalo have recently pivoted toward consumer video-based learning. Although the goal of creating "thousands of original, high-quality videos each week" seems unrealistic.

PPS reshuffle

2010-03-18T15:14:00.003Z

(Originally posted 28 January 2009).

PPS has been reshuffled (http://blogs.msdn.com/sharepoint/archive/2009/01/23/microsoft-business-intelligence-strategy-update-and-sharepoint.aspx). PPS has been a successful MSFT technology. It has taken MSFT from a standing start to a player in the competitive PM market, to the point where Gartner’s last Magic Quadrant for PM showed MSFT in the Visionary category. As with Content Management Server previously, MSFT will consolidate PPS Monitoring and Analytics into their flagship platform MOSS from mid 2009. PPS Planning will be withdrawn. The core MSFT BI stack remains unchanged.

Most customers will actually benefit from this development. Customers that want PPS Monitoring and Analytics (the bulk of those that have any interest in PPS) should see a reduction in licensing costs: a MOSS Enterprise CAL is cheaper than a PPS CAL. Many customers will already own it.

Why did this happen though?

The target market for PPS Planning (in its current form) is saturated. Most organizations that need end-to-end PM already have solutions. MSFT had only a modest share of the PM application market. SAP, IBM and Oracle between them took around half. The market was also fragmented at the SME end with around half again being smaller vendors often based upon the MSFT BI stack e.g. Calumo.

Customers in newer PM sectors e.g. Insurance/Legal and the SME market were gradually beginning to adopt PPS Planning; but working with PPS models was arguably more of a technical activity than they were used to. PPS Planning lacked web-based data entry. The Excel add-in could be slow when connecting to PPS from Excel or publishing drafts. PPS was well served with complex financial consolidation options but this is a niche activity (MSFT make revenue from CAL licenses after all). MSFT needed to democratize the PM market but this needed associated change in business practice with planning activities becoming both decentralized and collaborative. It was unlike the other products in the MSFT BI suite and ultimately did not drive forward the core MSFT proposition of “BI for the masses” (http://www.microsoft.com/presspass/features/2009/jan09/01-27KurtDelbeneQA.mspx).

Finally, there's maybe an alliance angle. MSFT jointly launched Duet with SAP, allowing easy interaction with SAP and MSFT Office environments (esp. Excel and Outlook). There could possibly be a strategic direction here to add financial planning and consolidation to a future release of Duet (or some future combination of Duet, Gemini and/or Dynamics). This would tightly integrate MSFT and SAP at a PM level. What would be in this for SAP is immediately unclear given their acquisition of OutlookSoft (and incorporation into their own product suite as SAP Business Planning & Consolidation [BPC]) but there has been a significant rise in the last year of existing SAP customers wanting to bolt-on PPS/SSRS onto their SAP deployments; basically because it is difficult for in-house resources to extract the data from SAP and present it themselves or because quotes from SIs are routinely in the tens of thousands of dollars per report. Partnering with MSFT here and using PPS Monitoring and Analytics from within MOSS may improve SAP client satisfaction around data access. MSFT would similarly benefit from increased enterprise access.

Universal enterprise UX - Part 1 (Concept)

2010-03-18T15:11:00.006Z

(Originally posted 6 November 2008).

From a UX perspective, portals are a great way of centralising; personalising and publishing the various functions that a user needs to undertake operationally. They are ideal for combining both structured and unstructured data and contextualising between the two. We’re all essentially information workers and users do similar information-centric operations during their day e.g. browsing, analysing, contextualising, starting new events (based upon old events) and data entry. There should be the same UX available for them to do these actions. Not just a similar portal but the same (configurable) web parts.

Where are these gadgets? SAP has a clear concept of UX reuse that plays partly in this space but where are they for other platforms? There are a finite dozen or so UX interactions (Find, Alert, Link, Flag, Copy, Browse, Cascade, CRUD, Audit, Confirm, Search and Analyse) that can be mapped to six or less web parts. Two of these web parts would account for most UX interactions i.e. Web part 1 (Find, Alert, Link, Flag and Copy) and Web part 2 (Browse and Cascade). Web part 1 would mainly deal with lists and Web part 2 would mainly deal with hierarchies.

These could be mix and matched with social networking and content management web parts. That’s just six or so interrelated web parts that could handle the majority of bespoke operational applications today. For example, an order processing system user would be able to search for a particular user, see all their past orders (and correspondence), analyse their propensity to cross/up-sell and take or query their order and flag particular customers for a follow-up call. When the IT department roll-on new functions e.g. recording user feedback at POS, users will know how they work as the web parts would behave the same as existing functions. For the back office, ongoing development, training and testing of incremental functionality would be reduced. MOSS is already treated seriously as an application development platform (http://www.andrewconnell.com/blog/archive/2007/09/24/6116.aspx) as it handles the plumbing every application needs. Some organisations have worked on solutions where they have deployed a similar concept for customers (reusing the same web parts each time) but this approach only really pays dividends with incremental development or new solutions that use the same UX web parts. Why is the universal enterprise UX not more prevalent (?).