Where did those mash-up tools go?

Just eighteen months ago, mash-up tools were big. The promise of building applications quickly, with minimal development and with context directly reflected within the application (as they are made by SMEs rather than IT resources) remains appealing. They looked to be the perfect tool for civic activists and knowledge workers alike. They were high in Garner’s top ten technologies to watch. The enterprise was starting to take them seriously as a mechanism to reduce crippling data integration challenges and consumers - bored on a diet of pushing links - thought they would be fun and/or a showcase for themselves in much the same way as blogs have been.

Now, the wind has shifted and MSFT’s Popfly and Google’s Mash-up Editor are both gone. Other niche vendors e.g. Sprout Builder similarly disappeared. Of the big players, Intel Mash-up Maker and Yahoo Pipes continue. All of them attempt/have attempted to straddle the void between consumer and enterprise spaces. This is an important distinction since – mash-ups even within the enterprise rely upon an ad-hoc passionate approach rather than formal development. They are typically built at home by passionate non-programmers who want to invest time in a single non-niche tool so whatever they learn is portable (work and social/other organisations etc). Even more so than blogging (which also uses one tool across both domains) a single tool is required - as more learning investment is required. With the exception of SAP’s Visual Composer (if you are an SAP shop), none of the tools mentioned have been particularly successful in either space let alone both. Why is this?

1) No UX standards. For both enterprise and consumer, there are no standard for widgets (or gadgets or web-parts or whatever else you call discrete self-contained UX functions). There are standards for business cards (vCard) – why not widgets?
2) Slow linked open data adoption. More of a consumer inhibitor at the moment. Linked Data is a core component of the Semantic Web vision that uses a specific set of current technologies. It provides a way to readily mix-and-match data in a meaningful way and so is a key enabling technology for mash-ups. Sig.ma is a simple mash-up tool for RDF data. Unlike other data-based mashups which tend to be query-based, Sig.ma is search-based. You enter a search term, the search engine gets your data, you remove the bits that are not relevant and (if you like) re-publish the data again as RDF (or other formats). This is perhaps too simplistic for users right now but it is evolving and could become a potent research tool. Any mash-ups that rely upon open linked data (ideally the best data of all) suffer from a lack of it; although this is changing as Government initiatives in particular publish RDF data for transparency reasons.
3) Insular data integration. Although the various flavours (SOA/ETL/EII/EAI) have been core CIO agenda topics for over five years, they have been mainly confined to the particular enterprise itself; especially in the narrow form of web services and have been of limited success even there. Extranet take-up, where common data is shared between parties in the supply chain, has been leisurely and this is precisely where mash-ups are needed. Very few organisations treat meta-data with the same focus as data. This means, mash-ups have trouble vouching for data currency and lineage which detracts from user take-on. It is possible that the solution to data integration will be the Semantic Web and a greater openness of organisations to share data. If so, we will be waiting some years yet.
4) Industry standards. Have been slow to be adopted. A notable exception here is XBRL for common reporting.
5) SSO. This is not exactly an issue for consumer mash-ups; assuming you are using open linked data but it is still a huge inhibitor to data integration for many organisations.
6) Blogging comparison. Although superficially similar to blogging - mashing (if you are going to do it properly) requires a thorough data understanding and a lot more effort than committing stream-of-conscious thoughts before they float away into the ether (or linking to other peoples work). Blogging is simply an easier way to achieve microcelebrity and also - because the majority of posts are written in the first-person (I think...) they can be defended (if need be) by the simple statement - "These are my opinions". This is a segueway into a whole minefield of philosophy, politics and culture that is best left alone. People mainly do. Only a small proportion will directly challenge someone's written thoughts. Publishing a mashup however - where you are vouching for the legitimacy of the data opens you up to direct challenge (people may have provably better data) and so people resist it. Only when the number of single versions of truth in the world become smaller and more consolidated will this situation change.

Popfly showed early promise as a learning tool but never really got past being a Silverlight showcase. Its focus was on looking good (geo mash-ups and slick drag/drop) rather than data integration. It did not use RDF at all. MSFT have been slow to utilise semantic web approaches in general. Some of their media management technologies use RDF in the background but their main focus has been around semantic search through the Semantic Engine. This initiative uses recently acquired Powerset technologies and will be released through SQL Server. PowerPivot has been significantly downsized from its original Project Gemini remit which would have provided not just a potent reporting mash-up environment but the management and support processes and infrastructure to QA and promote the mash-ups throughout the enterprise. This latter point is a key inhibitor to mash-up growth in general.

There are still signs of life in the mash-up tool space. Dapper is advertising focused. NetVibes is portal focussed. Alchemy API takes a content management/annotation approach (similar to Intel Mash-up Maker). Birst takes an analytic portal approach. Jackbe looks interesting; it appears to take a sales analytics approach. Snaplogic is not exactly a mash-up tool but it certainly takes a non-technical approach to data integration. None of them really play in that sweet-spot between enterprise and consumer though.

The parallel economic downturn has influenced mash-up take-on. The enterprise essentially stopped unproven development and consumers have yet to be sold on the concept but let’s face it – the focus on one or two drop-downs for configuration and Google Maps didn’t help either. The future of mash-ups is secure because it is the future of building useful applications quickly by SMEs and that will always be desirable. Five back-end data sources (database, RSS etc.) linked to five middle-ware components (aggregation, integration etc.) and five front-end components (analytics, data entry etc.) generates 125 possible combinations of application straight off the bat. Adding tailoring through filtering, personalisation and general configuration takes in into the thousands. This simple logic guarantees a future at least in the enterprise.

Whether the name “mash-up” has been tainted by its recent hiatus and – like its raison d’etre will need to resurface as part of something new remains to be seen. We can be sure though that the next generation of mash-up activity needs to be three things in order to stick around; interactive, data focused and usable in both enterprise and consumer spaces.

Floating up to the Spatial Web

(Originally posted 20 April 2009).

Google Street View is an option in Google Earth that shows sequential 2D images down the world’s main streets. This tool provides for a simple 3D effect if you traverse up or down a particular street. Photosynth is MSFT Live Labs software that creates a 3D effect from multiple 2D images of the same scene taken from different angles. You can zoom in/move around the scene if there are enough 2D images of it available. It is more sophisticated than Street View since it is not tied to a particular vector i.e. a street and also because it performs image extrapolation to complete partial views.

Moving through alternate realities like this holds a natural attraction for people; training, POS, medical diagnosis, national security, gaming, film-making, virtual tourism and urban planning would all immediately benefit. Photographs of existing places and objects are increasing at a huge rate through (camera phone) image uploads to social networking sites and the thousands of commercial and Government camera installations throughout the world. There will surely eventually be blanket photograph coverage of pretty much everything everywhere (and once that is achieved - at every time). Geo-tagging will help connect these pictures together.

These technologies basically link photographs of existing scenes up. Your brain mainly adds missing spatial orientation. There is no 3D model of structures behind them unlike say as with Google Sketch-up, AutoCAD or the various First Person Shoot-em-up (FPS) games e.g. Call of Duty etc.

Does there need to be a 3D model behind them? You could ignore surfaces/obstacles, simply walking through them as a disembodied ghost. This might be acceptable for many applications but for gaming it will not. 3D models are time consuming to plot/maintain with changing actual landscapes. The reason why models are created now is primarily for edge detection e.g. you see a wall and because the 3D model has been defined at design-time to identify it as a wall, you are prevented from moving through it. It is much better to do edge detection at run-time using some algorithm to identify the wall as an edge (or barrier), preventing it from being moved through. Gaming will likely be the driver for this new standard. This will be accessible through some cloud-based service and other applications will simply adopt it.

Development of the Internet has shown us that competing infrastructure technologies will co-exist for a while but they will eventually “float-up” to the highest common denominator e.g. the most extensible, cost-effective, egalitarian and “fit-for-purpose” technology. Applications built on selected-out infrastructure will eventually switch to the new standard. It is easy to see a gaming driven cloud-based service, regularly updated with near-real time images as being adopted by all other applications requiring spatial understanding.

In this way, we will see a single alternate reality develop where synergies and opportunities are created by markets colliding (as with actual reality) and the “Spatial Web” described last year by MSFT’s Craig Mundie will become normal (if not “real”!).

UPDATE: Swiss Computer Scientists use Flickr to do this now http://www.openculture.com/2010/12/3d_rome_was_built_in_a_day.html.

Virtual Earth & SSRS Integration

(Originally posted 27 March 2009).

Virtual Earth integration with SSRS can be a challenge due to uncertainty around approach and a general lack of resources on the topic. Developers can use three Virtual Earth APIs: the MapPoint Web Services, the Virtual Earth Map Control and the Virtual Earth Web Services. SSRS cannot currently use the Virtual Earth Map Control since this allows users to make requests via JavaScript to an AJAX map object and SSRS will not allow JavaScript to run (as otherwise it will not be able to export to PDF amongst other things). This would have been preferable since it affords a level of interactivity e.g. zooming-in on tiles. This leaves either the MapPoint Web Services or the Virtual Earth Web Services. Both methods effectively work the same in that points to plot are passed to the web service, rendered on the map server-side and passed back to the browser as an image. MSFT no longer develops MapPoint Web Services (although existing applications built using MapPoint Web Services will remain functional) so if you want Virtual Earth integration with SSRS, your only real choice is Virtual Earth Web Services. Here is a snippet on how to pass coordinates (Longitude/Latitude) to Virtual Earth. This should display a pushpin on the map (The Imagery Service is used).

You will need a Virtual Earth Platform Developer account and this itself will need a Windows Live ID. Once you have this, the SSRS report should call a function in a C# custom assembly, passing in an MDX query string to retrieve co-ordinates. The SSRS report only calls the custom assembly. The output of the function will be a URL for the map image to display on the report. The custom assembly queries the database/cube to retrieve co-ordinates for the points to plot. It should then call Virtual Earth Web Services passing through the co-ordinates to be plotted. Finally, the GetBestMapView method should be used within the assembly to scale the image appropriately e.g. if all points are plotted in the South East of England, the image needs to zoom in to the South East of England and not the whole of the UK.

Universal enterprise UX - Part 1 (Concept)

(Originally posted 6 November 2008).

From a UX perspective, portals are a great way of centralising; personalising and publishing the various functions that a user needs to undertake operationally. They are ideal for combining both structured and unstructured data and contextualising between the two. We’re all essentially information workers and users do similar information-centric operations during their day e.g. browsing, analysing, contextualising, starting new events (based upon old events) and data entry. There should be the same UX available for them to do these actions. Not just a similar portal but the same (configurable) web parts.

Where are these gadgets? SAP has a clear concept of UX reuse that plays partly in this space but where are they for other platforms? There are a finite dozen or so UX interactions (Find, Alert, Link, Flag, Copy, Browse, Cascade, CRUD, Audit, Confirm, Search and Analyse) that can be mapped to six or less web parts. Two of these web parts would account for most UX interactions i.e. Web part 1 (Find, Alert, Link, Flag and Copy) and Web part 2 (Browse and Cascade). Web part 1 would mainly deal with lists and Web part 2 would mainly deal with hierarchies.

These could be mix and matched with social networking and content management web parts. That’s just six or so interrelated web parts that could handle the majority of bespoke operational applications today. For example, an order processing system user would be able to search for a particular user, see all their past orders (and correspondence), analyse their propensity to cross/up-sell and take or query their order and flag particular customers for a follow-up call. When the IT department roll-on new functions e.g. recording user feedback at POS, users will know how they work as the web parts would behave the same as existing functions. For the back office, ongoing development, training and testing of incremental functionality would be reduced. MOSS is already treated seriously as an application development platform (http://www.andrewconnell.com/blog/archive/2007/09/24/6116.aspx) as it handles the plumbing every application needs. Some organisations have worked on solutions where they have deployed a similar concept for customers (reusing the same web parts each time) but this approach only really pays dividends with incremental development or new solutions that use the same UX web parts. Why is the universal enterprise UX not more prevalent (?).