Building Killer Apps with Big Data

One thing that gets lost in the general Big Data hubbub is the critical question of apps. Big Data can provide stunning business insights but unless those insights are embodied into an application that can galvanize new business behaviors they are not worth much.

VMware has been a thought leader in the area of cloud application platforms for some time. Now we are turning our attention to the intersection of Big Data and Cloud Computing.

What does it take to build applications that can move easily between private and public cloud while accessing data inside and outside of the firewall?

In particular, what are the best practices for building cloud applications that leverage big data? Here are some of our initial thoughts:

1. Lightweight services: REST is the new SOA - lightweight services form the basis for supporting web front ends while pub/sub messaging like RabbitMQ forms the basis for back end workflow and transactions.
2. Mobile-first UI: the twitter bootstrap library finally enables developers to build HTML5 apps for Mobile devices that scale beautifully to tablet and browser-based desktops.
3. Fast data: scaling the front end of the application often requires in-memory data management. The easiest way to interact with core data is through a SQL interface such as SQLFire.
4. Big Data: knowing what is going on right now takes fast data, knowing what to do about it takes access to large amounts of historical data. The key here is to provide integration between the two data sources so that the data warehouse is kept as up to date as possible with the in-memory database.
5. Application-level management: managing application performance as a series of logical tiers rather than physical instances eliminates a great deal of complexity for systems admins.
6. Cloud deployment: automated, dev/ops solutions like Application Director take the black magic out of large scale systems deployment, collapsing a multi-day deployment sequence into a few minutes of scripted wonder. 
7. Elastic scaling: a core value of cloud computing environment like Cloud Foundry is sizing the compute resources to the task at hand - when demand is high, the resources scale up and vice versa.
8. Self healing: cloud means never having to say you're sorry that your web site went down because a component croaked and couldn't restart - again, Cloud Foundry comes to the rescue.

I will be discussing how to build killer apps for big data at the GigaOm Structure conference at the end of this month along with Tom Roloff, COO, EMC Consulting.