Big Data And The Open Source Model - Can This Marriage Be Saved?

It is amazing how many open source software companies out there are trying to get hit by the same $1B bolt of lightning that hit MySQL without realizing that the MySQL result is not repeatable.

Looking at the current batch of big data high flyers, from TenGen to Cloudera to Horton Works, each seems to be vying for the same kind of ubiquitous usage that enabled MySQL to get a more than 20x multiple. What they don't realize is that the failure of early open source acquisitions to deliver substantial value to owners has made buyers much more wary.

Companies like MySQL were valued based on a mystical belief that downloads could be monitized (not unlike the similarly wishful belief in monetizing eyeballs that motivated disastrous dot com acquisitions in the 90s). Moving forward, open source companies will be valued the old-fashioned way: by the viability of their business model.

Here are the top three places most big data open source companies are missing the boat:

1. Prioritizing business model behind buzz: although buzz is critical for adoption growth, a viable business model trumps all in positioning a company for IPO or acquisition. First and foremost, this means being able to charge significant prices for add-on product pieces that customers want, such as security, clustering and monitoring.
2. Confusing services with sales: low margin services revenues are no substitute for high quality license revenues. More importantly, companies that build up large services teams often neglect to fully integrate their product, as product integration provides a driver for services engagement. This lack of product maturity in turn prevents customers from being willing to pay much for the product itself - a classic vicious cycle.
3. Hoping for a desperate buyer: companies that purchased open source players have by and large to translate open source leadership into commercial market share. The open source downloads generate lots of buzz but little license revenue, saddling their owners with an expensive, services-led business. In the immortal words of Mitt Romney, hope is not a strategy (although it *did* turn out to be an ok strategy for the incumbent in that case).

The Genius, The Conductor and The Bureaucrat

No, this is not a joke about three guys walking into a bar but the result of some recent musing about how the art of management is practiced in Silicon Valley.

The classic Silicon Valley stories often feature what Jim Collins calls "the genius with a thousand helpers" (from his book Good to Great). Steve Jobs, Larry Ellison and many other valley icons were known for their vice-like control over all aspects of their business.

When that Genius individual really is the smartest person in the world, you get the iPhone. When they are not, you get Palm's WebOs. Working for a boss who always has to be the smartest man in the room is a humbling experience but at least you know where you stand - at the bottom.

The contrast to the Genius is the conductor, a person who - without playing an instrument themselves - is judged purely on their ability to draw great performances out of others. This is the idealized, servant CEO that is touted in all of the business school texts but seen much less frequently in the wild.

Examples of the Conductor style of leadership would include people like Paul Maritz of VMware. In my experience, there is nothing in the work world that beats the thrill of working with a committed team on big, hairy, audacious goals where the person leading the charge is focused purely on helping the team win.

The third category is the Bureaucrat. The thing to remember about Bureaucrats is that what they are best at producing is more Bureaucrats. These are people who are always overwhelmed with work but never make decisions that would offload that work. In a way, they follow the same model as the Genius, in that all decisions have to come through them.

The goal for all CEOs should be to aspire to play the Conductor role, while realizing that it is human nature to slip into Genius and Bureaucrat now and again.

PaaS *Is* Automation

Cloud computing has a challenge endemic to many Silicon valley advances - a great technology triumph somewhat disconnected from a clear business benefit.

The developer version of cloud computing is PaaS (Platform as a Service). Like cloud computing in general, PaaS has struggled to articulate clearly why it deserves to capture the hearts and minds of enterprise developers

Hipper web developers have had no such hesitations and have collectively leapt to cloud computing platforms like Heroku and Cloud Foundry for Ruby on Rails.

This difference in adoption tells an important story. By and large, enterprise developers have spent years building highly automated toolchains centered around tools like Eclipse and Clearcase and targeting both desktop and web clients.

In contrast, platforms like Ruby on Rails were designed for web deployment and web tooling, so fit the online/PaaS model more naturally.

I would argue that from an enterprise developer's point of view, PaaS is just about automation. When a PaaS appears with associated tooling that makes it easier for enterprise developers to do their work in the cloud than on their desktop, we will see a big spike in adoption. At a minimum, this will require the following:

1. Seamless integration with existing tools: developers will want an easy on-ramp that doesn't require them to abandon existing tools like Eclipse, Clearcase
2. Automated build and test: this is where an integrated cloud tool chain can really rock, but only if it is easier to use than existing internal solutions.
3. Opinionated client stack: one of the biggest things holding Java back for full web development is that Java is not opinionated about how to build a client stack in the way that for example Ruby on Rails is. This means every development team has to come up with its own scaffolding and build process, making it difficult to deliver an automation solution that satisfies.

Of these three, the third is the real show stopper. More on this later.

ps. I thought briefly about coining a new acronym, PaaS *Is* Automation Stupid, but decided that we have enough acronyms in this space.

Evolutionary and Revolutionary Clouds

Now that we are a couple of years into the great cloud journey is it pretty clear that the big bang theory of cloud conversion is ain't happening.

Yes, ISVs are moving rapidly to the SaaS model and it would be hard to find a software startup who is *not* starting in the cloud, but enterprise adoption of the public cloud is happening at a more stately pace.

In large part this is due to the simplification required to make public clouds efficient and the complexity that characterizes most enterprise IT environments.  To put it differently, the public cloud makes app deployment simple by pruning app deployment options to the point that few enterprise applications can fit.

Moving forward, I see two paths for cloud adoption: evolutionary and revolutionary.

• Revolutionary cloud: Public clouds like Amazon EC2 and CloudFoundry.com represents a revolutionary leap forward for companies that are willing/able to abandon their current platforms. The revolutionary cloud offers a high degree of operational productivity at the expense of service choice (e.g., you can have any color you want as long as its black).
• Evolutionary cloud: public/private clouds like VMware's vCloud Director enable enterprises to get cloud benefits (public/private deployment, low upfront cost, elastic scaling, self-healing) without having to make major changes to their application architecture. The evolutionary cloud offers a lower level of productivity with a greater range of choice (e.g., you trade of productivity for flexibility).

Over time, the revolutionary cloud will offer more choice and flexibility while the evolutionary cloud will offer higher automation. Some questions for enterprise developers to answer as they move along this path include:

1. How much control do I have over the deployed application environment? The more flexible the deployment environment, the easier it is to move that application to the public cloud.
2. How do I move applications between different clouds? Having a way to move applications between evolutionary and revolutionary cloud architectures is just as important as being able to move apps between different flavors of public clouds

Building Killer Apps with Big Data

One thing that gets lost in the general Big Data hubbub is the critical question of apps. Big Data can provide stunning business insights but unless those insights are embodied into an application that can galvanize new business behaviors they are not worth much.

VMware has been a thought leader in the area of cloud application platforms for some time. Now we are turning our attention to the intersection of Big Data and Cloud Computing.

What does it take to build applications that can move easily between private and public cloud while accessing data inside and outside of the firewall?

In particular, what are the best practices for building cloud applications that leverage big data? Here are some of our initial thoughts:

1. Lightweight services: REST is the new SOA - lightweight services form the basis for supporting web front ends while pub/sub messaging like RabbitMQ forms the basis for back end workflow and transactions.
2. Mobile-first UI: the twitter bootstrap library finally enables developers to build HTML5 apps for Mobile devices that scale beautifully to tablet and browser-based desktops.
3. Fast data: scaling the front end of the application often requires in-memory data management. The easiest way to interact with core data is through a SQL interface such as SQLFire.
4. Big Data: knowing what is going on right now takes fast data, knowing what to do about it takes access to large amounts of historical data. The key here is to provide integration between the two data sources so that the data warehouse is kept as up to date as possible with the in-memory database.
5. Application-level management: managing application performance as a series of logical tiers rather than physical instances eliminates a great deal of complexity for systems admins.
6. Cloud deployment: automated, dev/ops solutions like Application Director take the black magic out of large scale systems deployment, collapsing a multi-day deployment sequence into a few minutes of scripted wonder. 
7. Elastic scaling: a core value of cloud computing environment like Cloud Foundry is sizing the compute resources to the task at hand - when demand is high, the resources scale up and vice versa.
8. Self healing: cloud means never having to say you're sorry that your web site went down because a component croaked and couldn't restart - again, Cloud Foundry comes to the rescue.

I will be discussing how to build killer apps for big data at the GigaOm Structure conference at the end of this month along with Tom Roloff, COO, EMC Consulting.