I had the privilege of attending the Keybanc Emerging Technology Summit this week in San Francisco as a member of their Mosaic group of technology experts. Buy-side investors asked me and my peers questions regarding trends that were impacting Amazon, Google, VMware, Azure, and other public companies. It was a blast being in sessions with the likes of John Willis of DevOps Handbook fame and Chris Wahl, the chief technologist at Rubrik, and others. I learned a lot from these and other peers and from the lines of questioning of many investors.
One theme that seemed to be under-appreciated at the conference that I thought was worth sharing here was the rise of the rest, by which I mean the emergence of best-of-breed solutions in the form of public companies like Okta, MongoDB, Elastic, Twilio, Atlassian and unicorns such as Hashicorp, Databricks, Datastax, Confluent / Kafka, GitLab, PagerDuty and others.
A few years ago, when discussing infrastructure investments with major investors in the valley, the feedback I received was almost always:
Once again, entrepreneurs were well served by listening to investor consensus — and doing the opposite.
How did we get here? What drivers allowed for this flowering of innovation and are they sustainable? Or will we, in the end, all end up working (if we are lucky) for Mr. Bezos?
Drivers:
Don’t tell the engineers how to do their job — focus on outcomes.
The rise of DevOps as a more-effective and humane approach to building and running software has brought with it admonitions from practitioners and thought leaders like John Willis that bosses should not pick the tools. This has meant that engineers have been able to pick the Amazon solution or whichever other solution that best fits what they need to do.
APIs Make Good Neighbors.
My two-pizza team is fine with collaborating with you and your two-pizza team as long as there is a wall between us, or at least a documented set of APIs that we both agree to. This level of clarity allows solutions to be frequently selected by these small teams because it avoids the need for meetings to argue over tools — meetings to negotiate vendor supplied tools and then the least common denominator configurations for those tools are anti-patterns.
Containers Contain Dependencies.
In a sense, what APIs do for limiting my dependency on the teams building other pieces of the application, containers do by limiting my dependency on bespoke dev and staging environments. If you and your team can build your code into a set of containers then you can be much more confident than ever before that they will work in operations. Once again — meeting avoided! Freedom achieved! No need to believe in the magic of a single vendor’s all-in-one suite to do most everything.
Kubernetes Ubiquitous Kubernetes
It turns out that while containers are great — if you are stuck with several different orchestration and operating environments that you need to either build or at least operate — you may feel beholden to just run the opinionated flavors of PaaS they have. For example, if your organization buys a big Pivotal subscription then you get extensive support for their flavors of software. That’s what you are expected to use; for example, instead of running your own PostgreSQL (still the favorite general purpose DB according to the 2018 StackOverflow survey) you might be told to use the modified Greenplum fork of PostgreSQL.
Perhaps just as important, open source projects couldn’t be sure what orchestration platform to support, not just as a target but also as a platform to deliver functionality. Now that Kubernetes has become the lingua franca of container operations, solutions like OpenEBS can use Kubernetes itself to deliver functionality, and hence are far easier to grok than external, shared-everything black box systems that were built well before Kubernetes and require special skills to understand, deploy and operate.
Communities Matter — Open Source
More than ever, engineers want technologies to do the one job they are expected to do (yes Unix like!). When these technologies behave strangely, they want to talk to the engineers that helped build them to figure out what is going on. This is one reason Open Source is winning — it is significantly more credible to many engineers not just because they can see the code, but also because they can directly communicate with the coders and learn with them better ways of solving the problems they face.
On the other hand, AWS obviously is credible as well. However, as Amazon has grown to become one of the largest companies in the world, even their hiring of hundreds of developer advocates has not fixed what is an inevitable issue — the folks writing the code are not hanging out on Slack as is the case with a successful open source project or even a SaaS start-up. Engineers trust engineers.
In summary, the Chevy v. the Tesla
While Amazon is willing to invest considerable resources to attack the apparent long-tail — such as running a Satellite Ground Station as a Service — and overall has compiled what many perceive to be the greatest run of software innovation of any large company ever — they still cannot expect to be the very best at everything they do. Instead, their value proposition often comes down to the fact that it works and fits into their existing billing relationships.
If you want a scale out in-memory database you can either select Redis from Redis Labs , which seems to be the best performing solution with open source and community benefits (Redis = Tesla), or you can grab Redis/Amazon ElastiCache as a service (yes, Amazon is running the open source Redis as a service without paying Redis or contributing back the software they use to keep it up and running — hence the strip mining narrative that arises now and again).
Similarly, if you want SQL based DBs, Amazon is more than willing to run existing open source code from communities such as PostgreSQL or MySQL for you. What you trade in cost and flexibility, you may gain in ease of use.
Yet another example is Spark. Once again Amazon has taken the solution and will run it for you as one of their EMR solutions: https://aws.amazon.com/emr/features/spark/ (Chevy)
DataBricks, on the other hand, has successfully added software as a service to the Spark project they helped create that is sufficient to build a mega unicorn:
https://databricks.com/spark/comparing-databricks-to-apache-spark (Tesla)
If you want block storage that works and is well-integrated into your AWS environment, you pick EBS. If you want block storage that is natively integrated into Kubernetes and runs the same wherever Kubernetes runs — whether running on EBS itself or on direct-attached or on ye olde legacy SAN — you pick OpenEBS (definitely Tesla). Here is a case, like Minio in object storage, where open source communities are taking a product category that Amazon helped create and making it open source. This is yet another flavor of best-in-class solutions flowering despite, and maybe even because of, the rise of AWS.
What would bring us back to a time of less innovation?
I leave it to you to determine whether the trends such as DevOps, APIs, containers, Kubernetes, and Open Source will persist. I certainly believe that they will because they deliver a competitive advantage to organizations that find software to be an important part of their value proposition.
On the other hand, if we swing back towards centralization based on illusions such as it being faster moving or a lack of security in less centrally controlled organizations, then IT may again become the provider or least common denominator solutions, and top-down sales may proliferate again. Ironically the very motion that Amazon once fought against or flanked and avoided — the top-down sales motion — may now be their best chance at retaining account control and outsized sales growth in competition with best of breed solutions.
The flywheel spins faster….
With this, I hope you agree that we’ve entered a golden age of software development and operations. The advocacy and vision of DevOps visionaries combined with many trends, including many more than those I mentioned here, as practiced by millions of dedicated practitioners have led us to a world in which engineers are increasingly free to decide which solutions to use. As a result, we see new solutions making it into the hands of practitioners faster than ever before because they do a better job of solving real-world problems than more complete suites from Amazon, Pivotal or other all-in-one solution providers. Demand for better solutions creates more supply, which enables these organizations to move faster and afford more best of class solutions, and so on. The flywheel spins faster.
Your feedback and insights would certainly be appreciated. Respond to me via Twitter, our OpenEBS Slack channel, or start a thread and argue about it on Hacker News or any other outlet. We’re always listening and look forward to learning from you.
This article was first published on Mar 1, 2019 on MayaData's Medium Account.