Paradigm Shifts and the Future of Data Infrastructure with Snowflake, Neo4j, and Cockroach Labs

We recently brought together Denise Persson, CMO @ Snowflake, Emil Eifrem, CEO @ Neo4j, and Spencer Kimball, CEO & Co-founder @ Cockroach Labs, to discuss the future of data infrastructure in the Cloud.

In undergoing a sudden shift this year to the cloud of the future, these three leaders discussed 3 main paradigm shifts:

#1 Everything is moving to the cloud

#2 Data is becoming a strategic asset

#3 We are in the midst of a big generational shift when it comes to data infrastructure

The truth is, most companies are already falling behind with the ramifications of 2020 and need to move quickly.

So how do we change our businesses by fundamentally exploiting the benefits of the cloud?

#1 You shouldn’t go with a cloud-specific platform service

Instead, opt for the independent service and be aware of company trade-offs.

#2 Avoid having any one vendor have significant leverage over your company

If you place all your eggs in one basket, it will be much harder to migrate to different platforms in case of change or exploitation.

As we grow and expand into this new cloud era, several companies have grown at a much faster rate. Skill sets required are shifting as the world becomes more automated and reliant on technology.

Compared to ten years ago, what are the new skill sets required for data technicians?

According to Cockroach CEO and Co-founder, Spencer Kimball, skill sets in some ways have actually been allowed to narrow. It once would take a specialist who knew the vast details of technology, now you just need to know the platform and how to manipulate data to reach your goals. Due to this, there has been a significant increase in the rate at which companies grow. Companies can get a lot further much faster because they are no longer responsible for as much of the deployment puzzle.

Additionally, business aspects are now much more involved in technology decisions. There is a clear demarcation when it comes to AI and ML and it really is divided into two stages:

#1: Where we once only saw “geeks” caring about data, it has branched into big data that everyone cares about. Eifrem even believes, “Data is the new oil”.

#2: Next, AI and machine learning came along and every single business executive ever wanted to digitally transform into a machine learning company. Many business owners didn’t even care about graphs five years ago.

These make it much harder to be a technologist with business aspects being constantly involved and relevant. However, today we are seeing many best of breed stacks. So how does this play into everything?

First, you must be aware of the considerations. We’re dealing with best of breed tech approaches vs. “I’ll just do whatever is built into my platform.” They are different versions of trade-offs and all depend on the risks you’re willing to take on. In theory, it enables things to be cheaper with plug and play services, and you can experiment with a much smaller radius in case something goes wrong.

According to CEO Kimball, there are three stages for this:

#1 SQL has been evolving for 40 years

Architectures were once monolithic. They all sat in one place and had to be scaled up.

#2 When web happened, a lot of innovation was necessary

Data building needed to occur that could handle global scales, web scales, and further. This kicked off the NoSQL movement.

#3 No scalability redundancy

We are now entering a new phase, where the relational SQL model is being married with NoSQL scalability redundancy.

Overall, there has been a remarkably fast evolution. This poses the question, How important is it for data infrastructure to be fully cloud native at this time with more apps moving to the cloud?

The truth is, you can’t switch fully to the cloud with a snap of a finger. It is super important for any new technology including databases to be more sympathetic to underlying cloud primitives. It is crucial you build a bridge for big enterprises and meet them where they are. If you aren’t planning for net new starts and increasing migrations, you’re going to miss the boat.

So where does open source play in the future?

It remains incredibly important. Open source took over because it is a far more efficient consumption model. You are now consuming services, and developers can work on API without having to actually understand how to run it. Some of the best services are based on open source and the best service has extremely powerful free tiers. There is a tight interplay between cloud and open source when it comes to developer-facing products like databases

The most important key takeaways:

#1 The digital transformation age has passed.

#2 Data Infrastructure is the backbone of modern business.

#3 Successful technology companies are developer and practitioner centric.