Data Lakes and the Responsive 21st Century Corporation

Data Lakes and the Responsive 21st Century Corporation

Yammer’s founder is introducing a new way of organizing the corporation. His ideas originate from earlier ideas from the Lean and Agile methodologies of software development. In his Responsive Manifesto he drives a case for a new kind of efficiency that will drive the successful workplaces of the future.

In a article “How Yammer’s Co-founder Impressed Bill Gates“:

Flash forward to 2015, when the future is more unpredictable than ever. The connectivity we’ve achieved over the last decade has changed everything. “We moved from a world of information scarcity to a world of information ubiquity,” Pisoni says. Consumers are learning, sharing, adapting — and changing their expectations more rapidly. “The world formed a giant network. And that has accelerated the pace of change to crescendo.”

By breaking down hierarchy and conducting smaller-scale, cheaper experiments, you can dramatically reduce the cost of failure and ultimately make your process both more responsive and more efficient.

The Responsive Manifesto
declared the following principles:

  • Purpose over Profit
  • Empowering over Controlling
  • Emergence over Planning
  • Networks over Hierarchies
  • Adaptivity over Efficiency
  • Transparency over Privacy

So let me explain the value of a Data Lake strategy in the context of enabling a more responsive organization:

Empowering over Controlling

The legacy Database administration organization limited the control and understanding of data to a few experts. This lead to an extremely time consuming and rigid process that required the continuous participation of the gatekeepers of the data.

Today, circumstances and markets change rapidly as information flows faster. A Data Lake’s self-service capability enables employees with the best insight and decision-making ability to easily access data of the company to gain better insight. Rather than controlling data through process and hierarchy, you achieve better results by empowering people at the edges.

Emergence over Planning

In a highly unpredictable environment, plans start losing value the moment they’re finished. Embracing agile methods that encourage experimentation and fuel rapid learning is a much better investment that spending too much time upfront planning.

In Data Lake’s the upfront investment typically found in Data Warehouse deployments of designing a universal canonical schema is done away with. The costs to continuously update this schema and corresponding ETL scripts in the an rapidly changing environment is removed. Data is ingested in a Data Lake in the most automated way possible. This empowers people at the edge to rapidly gain insight on new data.

Networks over Hierarchies

Data Lakes provide technology and connectivity to increase the ability to self-organize, collaborating more easily across internal and external organizational boundaries. Typical enterprise “Data Silos” are demolished as all data is made available in a single BigData store.

Adaptivity over Efficiency

A Data Lake is designed for change and continuous learning. Rather than seeking consistency, adaptive systems increase learning and experimentation, in the hopes that one novel idea, product, or method will be the one we need in the new world.

Transparency over Privacy

An enterprise has its data guarded by many different organizations. Data is hard to come by and hard to disseminate across the organization. A Data Lake provides access to data across silos because it is impossible to predict which data might be useful.

2 thoughts on “Data Lakes and the Responsive 21st Century Corporation

  1. Very nicely put.

    You clearly understand the concept of the Data Lake concept and its advantages, and are able to communicate it well.

    James Dixon
    Lord of the 1s and 0s, Pentaho.

    1. Thanks for the compliment.

      The notion of a “Data Lake” that goes beyond just a tactical approach to data integration and an entire agile methodology has promise indeed.

      Hope to be able to discuss this further with you.

Leave a Reply

Your email address will not be published. Required fields are marked *