The Lambda Architecture

Nathan Marz gave this idea of Lambda Architecture as a holistic solution for Big Data Platform. It is not something groundbreaking, but the beauty lies in the way he has combined Batch Processing and Stream Processing into a single unified architecture and laid down some principles on which Big Data solutions may be based on.

Since its introduction (by Marz himself while working in Twitter) Lambda Architecture has been a popular topic of discussion in the Big Data circles. Many attribute to it as an architecture for real time data processing, but the real time aspect is just one part of the whole architecture as we will see in the following slides.

1. Background and Principles

2. Lambda Architecture Executive Summary

3. Analysis

4. Extending on top of Lambda

As I have shown briefly above, Lambda Architecture can easily be extended to integrate further capabilities such as complex event processing into the same solution. The magic lies in the ways one can generate the views which can be exploited to get real time, quasi real time or non real time insights into data.

Going further we can see that the Lambda Architecture also gives rise to two distinct kinds application platforms. One which allows the user to build and integrate the 'Platform Services' for the Batch and Speed Layers and another one which requires user to build the actual 'Application Services' which can exploit the views in appropriate manner and offer the final business logic. It needs to be seen how easily such platforms can be integrated together and may be a universal computing platform for Big Data will finally become a reality.