It\u2019s been some time now since Nathan Marz wrote the first Lambda Architecture post<\/a>. What has happened since then? How has the community reacted to such a concept? What are the architectural trends in the Big Data space, \u00a0as well as the challenges and remaining problems?<\/p>\n Despite the attractiveness of a dual batch \/ real-time architecture, there exists a wide variety of problems in Big Data for which a batch layer is good enough<\/b>, and I think it will continue to be so.<\/p>\n The consolidated adoption of Hadoop,<\/strong>\u00a0together with the dramatic improvement of its available tools, makes it today often the main architectural requirement for solving many Big Data challenges. With SQL-on-Hadoop tools such as Impala<\/a>\u00a0or Apache Drill<\/a>\u00a0it is no longer the case that data entering Hadoop is not rapidly actionable. Operational applications \u2013 e.g. involving recommendations, long-history analytics, client segmentation \u2013 benefit from a richer ecosystem, resulting in shorter development cycles; as well as improved hardware with better underlying software \u2013 therefore shorter batch cycle times<\/strong>. More frameworks to choose (Spark<\/a>, Stratosphere<\/a>) and future paradigms like Tez<\/a>\u00a0(remember Dryad<\/a>?) complete the batch-processing scenario.<\/p>\n (The story of Spiderio\u2019s architectural evolution came into my mind<\/a>, switching from real-time to batch at some point.)<\/p>\n Targeted to businesses where real-time is crucial, we are starting to see interesting real-time solutions such as Druid<\/a>. Stream processing and NoSQL-centric solutions remain a trend (remember Facebook\u2019s move to HBase<\/a>?). Storm improved its default API by adding Trident<\/a>, which adds exactly-once semantics \u2013 and Google announced Millwheel<\/a>, which, as usual, seems to take the state-of-the-art one step further. Add Spark Streaming<\/a> to the mix.<\/p>\n Still c<\/strong>hallenging to think about the human fault-tolerancy<\/b> guarantees of such systems, their complexity and their overall price. And interesting to see how they kind of meet in the middle<\/i>, incorporating batch storage \/ indexing<\/b> sometimes as part of their architecture.<\/p>\n What seemed once impossible, is starting to be achieved: developing Lambda Architectures using a \u201cunified\u201d framework<\/b> which makes the same code available to both the real-time and batch layer, and combines the results of both layers transparently. Two tools are in the scope for achieving that:<\/p>\n With Lambdoop not yet released and Summingbird being a very recent release, the usefulness of these frameworks is yet to be seen. It might be the case that they are too complex or restrictive<\/b>, or that they evolve naturally into something very usable. By now the kind of operations that one can implement using these frameworks seem somehow restricted by the nature of the frameworks themselves, and the final data serving one can expect is restricted to pure key\/value serving<\/b>.<\/p>\n What still seems to me the most common case of Lambda Architecture, is a \u201cliteral\u201d implementation of it where each layer operates independently and results are merged depending on the business nature<\/b>.<\/p>\n When things like complex documents and search-engines<\/strong> come into play, it becomes harder to reason about an hypothetic \u201cunified\u201d framework that would provide the guarantees and semantics of a batch-processing layer and the immediateness of a speed layer. The guys at Trovit made their system coordinate a batch and speed layer using Zookeeper<\/a>.<\/p>\n As another example, at Datasalt we implemented analytics for Ad Networks<\/a> using Splout SQL<\/a>as the batch read-only datastore. This allows us to precalculate less things and calculate real-time aggregations on-the-fly i.e.\u00a0ad-hoc drill-downs<\/strong>. Tied with Splout, it is very easy to deploy a real-time SQL layer that compliments it using well-proven technologies such as MySQL, keeping in mind that the hardest problems are already solved by the batch layer.<\/p>\n I also remember the guys at GameDuell somehow managed to mix Hadoop and VoltDB bi-directionally<\/a> with quick, few-minutes model re-calculations.<\/p>\n With things like Yahoo\u2019s Storm-Hadoop for YARN<\/a>, and Big Data tools being nowadays easier and easier to install and manage (e.g. Cloudera Manager or Apache Ambari making life much easier) it will become less of a pain to deploy an open-source Big Data architecture, implement batch-processing and colocate it with real-time processing. What\u2019s more, as stated, many cases will still be easily solvable just by batch processing. And as we have seen, newer real-time processing frameworks may already provide re-indexing \/ re-processing semantics underneath.<\/p>\n Because of the many possible use cases, data natures and serving requirements, it is unclear to me whether frameworks like Summingbird or Lambdoop will be key to implementing Lambda Architectures. But who knows, m<\/strong>aybe the best is yet to come!<\/b><\/p>\n source: datasalt.com<\/p>\n","protected":false},"excerpt":{"rendered":" Lambda Architecture: A state-of-the-art It\u2019s been some time now since Nathan Marz wrote the first Lambda Architecture post. What has happened since then? How has the community reacted to such a concept? What are the architectural trends in the Big Data space, \u00a0as well as the challenges and remaining problems? Big Data: Batch processing-only Despite\u2026 Read More »<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-614","post","type-post","status-publish","format-standard","hentry","category-architecture"],"_links":{"self":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/614","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/comments?post=614"}],"version-history":[{"count":1,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/614\/revisions"}],"predecessor-version":[{"id":615,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/614\/revisions\/615"}],"wp:attachment":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/media?parent=614"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/categories?post=614"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/tags?post=614"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Big Data: Batch processing-only<\/h3>\n
Big Data: Real-time processing-only<\/h3>\n
\u201cUnified\u201d Lambda Architectures<\/h3>\n
\n
\u201cFree\u201d Lambda Architectures<\/h3>\n
The future<\/h3>\n