{"id":616,"date":"2015-04-22T10:48:27","date_gmt":"2015-04-22T10:48:27","guid":{"rendered":"http:\/\/adriangrigoras.com\/blog\/?p=616"},"modified":"2015-04-22T10:49:21","modified_gmt":"2015-04-22T10:49:21","slug":"lambda-architecture","status":"publish","type":"post","link":"https:\/\/adriangrigoras.com\/blog\/lambda-architecture\/","title":{"rendered":"Lambda Architecture"},"content":{"rendered":"<p><a href=\"https:\/\/twitter.com\/nathanmarz\">Nathan Marz<\/a> came up with the term Lambda Architecture (LA) for a generic, scalable and fault-tolerant data processing architecture, based on his experience working on distributed data processing systems at Backtype and Twitter.<\/p>\n<p>The LA aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being able to serve a wide range of workloads and use cases, and in which low-latency reads and updates are required. The resulting system should be linearly scalable, and it should scale out rather than up.<\/p>\n<p>Here\u2019s how it looks like, from a high-level perspective:<\/p>\n<p><a href=\"http:\/\/adriangrigoras.com\/blog\/wp-content\/uploads\/2015\/04\/la-overview_small.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-618\" src=\"http:\/\/adriangrigoras.com\/blog\/wp-content\/uploads\/2015\/04\/la-overview_small.png\" alt=\"la-overview_small\" width=\"600\" height=\"342\" srcset=\"https:\/\/adriangrigoras.com\/blog\/wp-content\/uploads\/2015\/04\/la-overview_small.png 600w, https:\/\/adriangrigoras.com\/blog\/wp-content\/uploads\/2015\/04\/la-overview_small-300x171.png 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<ol>\n<li>All <strong>data<\/strong> entering the system is dispatched to both the batch layer and the speed layer for processing.<\/li>\n<li>The <strong>batch layer<\/strong> has two functions: (i) managing the master dataset (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views.<\/li>\n<li>The <strong>serving layer<\/strong> indexes the batch views so that they can be queried in low-latency, ad-hoc way.<\/li>\n<li>The <strong>speed layer<\/strong> compensates for the high latency of updates to the serving layer and deals with recent data only.<\/li>\n<li>Any incoming <strong>query<\/strong> can be answered by merging results from batch views and real-time views.<\/li>\n<\/ol>\n<p>Source:\u00a0lambda-architecture.net<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Nathan Marz came up with the term Lambda Architecture (LA) for a generic, scalable and fault-tolerant data processing architecture, based on his experience working on distributed data processing systems at Backtype and Twitter. The LA aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being\u2026 <span class=\"read-more\"><a href=\"https:\/\/adriangrigoras.com\/blog\/lambda-architecture\/\">Read More &raquo;<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-616","post","type-post","status-publish","format-standard","hentry","category-architecture"],"_links":{"self":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/616","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/comments?post=616"}],"version-history":[{"count":2,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/616\/revisions"}],"predecessor-version":[{"id":619,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/616\/revisions\/619"}],"wp:attachment":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/media?parent=616"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/categories?post=616"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/tags?post=616"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}