{"id":477,"date":"2015-02-10T09:23:15","date_gmt":"2015-02-10T09:23:15","guid":{"rendered":"http:\/\/adriangrigoras.com\/blog\/?p=477"},"modified":"2015-03-17T14:42:10","modified_gmt":"2015-03-17T14:42:10","slug":"vinted-architecture-keeping-busy-portal-stable-deploying-times-day","status":"publish","type":"post","link":"https:\/\/adriangrigoras.com\/blog\/vinted-architecture-keeping-busy-portal-stable-deploying-times-day\/","title":{"rendered":"Keeping A Busy Portal Stable By Deploying Several Hundred Times Per Day"},"content":{"rendered":"<p>An interesting architecture with alternatives to big software providers.<\/p>\n<p><a href=\"https:\/\/www.vinted.com\/\">Vinted<\/a> is a peer-to-peer marketplace to sell, buy and swap clothes. It allows members to communicate directly and has the features of a social networking service.<\/p>\n<p>Started in 2008 as a small community for Lithuanian girls, it developed into a worldwide project that serves over 7 million users in 8 different countries, and is growing non-stop, handling over 200 M requests per day.<\/p>\n<h2>Stats<\/h2>\n<ul>\n<li>7 million users and growing<\/li>\n<li>2.5 million monthly active users<\/li>\n<li>200 million requests \/ day (pageviews + API calls)<\/li>\n<li>930 million photos<\/li>\n<li>80 TB raw space for image storage<\/li>\n<li>60 TB HDFS raw space for analytics data<\/li>\n<li>70% of traffic comes from mobile apps (API requests)<\/li>\n<li>3+ hours \/ week time spent per member<\/li>\n<li>25 million listed items<\/li>\n<li>Over 220 servers:\n<ul>\n<li>47 for internal tools (vpn, chef, monitoring, development, graphing, build, backups, etc.)<\/li>\n<li>38 for the Hadoop ecosystem<\/li>\n<li>34 for image processing and storage<\/li>\n<li>30 for Unicorn and microservices<\/li>\n<li>28 for MySQL databases, including replicas<\/li>\n<li>19 for the Sphinx search<\/li>\n<li>10 for resque background jobs<\/li>\n<li>10 for load balancing with Nginx<\/li>\n<li>6 for Kafka<\/li>\n<li>4 for Redis<\/li>\n<li>4 for email delivery<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Technology Stack<\/h2>\n<h3>Third Party Services<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/\">GitHub<\/a> for code, issues and discussions<\/li>\n<li><a href=\"http:\/\/newrelic.com\/\">NewRelic<\/a> for monitoring app response times<\/li>\n<li><a href=\"https:\/\/www.cloudflare.com\/\">CloudFlare<\/a> for DDoS protection and DNS<\/li>\n<li><a href=\"http:\/\/aws.amazon.com\/\">Amazon Web Services<\/a> (SES, Glacier) for notifications and long term backups<\/li>\n<li><a href=\"https:\/\/slack.com\/\">Slack<\/a> for work conversations and &#8220;chat ops&#8221;<\/li>\n<li><a href=\"https:\/\/trello.com\/\">Trello<\/a> for tasks<\/li>\n<li><a href=\"http:\/\/pingdom.com\/\">Pingdom<\/a> for uptime monitoring<\/li>\n<\/ul>\n<h3>Core App And Services<\/h3>\n<ul>\n<li><a href=\"https:\/\/www.ruby-lang.org\/\">Ruby<\/a> for services, scripts and the main app<\/li>\n<li><a href=\"http:\/\/rubyonrails.org\/\">Rails<\/a> for the main app and internal apps<\/li>\n<li><a href=\"http:\/\/unicorn.bogomips.org\/\">Unicorn<\/a> to serve the main app<\/li>\n<li><a href=\"http:\/\/www.percona.com\/software\/percona-server\/ps-5.6\">Percona MySQL Server<\/a> as main database<\/li>\n<li><a href=\"http:\/\/sphinxsearch.com\/\">Sphinx Search<\/a> for full-text search and for reducing load to MySQL (testing <a href=\"http:\/\/www.elasticsearch.org\/\">ElasticSearch<\/a>)<\/li>\n<li><a href=\"http:\/\/capistranorb.com\/\">Capistrano<\/a> for deployments<\/li>\n<li><a href=\"http:\/\/jenkins-ci.org\/\">Jenkins<\/a> for running tests, deployments and various other tasks<\/li>\n<li><a href=\"http:\/\/memcached.org\/\">Memcached<\/a> for caching<\/li>\n<li><a href=\"http:\/\/resquework.org\/\">Resque<\/a> for background jobs<\/li>\n<li><a href=\"http:\/\/redis.io\/\">Redis<\/a> for Resque and Feed<\/li>\n<li><a href=\"http:\/\/www.rabbitmq.com\/\">RabbitMQ<\/a> for passing messages to services<\/li>\n<li><a href=\"http:\/\/www.haproxy.org\/\">HAproxy<\/a> for high availability and load balancing<\/li>\n<li><a href=\"http:\/\/www.gluster.org\/\">GlusterFS<\/a> for distributed storage<\/li>\n<li><a href=\"http:\/\/nginx.org\/\">Nginx<\/a> to serve web requests<\/li>\n<li><a href=\"http:\/\/zookeeper.apache.org\/\">Apache Zookeeper<\/a> for distributed locking<\/li>\n<li><a href=\"https:\/\/github.com\/facebook\/flashcache\/\">Flashcache<\/a> for increased I\/O throughput<\/li>\n<\/ul>\n<h3>Data Warehouse Stack<\/h3>\n<ul>\n<li><a href=\"http:\/\/clojure.org\/\">Clojure<\/a> for data ingestion services<\/li>\n<li><a href=\"http:\/\/kafka.apache.org\/\">Apache Kafka<\/a> for storing in-flight data<\/li>\n<li><a href=\"https:\/\/github.com\/linkedin\/camus\">Camus<\/a> for offloading data from Kafka to HDFS<\/li>\n<li><a href=\"https:\/\/hive.apache.org\/\">Apache Hive<\/a> as SQL-on-Hadoop solution<\/li>\n<li><a href=\"http:\/\/www.cloudera.com\/content\/cloudera\/en\/products-and-services\/cdh.html\">Cloudera CDH<\/a> Hadoop distribution<\/li>\n<li><a href=\"http:\/\/www.cloudera.com\/content\/cloudera\/en\/products-and-services\/cdh\/impala.html\">Cloudera Impala<\/a> as low latency SQL-on-Hadoop solution<\/li>\n<li><a href=\"https:\/\/spark.apache.org\/\">Apache Spark<\/a> in experimentation phase<\/li>\n<li><a href=\"http:\/\/oozie.apache.org\/\">Apache Oozie<\/a> as workflow scheduler (investigating alternatives)<\/li>\n<li><a href=\"https:\/\/github.com\/twitter\/scalding\">Scalding<\/a> for data transformations<\/li>\n<li><a href=\"http:\/\/avro.apache.org\/\">Avro<\/a> for data serialization<\/li>\n<li><a href=\"http:\/\/parquet.incubator.apache.org\/\">Apache Parquet<\/a> for data serialization<\/li>\n<\/ul>\n<h3>Servers And Provisioning<\/h3>\n<ul>\n<li>Mostly bare metal<\/li>\n<li>Multiple data centers<\/li>\n<li><a href=\"http:\/\/www.centos.org\/\">CentOS<\/a><\/li>\n<li><a href=\"https:\/\/www.chef.io\/\">Chef<\/a> for provisioning nearly everything<\/li>\n<li><a href=\"http:\/\/berkshelf.com\/\">Berkshelf<\/a> for managing Chef dependencies<\/li>\n<li><a href=\"http:\/\/kitchen.ci\/\">Test Kitchen<\/a> for running infrastructure tests on VMs<\/li>\n<li><a href=\"http:\/\/www.ansible.com\/\">Ansible<\/a> for rolling upgrades<\/li>\n<\/ul>\n<h3>Monitoring<\/h3>\n<ul>\n<li><a href=\"http:\/\/www.nagios.org\/\">Nagios<\/a> for usual ops monitoring<\/li>\n<li><a href=\"http:\/\/www.cacti.net\/\">Cacti<\/a> for graphs and capacity planning<\/li>\n<li><a href=\"https:\/\/www.graylog2.org\/\">Graylog2<\/a> for app logs<\/li>\n<li><a href=\"http:\/\/logstash.net\/\">Logstash<\/a> for server logs<\/li>\n<li><a href=\"http:\/\/www.elasticsearch.org\/overview\/kibana\/\">Kibana<\/a> for querying logstash<\/li>\n<li><a href=\"https:\/\/github.com\/etsy\/statsd\">Statsd<\/a> for collecting real-time metrics from the app<\/li>\n<li><a href=\"http:\/\/graphite.wikidot.com\/\">Graphite<\/a> for storing real-time metrics from the app<\/li>\n<li><a href=\"http:\/\/grafana.org\/\">Grafana<\/a> for creating beautiful dashboards with app metrics<\/li>\n<li><a href=\"https:\/\/hubot.github.com\/\">Hubot<\/a> for chat-based monitoring<\/li>\n<\/ul>\n<h2>Architecture Overview<\/h2>\n<ul>\n<li>Bare metal servers are used as a cheaper and more powerful alternative to cloud providers.<\/li>\n<li>Servers hosted in 3 different data centers across Europe and the US.<\/li>\n<li>Nginx frontends route HTTP requests to unicorn workers and do SSL termination.<\/li>\n<li><a href=\"http:\/\/clusterlabs.org\/\">Pacemaker<\/a> is used to ensure high availability of Nginx servers.<\/li>\n<li>In different countries, every Vinted portal has its own separate deployment and set of resources.<\/li>\n<li>To increase machine utilization, most services that belong to multiple portals may run alongside each other on a single bare metal server. Chef recipes take care of unique port allocation and other separation concerns.<\/li>\n<li>MySQL sharding is avoided by having a separate database for each portal.<\/li>\n<li>In our largest portal, functional sharding is already happening, and a point where the single largest table will not fit into the server is months away.<\/li>\n<li>Caching in Rails app uses custom mechanism with L2 cache that prevents spikes when the main cache is expired. Several caching strategies are used:\n<ul>\n<li>remote (in memcached)<\/li>\n<li>local (in unicorn worker memory)<\/li>\n<li>semi-local (in unicorn worker memory, falling back to memcached when local cache expires).<\/li>\n<\/ul>\n<\/li>\n<li>Several microservices are built around the core rails app, all with a clear purpose, like sending iOS push notifications, storing and serving brand names, storing and serving hashtags.<\/li>\n<li>Microservices can be either per-portal, per-datacenter or global.<\/li>\n<li>Multiple instances of each microservice are running for high availability.<\/li>\n<li>HTTP-based microservices are balanced by Nginx.<\/li>\n<li>Custom feeds for each member are implemented using Redis.<\/li>\n<li>Catalog pages are loaded from the Sphinx index rather than MySQL when filters are used (custom size, color, etc.).<\/li>\n<li>Images are processed by a separate Rails app.<\/li>\n<li>Processed images are stored in GlusterFS.<\/li>\n<li>Images are cached after 3rd hit, since the first few hits usually come from the uploader and an image may be rotated.<\/li>\n<li>Memcached instances are sharded using <a href=\"https:\/\/github.com\/twitter\/twemproxy\">twemproxy<\/a>.<\/li>\n<\/ul>\n<h2>Team<\/h2>\n<ul>\n<li>over 150 full-time employees<\/li>\n<li>30 developers (backend, frontend, mobile)<\/li>\n<li>5 site reliability engineers<\/li>\n<li>Main HQ in Vilnius, Lithuania<\/li>\n<li>Offices in the USA, Germany, France, the Czech Republic and Poland<\/li>\n<\/ul>\n<h2>How The Company Works<\/h2>\n<ul>\n<li>Nearly all information is open to all employees<\/li>\n<li>Using GitHub for almost everything (including to discuss company issues)<\/li>\n<li>Real-time discussions in Slack<\/li>\n<li>Everyone is free to participate in things that interest them<\/li>\n<li>Autonomous teams<\/li>\n<li>No seniority levels<\/li>\n<li>Cross-functional development teams<\/li>\n<li>No enforced process, teams decides how to manage themselves<\/li>\n<li>Teams work on high-level problems and decide how to tackle them<\/li>\n<li>Each team decides how, when and where they work<\/li>\n<li>Teams hire for themselves when a need arises<\/li>\n<\/ul>\n<h2>Development Cycle<\/h2>\n<ul>\n<li>Developers branch from master.<\/li>\n<li>Changes become pull requests in GitHub.<\/li>\n<li>Jenkins runs <a href=\"https:\/\/github.com\/mmozuras\/pronto\">Pronto<\/a> to do static code and style checks, runs all tests on pull request branches and updates the GitHub pull request with a status.<\/li>\n<li>Other developers review the change, add comments.<\/li>\n<li>Pull requests may be updated multiple times with various fixes.<\/li>\n<li>Jenkins runs all tests after each update.<\/li>\n<li>Git history is cleaned up before merging with master to keep master history concise.<\/li>\n<li>Accepted pull requests are merged into master.<\/li>\n<li>Jenkins runs master build with all tests and triggers deployment jobs to roll out the new version.<\/li>\n<li>Several minutes later, after the code has reached master branch, it is applied in production.<\/li>\n<li>Pull requests are usually small.<\/li>\n<li>Deployments happen ~300 times per day.<\/li>\n<\/ul>\n<h2>Avoiding Failures<\/h2>\n<p>Deploying hundreds of times per day does not mean everything always has to be broken, but keeping the site stable requires some discipline.<\/p>\n<ul>\n<li>Deployments do not happen if tests are broken. There is no way to override this, master has to be green in order to deploy.<\/li>\n<li>Automatic deployments are locked every night and during weekends.<\/li>\n<li>Anyone can lock deployments manually from a Slack chat.<\/li>\n<li>When deployments are locked, it is possible to &#8220;force&#8221; a deployment manually, from the Slack chat.<\/li>\n<li>The team is very picky when it comes to reviewing code. The quality bar is set high. Tests are mandatory.<\/li>\n<li>Before Unicorn reloads code, a warmup is done in production. It includes several requests to various critical parts of the portal.<\/li>\n<li>During warmup, if any one request does not return 200 OK, the old code remains and deployment fails.<\/li>\n<li>Occasionally a problem is not caught by tests \/ warmup, and broken code is released to production.<\/li>\n<li>Errors are streamed to graylog server.<\/li>\n<li>Alarms are trigerred if the error rate exceeds a threshold.<\/li>\n<li>Error rate alarms and failed build notifications are reported instantly to a Slack chat.<\/li>\n<li>All errors contain extra metadata: Unicorn worker host and pid, HTTP request details, git revision of the code that caused the problem, error backtrace.<\/li>\n<li>Some types of &#8220;Fatal&#8221; errors are also reported directly to a Slack chat.<\/li>\n<li>Each deployment log contains GitHub diff URL with all the changes.<\/li>\n<li>If something breaks in production, it is easy to pinpoint the problem quickly due to a small changeset and instant error feedback.<\/li>\n<li>Deployment details are reported to NewRelic, which makes it easy to troubleshoot introduction of performance bottlenecks.<\/li>\n<\/ul>\n<h2>Reducing Time To Production<\/h2>\n<p>There is a big focus on both fast and stable releases, the team is always working to keep build and deployment times as short as possible.<\/p>\n<ul>\n<li>Full test suite contains &gt;7000 tests and runs in ~3 minutes.<\/li>\n<li>New tests are added constantly.<\/li>\n<li>Jenkins runs on a bare metal machine with 32 CPU, 128G RAM and SSD drives.<\/li>\n<li>Jenkins splits all tests into multiple chunks and runs them in parallel, aggregating final results.<\/li>\n<li>Tests would run over 1 hour on an average machine without parallelisation.<\/li>\n<li>Rails assets are pre-built in Jenkins for every commit in master branch.<\/li>\n<li>During deployment, prebuilt assets are downloaded from jenkins and uploaded to all target servers.<\/li>\n<li>BitTorrent is used when uploading a release to target servers.<\/li>\n<\/ul>\n<h2>Running Live Database Migrations<\/h2>\n<p>We can change the structure of our production MySQL databases without downtime, in most cases even at rush hour.<\/p>\n<ul>\n<li>Database migrations can happen during any deployment<\/li>\n<li>Percona toolkit&#8217;s pt-online-schema-change is used to run alters on a copy of the table, while keeping it in sync with the original, and to switch the copy with the original when the change is done<\/li>\n<\/ul>\n<h1>Operations<\/h1>\n<h2>Server Provisioning<\/h2>\n<ul>\n<li>Chef is used to provision nearly every aspect of our infrastructure.<\/li>\n<li>SRE team does pull requests for infrastructure changes just like Development teams do for code changes.<\/li>\n<li>Jenkins is verifying all Chef pull requests, running cross dependency checks, foodcritic, etc.<\/li>\n<li>Chef code changes are reviewed in GitHub by the team.<\/li>\n<li>Developers can make infrastructure change pull requests to Chef repository.<\/li>\n<li>When pull requests are merged, Jenkins uploads changed Chef cookbooks and applies them in production.<\/li>\n<li>Plans to spawn DigitalOcean droplets to run Chef kitchen tests with Jenkins.<\/li>\n<li>Jenkins itself is configured with Chef, not using the web UI.<\/li>\n<\/ul>\n<h2>Monitoring<\/h2>\n<ul>\n<li>Cacti graphs server side metrics<\/li>\n<li>Nagios checks for everything that can be up or down<\/li>\n<li>Nagios health checks for some of our apps and services<\/li>\n<li>Statsd \/ Graphite for tracking app metrics, like member logins, signups, database object creation<\/li>\n<li>Grafana used for nice dashboards<\/li>\n<li>Grafana dashboards can be described and created dynamically using Chef<\/li>\n<\/ul>\n<h2>Chat Ops<\/h2>\n<ul>\n<li>Hubot sits in Slack.<\/li>\n<li>Slack rooms are subscribed to various event notifications via <a href=\"https:\/\/github.com\/spajus\/hubot-pubsub\">hubot-pubsub<\/a>.<\/li>\n<li>#deploy room shows information about deployments, with links to GitHub diffs, Jenkins console outputs, etc. Right here deployments can be triggered, locked and unlocked with simple chat commands.<\/li>\n<li>#deploy-fail room shows only deployment failures.<\/li>\n<li>#failboat room shows announcements about error rates and individual errors in production.<\/li>\n<li>There multiple #failboat-* rooms that give information about cron failures, stuck resque jobs, nagios warnings, newrelic alerts.<\/li>\n<li>Twice a day Graylog errors are crunched and cross-checked with GitHub, reports are generated and presented to the development team.<\/li>\n<li>If someone mentions you in GitHub, Hubot gives you a link to the issue in a private message on Slack.<\/li>\n<li>When an issue or pull request is created in GitHub, teams that are mentioned receive pings in their appropriate Slack rooms.<\/li>\n<li>Graphite graphs can be queried and displayed in Slack chat via Hubot.<\/li>\n<li>You can ask Hubot questions about infrastructure, i.e. as about the machines where a certain service is deployed. Hubot queries the Chef server to get the data.<\/li>\n<li>Developers use Hubot notifications for certain events that happen in our app. For example, customer the support chatroom is automatically notified about possible fraud cases.<\/li>\n<li>Hubot is integrated with the office <a href=\"https:\/\/www.netatmo.com\/\">netatmo<\/a> module and can tell everyone what CO2, noise, temperature and humidity levels are.<\/li>\n<\/ul>\n<h1>Data warehouse stack<\/h1>\n<ul>\n<li>We ingest data from two sources:\n<ul>\n<li>Website event tracking data: impressions, clicks etc.<\/li>\n<li>Database (MySQL) changes.<\/li>\n<\/ul>\n<\/li>\n<li>Most event tracking data is published to an HTTP endpoint from JavaScript\/Android\/iOS front-end apps. Some events are published to a local UDP socket by our core Ruby on Rails application. We chose UDP in order to avoid blocking the request.<\/li>\n<li>There&#8217;s a service that listens on the UDP socket for new events, buffers them and periodically emits to a raw events&#8217; topic in Kafka.<\/li>\n<li>A stream processing service consumes the raw events topic, validates the events, encodes them in Avro format and emits to event-specific topics.<\/li>\n<li>We keep our Avro schemas of events in a separate centralized schema registry service.<\/li>\n<li>Invalid events are placed into a separate Kafka topic for possible correction.<\/li>\n<li>We use LinkedIn&#8217;s Camus job to offload events from event-specific topics to HDFS incrementally. Time to Hadoop for an event is usually up to an hour.<\/li>\n<li>Using Hive and Impala for ad-hoc queries and data analysis.<\/li>\n<li>Working on Scalding-based solution for data processing.<\/li>\n<li>Reports run from our in-house grown OLAP-like reporting system written in Ruby.<\/li>\n<\/ul>\n<h1>Lessons Learned<\/h1>\n<h2>Product Development<\/h2>\n<ul>\n<li>Invest in code quality.<\/li>\n<li>Cover absolutely everything with tests.<\/li>\n<li>Release small changes.<\/li>\n<li>Use feature switches to deploy unfinished features to production.<\/li>\n<li>Do not keep long running branches of code when developing something.<\/li>\n<li>Invest in a fast release cycle.<\/li>\n<li>Build the product mobile-first.<\/li>\n<li>Design API very well from the very beginning, it is difficult to change it later.<\/li>\n<li>Including sender hostname and app name in RabbitMQ messages can be a life saver.<\/li>\n<li>Do not guess or make assumptions, do <a href=\"https:\/\/github.com\/vinted\/ab\">AB<\/a> testing and check the data.<\/li>\n<li>If you plan on using Redis for something big, think about sharding from very beginning.<\/li>\n<li>Never use forked and slightly modified ruby gems if you are planning on keeping your Rails up to date.<\/li>\n<li>Make sharing your content through social networks as easy as possible.<\/li>\n<li>Search engine optimization is a full-time job.<\/li>\n<\/ul>\n<h2>Infrastructure And Operations<\/h2>\n<ul>\n<li>Deploy often to increase system stability. It may sound counter-intuitive at first.<\/li>\n<li>Automate everything you can.<\/li>\n<li>Monitor everything that can be down.<\/li>\n<li>Network switch buffers matter.<\/li>\n<li>Capacity planning is hard.<\/li>\n<li>Think about HA from the start.<\/li>\n<li>RabbitMQ cluster is not worth the effort.<\/li>\n<li>Measure and graph everything you can: HTTP response times, Resque queue sizes, Model persistence rates, Redis response times. You cannot improve what you cannot measure.<\/li>\n<\/ul>\n<h2>Data Warehouse Stack<\/h2>\n<ul>\n<li>There&#8217;s a myriad of tools in the Hadoop ecosystem. It&#8217;s hard to pick the right one.<\/li>\n<li>If you are writing a user defined function (UDF) for Hive it&#8217;s time to consider a non-SQL solution for data transformations.<\/li>\n<li>The &#8220;Hadoop application&#8221; notion is vague. Oftentimes we need to glue our application components together manually.<\/li>\n<li>Everyone writes their own workflow manager. And for a good reason.<\/li>\n<li>Distributed system monitoring is tough. Anomaly detection and graphing helps.<\/li>\n<li>The core Hadoop infrastructure (HDFS, YARN) is rock solid but the newer tools usually have unpleasant nuances.<\/li>\n<li>Kafka is amazing.<\/li>\n<\/ul>\n<p>source:\u00a0highscalability.com<\/p>\n","protected":false},"excerpt":{"rendered":"<p>An interesting architecture with alternatives to big software providers. Vinted is a peer-to-peer marketplace to sell, buy and swap clothes. It allows members to communicate directly and has the features of a social networking service. Started in 2008 as a small community for Lithuanian girls, it developed into a worldwide project that serves over 7\u2026 <span class=\"read-more\"><a href=\"https:\/\/adriangrigoras.com\/blog\/vinted-architecture-keeping-busy-portal-stable-deploying-times-day\/\">Read More &raquo;<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-477","post","type-post","status-publish","format-standard","hentry","category-architecture"],"_links":{"self":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/477","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/comments?post=477"}],"version-history":[{"count":2,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/477\/revisions"}],"predecessor-version":[{"id":479,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/posts\/477\/revisions\/479"}],"wp:attachment":[{"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/media?parent=477"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/categories?post=477"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/adriangrigoras.com\/blog\/wp-json\/wp\/v2\/tags?post=477"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}