Notes from Microservices at Hudl

On Wednesday, I went to presentation by the CTO of Hudl on how they do microservices. This is their story. (Most interesting stuff in bold)

  • Initially led to microservices due to organizational issues-whenever a new feature or change needed to be made, it took several different teams. Lack of developer autonomy.
  • Modelled themselves after the Spotify Squad + Tribe structure
  • Every 6 months or so, everyone in their 350 person engineering org restructures to a new squad
  • Their stack-AWS. ELB->NGINX->IIS (runs all of their apps in various clusters, nginx knows which to go to)->Redis+MongoDB
  • Each team is responsible for their own DBs (and all aspects of their cluster) however an infrastructure team exists to spin up and manage their MongoDB + Redis.
  • Using Eureka for service discovery. They decorate it with a routing object to tell NGINX what routes are available. A little tool called Sparkplug propagates from Eureka to an NGINX config
  • They have a tool called Bifrost that handles routing + load balancing between clusters. Basically it’s an API routing layer.
  • For each cluster, they run two app pools. This allows them to update one app pool, graceful switchover to it, and if it fails, instantly rollback to the old one.
  • Mjolnir offer circuit breaking and transport similar to Netflix’s Hystrix
  • Deployments-You can read about them here but basically you create a new branch, CI picks it up, runs the tests, uploads it to S3. Then the developer goes in and clicks “Deploy!” and it gets put onto a test environment. They automatically add a route53 entry of clustername.something.com to make it easy to find your test environment.
  • Updates to test environments take seconds
  • When you’re working on local dev, that Sparkplug thing works locally by either routing your app to hit local versions of apps you have running or pointing to a test environment with the master of all of the other microservices running. You can even point your app to point at another person’s running environments!
  • To build all these amazing development tools + deployment, it took a team of 4 developers 6 months to do the initial work and they still have a 10 person dedicated tools/infrastructure team.
  • It’s the responsible of the microservice builder to supply client libraries. They all use .NET, so they autogenerate clients with some .NET tool.
  • Their deploys take 30 seconds to all their boxes
  • Interesting-they don’t do offline development. No one pulls all the data to their local boxes (they’re a video company afterall). You have to be hooked to the internet.
  • They do evented communication with a single, namespaced SQS

 

 

 

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *