GitHub Status - Incident with GitHub.com
Incident report on this week's very long GitHub outage. tl;dr: service meshes still suck.
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.
Interestingly, https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/ blames Azure "capacity failures", which isn't quite matching up with "istio autoscaling broke". Interesting mismatch there.
Tags: azure github capacity scalability autoscaling istio kubernetes outages post-mortems postmortems service-meshes