Istio: 503’s with UC’s and TCP Fun Times
The istio service mesh for K8S has a bit of difficulty with idle TCP connections from the upstream closing “prematurely”. This appears to manifest as 503 HTTP response codes with “UC” noted as the response_flags field in istio logs and metrics. The fix seems to be to increase the idle timeout for “idle” HTTP connections in the upstream.
(tags: istio kubernetes k8s eks http tcp timeouts connection-pools networking)
-
“A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster. You get the full power of Wireshark with minimal impact on your running pods. When working with micro-services, many times it’s very helpful to get a capture of the network activity between your micro-service and it’s dependencies. ksniff use kubectl to upload a statically compiled tcpdump binary to your pod and redirecting it’s output to your local Wireshark for smooth network debugging experience.” This would be an absolutely vital piece of software once you get into the nitty-gritty of debugging TCP issues in K8S; I’ve been on the verge of needing a packet capture once or twice, but managed to just about avoid it so far. I’ll be keeping this one in the back pocket.
(tags: debugging kubernetes network networking packet-captures tcpdump wireshark ops k8s eks sniffing kubectl)