Splunk

Splunk Universal Forwarder is a logging solution that "provides reliable, secure data collection from remote sources and forward that data into Splunk software for indexing and consolidation".

Splunk forwarder is a potential factor for the failure of applications pods to come back up. Even though the node metrics may miss CPU starvation, watching the docker stats on the node will show the forwarder spiking to very high CPU usage.

The solution is to add service CPU limit and liveness probe initial delay.