The result? A telemetry system that survives real network partitions, overloaded exporters, and misconfigured rules. And a team that actually knows how to debug their monitoring stack under pressure.
| Risk | Mitigation | | --- | --- | | PCE accidentally runs on production | Use namespace isolation, explicit --chaos.enabled=false flag in prod. | | Permanent data loss | Run against a replica Prometheus with --storage.tsdb.retention.time=6h . | | Alert fatigue | Notify a separate “chaos channel” during experiments. | | Controller plane overload | Limit chaos duration (e.g., 5 minutes max). | prometheus chaos edition
In short: How to Run Prometheus Chaos Edition (Step-by-Step) The result
@app.route('/metrics') def metrics(): if random.random() < 0.2: # 20% of the time return "malformed_metric{ invalid syntax", 200 return Response(real_metrics(), mimetype='text/plain') | Risk | Mitigation | | --- |