I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take.

We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/

Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.