Make PR message on pushes configurable (#10664)