Empirical evidence that serious misalignment can emerge from seemingly benign reward misspecification.