Tracking buffers of mutation and noise to the genome