Measuring Environmental Quantum Noise Exhibiting a Nonmonotonic Spectral Shape