Neuronal circuits are shaped by experience during time windows of increased plasticity in postnatal development. In the auditory system, the critical period for the simplest sounds-pure frequency tones-is well defined. Critical periods for more complex sounds remain to be elucidated. We used in vivo electrophysiological recordings in the mouse auditory cortex to demonstrate that passive exposure to frequency modulated sweeps (FMS) from postnatal day 31 to 38 leads to long-Term changes in the temporal representation of sweep directions. Immunohistochemical analysis revealed a decreased percentage of layer 4 parvalbumin-positive (PV⁺) cells during this critical period, paralleled with a transient increase in responses to FMS, but not to pure tones. Preventing the PV⁺ cell decrease with continuous white noise exposure delayed the critical period onset, suggesting a reduction in inhibition as a mechanism for this plasticity. Our findings shed new light on the dependence of plastic windows on stimulus complexity that persistently sculpt the functional organization of the auditory cortex.