Congenital amusia, characterized by a severe problem in detecting anomalies in melodies, is a lifelong disorder that has been ascribed to an acoustical pitch deficit. In the present study, we investigated how the perception of a duration is altered when it is bounded by tones varying in pitch. The results show that temporal accuracy is impaired by pitch variations as small as a quarter of a semitone in control participants, whereas it is impaired only when pitch variations are increased to 4 semitones in congenital amusics. Furthermore, control participants associate intervals bounded by low- and high-pitched tones with long and short durations, respectively. Amusic participants do not make this connection, even with large pitch differences, pointing to a deficit in pitch-time integration. Thus, our results are consistent with the notion that congenital amusia is linked to a neurogenetic anomaly that impairs pitch processing, independently of task factors.