White matter (WM) degeneration in Alzheimer's disease (AD) and mild cognitive impairment (MCI) may be a key indicator of early damage in AD. Here, we analyzed WM diffusion tensor data using Tract-Based Spatial Statistics in conjunction with mixed-effects models. Four indices of diffusion were assessed in 61 healthy control, 19 non-amnestic MCIs, 14 amnestic MCIs, and 9 AD patients. The aim of the study was to use advanced mixed-effects models to investigate the retrogenesis hypothesis of AD, which suggests that tracts that are late to myelinate in ontogenetic development are the earliest to be affected in AD. Our results show that a number of late-myelinating pathways, including the parahippocampal region and the inferior longitudinal fasciculus, were predominantly affected by changes in WM volume. Conversely, early-myelinating pathways were found to be affected by a combination of both WM and gray matter (GM) atrophy. A model of the entire WM structure of the brain returned GM models for two indices of diffusion, suggesting that more complex regional landscapes of diffusion lie hidden beneath a global analysis of the entire brain. Our results warn against an explanation of white matter damage that points simply to one of two mechanisms: secondary degeneration or direct damage of myelin. We suggest that tracts may be affected by both mechanisms, with the balance depending on whether tracts are early or late-myelinating. A greater understanding of the pattern of WM changes in AD may prove useful for the early detection of AD.