Multimodal imaging enables sensitive measures of the architecture and integrity of the human brain, but the high-dimensional nature of advanced brain imaging features poses inherent challenges for the analyses and interpretations. Multivariate age prediction reduces the dimensionality to one biologically informative summary measure with potential for assessing deviations from normal lifespan trajectories. A number of studies documented remarkably accurate age prediction, but the differential age trajectories and the cognitive sensitivity of distinct brain tissue classes have yet to be adequately characterized. Exploring differential brain age models driven by tissue-specific classifiers provides a hitherto unexplored opportunity to disentangle independent sources of heterogeneity in brain biology. We trained machine-learning models to estimate brain age using various combinations of FreeSurfer based morphometry and diffusion tensor imaging based indices of white matter microstructure in 612 healthy controls aged 18–87 years. To compare the tissue-specific brain ages and their cognitive sensitivity, we applied each of the 11 models in an independent and cognitively well-characterized sample (n = 265, 20–88 years). Correlations between true and estimated age and mean absolute error (MAE) in our test sample were highest for the most comprehensive brain morphometry (r = 0.83, CI:0.78–0.86, MAE = 6.76 years) and white matter microstructure (r = 0.79, CI:0.74–0.83, MAE = 7.28 years) models, confirming sensitivity and generalizability. The deviance from the chronological age were sensitive to performance on several cognitive tests for various models, including spatial Stroop and symbol coding, indicating poorer performance in individuals with an over-estimated age. Tissue-specific brain age models provide sensitive measures of brain integrity, with implications for the study of a range of brain disorders.