As climate change drives increased drought in many forested regions, mechanistic understanding of the factors conferring drought tolerance in trees is increasingly important. The dendrochronological record provides a window through which we can understand how tree size and traits shape growth responses to droughts. We analyzed tree-ring records for 12 species in a broadleaf deciduous forest in Virginia (USA) to test hypotheses for how tree height, microenvironment characteristics, and species’ traits shaped drought responses across the three strongest regional droughts over a 60-yr period. Drought tolerance (resistance, recovery, and resilience) decreased with tree height, which was strongly correlated with exposure to higher solar radiation and evaporative demand. The potentially greater rooting volume of larger trees did not confer a resistance advantage, but marginally increased recovery and resilience, in sites with low topographic wetness index. Drought tolerance was greater among species whose leaves lost turgor (wilted) at more negative water potentials and experienced less shrinkage upon desiccation. The tree-ring record reveals that tree height and leaf drought tolerance traits influenced growth responses during and after significant droughts in the meteorological record. As climate change-induced droughts intensify, tall trees with drought-sensitive leaves will be most vulnerable to immediate and longer-term growth reductions.