Plants face climatic stress and herbivore pressure. Herbivore pressure is highest in a conspecific or phylogenetically proximate neighbourhood because such a neighbourhood may be a reservoir of co-evolutionary shared phytophagous species. We hypothesize that, first, trees in phylogenetically proximate neighbourhood adapt by increased anti-herbivores defences. Second, a strong investment in such anti-herbivores defences could be traded off against the ability to tolerate climatic stress, and a phylogenetically proximate neighbourhood might hence indirectly select against climatic-stress tolerance. Finally, enemy pressure in a phylogenetically proximate neighbourhood might interfere with the selection by climate for defences. We studied oak (Quercus petraea) genotypes from provenances of contrasting phylogenetic neighbourhoods and climates in a common garden. We measured traits related to herbivory and climatic-stress tolerance. We found that genotypes from the populations that had evolved in phylogenetically proximate neighbourhood have tougher leaves and are better defended against leaf miners than genotypes that evolved in phylogenetically distant neighbourhood, but without inducing cost to climatic stress tolerance. We found that in phylogenetically proximate neighbourhood, the selection pressure for chemical traits promoting climatic tolerance are opposites from that in a phylogenetically distant neighbourhood. The results suggest that tree populations are not passive victims of herbivory pressure in a closely related neighbourhood but can adapt. This adaptation has no costs in terms of climatic-stress tolerance but interferes with the way populations adapt to climate.