Habitat fragmentation is altering species interactions worldwide. However, the mechanisms underlying the response of network specialization to habitat fragmentation remain unknown, especially for multi-trophic interactions. We here collected a large dataset consisting of 2670 observations of tri-trophic interactions among plants, sap-sucking aphids and honeydew-collecting ants on 18 forested islands in the Thousand Island Lake, China. For each island, we constructed an antagonistic plant-aphid and a mutualistic aphid-ant network, and tested how network specialization varied with island area and isolation. We found that both networks exhibited higher specialization on smaller islands, while only aphid-ant networks had increased specialization on more isolated islands. Variations in network specialization among islands was primarily driven by species turnover, which was interlinked across trophic levels as fragmentation increased the specialization of both antagonistic and mutualistic networks through bottom-up effects via plant and aphid communities. These findings reveal that species on small and isolated islands display higher specialization mainly due to effects of fragmentation on species turnover, with behavioural changes causing interaction rewiring playing only a minor role. Our study highlights the significance of adopting a multi-trophic perspective when exploring patterns and processes in structuring ecological networks in fragmented landscapes.