With increasing awareness of the impact of food on the climate, consumers are gradually changing their dietary habits towards a more plant-based diet. While acceptable products have been developed in meat analogues and non-fermented dairy products, alternative fermented dairy products such as yogurt and particularly ripened hard and semi-soft cheese products are not yet satisfactory. Since the cheese category has such a broad range of flavors and applications, it has proven complicated to find plant-based sources able to mimic them in terms of texture, meltability, ripening and flavor. Moreover, plant-based dairy alternatives do not provide the same nutritional supply. New technological approaches are needed to make cheese production more sustainable, which should be integrated in the already existing conventional cheese production to ensure a fast and cost-efficient transition. This can be tackled by incorporating plant-based components into the milk matrix, creating so-called “hybrid cheeses”. This review will discuss the challenges of both animal- and plant-based cheese products and highlight how the combination of both matrices can associate the best properties of these two worlds in a hybrid product, reviewing current knowledge and development on the matter. Emphasis will be drawn to the selection and pre-processing of raw materials. Furthermore, the key challenges of removing the off-flavors and creating a desirable cheese flavor through fermentation will be discussed.