Background: It is of crucial importance to be able to discriminate unipolar disorder (UD) from bipolar disorder (BD), as treatments, as well as course of illness, differ between the two disorders. Aims: To investigate whether voice features from naturalistic phone calls could discriminate between (1) UD, BD, and healthy control individuals (HC); (2) different states within UD. Methods: Voice features were collected daily during naturalistic phone calls for up to 972 days. A total of 48 patients with UD, 121 patients with BD, and 38 HC were included. A total of 115,483 voice data entries were collected (UD [n = 16,454], BD [n = 78,733], and HC [n = 20,296]). Patients evaluated symptoms daily using a smartphone-based system, making it possible to define illness states within UD and BD. Data were analyzed using random forest algorithms. Results: Compared with BD, UD was classified with a specificity of 0.84 (SD: 0.07)/AUC of 0.58 (SD: 0.07) and compared with HC with a sensitivity of 0.74 (SD: 0.10)/AUC = 0.74 (SD: 0.06). Compared with BD during euthymia, UD during euthymia was classified with a specificity of 0.79 (SD: 0.05)/AUC = 0.43 (SD: 0.16). Compared with BD during depression, UD during depression was classified with a specificity of 0.81 (SD: 0.09)/AUC = 0.48 (SD: 0.12). Within UD, compared with euthymia, depression was classified with a specificity of 0.70 (SD 0.31)/AUC = 0.65 (SD: 0.11). In all models, the user-dependent models outperformed the user-independent models. Conclusions: The results from the present study are promising, but as reflected by the low AUCs, does not support that voice features collected during naturalistic phone calls at the current state of art can be implemented in clinical practice as a supplementary and assisting tool. Further studies are needed.