The aim of this study was to design a new deep learning framework for end-to-end processing of polysomnograms. This framework can be trained to analyze whole-night polysomnograms without the limitations of and bias towards clinical scoring guidelines. We validated the framework by predicting the age of subjects. We designed a hierarchical attention network architecture, which can be pre-trained to predict labels based on 5-minute epochs of data and fine-tuned to predict based on whole-night polysomnography recordings. The model was trained on 511 recordings from the Cleveland Family study and tested on 146 test subjects aged between 6 to 88 years. The proposed network achieved a mean absolute error of 7.36 years and a correlation to true age of 0.857. Sleep can be analyzed using our end-to-end deep learning framework, which we expect can generalize to learning other subject-specific labels such as sleep disorders. The difference in the predicted and chronological age is further proposed as an estimate of biological age.