TY - GEN

T1 - Prediction of Patient Demographics using 3D Craniofacial Scans and Multi-view CNNs

AU - Hanif, Umaer

AU - Paulsen, Rasmus R.

AU - Leary, Eileen B.

AU - Mignot, Emmanuel

AU - Jennum, Poul

AU - Sorensen, Helge B.D.

N1 - Publisher Copyright: © 2020 IEEE.

PY - 2020

Y1 - 2020

N2 - 3D data is becoming increasingly popular and accessible for computer vision tasks. A popular format for 3D data is the mesh format, which can depict a 3D surface accurately and cost-effectively by connecting points in the (x, y, z) plane, known as vertices, into triangles that can be combined to approximate geometrical surfaces. However, mesh objects are not suitable for standard deep learning techniques due to their non-euclidean structure. We present an algorithm which predicts the sex, age, and body mass index of a subject based on a 3D scan of their face and neck. This algorithm relies on an automatic pre-processing technique, which renders and captures the 3D scan from eight different angles around the x-axis in the form of 2D images and depth maps. Subsequently, the generated data is used to train three convolutional neural networks, each with a ResNet18 architecture, to learn a mapping between the set of 16 images per subject (eight 2D images and eight depth maps from different angles) and their demographics. For age and body mass index, we achieved a mean absolute error of 7.77 years and 4.04 kg/m2 on the respective test sets, while Pearson correlation coefficients of 0.76 and 0.80 were obtained, respectively. The prediction of sex yielded an accuracy of 93%. The developed framework serves as a proof of concept for prediction of more clinically relevant variables based on 3D craniofacial scans stored in mesh objects.

AB - 3D data is becoming increasingly popular and accessible for computer vision tasks. A popular format for 3D data is the mesh format, which can depict a 3D surface accurately and cost-effectively by connecting points in the (x, y, z) plane, known as vertices, into triangles that can be combined to approximate geometrical surfaces. However, mesh objects are not suitable for standard deep learning techniques due to their non-euclidean structure. We present an algorithm which predicts the sex, age, and body mass index of a subject based on a 3D scan of their face and neck. This algorithm relies on an automatic pre-processing technique, which renders and captures the 3D scan from eight different angles around the x-axis in the form of 2D images and depth maps. Subsequently, the generated data is used to train three convolutional neural networks, each with a ResNet18 architecture, to learn a mapping between the set of 16 images per subject (eight 2D images and eight depth maps from different angles) and their demographics. For age and body mass index, we achieved a mean absolute error of 7.77 years and 4.04 kg/m2 on the respective test sets, while Pearson correlation coefficients of 0.76 and 0.80 were obtained, respectively. The prediction of sex yielded an accuracy of 93%. The developed framework serves as a proof of concept for prediction of more clinically relevant variables based on 3D craniofacial scans stored in mesh objects.

U2 - 10.1109/EMBC44109.2020.9176333

DO - 10.1109/EMBC44109.2020.9176333

M3 - Article in proceedings

C2 - 33018384

AN - SCOPUS:85091025695

T3 - Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS

SP - 1950

EP - 1953

BT - 42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society

PB - IEEE

T2 - 42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society, EMBC 2020

Y2 - 20 July 2020 through 24 July 2020

ER -