A frequently used treatment strategy in locally advanced rectal cancer (RC) is neoadjuvant therapy followed by surgery. Patients treated with neoadjuvant therapy achieve varying pathological response, and currently, predicting the degree of response is challenging. This study examined the association between digitally assessed histopathological features in the diagnostic biopsies and pathological response to neoadjuvant therapy, aiming to find potential predictive biomarkers. 50 patients with RC treated with neoadjuvant chemotherapy and/or radiotherapy followed by surgery were included. Deep learning-based digital algorithms were used to assess the epithelium tumor area percentage (ETP) based on H&E-stained slides, and to quantify the density of CD3+ and CD8+ lymphocytes, as well as the CD8+/CD3+ lymphocyte percentage, based on immunohistochemically stained slides, from the diagnostic tumor biopsies. Pathological response was assessed according to the Mandard method. A good pathological response was defined as tumor regression grade (TRG) 1–2, and a complete pathological response was defined as Mandard TRG 1. Associations between the ETP and lymphocyte densities in the diagnostic biopsies and the pathological response were examined. The density of CD8+ lymphocytes, and the CD8+/CD3+ lymphocyte percentage, were associated with both good and complete response to neoadjuvant therapy, while the density of CD3+ lymphocytes was associated with complete response. The ETP did not correlate with response to neoadjuvant therapy. It is well-known that infiltration of lymphocytes in colorectal cancer is a prognostic biomarker. However, assessment of CD8+ and CD3+ lymphocytes in the diagnostic tumor biopsies of patients with RC may also be useful in predicting response to neoadjuvant therapy.