The development of vaccine candidates for COVID-19 has been rapid, and those that are currently approved display high efficacy against the original circulating strains. However, recently, new variants of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) have emerged with increased transmission rates and less susceptibility to vaccine induced immunity. A greater understanding of protection mechanisms, including antibody longevity and cross-reactivity towards the variants of concern (VoCs), is needed. In this study, samples collected in Denmark early in the pandemic from paucisymptomatic subjects (n = 165) and symptomatic subjects (n = 57) infected with SARS-CoV-2 were used to assess IgG binding and inhibition in the form of angiotensin-converting enzyme 2 receptor (ACE2) competition against the wild-type and four SARS-CoV-2 VoCs (Alpha, Beta, Gamma, and Omicron). Antibodies induced early in the pandemic via natural infection were cross-reactive and inhibited ACE2 binding of the VoC, with reduced inhibition observed for the Omicron variant. When examined longitudinally, sustained cross-reactive inhibitory responses were found to exist in naturally infected paucisymptomatic subjects. After vaccination, receptor binding domain (RBD)-specific IgG binding increased by at least 3.5-fold and inhibition of ACE2 increased by at least 2-fold. When vaccination regimens were compared (two doses of Pfizer-BioNTech BNT162b2 (n = 50), or one dose of Oxford-AstraZeneca ChAdOx1 nCoV-19 followed by Pfizer-BioNTech BNT162b2 (ChAd/BNT) (n = 15)), higher levels of IgG binding and inhibition were associated with mix and match (ChAd/BNT) prime-boosting and time since vaccination. These results are particularly relevant for countries where vaccination levels are low.