Quantum sensing represents a significant research direction with Quantum Technologies, particularly promising in the acoustic frequency regime. This potential offers a wide range of scientific applications, including the detection of magnetic fields generated by the brain’s activity, heartbeat, and the measurement of weak forces such as gravitational wave signals emitted by extreme astronomical events.

A principle challenge in enhancing the sensitivity of current gravitational wave detectors is managing two competing types of quantum noise; shot noise: which arises from the uncertainty associated with the arrival of photons, and quantum backaction noise: which results from the transfer of photon momentum to the test mass object as radiation pressure during the interaction. These noises, arise from the quantum nature of light, scale differently with the light power, and dominate at different frequencies. Their broadband reduction requires the injection of a squeezed vacuum source, with frequency-dependent rotation of the squeezed quadrature presently accomplished via a complex filter cavity.

This thesis explores an alternative approach for achieving broadband quantum noise reduction using polarized cesium atoms prepared in a higher energy ground state, oriented in an effective negative mass reference frame. Conditional reduction of broadband quantum noise is possible once the EPR entangled states are detected, with one arm positioned in a negative mass reference frame.

In this thesis, we report on the construction and characterization of an interactionenhanced atomic system, consisting of polarized cesium atoms confined with a 2mm*2mm*80mm channel. These atoms are manipulated by a homogeneous, home-built magnetic coil system with an intrinsic decay of 30Hz, and are uniformly probed by a spatially shaped top hat beam. This setup has demonstrated quantum noise-limited performance and a quantum nondemolition (QND) based backaction dominance with a quantum cooperativity of approximately 3, across a wide range of Larmor frequencies. This is evidenced by the observed ponderomotive squeezing from - 4.9 dB at 1 MHz down to the upper acoustic frequency at 18 kHz. Additionally, experimental investigations into the virtual frequency shift of the atomic spin oscillator, facilitated by ponderomotive squeezing, have been conducted.

At lower acoustic frequencies, the dominance of quantum backaction noise is com- promised by various classical noise sources and additional atomic spin noise at nearly DC levels. Experimentally investigating and mitigating these noise sources have enabled us to maintain the -3 dB squeezing down to 3 kHz, extending to -1.2 dB slightly below 1 kHz, thus bringing our proof-of-principle experiments closer to the gravitational wave bandwidth.

The thesis concludes with the discussion of our parallel achievement of ∼ - 7 dB nondegenerate entangled sources, which bridges the gap between the atomic system and gravitational wave detection, alongside the theoretical predictions for the broadband quantum noise reduction optimized using the well-calibrated experimental parameters. Additionally, I will also present our ’last-minute’ preliminary experimental achievement of broadband quantum noise reduction at 50 kHz with the joint measurement of the hybrid systems. This results sets stage for our ongoing proof-of-principle frequency-dependent entangled source, aimed at broadband quantum noise reduction in the acoustic frequency regime. Furthermore, the established hybrid system signifies a step towards quantum-enhanced magnetic sensing and the potential for quantum entanglement and teleportation with this hybrid entangled light-atomic system.