Context cue is important in object classification. By embedding the depth context cue of image attributes into kernel descriptors, we propose a new set of depth image descriptors called depth context kernel descriptors (DCKD) for RGB-D based object classification. The motivation of DCKD is to use the depth consistency of image attributes defined within a neighboring region to improve the robustness of descriptor matching in the kernel space. Moreover, a novel joint spatial-depth pooling (JSDP) scheme, which further partitions image sub-regions using the depth cue and pools features in both 2D image plane and the depth direction, is developed to take full advantage of the available depth information. By embedding DCKD and JSDP into the standard object classification pipeline, we achieve superior performance to state-of-the-art methods on RGB-D benchmarks for object classification and scene recognition.