We propose and demonstrate a multicolor single-pixel digital holography technique. The intensity and phase images of an object are simultaneously obtained from the time-sequence intensity data captured using a single-pixel photodetector. Moreover, phase-structured light and phase-shifting interferometry are implemented using only a spatial light modulator without any mechanical movements. The adopted configuration dramatically simplifies the optical setup by removing both imaging optics and the resultant chromatic aberration. The effectiveness of the proposed technique is demonstrated numerically and experimentally.