Black holes (BHs) play a central role in physics. However, gathering observational evidence for their existence is a notoriously difficult task. Current strategies to quantify the evidence for BHs all boil down to looking for signs of highly compact, horizonless bodies. Here, we study particle creation by objects which collapse to form ultracompact configurations, with the surface at an areal radius R = R-f satisfying 1 -(2M/R-f) = epsilon(2) << 1 with M the object mass. We assume that gravitational collapse proceeds in a "standard" manner until R = R-f + 2M epsilon(2 ss), where ss > 0, and then slows down to form a static object of radius Rf. In the standard collapsing phase, Hawking-like thermal radiation is emitted, which is as strong as the Hawking radiation of a BH with the same mass but lasts only for similar to 40(M/M-circle dot) [44 + ln(10(-19)/epsilon)] mu s. Thereafter, in a very large class of models, there exist two bursts of radiation separated by a very long dormant stage. The first burst occurs at the end of the transient Hawking radiation and is followed by a quiescent stage which lasts for similar to 6 x 10(6) (epsilon/10(-19))-1(M/M-circle dot)yr. Afterwards, the second burst is triggered, after which there is no more particle production and the star is forever dark. In a model with ss = 1, both the first and second bursts outpower the transient Hawking radiation by a factor similar to 10(38) (epsilon/10(-19))(-2).