They may not have colonized many planets. They may have been wiped out so easily because of some kind of minimalistic choice on their part.
Strange that was also mentioned in Silentium,
Seems we are the exception when it comes to exploding population numbers, least for a evolved mamalian species so not every race has to spread over every inch of the galaxy so.......
as a race are we hoarders?