The minimum power guaranteed by the USB standard is very low, significantly below what is required by Galileo. Many USB hosts provide more power than the standard but your mileage may vary - a lot. Note Galileo is not the only USB device having USB power problems.
The most likely "damage" when powering with USB only is: the Galileo firmware is trying to update some variables in the SPI flash while booting and not getting enough power to do that, leaving the SPI flash and firmware in an inconsistent state: "bricking" the board. In the worst cases the firmware can only be recovered with a Dediprog SPI flash programmer (which costs a bit more than the board itself). For more about Dediprog see the BSP build guide.