You will rarely see speaker damage from an amplifier, high or low power, that isn’t pushed to the point it sounds pretty awful.
Having more power available than your speakers are rated for can make a difference in their response to transients, usually not audible until you have the volume way up. Most folks will never hear the difference either.
I used a 600 watt amp feeding speakers rated for 100 to 250 watts with no problems for years. I rarely had the speakers playing really loud, didn’t want to blow out my ears worse than they were.
Going the other way I used a 5 watt amp to drive speakers that had a recommended power of 60 Watts. They sounded pretty good at lower volume levels but you could really hear the distortion if you cranked them up.
If you dig around the Internet you can find a lot of good discussion on speaker efficiency and power handling versus amplifier power. You will have to sort out the good ones from the bad but it can be fun reading.