Dust Attack

My hardware wallet does not have coin control. I now have a couple of unknown dust deposits.

Do I need to be concerned and move to a wallet with coin control to isolate the dust? What can actually happen if I spend that dust. There are plenty of phishing attacks we already have to deal with from data leaks. So following my dust opens up more phishing attempts, but in and of itself it can’t cause issues can it?

Would this concern you enough to switch wallets?