I hate that saying. It dismisses the feelings/actual pain/problems of the person it is being said to.
What I really hate about it is that someone, somewhere actually is the worst off so why isn't the person saying it trying to find them and help them?
It's just said to shut someone down instead of helping them or helping them deal with their situation.