Honestly, lately I've been setting all users to ignore at start of my games, and my games are more enjoyable. (Expect a feature request from me on this!)
Especially since I've seen a few times a players keeps asking me to ally over and over, and then I can't close that invite windows. Games still playing fine, but half my screen is blocked by the invite window. I have to close out the game and rejoin to solve.