Each player chooses a color. Then exile each permanent unless it's colorless or it's only the color its controller chose.
"I don't know what's worse—that my family was taken, or that I wasn't." —Elsen, Sea Gate survivor