(1) It assumes a us-versus-them worldview of everyone versus persons of European descent.
(2) It defines people not by who they are, but who they're NOT.
(3) It assumed caucasians are "colourless," which is really pretty arbitrary
(4) It sets caucasians as the benchmark by which other people are defined (ie, having more "colour" than them)
(5) As opposed to more accurate terms like "minorities" it cannot be used to describe minority demographic tensions in places where both groups might be considered "of colour" such as the numourous ethnic tensions throughout Africa and well, anywhere else in the world pretty much.
My friend asked me what term I would use in place of "of colour," to which I said "minorities?" She proceeded claim that phrase was "dehumanizing" because it was a cold statistic that described people as not being the majority or some such (which she seemed to allege was an inhernetly pejorative prospect). I think this is really silly. Minority political groups never flinch at being called the political "minority," and their primary interest is in seeing that they are taken seriously and so forth. The argument went round and round but I really don't see how describing people by how much melanin they have in their skin compared to persons of European descent is exceedingly questionable.
I know that some scholars in related majors have espoused the "of colour" term, and their convoluted justifications are served up assembly-line style to students like my friend in liberal majors. My hypothesis, however, is that this term originated among people of minority groups who didn't think too hard about it and did in fact take a primarily us-versus-them view of "white people," but then scholars who were proud of their minority groups (which is fine) and wanted to justify everything they did (which is questionable) came up with their convoluted explanations of how "of colour" somehow makes sense as a politically viable term. And while in these liberal majors they spend their time stringently criticizing mainstream culture (which is fine), I think they are extremely reluctant to criticize one another, leading to a large scale groupthink scenario where you're a traitor to the revolution if you dare question the term "people of colour."