Group Members should be greyed out if they aren't able to be removed.

If you have users who's group membership is being controlled by an external resource (right now the only thing in core that does it is enrol_cohort (and it should be done by enrol_meta)) such that xx_yy_allow_group_member_remove returns false for a given itemid/user/group combination, that user shouldn't even be a valid choice in the group_members_selector.

While cohorts (and meta) are the only plugins that currently makes use of the group_members.component field in core, we (and likely others) have custom modules that also make use of this.