If you have users who's group membership is being controlled by an external resource (right now the only thing in core that does it is enrol_cohort (and it should be done by enrol_meta)) such that xx_yy_allow_group_member_remove returns false for a given itemid/user/group combination, that user shouldn't even be a valid choice in the group_members_selector.

While cohorts (and meta) are the only plugins that currently makes use of the group_members.component field in core, we (and likely others) have custom modules that also make use of this.