This was noticed on MDL-17929 by stronk7 (see comment), but the code there was copy-pasted from the function enrol_cohort_create_new_group() introduced in issue MDL-49380
When looking for existing group we look globally instead of within the given course.