Create a single topic forum and activate groups.
If separate groups are active students can't answer to the initial post.
If visible groups are active, students can answer in all groups even if they are not enrolled in the group.
I think there are several options to solve this:
In single topic forum the initial post is published separately for each group so group members can answer in their own group only.
Alternative: In single topic forum group settings are disabled.