If a user has role A with "mod/forum:allowforcesubscribe" set to "Prevent" in a course and role B in a parent category that user will still be forced subscribed to forums.

According to the way capabilities are calculated there is an even number of Allow and Prevent, so the capability check should fail: http://docs.moodle.org/20/en/How_permissions_are_calculated

Looking at the code, the problem seems to be mod/forum/lib.php: forum_get_potential_subscribers and the call to get_enrolled_sql. The get_enrolled_sql function doesn't seem to take into account "Prevent" in filtering out users.

The code for get_enrolled_sql is very complex, so I am not sure I can provide a patch. If that is the expected behavior of get_enrolled_sql, then it should be documented that the $withcapability only restricts capabilities with "Prohibit". Also, the forum checks should maybe use some other method of obtaining forced subscribers with a more rigorous capability check.