Steps:
Create an essay question type.
Enter the required details and set the following field values: Allow attachments = 'Unlimited' and Require attachments = '3'.
Save changes.
Now edit the question and set Allow attachments = 'No'
Expected result: User should be allowed to set the Allow attachments = 'No'
Actual result: Error message is displayed - 'You cannot require more attachments than you allow.'
and is in the hidden fields which can seen by again setting the Allow attachments = 'Unlimited' or any other value..

The alternative workaround is to first set Require attachments = 'Attachments are optional' and save changes. Then edit the question and set Allow attachments = 'No', the changes can be saved and works fine.