I've been unable to verify if this problem persists in newer versions that 1.6.5 - but I should be able to soon
If a course has a large (but not ridiculous) number of quiz attempts (order of 1000 for a quiz with about 25 questions) backup and/or restore is severely compromised. It seems that building the XML for those attempts burns up crazy amounts of memory (we went up to half a gig before giving up). A similar problem is exhibited trying to restore the file - this has been verified as a problem in 1.8 STABLE.
IMHO, backup and restore shouldn't ever use that amount of memory, although I appreciate the solution will be complex.