Details
Improvement
Description
While researching about http://docs.moodle.org/en/Development:Backup_2.0_-_Improve_XML_parsing and some important bugs like MDL-15489 one of the things being planned for 2.0 is to split the current moodle.xml file into bunch of smaller files like:
- info
- course_header
- course_structure
- users
- modules (one file per activity)
- blocks
- ...
This will allow the parser to perform it's job (reading contents from XML and pushing them to in-memory structures) really quicker than current approach, where the whole moodle.xml file in parsed up to 19 times (19 is the number of "TODO"s - parts present in a moodle.xml that are parsed separately).
While that split is going to be done in Moodle 2.0 backup (in order to allow restore to handle directly those smaller files), I thought that, perhaps it could be interesting also to perform some split in Moodle 1.9.
After some (a lot!) of coding and tests, comparing results, I've ended with:
1) One parser (moodle_splitter_parser) able to split the moodle.xml file into 19 smaller files.
2) Hack the main restore parser (MoodleParser) to use those split files instead of the original moodle.xml one
3) Put all that code under one experimental $CFG->experimentalsplitrestore configuration setting
While results aren't noticeable in small backups, processing one 50MB files, spent initially 50 seconds performing the split, but later, the whole restore ended 350 seconds quicker, so we saved 300 seconds of non-useful parsing time. Note results can a lot, depending of the contents of the backup, but in any case, the split time should be always smaller than the saved time.
So, I'm going to commit this both to 19_STABLE and HEAD (for the "legacy" Moodle 1.9 => 2.0 restore). It would be great to have people using and testing it, in order to get some feedback to promote the split strategy to be enabled always. I've restored at least 10 different courses of all sort of types and sizes and everything is working ok here. Let's see how it evolves.
Ciao 
Attachments
Issue Links
| This issue has a non-specific relationship to: | ||||
| MDL-18482 | Restore XML parsing: Improve for speed by skipping non-selected modules |
|
|
|
Activity
- All
- Comments
- History
- Activity
- Source
- Test Sessions
Code has been committed both to 19_STABLE and HEAD, with one "experimental" setting (under Admin) controlling it 8defaults to disabled).
Ray, it depends of what do you mean by "incompatible":
- I'm 100% sure that Moodle 1.9.x backups will restore in 2.0 without problems. No doubt about that. It's a primary objective.
- I'm 90% sure that Moodle 2.0 backups will have a different internal format (see http://docs.moodle.org/en/Development:Backup_2.0 - specially the ongoing research about parsing). My current idea is that Moodle 2.0 won't have one "monolithic" moodle.xml file containing all the backup info, but a bunch of smaller files containing "specialised parts". That will be a huge improvement both for backup and restore.
Resolving this as fixed... ciao
- I'm 100% sure that Moodle 1.9.x backups will restore in 2.0 without problems. No doubt about that. It's a primary objective.
- I'm 90% sure that Moodle 2.0 backups will have a different internal format (see http://docs.moodle.org/en/Development:Backup_2.0 - specially the ongoing research about parsing). My current idea is that Moodle 2.0 won't have one "monolithic" moodle.xml file containing all the backup info, but a bunch of smaller files containing "specialised parts". That will be a huge improvement both for backup and restore.
Eloy, thanks a lot for this improvement
Information added to the documentation wiki:
http://docs.moodle.org/en/Backup_and_restore_FAQ
http://docs.moodle.org/en/Moodle_1.9.5_release_notes
Information added to the documentation wiki:
http://docs.moodle.org/en/Backup_and_restore_FAQ
http://docs.moodle.org/en/Moodle_1.9.5_release_notes 
Attached basic patch that implements the splitter, based on $CFG->experimentalsplitrestore
I've optimised the splitter in every bit I've been able to do so. Has a good throughput right now, IMO (and a small memory usage).
Missing bits:
Ciao
- admin setting frontend.
- lang string.
Ciao