I made a patch for MDL-37132 which seems to be the tip of the iceberg in a bunch of potential improvements and refactoring around file downloading scalability and performance.

csv, ods, excel file downloads in Moodle are almost universally bad at scale, the solution in MDL-37132 is an improvement to the csvlib to stream from a DB RecordSet or Iterator, through an optional transforming function, and then encoded as csv line by line and flushed continually over the network giving it a fixed memory footprint and almost instantaneous download start. But this is just one download option in one module.

Almost every large client of ours has been burnt by this bug in different places depending on what parts of moodle they happen to be using massively.

I'd like to take this solution further and also add streaming download support for the other download types, eg .ods and excel which will be a fairly major refactor of many places in moodle as it's a fundamental change in API. But if we're going to do this, then I think we could go an extra extra step and abstract away the various formats into a new plugin type. ie a module like mod_feedback, or the grade book just says 'I have a RecordSet or an Iterator, please present this as a bunch of alternative downloads to the user'. Then if I install a new 'format' like say RDF, or JSON, then every report or export in moodle gets this new format for free.

So a new api something this which renders the dropdown with all the different export options:

report.php:

$OUTPUT->download_as_chooser(
get_string('exportgrades', 'feedback'), // Name on button dropdown
new moodle_url('export.php', array('id' => $id)), // base url
'type', // optional name of query param which will have download format

export.php:

$format = required_param('type', PARAM_ALPHANUM);

$rs = $DB->get_record_set(...);

$download = new downloader();
$download->set_filename('feedback');
$download->set_key_names(array('name' => get_string('name') ...... ); // eg human readable csv header names
$download->download_from_resultset($rs, $callback, $type);

Under the hood we'd have a core set of 'formats' something like:

/format/csv/
/format/ods/
/format/xls/

These would either wrap existing API's or reimplement as needed. In particular I am thinking of pulling in Spout as a dependency which does streaming ods, csv, and excel:

https://github.com/box/spout

And then as bonus points I'd also do a json and rdf export but these would probably only go into the plugin dir and not into core.

There are probably other considerations:

• would a 'format' also potentially provide a parse to accept incoming uploads? Ideally yes, but this would be optional for each format. The core ones should implement this. Is there any grade export / import code to consider or align with?
• each format would generally be content agnostic, but we'd still pass the type into the callback so the module has an opportunity to transform the data if needed for a particular format.
• some downloads, eg the feedback excel, are special snowflakes with text formatting and worksheets. These would need a case by case decision to keep or refactor.

Mentioning the big commiters around this: timhunt skodak nicolasconnault