Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69549

Create course content export API

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Setup

      1. Log in as admin and create a course
      2. Edit the course settings and:
        1. set a course summary, which includes some inline image
        2. include a Course image file
      3. Create a range of activities, including:
        1. A folder using both files and subfolders (in the Content -> Files section), and with a Description which includes inline images
        2. A forum whose Description contains text and some inline images (i.e. in the Description use the Atto Insert image button)
        3. An assignment whose Description contains text, and some inline images
        4. A page, whose Description contains text, and some inline images, and whose Content contains different text and images
        5. A quiz, whose Description contains text, and some inline images
        6. A page, whose Description contains text, and some inline images
      4. Edit several of the sections and set a Summary, including some inline images
      5. Make note of the ID for that course.
      6. Download testexport.php and save it to your moodle root folder.

      Standard tests

      1. Run the script in your browser, and adding the courseid to the URL, for example:

        http://localhost/sm/testexport.php?courseid=4
        

        1. Confirm that a new zip file was downloaded to your computer
        2. Confirm that the filename contained the word test_ and a number
      2. Unzip the zip file and expand the contents
        1. Confirm that there is an _course folder, which contains a summary and section folders
        2. Confirm that those summary/section folders contain all of the files used in the course summary, and the course sections
        3. View the index.html in your browser
          1. Confirm that all images were shown inline
      3. Open each of the activities in the index.html
        1. Confirm that the Description matches the description in the course, including any inline images
          Note: Any content which would be filtered is not filtered - i.e. MathJax will not be rewritten the same as in the course
      4. For the Folder activity:
        1. Confirm that all of the files in the folder are present
      5. For the Resource activity:
        1. Confirm that all of the files in the folder are present
      6. For the Page activity
        1. Confirm that the page Content is also shown

      Fancy tests

      1. Look at the size of the files that you downloaded in the previous test, and choose a value in the middle
        i.e. if your files are 98KB, 154KB, 200KB, 300KB; choose a value of about 160KB
        Make a note of those files
      2. Add a maxkb value to the URL, for example:

        http://localhost/sm/testexport.php?courseid=4&maxkb=160
        

        1. Confirm that the zip downloaded
      3. Unzip it and view the index.html in your browser
      4. Check it all back again as above, but for any file which is bigger than the size you chose:
        1. Confirm that it opens up on the live site
      Show
      Setup Log in as admin and create a course Edit the course settings and: set a course summary, which includes some inline image include a Course image file Create a range of activities, including: A folder using both files and subfolders (in the Content -> Files section), and with a Description which includes inline images A forum whose Description contains text and some inline images (i.e. in the Description use the Atto Insert image button) An assignment whose Description contains text, and some inline images A page, whose Description contains text, and some inline images, and whose Content contains different text and images A quiz, whose Description contains text, and some inline images A page, whose Description contains text, and some inline images Edit several of the sections and set a Summary, including some inline images Make note of the ID for that course. Download testexport.php and save it to your moodle root folder. Standard tests Run the script in your browser, and adding the courseid to the URL, for example: http://localhost/sm/testexport.php?courseid=4 Confirm that a new zip file was downloaded to your computer Confirm that the filename contained the word test_ and a number Unzip the zip file and expand the contents Confirm that there is an _course folder, which contains a summary and section folders Confirm that those summary/section folders contain all of the files used in the course summary, and the course sections View the index.html in your browser Confirm that all images were shown inline Open each of the activities in the index.html Confirm that the Description matches the description in the course, including any inline images Note: Any content which would be filtered is not filtered - i.e. MathJax will not be rewritten the same as in the course For the Folder activity: Confirm that all of the files in the folder are present For the Resource activity: Confirm that all of the files in the folder are present For the Page activity Confirm that the page Content is also shown Fancy tests Look at the size of the files that you downloaded in the previous test, and choose a value in the middle i.e. if your files are 98KB, 154KB, 200KB, 300KB; choose a value of about 160KB Make a note of those files Add a maxkb value to the URL, for example: http://localhost/sm/testexport.php?courseid=4&maxkb=160 Confirm that the zip downloaded Unzip it and view the index.html in your browser Check it all back again as above, but for any file which is bigger than the size you chose: Confirm that it opens up on the live site
    • Affected Branches:
      MOODLE_310_STABLE
    • Fixed Branches:
      MOODLE_310_STABLE
    • Pull 3.10 Branch:
      MDL-69549-310-7
    • Pull Master Branch:
      MDL-69549-master-7
    • Story Points:
      2
    • Sprint:
      International 4.0 - Sprint 5, International 4.0 - Sprint 6, International 4.0 - Sprint 7

      Description

      API details

      The proposed API for course export is a part of a new API for content.

      The content API will include features for:

      1. identifying, locating, and controlling access of files within a component, or subsystem, which are a part of the File Storage API, or are served via one of the pluginfile.php endpoints (MDL-66006 / MDL-69708)
      2. identifying, locating, and describing text areas within Moodle; and
      3. exporitng content stored within a component, or subsystem.

      The new API will be the content API, and will not be a subsystem as it is expected to become more and more integral to parts of Moodle.

      All core classes will be located in the core\content namespace, with subsystems and components placing their code and extensions of this API in [frankenstyle_component]\content.

      The Content Export API will be located within the core\content\export namespace, and with subsystems and components placing their code and extensions of this API in [frankenstyle_component]\content\export.

      API Components

      The API is made up of several core component types:

      1. The zipwriter, located in core\content\export\zipwriter is a wrapper around the ZipStream\ZipStream library and adds additional functionality and business logic related specifically to the export of content. It also includes helpers to add all pluginfiles which relate to a textarea, rewriting the @@PLUGINFILE@@ references in that textarea to point to the exported files.
      2. Exporters, located in core\content\export\exporters are the classes which perform the work of collecting all relevant content together, and adding it to the zipwriter for a given context;
      3. Exportable Items, located in core\content\export\exportable_items and extending the abstract core\content\export\exportable_item class are helpers which describe the content being exported and prepare it for export. These cover cases such as the export of a single stored_file object, all stored_file instances within a specific filearea in the file_storage API, and the ecport of a text field within the database. These exportable_items allow an indivdual component (subsystem or plugin) to describe in an abstract way the content that they include. Each exportable item additionally includes the logic to actually add the content it describes to the zipwriter archive.
      4. Items which have been exported, located in core\content\export\exported_item which provide a generic way for a component to describe what was exported in such a way that can be used for a human-readable HTML file. This includes features such as a title, shown to the user, a list of files, and any string content (typically a text field like a description) that relates to the exported item.

      This API is designed to support the export of course content including activities. At a sutiable time in the future it would be possible to extend it to support the export of blocks within a course, and other parts of the site, including course categories, or any other arbitrary context.

      Exporters

      The exporters included in core\content\export\exporters as standard include:

      1. The abstract component_exporter which all other exporters must extend.
        It includes helpers used to fetch the context being exported, the name of the component being exported, and the zipwriter.
        It also includes helpers to export a set of exportable items.
        It describes a get_exportables() function which, by default, returns an empty array.
      2. The abstract mod_instance_exporter, which extends the abstract component_exporter, and is intended to be extended by activities implementing the API.
        For example, the mod_page component defines the mod_page\content\exporter which extends the mod_instance_exporter.
        This class adds helpers for fetching the cm_info and activity name.
      3. The mod_exporter which is used to export generic data which is defined in core.
        This is required as the "intro" feature of activities is not an instance feature, but a part of of the general API for activities.
        For an activity support the intro field, it just has to declare that it supports the feature and core handles the rest.
        This class does not define its own list of exportables, but rather takes in the list of exportables from an activity and handles their export.
        It is a feature of the core\content\export API itself, and is not a part of the public API. It is declared final and cannot be extended. It is only called by the core API.
        Essentially the calling code would look a little like the following:

        foreach ($contexts as $context) {
          $component = get_component_from_context($context); // Some made up function to get the mod_[name] from an activity context.
          $exportables = [];
         
          // Fetch the exportables that the activity instance defines.
          $classname = component_exporter::get_classname_for_component($component);
          if (class_exists($classname)) {
            $exporter = new $classname($context, $component, $user, $archive);
            $exportables = $exporter->get_exportables();
          }
         
          // Pass to the core mod_exporter to handle the remaining export.
          $modexporter = new \core\content\export\exportables\mod_exporter($context, $component, $user, $archive);
          $modexporter->export_exportables($exportables);
        }
        

        The export_exportables function is responsible for fetching the module intro, should the activity support it, adding all exportable content defined in the activity to the zipwriter, and generating a human-readable module index within the zip archive.

      4. The course_exporter, which is used to export a whole course.
        Again this is not a part of the public API, and only called by core\content itself.
        It does not return anything in the get_exportables() function but defines an export_course function which takes a list of all contexts which were successfully included in the export.
        This function adds the course index to the zip stream, which includes all course sections, and links to each activity within the course.
        Where an activity was included in the downloaded content, all links to the activity point to the module index created by the module_exporter.
        Where an activity was not included in the downloaded content, links will point to the live site. This will be used in the future to support cases where a specific activity has been configured to not be exported. This allows the course structure to remain and all data to be shown in context, but for those activities not exported to point to the live site instead.
      Course module exporters

      Any individual activity/course module can define its own exporter.

      Those included in this patch are for:

      1. mod_page - to export the Content part of a Page activity;
      2. mod_folder - to export all files and folders within the activity; and
      3. mod_resource - to export the single file within the activity.

      Each activity extends the core\content\export\exporters\mod_instance_exporter class, must override the get_exportables function, and have that function return a list of exportable_item instances.

      Exportable items

      Exportable items are used to describe all content to be exported in an abstract and simplified way so as to reduce duplication across the codebase.

      All content is exported as either Text, or a collection of Files. No other types are supported or expected at this time.

      Three standard item types are defined in core, but where these do not fit a component is able to write its own exportable_item type.

      Most activities will likely export an exportable_textarea, which takes a table name, field name, and row ID.
      For text areas which can have files embedded it also takes an optional filearea, and itemid which are used to fetch the file from the File Storage API, and an optional pluginfileitemid which is the itemid used in the /pluginfile.php/ URL.
      When adding to the archive, all files in the filearea/itemid will be added to the zip archive. As each file is added, the content will be rewritten to find any use of @@PLUGINFILE@@ in the text which relates to that specific file, and rewrite it to point to the version stored in the zip archive.
      After all files have been exported and the content is rewritten, any remaining uses of @@PLUGINFILE are rewritten to point to the live site, using the tokenpluginfile.php endpoint which allows viewing without requiring a current login.

      For activities which purely export files, two additional types have been defined:

      1. the exportable_stored_file, which allows for a single file to be exported; and
      2. the exportable_filearea, which allows for all files within a filearea/itemid combination for a specific component + context combination to be exported
      Exported items

      In order that exported content can be adequately described in a human-readable format, all exported content must be described in an exported_item.

      This includes a Title, any string content, and two lists of files. It is consumed by the course_exporter and mod_exporter to create template data used in the Mustache templates.

        Attachments

        1. exporter.php
          4 kB
        2. MDL-69549_indexhtml.jpg
          MDL-69549_indexhtml.jpg
          19 kB
        3. MDL-69549_zipfile.jpg
          MDL-69549_zipfile.jpg
          19 kB
        4. Screenshot1.png
          Screenshot1.png
          2.33 MB
        5. Screenshot2.png
          Screenshot2.png
          516 kB
        6. Screenshot3.PNG
          Screenshot3.PNG
          187 kB
        7. testexport.php
          1.0 kB

          Issue Links

            Activity

              People

              Assignee:
              dobedobedoh Andrew Nicols
              Reporter:
              dobedobedoh Andrew Nicols
              Peer reviewer:
              Simey Lameze
              Integrator:
              Adrian Greeve
              Tester:
              Gladys Basiana
              Participants:
              Component watchers:
              Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Sara Arjona (@sarjona)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                9/Nov/20

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 week, 4 days, 2 hours, 40 minutes
                  1w 4d 2h 40m