Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-49840

Bundle composer.lock in git

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      1. Remove vendor directory:

        rm -rf vendor
        

      2. Run basic behat init:

        php admin/tool/behat/cli/init.php
        

        • Confirm that composer installs dependencies
        • Confirm that dependencies installed are the exact versions specified in composer.lock
        • Confirm that composer.lock has not changed
      3. Run same init again

        php admin/tool/behat/cli/init.php
        

        • Confirm that an informative message appeared only - no changes to composer dependencies etc.
      4. Run parallel init

        php admin/tool/behat/cli/init.php --parallel=2
        

        • Confirm that an informative message appeared only - no changes to composer dependencies etc.
      5. Confirm that the composer message was only seen once
      6. Remove vendor directory:

        rm -rf vendor
        

      7. Run parallel init

        php admin/tool/behat/cli/init.php --parallel=2
        

        • Confirm that composer installs dependencies
        • Confirm that dependencies installed are the exact versions specified in composer.lock
        • Confirm that composer.lock has not changed
      8. Confirm that the composer message was only seen once
      9. Remove vendor directory:

        rm -rf vendor
        

      10. Run PHPUnit init:

        php admin/tool/phpunit/cli/init.php
        

        • Confirm that composer installs dependencies
        • Confirm that dependencies installed are the exact versions specified in composer.lock
        • Confirm that composer.lock has not changed
      11. Run PHPUnit init:

        php admin/tool/phpunit/cli/init.php
        

        • Confirm that an informative message appeared only - no changes to composer dependencies etc.
      Show
      Remove vendor directory: rm -rf vendor Run basic behat init: php admin/tool/behat/cli/init.php Confirm that composer installs dependencies Confirm that dependencies installed are the exact versions specified in composer.lock Confirm that composer.lock has not changed Run same init again php admin/tool/behat/cli/init.php Confirm that an informative message appeared only - no changes to composer dependencies etc. Run parallel init php admin/tool/behat/cli/init.php --parallel=2 Confirm that an informative message appeared only - no changes to composer dependencies etc. Confirm that the composer message was only seen once Remove vendor directory: rm -rf vendor Run parallel init php admin/tool/behat/cli/init.php --parallel=2 Confirm that composer installs dependencies Confirm that dependencies installed are the exact versions specified in composer.lock Confirm that composer.lock has not changed Confirm that the composer message was only seen once Remove vendor directory: rm -rf vendor Run PHPUnit init: php admin/tool/phpunit/cli/init.php Confirm that composer installs dependencies Confirm that dependencies installed are the exact versions specified in composer.lock Confirm that composer.lock has not changed Run PHPUnit init: php admin/tool/phpunit/cli/init.php Confirm that an informative message appeared only - no changes to composer dependencies etc.
    • Affected Branches:
      MOODLE_26_STABLE, MOODLE_27_STABLE, MOODLE_28_STABLE, MOODLE_29_STABLE
    • Fixed Branches:
      MOODLE_29_STABLE
    • Pull Master Branch:
      MDL-49840-master

      Description

      Composer has two commands which we use:

      1. self-update - used to update composer.phar
      2. update - used to update dependencies all dependencies

      We are arguably using this latter one incorrectly.

      There are actually two commands used for installing dependencies, install and update, and they differ slightly but importantly. The difference is obvious - update updates the dependencies, install only installs them, but the semantic difference is huge.

      Composer tracks the installed versions using a lock file - composer.lock. When we call composer update, it performs a number of tasks;

      1. fetches the dependency tree for each required package
      2. calculates the optimal versions of all packages
      3. updates composer.lock with these details
      4. installs them

      composer update is slow. It can be very slow, especially on poorer connections. The reason is that it has to build a complete dependency tree, and work out which versions are compatible with which other versions. It is unable to use local caches for this stage because it must deal with the latest available information all the time. This means going back to packagist and finding that out.

      composer install on the other hand is pretty fast. It fetches the dependency tree and version information from composer.lock first, and then just installs the specified version. It does not have to try and build a dependency tree. It does not need to speak to the packagist repostiory, and it can use the local composer cache for all operations.

      So what should we be doing in Moodle?
      Well, IMO since we ship a composer.json with a list of mostly-fixed versions, we should:

      1. stop ignoring composer.lock
      2. have an integration task which calls `composer update` to update the dependencies and lock at a pre-defined period each week (e.g. Wednesday 6am Perth time before testing starts)
      3. modify testing_update_composer_dependencies() to call composer install instead of composer update
      4. always call install (except when running a parallel behat child)

      At present we only seem to call the testing_update_composer_dependencies() sometimes - we only actually call it if we bump a moodle version and invalidate behat. In reality composer version changes do not equate to behat needing to be run again. We should call it always and update to the versions stored in composer.lock. If we currently bump composer.json in a commit, but it's after we've run the moodle versions there will be inconsistent versions going around.

      What does this mean for Moodle?

      1. an extra tracked file (oh woe)
      2. a task (manual or automated) to update dependencies at some point - either before integration starts, before testing starts, or at a set point each week
      3. guaranteed versions for all people running tests at the same git commit
      4. we can make better use of the composer on-disk cache

      For comparison on my home connection:

      2020 sm:master> time composer update
      Loading composer repositories with package information
      Updating dependencies (including require-dev)
      Nothing to install or update
      Generating autoload files
       
      real	0m32.089s
      user	0m12.423s
      sys	0m0.420s
      

      2019 sm:master> time composer install
      Loading composer repositories with package information
      Installing dependencies (including require-dev) from lock file
      Nothing to install or update
      Generating autoload files
       
      real	0m0.680s
      user	0m0.504s
      sys	0m0.141s
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  11/May/15