Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Component/s: download.moodle.org
    • Labels:
      None
    • Rank:
      46777

      Description

      The basic idea is that, instead of blindly generating the packages daily (no matter if they change or no - current approach), they only will be generated when there are effective changes triggering the package.

      This can be easily implemented by implementing one "post-receive" hook into one (target) repository that will launch the whole packaging/sync process by pushing changes from another (source) repository.

      Right now both the source and target repositories will be @ download.moodle.org, and pushing will happen in a cron-based way (MDLSITE-2045), but it's pretty easy to move the source repository to, say, Jenkins, and have the packaging controlled from there (with all the logs/facilities available there). Only some properly-restricted git write access (over SSH) would be necessary to achieve that.

      Also, the hook must be able to detect, based on some meta-information file, if the builds for some branch are stopped for some reason (old branches, for example), in order to be able to skip some packaging by configuration.

      The git hook will use the general packager created @ MDLSITE-2046, with some post-process to adjust it to download.moodle.org particularities.

      Ciao

        Issue Links

          Activity

          Hide
          Eloy Lafuente (stronk7) added a comment -

          Amended by setting git config's tar.umask to 0022, so it will generate permissions as previously was (MDLSITE-2061).

          Also, prevented the listing of the packager ".manifest" files in the directory indexes via .htaccess.

          Show
          Eloy Lafuente (stronk7) added a comment - Amended by setting git config's tar.umask to 0022, so it will generate permissions as previously was ( MDLSITE-2061 ). Also, prevented the listing of the packager ".manifest" files in the directory indexes via .htaccess.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Confirmed that stats are supporting the new moodle_BRANCH_date/tag name schema.

          Show
          Eloy Lafuente (stronk7) added a comment - Confirmed that stats are supporting the new moodle_BRANCH_date/tag name schema.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Added --branch support to the generic git-package packager. That will allow us to be be more exact/deterministic here when invoking the packager for a commit present in multiple branches.

          Show
          Eloy Lafuente (stronk7) added a comment - Added --branch support to the generic git-package packager. That will allow us to be be more exact/deterministic here when invoking the packager for a commit present in multiple branches.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Added prefix.executes support to the generic git-package packager. That will allow us to execute any arbitrary code before proceeding with packaging.

          Show
          Eloy Lafuente (stronk7) added a comment - Added prefix.executes support to the generic git-package packager. That will allow us to execute any arbitrary code before proceeding with packaging.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          First complete execution @ download.moodle.org successful. Steps executed were:

          cd /moodle/git_repositories/moodle.git
          git remote update origin
          git push package
          /usr/local/bin/mirror-all-to-sf.pl
          

          That ended with all the weekly packages generated automatically in their directories, apparently ok.

          Next steps:

          0) Look for the cleaner of old files, I have not been able to find it yet, and surely it needs some adjusting for the new file names.
          1) Instead of manual execution, proceed with the script above by polling (using git ls-remote) the moodle.git "main" repository from cron.
          2) Send results of the automated execution by email.
          3) Minor improvements to moodle-package so it will support manual execution better.
          4) Re-introduce the main manifest where each branch is defined properly (generation enabled, show @ download main page... and other details aimed to automate the index page output.

          Once everything above is done we'll proceed with the windows generators (MDLSITE-2048) and the releases info for the updates API (MDLSITE-2049).

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - First complete execution @ download.moodle.org successful. Steps executed were: cd /moodle/git_repositories/moodle.git git remote update origin git push package /usr/local/bin/mirror-all-to-sf.pl That ended with all the weekly packages generated automatically in their directories, apparently ok. Next steps: 0) Look for the cleaner of old files, I have not been able to find it yet, and surely it needs some adjusting for the new file names. 1) Instead of manual execution, proceed with the script above by polling (using git ls-remote) the moodle.git "main" repository from cron. 2) Send results of the automated execution by email. 3) Minor improvements to moodle-package so it will support manual execution better. 4) Re-introduce the main manifest where each branch is defined properly (generation enabled, show @ download main page... and other details aimed to automate the index page output. Once everything above is done we'll proceed with the windows generators ( MDLSITE-2048 ) and the releases info for the updates API ( MDLSITE-2049 ). Ciao
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Adding a brief schema about how things get packaged:
          (note that moodle-package can also be executed manually if some impeding re-packaging is needed):

          # This is the schema of the whole flow
          #
          #     moodle-package-detect-changes (shell script) ---------------> sourceforge sync
          #          (polls moodle.git via cron)
          #                      |
          #                    calls
          #                      |
          #      moodle-package-update-local (shell script) ----------------> mail execution
          #    (updates moodle.git and pushes to package.git)
          #                      |
          #                    invokes
          #                      |
          #        post-receive-moodle-package (package.git hook)
          #                      |
          #                    calls
          #                      |
          #           moodle-package (shell script)
          # (high level packager, prepares env, filenames, aliases, clean old..)
          #                      |
          #                    uses
          #                      |
          #            git package (git command)
          # (low level packager, supports includes, exludes and executes)
          
          Show
          Eloy Lafuente (stronk7) added a comment - - edited Adding a brief schema about how things get packaged: (note that moodle-package can also be executed manually if some impeding re-packaging is needed): # This is the schema of the whole flow # # moodle- package -detect-changes (shell script) ---------------> sourceforge sync # (polls moodle.git via cron) # | # calls # | # moodle- package -update-local (shell script) ----------------> mail execution # (updates moodle.git and pushes to package .git) # | # invokes # | # post-receive-moodle- package ( package .git hook) # | # calls # | # moodle- package (shell script) # (high level packager, prepares env, filenames, aliases, clean old..) # | # uses # | # git package (git command) # (low level packager, supports includes, exludes and executes)
          Hide
          Eloy Lafuente (stronk7) added a comment -

          First fully automated packaging of weeklies executed ok some mins ago, yay!

          Let's see how minor releases are generated later today...
          (going to work with MDLSITE-2048 and MDLSITE-2049 right now)

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - First fully automated packaging of weeklies executed ok some mins ago, yay! Let's see how minor releases are generated later today... (going to work with MDLSITE-2048 and MDLSITE-2049 right now) Ciao
          Hide
          Matteo Scaramuccia added a comment -

          Hi Eloy,
          is it possible to preserve someway the time info of each commit to let the files appear to be changed based on their time modified attribute?
          More about this use case in https://moodle.org/mod/forum/discuss.php?d=219224#p956193 and below.

          TIA,
          Matteo

          Show
          Matteo Scaramuccia added a comment - Hi Eloy, is it possible to preserve someway the time info of each commit to let the files appear to be changed based on their time modified attribute? More about this use case in https://moodle.org/mod/forum/discuss.php?d=219224#p956193 and below. TIA, Matteo
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Ah, Matteo, thanks for the link, I saw your post after finishing mine!

          as commented in the forum any attempt to calculate or maintain the file modification dates can lead to strange/nonsense/invented results.

          While CVS was able to keep file modifications, because it was 100% lineal, git is distributed by definition and one commit from 2 years ago can land today and all sort of combinations. Not to talk about how to "play" with those modification dates can affect to other progressive tools using that information to decide what to do (luckily we don't depend of "make" or other tools).

          I'm pretty sure that the git guys were aware of that problem and that's the cause they (on purpose) don't store/maintain the file modification at all. Instead, git itself can be used to ask for any change between 2 moments, with any granularity (stats, per file, with details...). But not based on file modification date, not at all.

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Ah, Matteo, thanks for the link, I saw your post after finishing mine! as commented in the forum any attempt to calculate or maintain the file modification dates can lead to strange/nonsense/invented results. While CVS was able to keep file modifications, because it was 100% lineal, git is distributed by definition and one commit from 2 years ago can land today and all sort of combinations. Not to talk about how to "play" with those modification dates can affect to other progressive tools using that information to decide what to do (luckily we don't depend of "make" or other tools). I'm pretty sure that the git guys were aware of that problem and that's the cause they (on purpose) don't store/maintain the file modification at all. Instead, git itself can be used to ask for any change between 2 moments, with any granularity (stats, per file, with details...). But not based on file modification date, not at all. Ciao
          Hide
          Matteo Scaramuccia added a comment -

          TNX for taking the time to comment it here too!

          Show
          Matteo Scaramuccia added a comment - TNX for taking the time to comment it here too!
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Closing this as fixed, the only remaining point in the TODO above is to implement the cleaner of old weekly files, and that has been delegated to MDLSITE-2086.

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Closing this as fixed, the only remaining point in the TODO above is to implement the cleaner of old weekly files, and that has been delegated to MDLSITE-2086 . Ciao

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development