Moodle
  1. Moodle
  2. MDL-35099

Convert hidden search/replace script into a proper core admin tool

    Details

    • Affected Branches:
      MOODLE_23_STABLE
    • Rank:
      43724

      Description

      Can someone please convert the very useful admin/replace.php script into a proper admin tool in core so it can be found and used?

      http://docs.moodle.org/23/en/Search_and_replace

        Issue Links

          Activity

          Hide
          Martin Dougiamas added a comment - - edited

          Note this script was moved into admin/tool/replace.php in 2.2.

          The primary use of it was to fix the current site URLs in texts. ie if you have moved the Moodle site, you need to fix any references to the old site.

          Since it's not using any callbacks to plugins to let them decide how to treat fields, it needs to be very restricted with good sanity checking so that only URLs can be replaced, with lots of 'Are you sure?' warnings.

          Show
          Martin Dougiamas added a comment - - edited Note this script was moved into admin/tool/replace.php in 2.2. The primary use of it was to fix the current site URLs in texts. ie if you have moved the Moodle site, you need to fix any references to the old site. Since it's not using any callbacks to plugins to let them decide how to treat fields, it needs to be very restricted with good sanity checking so that only URLs can be replaced, with lots of 'Are you sure?' warnings.
          Hide
          Luis de Vasconcelos added a comment - - edited

          Yes! Please!

          And please can support be added for all the database platforms that Moodle is supported on, including MSSQL. It seems that the script currenly only works on MySQL and Postgres. See MDL-26597.

          Trying to maintain multiple copies of a Moodle website (PROD, DEV, QA, DR, etc.) is really quite difficult without this Replace tool. And being able to do that without having to resort to running hacky and dangerous "search and replace" sql commands in the database will really earn Moodle some points in the business/corporate training space.

          Show
          Luis de Vasconcelos added a comment - - edited Yes! Please! And please can support be added for all the database platforms that Moodle is supported on, including MSSQL. It seems that the script currenly only works on MySQL and Postgres. See MDL-26597 . Trying to maintain multiple copies of a Moodle website (PROD, DEV, QA, DR, etc.) is really quite difficult without this Replace tool. And being able to do that without having to resort to running hacky and dangerous "search and replace" sql commands in the database will really earn Moodle some points in the business/corporate training space.
          Hide
          Luis de Vasconcelos added a comment -

          Linking to related issue - MDL-26597 (admin/replace.php fails under MSSQL).

          Show
          Luis de Vasconcelos added a comment - Linking to related issue - MDL-26597 (admin/replace.php fails under MSSQL).
          Hide
          Dan Poltawski added a comment -

          My -1 to make this 'more supported', its hacking the DB and if we make it more prominent we make ourselves more liable for the consequences (GPL get out clause aside )

          Show
          Dan Poltawski added a comment - My -1 to make this 'more supported', its hacking the DB and if we make it more prominent we make ourselves more liable for the consequences (GPL get out clause aside )
          Hide
          Martin Dougiamas added a comment -

          My +2 to override your -1 ... it's essential functionality for real Moodle maintenance and otherwise we give no clues to admins about it's necessity (and beginners can stuff things up worse writing raw SQL).

          Show
          Martin Dougiamas added a comment - My +2 to override your -1 ... it's essential functionality for real Moodle maintenance and otherwise we give no clues to admins about it's necessity (and beginners can stuff things up worse writing raw SQL).
          Hide
          Luis de Vasconcelos added a comment -

          I agree with Martin. Without this feature maintaining multiple copies of a Moodle website (example: PROD, PRE-PROD, QA, DEV, DR, etc.) is a big pain - all url's in the DEV/QA/PRE-PROD etc databases still point to the origional PROD url's when you restore a copy of the PROD site to those NON-PROD environments. In our case the DEV and QA boxes do not have internet access so the links to the prod URL's are all broken on the NON-PROD sites. And we want the links to point to appropriate environments. We need a workable way to restore our Moodle PROD site onto the non-prod environments without too much fuss or having to resort to dangerous "update dev_db_tables set urls = dev_urls where urls = prod_urls" scripts. This is essential for testing new versions of Moodle against our existing production data.

          Show
          Luis de Vasconcelos added a comment - I agree with Martin. Without this feature maintaining multiple copies of a Moodle website (example: PROD, PRE-PROD, QA, DEV, DR, etc.) is a big pain - all url's in the DEV/QA/PRE-PROD etc databases still point to the origional PROD url's when you restore a copy of the PROD site to those NON-PROD environments. In our case the DEV and QA boxes do not have internet access so the links to the prod URL's are all broken on the NON-PROD sites. And we want the links to point to appropriate environments. We need a workable way to restore our Moodle PROD site onto the non-prod environments without too much fuss or having to resort to dangerous "update dev_db_tables set urls = dev_urls where urls = prod_urls" scripts. This is essential for testing new versions of Moodle against our existing production data.
          Hide
          Dan Poltawski added a comment -

          From a brief discussion in the dev chat, this doesn't even deal with the html block (i.e. serialised block data, so for any real site it doesn't really work). So, I counter add another -1 :-P

          Show
          Dan Poltawski added a comment - From a brief discussion in the dev chat, this doesn't even deal with the html block (i.e. serialised block data, so for any real site it doesn't really work). So, I counter add another -1 :-P
          Hide
          Aparup Banerjee added a comment -

          my 2 paisa:
          would a proper solution here be some site wide (low level so that cli, ws, renderers etc could all go thru this) filter that does a configurable data(urls etc) regex in the use case of replacing urls? This would cater to on the fly 'compatible' DB data. I'm just thinking lots of s/search/replace mappings in a filter.

          It would avoid DB hacking if its on the fly and its not really tampering with stored data but a filter that can always change.
          Eventually it could have a 'Make it so, for real' button to do the DB replace if needed (for more realistic performance tests?)

          Show
          Aparup Banerjee added a comment - my 2 paisa: would a proper solution here be some site wide (low level so that cli, ws, renderers etc could all go thru this) filter that does a configurable data(urls etc) regex in the use case of replacing urls? This would cater to on the fly 'compatible' DB data. I'm just thinking lots of s/search/replace mappings in a filter. It would avoid DB hacking if its on the fly and its not really tampering with stored data but a filter that can always change. Eventually it could have a 'Make it so, for real' button to do the DB replace if needed (for more realistic performance tests?)
          Hide
          Paul Holden added a comment -

          Doesn't the html block deal with it by implementing the block_html_global_db_replace method[1] which gets called by db_replace...?

          [1] http://git.moodle.org/gw?p=moodle.git;a=blame;f=blocks/html/lib.php;h=03d58ca27ceeefbbde6ebfab1518deb3b1f74753;hb=refs/heads/master#l71

          Show
          Paul Holden added a comment - Doesn't the html block deal with it by implementing the block_html_global_db_replace method [1] which gets called by db_replace...? [1] http://git.moodle.org/gw?p=moodle.git;a=blame;f=blocks/html/lib.php;h=03d58ca27ceeefbbde6ebfab1518deb3b1f74753;hb=refs/heads/master#l71
          Hide
          Dan Poltawski added a comment -

          Yep, seems we were wrong on the html block comment, ignore me

          Show
          Dan Poltawski added a comment - Yep, seems we were wrong on the html block comment, ignore me
          Hide
          Owen McKenna added a comment -

          In my experience the current script is extremely slow. Perhaps the new/improved version will permit each component to run in isolation (db search/replace, rebuild course cache, blocks search/replace)? Even then, running each step by itself, I have found it necessary to introduce hacks to reduce the dataset so that the script can run to completion. Moodlers with large sites must have some homebrew they can share, to beef-up this essential tool, surely!

          Show
          Owen McKenna added a comment - In my experience the current script is extremely slow. Perhaps the new/improved version will permit each component to run in isolation (db search/replace, rebuild course cache, blocks search/replace)? Even then, running each step by itself, I have found it necessary to introduce hacks to reduce the dataset so that the script can run to completion. Moodlers with large sites must have some homebrew they can share, to beef-up this essential tool, surely!
          Hide
          Bret Miller added a comment -

          In my experience, I have to use the search/replace tool to replace 4 different elements when I create refresh my test site from the live site. So a "proper" tool would have to:

          1. Store the following in data: URL, moodledata location, moodle files location, database name.
          2. At admin login, compare those items with what's now running.
          3. If different notify the admin and give him/her the option to "update the database" to match the new settings.
          4. When activated, search/replace all 4 of those items globally in the database.
          5. Work on all supported database systems.

          That would be a start. I agree that a "proper" tool would need to allow modules to choose how to do this in its own data, but to get that done would be a monumental task. There would need to be some way to know which table(s) were associated with which modules, do the global search/replace for modules that didn't support the filter, and skip those tables where the module does support it.

          Perhaps fixing this a little at a time would help it get better...

          Show
          Bret Miller added a comment - In my experience, I have to use the search/replace tool to replace 4 different elements when I create refresh my test site from the live site. So a "proper" tool would have to: 1. Store the following in data: URL, moodledata location, moodle files location, database name. 2. At admin login, compare those items with what's now running. 3. If different notify the admin and give him/her the option to "update the database" to match the new settings. 4. When activated, search/replace all 4 of those items globally in the database. 5. Work on all supported database systems. That would be a start. I agree that a "proper" tool would need to allow modules to choose how to do this in its own data, but to get that done would be a monumental task. There would need to be some way to know which table(s) were associated with which modules, do the global search/replace for modules that didn't support the filter, and skip those tables where the module does support it. Perhaps fixing this a little at a time would help it get better...

            People

            • Votes:
              28 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated: