Moodle

Read/Write splitting of DB operations.

Details

  • Type: New Feature New Feature
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: None
  • Component/s: Performance
  • Labels:
    None
  • Affected Branches:
    MOODLE_20_STABLE

Description

It would give an advantage on clusters if Moodle could allow slave (read-only) databases. Moodle could use master DB server for all write operations and could use slave DB server for all read operations. It will make clustering with Moodle simpler to setup and maintain.

DB layer 2.0 could make it possible (easier) to do.

Issue Links

Activity

Hide
Petr Škoda (skodak) added a comment -

yes it should be possible to write new database driver for this, but I do not think it will be implemented and maintained in official Moodle distribution; I think that better performance gains should be possible via clever shared memory usage

Show
Petr Škoda (skodak) added a comment - yes it should be possible to write new database driver for this, but I do not think it will be implemented and maintained in official Moodle distribution; I think that better performance gains should be possible via clever shared memory usage
Hide
jai gupta added a comment -

Shared memory is practical in cases where DB size is small compared to budget to buy memory. If shared memory is not clever enough it can actually lead to increase of response time. Separating read and writes operations could help people run Moodle at lower cost.

Drupal will also have native support for master-slave replication in its next release.
http://www.garfieldtech.com/blog/drupal7-database-update
http://buytaert.net/scaling-with-mysql-replication

They also have released a backport for its current version.
http://drupal.org/node/147160

I think it will really make Moodle scalable if it could work with multiple databases at the same time.

Show
jai gupta added a comment - Shared memory is practical in cases where DB size is small compared to budget to buy memory. If shared memory is not clever enough it can actually lead to increase of response time. Separating read and writes operations could help people run Moodle at lower cost. Drupal will also have native support for master-slave replication in its next release. http://www.garfieldtech.com/blog/drupal7-database-update http://buytaert.net/scaling-with-mysql-replication They also have released a backport for its current version. http://drupal.org/node/147160 I think it will really make Moodle scalable if it could work with multiple databases at the same time.
Hide
Petr Škoda (skodak) added a comment -

I think in shared memory should be stored data that is expensive to get from database, for example relative small and constant objects that require very complex queries - like guest access data.
Another our problem is repeated database queries such as get_record() inside looks - we still do a lot of these, I think that removing of these problems should have higher priority.

My personal priority is:
1/ eliminate inefficient queries - everybody benefits immediately, no extra set-up needed
2/ implement clever shared memory use - anybody with configured shared memory benefits
3/ read/write separation - probably in contrib only, only special master/slave set-ups would benefit

I believe that it would be much better to spend time on rewriting capability evaluation and enrolments in 2.0 instead of this project, sorry.

Show
Petr Škoda (skodak) added a comment - I think in shared memory should be stored data that is expensive to get from database, for example relative small and constant objects that require very complex queries - like guest access data. Another our problem is repeated database queries such as get_record() inside looks - we still do a lot of these, I think that removing of these problems should have higher priority. My personal priority is: 1/ eliminate inefficient queries - everybody benefits immediately, no extra set-up needed 2/ implement clever shared memory use - anybody with configured shared memory benefits 3/ read/write separation - probably in contrib only, only special master/slave set-ups would benefit I believe that it would be much better to spend time on rewriting capability evaluation and enrolments in 2.0 instead of this project, sorry.
Hide
jai gupta added a comment -

Right, I understand.

I am not tracking 2.0 status. Google docs spreadsheet shows 9.9 days left for Database API. If I want to take read/write splitting ahead then when would it be a good idea to start. I can start this as soon as all queries in Moodle 2.0 start using the new database API.

Show
jai gupta added a comment - Right, I understand. I am not tracking 2.0 status. Google docs spreadsheet shows 9.9 days left for Database API. If I want to take read/write splitting ahead then when would it be a good idea to start. I can start this as soon as all queries in Moodle 2.0 start using the new database API.
Hide
Petr Škoda (skodak) added a comment -

You can start now, the API is 99.9% finished and all core code is already using it for nearly a year.
Eloy is working on more unit tests, they should be in cvs soon. Current plan is to create missing native mssql drivers and fix oracle driver.

Show
Petr Škoda (skodak) added a comment - You can start now, the API is 99.9% finished and all core code is already using it for nearly a year. Eloy is working on more unit tests, they should be in cvs soon. Current plan is to create missing native mssql drivers and fix oracle driver.
Hide
jai gupta added a comment -

Great, thanks.

Show
jai gupta added a comment - Great, thanks.
Hide
Juan Segarra Montesinos added a comment -

Hi all

Maybe R/W sppliting is not the best for moodle. Basically in places where write and read operations are performed quickly and in asynchronous or semisynchronius configurations (mysql or postgres replication). We tried this with mysql-proxy and read-write sppliting script with semisynchronous replication, the slave delay made some odd results in the database and errors. But it'll be great to have a second opinion and experience

However I think that stats code is the best place that can benefit from R/W sppliting.

It'll be great to track your progress wih this

Regards

Show
Juan Segarra Montesinos added a comment - Hi all Maybe R/W sppliting is not the best for moodle. Basically in places where write and read operations are performed quickly and in asynchronous or semisynchronius configurations (mysql or postgres replication). We tried this with mysql-proxy and read-write sppliting script with semisynchronous replication, the slave delay made some odd results in the database and errors. But it'll be great to have a second opinion and experience However I think that stats code is the best place that can benefit from R/W sppliting. It'll be great to track your progress wih this Regards
Hide
jai gupta added a comment -

If mysql-proxy had been working fine that would have eliminated the need for Moodle (or any other app) to support R/W splitting.

http://forge.mysql.com/wiki/MySQL_Proxy_RW_Splitting
http://jan.kneschke.de/2007/8/1/mysql-proxy-learns-r-w-splitting

Show
jai gupta added a comment - If mysql-proxy had been working fine that would have eliminated the need for Moodle (or any other app) to support R/W splitting. http://forge.mysql.com/wiki/MySQL_Proxy_RW_Splitting http://jan.kneschke.de/2007/8/1/mysql-proxy-learns-r-w-splitting
Hide
Jason Cameron added a comment -

Are the R/W splitting operations still going to take place in the 2.x Moodle? I understand it may not be the best for Moodle, but it could have a major impact on the user's database depending on how much resources they have dedicated to it.

Thanks,
Jason

Show
Jason Cameron added a comment - Are the R/W splitting operations still going to take place in the 2.x Moodle? I understand it may not be the best for Moodle, but it could have a major impact on the user's database depending on how much resources they have dedicated to it. Thanks, Jason

People

Dates

  • Created:
    Updated: