Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.9
    • Fix Version/s: 1.8.9, 1.9.5
    • Component/s: Administration
    • Labels:
      None

      Description

      Implement a simple drop in script to help delete profiles that already have spam in them.

        Gliffy Diagrams

        1. spam.php
          0.2 kB
          Amr Hourani
        2. spamcleaner_sesskey.txt
          3 kB
          Martin Dougiamas
        3. spamcleaner.php
          13 kB
          Amr Hourani
        4. spamcleaner.php
          11 kB
          Amr Hourani

          Issue Links

            Activity

            Hide
            dougiamas Martin Dougiamas added a comment - - edited

            This is now available:

            http://cvs.moodle.org/contrib/tools/spamcleaner/spamcleaner.php?view=co

            Your improvements are very welcome! Please improve the script if you have write access already, or post ideas and patches here in the tracker.

            The main thing I think we could add next would be a built-in list of phrases to search for, or perhaps a way to process all the texts with an external spam-checking engine.

            Show
            dougiamas Martin Dougiamas added a comment - - edited This is now available: http://cvs.moodle.org/contrib/tools/spamcleaner/spamcleaner.php?view=co Your improvements are very welcome! Please improve the script if you have write access already, or post ideas and patches here in the tracker. The main thing I think we could add next would be a built-in list of phrases to search for, or perhaps a way to process all the texts with an external spam-checking engine.
            Hide
            dongsheng Dongsheng Cai added a comment -

            Akismet could help, a PHP lib is available too.

            Show
            dongsheng Dongsheng Cai added a comment - Akismet could help, a PHP lib is available too.
            Show
            dongsheng Dongsheng Cai added a comment - http://www.achingbrain.net/stuff/php/akismet
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            Add sesskey to protect it a bit.

            (just guessing if we could start storing somewhere a table of big hashes detected as spam in order to provide one webservice with that. nothing "intelligent" for now, just hash-check).

            I the future this could be evolved to anything more complex (own heuristic checker, gateway to "ask" other spam detectors...). But the ws should remain the same.

            Ciao

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - Add sesskey to protect it a bit. (just guessing if we could start storing somewhere a table of big hashes detected as spam in order to provide one webservice with that. nothing "intelligent" for now, just hash-check). I the future this could be evolved to anything more complex (own heuristic checker, gateway to "ask" other spam detectors...). But the ws should remain the same. Ciao
            Hide
            dougiamas Martin Dougiamas added a comment -

            Attached is a patch to add sesskey, though some YUI magic seems to be interfering when I try it. Dongsheng or Eloy, can you please look into it and update CVS?

            A further feature idea would be to examine config settings at the start of this script and offer to fix forceloginforprofiles etc for the admin.

            Show
            dougiamas Martin Dougiamas added a comment - Attached is a patch to add sesskey, though some YUI magic seems to be interfering when I try it. Dongsheng or Eloy, can you please look into it and update CVS? A further feature idea would be to examine config settings at the start of this script and offer to fix forceloginforprofiles etc for the admin.
            Hide
            awyatt A. T. Wyatt added a comment -

            I do not exactly know how to attach related tracker issues, but here are some things that bear on this discussion:

            Admin approval of email account creation/course enroll:
            http://tracker.moodle.org/browse/MDL-9624

            Completely remove user profiles (I can't see this one):
            http://tracker.moodle.org/browse/MDL-17065

            Preventing spam in user proviles:
            http://tracker.moodle.org/browse/MDL-17107
            (and of course this one http://tracker.moodle.org/browse/MDL-17144)

            Show
            awyatt A. T. Wyatt added a comment - I do not exactly know how to attach related tracker issues, but here are some things that bear on this discussion: Admin approval of email account creation/course enroll: http://tracker.moodle.org/browse/MDL-9624 Completely remove user profiles (I can't see this one): http://tracker.moodle.org/browse/MDL-17065 Preventing spam in user proviles: http://tracker.moodle.org/browse/MDL-17107 (and of course this one http://tracker.moodle.org/browse/MDL-17144 )
            Hide
            tsala Helen Foster added a comment -

            Thanks atw.

            (For future reference, you can add related issues by using the 'Link' link on the left of the page.)

            Show
            tsala Helen Foster added a comment - Thanks atw. (For future reference, you can add related issues by using the 'Link' link on the left of the page.)
            Hide
            amr Amr Hourani added a comment -

            Hi,
            I changed the above script:

            1- removed the rs_fetch_next_record($users) because it cause this error: Call to a member function FetchRow() on a non-object in C:\xampp\htdocs\vle\lib\dmllib.php on line 827

            2- added image search, because some spammers add images such as Ads and p0rn0

            3- enhanced sql queries, joining all conditions rather than making an query in loop

            4- removing all redundunt queries and make them in one function to avoid any future mess if you want work on this script

            5- showing one record for the user who meets more than 1 keyword.. the previous script was showing more than 1 record for any user who has more than 1 keyword in his profile.

            hope this helps..

            please upload it to CVS.

            Cheers
            Amr!

            Show
            amr Amr Hourani added a comment - Hi, I changed the above script: 1- removed the rs_fetch_next_record($users) because it cause this error: Call to a member function FetchRow() on a non-object in C:\xampp\htdocs\vle\lib\dmllib.php on line 827 2- added image search, because some spammers add images such as Ads and p0rn0 3- enhanced sql queries, joining all conditions rather than making an query in loop 4- removing all redundunt queries and make them in one function to avoid any future mess if you want work on this script 5- showing one record for the user who meets more than 1 keyword.. the previous script was showing more than 1 record for any user who has more than 1 keyword in his profile. hope this helps.. please upload it to CVS. Cheers Amr!
            Hide
            dongsheng Dongsheng Cai added a comment -

            Thanks your improvement, Amr.
            I modified your code to use get_recordset_sql, this will help to deal with large record sets.

            Show
            dongsheng Dongsheng Cai added a comment - Thanks your improvement, Amr. I modified your code to use get_recordset_sql, this will help to deal with large record sets.
            Hide
            dougiamas Martin Dougiamas added a comment -

            I reviewed the code today and made quite a lot of cleanups. Please upgrade if you are using this script!

            Show
            dougiamas Martin Dougiamas added a comment - I reviewed the code today and made quite a lot of cleanups. Please upgrade if you are using this script!
            Hide
            amr Amr Hourani added a comment -

            The new script doesnt include image search.

            Show
            amr Amr Hourani added a comment - The new script doesnt include image search.
            Hide
            dongsheng Dongsheng Cai added a comment -

            Hi, Amr, image searching is included in "Autodetect common spam patterns"

            Show
            dongsheng Dongsheng Cai added a comment - Hi, Amr, image searching is included in "Autodetect common spam patterns"
            Hide
            amr Amr Hourani added a comment -

            thanks Dongsheng, i noticed it

            Show
            amr Amr Hourani added a comment - thanks Dongsheng, i noticed it
            Hide
            ralfh Ralf Hilgenstock added a comment -

            Error message in 1.9.2 2080711
            require_js: yui_json - file not found.

            and

            The entry "<img" has as consequence that all emoticons in profiles are declared as spam.

            Show
            ralfh Ralf Hilgenstock added a comment - Error message in 1.9.2 2080711 require_js: yui_json - file not found. and The entry "<img" has as consequence that all emoticons in profiles are declared as spam.
            Hide
            ralfh Ralf Hilgenstock added a comment -

            'cialis' finds specialist as term in profile.

            Show
            ralfh Ralf Hilgenstock added a comment - 'cialis' finds specialist as term in profile.
            Hide
            jsherk Jeff Sherk added a comment -

            The Bad Behavior script can prevent spam accounts from ever even being created in the first place... this is an extremely effective script!!

            See
            http://tracker.moodle.org/browse/MDL-17162

            and
            http://www.bad-behavior.ioerror.us/

            Show
            jsherk Jeff Sherk added a comment - The Bad Behavior script can prevent spam accounts from ever even being created in the first place... this is an extremely effective script!! See http://tracker.moodle.org/browse/MDL-17162 and http://www.bad-behavior.ioerror.us/
            Hide
            dougiamas Martin Dougiamas added a comment -

            Thanks Jeff, that's a very interesting development for spambot detection, I'll add it to the main bug MDL-17107.

            We still need a good clean up script, because of old sites already affected as well as those affected by human spammers.

            TODO:

            • Use format_text when printing the user->description!
            • We need to add blog searching (posts table) and show the same user info (just replace the description with the blog content).
            • Can someone come up with a better image search that avoids smilies? (Will require regular expressions I think which will slow down the search a lot).
            Show
            dougiamas Martin Dougiamas added a comment - Thanks Jeff, that's a very interesting development for spambot detection, I'll add it to the main bug MDL-17107 . We still need a good clean up script, because of old sites already affected as well as those affected by human spammers. TODO: Use format_text when printing the user->description! We need to add blog searching (posts table) and show the same user info (just replace the description with the blog content). Can someone come up with a better image search that avoids smilies? (Will require regular expressions I think which will slow down the search a lot).
            Hide
            dongsheng Dongsheng Cai added a comment -

            TODOs are fixed, please review.

            Show
            dongsheng Dongsheng Cai added a comment - TODOs are fixed, please review.
            Hide
            ralfh Ralf Hilgenstock added a comment -

            Hi Dongsheng

            ver 1.12. finds smilies furthermore for me.

            Show
            ralfh Ralf Hilgenstock added a comment - Hi Dongsheng ver 1.12. finds smilies furthermore for me.
            Hide
            dongsheng Dongsheng Cai added a comment -

            Hi, Ralf, what's your search keyword, is it "<img"?
            I tested, and no smilies for me.

            Show
            dongsheng Dongsheng Cai added a comment - Hi, Ralf, what's your search keyword, is it "<img"? I tested, and no smilies for me.
            Hide
            timhunt Tim Hunt added a comment -

            Putting back 1.8.x fix version that got lost.

            Show
            timhunt Tim Hunt added a comment - Putting back 1.8.x fix version that got lost.
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            Hi, can this be considered resolved?

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - Hi, can this be considered resolved?
            Hide
            mhuntley Martin Huntley added a comment -

            I'm using Moodle 1.9.2+ (Build: 20080723).

            When I go to run spamcleaner.php, I get error:

            require_js: yui_json - file not found.

            Show
            mhuntley Martin Huntley added a comment - I'm using Moodle 1.9.2+ (Build: 20080723). When I go to run spamcleaner.php, I get error: require_js: yui_json - file not found.
            Hide
            dougiamas Martin Dougiamas added a comment - - edited

            Martin, are you updating via CVS? If so, make sure you use the create new directories option. On command-line CVS that is -d, for example:

            cvs -q update -dP

            You might need to upgrade to Moodle 1.9.4 anyway.

            Show
            dougiamas Martin Dougiamas added a comment - - edited Martin, are you updating via CVS? If so, make sure you use the create new directories option. On command-line CVS that is -d, for example: cvs -q update -dP You might need to upgrade to Moodle 1.9.4 anyway.
            Hide
            timhunt Tim Hunt added a comment -

            Closing. Spam-cleaner is implemented and works nicely.

            Show
            timhunt Tim Hunt added a comment - Closing. Spam-cleaner is implemented and works nicely.
            Hide
            tsala Helen Foster added a comment -

            Dongsheng, thanks for fixing, and thanks for everyone's comments.

            We now have a documentation page: http://docs.moodle.org/en/Spam_cleaner

            Show
            tsala Helen Foster added a comment - Dongsheng, thanks for fixing, and thanks for everyone's comments. We now have a documentation page: http://docs.moodle.org/en/Spam_cleaner
            Hide
            amr Amr Hourani added a comment -

            keywords separated for non english auto terms

            Show
            amr Amr Hourani added a comment - keywords separated for non english auto terms
            Hide
            dougiamas Martin Dougiamas added a comment - - edited

            Thanks Amr, but would it be possible to post your changes as a patch rather than a whole file? Makes it easier to see what you changed.

            Ideally it should be in a new bug too, since this one is closed.

            Show
            dougiamas Martin Dougiamas added a comment - - edited Thanks Amr, but would it be possible to post your changes as a patch rather than a whole file? Makes it easier to see what you changed. Ideally it should be in a new bug too, since this one is closed.
            Hide
            amr Amr Hourani added a comment -

            file diff and new lang file are here.

            Show
            amr Amr Hourani added a comment - file diff and new lang file are here.

              People

              • Votes:
                5 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  13/May/09