Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-36229

Some search engines missing in is_web_crawler()

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.2
    • Fix Version/s: 2.4
    • Component/s: Authentication
    • Labels:

      Description

      This means that some crawlers can't crawl closed sites. So far I've just got:

      • bing

      Any other suggestions welcome.

      The current list is...

      function is_web_crawler() {
          if (!empty($_SERVER['HTTP_USER_AGENT'])) {
              if (strpos($_SERVER['HTTP_USER_AGENT'], 'Googlebot') !== false ) {
                  return true;
              } else if (strpos($_SERVER['HTTP_USER_AGENT'], 'google.com') !== false ) { // Google
                  return true;
              } else if (strpos($_SERVER['HTTP_USER_AGENT'], 'Yahoo! Slurp') !== false ) {  // Yahoo
                  return true;
              } else if (strpos($_SERVER['HTTP_USER_AGENT'], '[ZSEBOT]') !== false ) {  // Zoomspider
                  return true;
              } else if (strpos($_SERVER['HTTP_USER_AGENT'], 'MSNBOT') !== false ) {  // MSN Search
                  return true;
              } else if (strpos($_SERVER['HTTP_USER_AGENT'], 'Yandex') !== false ) {
                  return true;
              } else if (strpos($_SERVER['HTTP_USER_AGENT'], 'AltaVista') !== false ) {
                  return true;
              }
          }
          return false;
      }

        Attachments

          Activity

            People

            Assignee:
            skodak Petr Skoda
            Reporter:
            dobedobedoh Andrew Nicols
            Integrator:
            Dan Poltawski
            Tester:
            Frédéric Massart
            Participants:
            Component watchers:
            Adrian Greeve, Jake Dallimore, Mathew May, Mihail Geshoski, Peter Dias
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Fix Release Date:
              3/Dec/12