Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-38170

SimplePie: Cannot read https feeds through local proxy (Squid, Privoxy)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.3, 2.4.1, 2.5
    • Fix Version/s: 2.3.5, 2.4.2
    • Component/s: RSS
    • Labels:
    • Testing Instructions:
      Hide

      NOTE: In order to test this you must be using an HTTP proxy. Unfortunately it's not possible to test otherwise. It is also possible that this problem might depend on exactly which HTTP proxy you are using (I don't know!) so for a proper test, please test with current Moodle first and ensure you get the 'Before fix' behaviour described below, then test a version with this patch applied to ensure the error goes away.

      See below for instructions on setting up a proxy if you don't have one.

      0. Ensure your server is configured correctly to use the proxy.

      1. On a course page, turn editing on and choose 'Remote RSS feeds' from the 'Add block' dropdown.'
      2. In the block, click the Edit icon, then click 'Add/edit feeds' and 'Add a new feed'.
      3. Paste in the feed URL https://zeustracker.abuse.ch/monitor.php?urlfeed=configs (NOTE - if you repeat this test, avoid caching problems by adding an extra parameter to this URL which you increment each time you test, for example &frog=1, &frog=2, etc.)
      4. Hit 'Add a new feed'.

      EXPECTED: Feed is added successfully.

      BEFORE FIX: The following error appears:

      Error loading this RSS feed (A feed could not be found at https://zeustracker.abuse.ch/monitor.php?urlfeed=configs. A feed with an invalid mime type may fall victim to this error, or SimplePie was unable to auto-discover it.. Use force_feed() if you are certain this URL is a real feed.)

      TO SET UP A PROXY:

      A. Install Privoxy. I installed the Windows version 3.0.20 successfully. I did not set it to run on startup. I used default settings except that:
      i. If you need to access it from a different machine, you may need to change the listen-address.
      ii. You may want to set debug = 1 so you can be sure it's working (look at the display in the Privoxy window to check it makes requests)

      B. Set up Moodle to use your Privoxy server. By default, Privoxy works on localhost:8118.

      C. When running the above test, ensure that the https URL you are using for testing is accessible WITHOUT a proxy (i.e. you're going through Privoxy but the request would work without using a proxy). In other words if you are using an internal network that requires proxy, place a test file on a suitable https server within your network. This is because otherwise you have to configure Privoxy to work through your actual proxy and when I tried to do that, I couldn't make it work (plus it makes things more complicated).

      Show
      NOTE: In order to test this you must be using an HTTP proxy. Unfortunately it's not possible to test otherwise. It is also possible that this problem might depend on exactly which HTTP proxy you are using (I don't know!) so for a proper test, please test with current Moodle first and ensure you get the 'Before fix' behaviour described below, then test a version with this patch applied to ensure the error goes away. See below for instructions on setting up a proxy if you don't have one. 0. Ensure your server is configured correctly to use the proxy. 1. On a course page, turn editing on and choose 'Remote RSS feeds' from the 'Add block' dropdown.' 2. In the block, click the Edit icon, then click 'Add/edit feeds' and 'Add a new feed'. 3. Paste in the feed URL https://zeustracker.abuse.ch/monitor.php?urlfeed=configs (NOTE - if you repeat this test, avoid caching problems by adding an extra parameter to this URL which you increment each time you test, for example &frog=1, &frog=2, etc.) 4. Hit 'Add a new feed'. EXPECTED: Feed is added successfully. BEFORE FIX: The following error appears: Error loading this RSS feed (A feed could not be found at https://zeustracker.abuse.ch/monitor.php?urlfeed=configs . A feed with an invalid mime type may fall victim to this error, or SimplePie was unable to auto-discover it.. Use force_feed() if you are certain this URL is a real feed.) TO SET UP A PROXY: A. Install Privoxy. I installed the Windows version 3.0.20 successfully. I did not set it to run on startup. I used default settings except that: i. If you need to access it from a different machine, you may need to change the listen-address. ii. You may want to set debug = 1 so you can be sure it's working (look at the display in the Privoxy window to check it makes requests) B. Set up Moodle to use your Privoxy server. By default, Privoxy works on localhost:8118. C. When running the above test, ensure that the https URL you are using for testing is accessible WITHOUT a proxy (i.e. you're going through Privoxy but the request would work without using a proxy). In other words if you are using an internal network that requires proxy, place a test file on a suitable https server within your network. This is because otherwise you have to configure Privoxy to work through your actual proxy and when I tried to do that, I couldn't make it work (plus it makes things more complicated).
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
    • Fixed Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE
    • Pull Master Branch:
      MDL-38170-master

      Description

      It's fairly rare for RSS/Atom feeds to be served over https, but does happen. Here's an example I found:

      https://zeustracker.abuse.ch/monitor.php?urlfeed=configs

      This feed will work fine when not using a proxy, but when using a proxy it fails. The reason for failure is that the resulting HTTP headers (when using a standard open-source Squid proxy) include two sets of status codes: 200 Connection Established, followed by the normal 200 OK. The system then fails to identify the headers (which follow the second status code) as headers.

      Docs for this appear to be at: http://muffin.doit.org/docs/rfc/tunneling_ssl.html

      (I'm investigating this issue further and will add more detail.)

      It was while investigating this issue that I found MDL-38168; as it happens, the feed in question was a local one, so fixing the bypass support solves that problem. But it ought to work for remote https feeds as well.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Votes:
                  2 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:
                    Fix Release Date:
                    11/Mar/13