Moodle
  1. Moodle
  2. MDL-38170

SimplePie: Cannot read https feeds through local proxy (Squid, Privoxy)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.3, 2.4.1, 2.5
    • Fix Version/s: 2.3.5, 2.4.2
    • Component/s: RSS
    • Labels:
    • Testing Instructions:
      Hide

      NOTE: In order to test this you must be using an HTTP proxy. Unfortunately it's not possible to test otherwise. It is also possible that this problem might depend on exactly which HTTP proxy you are using (I don't know!) so for a proper test, please test with current Moodle first and ensure you get the 'Before fix' behaviour described below, then test a version with this patch applied to ensure the error goes away.

      See below for instructions on setting up a proxy if you don't have one.

      0. Ensure your server is configured correctly to use the proxy.

      1. On a course page, turn editing on and choose 'Remote RSS feeds' from the 'Add block' dropdown.'
      2. In the block, click the Edit icon, then click 'Add/edit feeds' and 'Add a new feed'.
      3. Paste in the feed URL https://zeustracker.abuse.ch/monitor.php?urlfeed=configs (NOTE - if you repeat this test, avoid caching problems by adding an extra parameter to this URL which you increment each time you test, for example &frog=1, &frog=2, etc.)
      4. Hit 'Add a new feed'.

      EXPECTED: Feed is added successfully.

      BEFORE FIX: The following error appears:

      Error loading this RSS feed (A feed could not be found at https://zeustracker.abuse.ch/monitor.php?urlfeed=configs. A feed with an invalid mime type may fall victim to this error, or SimplePie was unable to auto-discover it.. Use force_feed() if you are certain this URL is a real feed.)

      TO SET UP A PROXY:

      A. Install Privoxy. I installed the Windows version 3.0.20 successfully. I did not set it to run on startup. I used default settings except that:
      i. If you need to access it from a different machine, you may need to change the listen-address.
      ii. You may want to set debug = 1 so you can be sure it's working (look at the display in the Privoxy window to check it makes requests)

      B. Set up Moodle to use your Privoxy server. By default, Privoxy works on localhost:8118.

      C. When running the above test, ensure that the https URL you are using for testing is accessible WITHOUT a proxy (i.e. you're going through Privoxy but the request would work without using a proxy). In other words if you are using an internal network that requires proxy, place a test file on a suitable https server within your network. This is because otherwise you have to configure Privoxy to work through your actual proxy and when I tried to do that, I couldn't make it work (plus it makes things more complicated).

      Show
      NOTE: In order to test this you must be using an HTTP proxy. Unfortunately it's not possible to test otherwise. It is also possible that this problem might depend on exactly which HTTP proxy you are using (I don't know!) so for a proper test, please test with current Moodle first and ensure you get the 'Before fix' behaviour described below, then test a version with this patch applied to ensure the error goes away. See below for instructions on setting up a proxy if you don't have one. 0. Ensure your server is configured correctly to use the proxy. 1. On a course page, turn editing on and choose 'Remote RSS feeds' from the 'Add block' dropdown.' 2. In the block, click the Edit icon, then click 'Add/edit feeds' and 'Add a new feed'. 3. Paste in the feed URL https://zeustracker.abuse.ch/monitor.php?urlfeed=configs (NOTE - if you repeat this test, avoid caching problems by adding an extra parameter to this URL which you increment each time you test, for example &frog=1, &frog=2, etc.) 4. Hit 'Add a new feed'. EXPECTED: Feed is added successfully. BEFORE FIX: The following error appears: Error loading this RSS feed (A feed could not be found at https://zeustracker.abuse.ch/monitor.php?urlfeed=configs . A feed with an invalid mime type may fall victim to this error, or SimplePie was unable to auto-discover it.. Use force_feed() if you are certain this URL is a real feed.) TO SET UP A PROXY: A. Install Privoxy. I installed the Windows version 3.0.20 successfully. I did not set it to run on startup. I used default settings except that: i. If you need to access it from a different machine, you may need to change the listen-address. ii. You may want to set debug = 1 so you can be sure it's working (look at the display in the Privoxy window to check it makes requests) B. Set up Moodle to use your Privoxy server. By default, Privoxy works on localhost:8118. C. When running the above test, ensure that the https URL you are using for testing is accessible WITHOUT a proxy (i.e. you're going through Privoxy but the request would work without using a proxy). In other words if you are using an internal network that requires proxy, place a test file on a suitable https server within your network. This is because otherwise you have to configure Privoxy to work through your actual proxy and when I tried to do that, I couldn't make it work (plus it makes things more complicated).
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
    • Fixed Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE
    • Pull 2.4 Branch:
      MDL-38170-m24
    • Pull Master Branch:
      MDL-38170-master
    • Rank:
      48001

      Description

      It's fairly rare for RSS/Atom feeds to be served over https, but does happen. Here's an example I found:

      https://zeustracker.abuse.ch/monitor.php?urlfeed=configs

      This feed will work fine when not using a proxy, but when using a proxy it fails. The reason for failure is that the resulting HTTP headers (when using a standard open-source Squid proxy) include two sets of status codes: 200 Connection Established, followed by the normal 200 OK. The system then fails to identify the headers (which follow the second status code) as headers.

      Docs for this appear to be at: http://muffin.doit.org/docs/rfc/tunneling_ssl.html

      (I'm investigating this issue further and will add more detail.)

      It was while investigating this issue that I found MDL-38168; as it happens, the feed in question was a local one, so fixing the bypass support solves that problem. But it ought to work for remote https feeds as well.

        Issue Links

          Activity

          Hide
          Dan Poltawski added a comment -

          Sam, one of your colleagues reported a similar problem - trying to find it.

          Show
          Dan Poltawski added a comment - Sam, one of your colleagues reported a similar problem - trying to find it.
          Hide
          Dan Poltawski added a comment -

          Actually it wasn't an OU person, for some reason I thought it was, I think this was the issue I was thinking of:
          https://tracker.moodle.org/browse/MDL-30648

          Show
          Dan Poltawski added a comment - Actually it wasn't an OU person, for some reason I thought it was, I think this was the issue I was thinking of: https://tracker.moodle.org/browse/MDL-30648
          Hide
          Sam Marshall added a comment -

          Code and unit test complete. I made the scope of this a bit wider by also handling other proxy issues such as those documented in MDL-30648.

          Show
          Sam Marshall added a comment - Code and unit test complete. I made the scope of this a bit wider by also handling other proxy issues such as those documented in MDL-30648 .
          Hide
          Sam Marshall added a comment -

          Please could somebody peer-review this. I have tested:

          a) Using the test script I wrote.
          b) Using the unit test, which includes my example and also the one Dan put in MDL-30648

          Show
          Sam Marshall added a comment - Please could somebody peer-review this. I have tested: a) Using the test script I wrote. b) Using the unit test, which includes my example and also the one Dan put in MDL-30648
          Hide
          Dan Poltawski added a comment -

          Hi Sam,

          In general i'm a bit uneasy about this solution, as it feels a bit like a 'hack'. It seems like we should be preventing the double headers from being 'remembered' by the curl class in the first place. But I think we've discussed that this isn't really easy to do with the way the curl stuff works.

          So I give it a +1 anyway, just acknowledging that it is a slightly hacky solution.

          You git commit message seems to be an interal one with a non- mdl issue number, so that needs to be fixed.

          [Y] Syntax
          [-] Output
          [Y] Whitespace
          [-] Language
          [-] Databases
          [Y] Testing
          [-] Security
          [-] Documentation
          [N] Git
          [Y] Sanity check

          +1 for integration once the commit message is sorted.

          Show
          Dan Poltawski added a comment - Hi Sam, In general i'm a bit uneasy about this solution, as it feels a bit like a 'hack'. It seems like we should be preventing the double headers from being 'remembered' by the curl class in the first place. But I think we've discussed that this isn't really easy to do with the way the curl stuff works. So I give it a +1 anyway, just acknowledging that it is a slightly hacky solution. You git commit message seems to be an interal one with a non- mdl issue number, so that needs to be fixed. [Y] Syntax [-] Output [Y] Whitespace [-] Language [-] Databases [Y] Testing [-] Security [-] Documentation [N] Git [Y] Sanity check +1 for integration once the commit message is sorted.
          Hide
          Sam Marshall added a comment -

          thanks Dan! (And oops, can't believe I used wrong format for the commit message - fixed.) Submitting for integration.

          Show
          Sam Marshall added a comment - thanks Dan! (And oops, can't believe I used wrong format for the commit message - fixed.) Submitting for integration.
          Hide
          Andrew Nicols added a comment -

          As an aside, useful testing notes:

          • purge caches after each time you successfully manage to download because the CURL class caches RSS feeds
          • you can use ssh -D PORT HOST to create a SOCKS proxy for testing
          Show
          Andrew Nicols added a comment - As an aside, useful testing notes: purge caches after each time you successfully manage to download because the CURL class caches RSS feeds you can use ssh -D PORT HOST to create a SOCKS proxy for testing
          Hide
          Eloy Lafuente (stronk7) added a comment -

          After trying to reproduce this by using like 30-40 free/public HTTP proxy servers (no luck, I got the error always, with and without the patch), I ended installing a squid proxy in my local network and only the I was able to reproduce it.

          I don't think this is SOCKS-related at all, but only HTTP-proxy related, isn't it? I bet it will work with SOCKS, thought it's 100% transparent and doesn't introduce any header at all.

          I've been using proxies from:

          They seem to work ok for normal requests, but not for the simplepie thingy. Perhaps they are adding other headers or...

          Anyway, Sam... given my tests... and the results of only being able to get it working for local squid proxies... do you think it needs more researching to cover other proxies or is enough to get local-squid fixed and forget?

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - After trying to reproduce this by using like 30-40 free/public HTTP proxy servers (no luck, I got the error always, with and without the patch), I ended installing a squid proxy in my local network and only the I was able to reproduce it. I don't think this is SOCKS-related at all, but only HTTP-proxy related, isn't it? I bet it will work with SOCKS, thought it's 100% transparent and doesn't introduce any header at all. I've been using proxies from: http://www.checkedproxylists.com http://spys.ru/en/ They seem to work ok for normal requests, but not for the simplepie thingy. Perhaps they are adding other headers or... Anyway, Sam... given my tests... and the results of only being able to get it working for local squid proxies... do you think it needs more researching to cover other proxies or is enough to get local-squid fixed and forget? Ciao
          Hide
          Sam Marshall added a comment -

          Eloy: I tested on Privoxy and was also able to reproduce the problem before fix, but not afterward, same deal. Don't know if Privoxy/Squid are actually based on shared code.

          I think it would still be worth including this as it is a clear improvement for some cases (I modified title slightly to describe). Personally I am not going to be able to test on any remote proxies but realistically, I doubt anyone points their Moodle install at a public proxy anyway!

          I've written better test instructions including how to install Privoxy for users who don't already have a local proxy server that causes this problem.

          Show
          Sam Marshall added a comment - Eloy: I tested on Privoxy and was also able to reproduce the problem before fix, but not afterward, same deal. Don't know if Privoxy/Squid are actually based on shared code. I think it would still be worth including this as it is a clear improvement for some cases (I modified title slightly to describe). Personally I am not going to be able to test on any remote proxies but realistically, I doubt anyone points their Moodle install at a public proxy anyway! I've written better test instructions including how to install Privoxy for users who don't already have a local proxy server that causes this problem.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Yesyes, agree it does not have sense to point to a public proxy at all, but they should be working (in fact, given the port, a bunch of them are squids too, I'haven't looked deeper).

          Anyway, np, I agree after the patch behavior is better for people using local/default squid/privoxy proxies... so integrating...

          Show
          Eloy Lafuente (stronk7) added a comment - Yesyes, agree it does not have sense to point to a public proxy at all, but they should be working (in fact, given the port, a bunch of them are squids too, I'haven't looked deeper). Anyway, np, I agree after the patch behavior is better for people using local/default squid/privoxy proxies... so integrating...
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Integrated (23, 24 & master), thanks!

          Show
          Eloy Lafuente (stronk7) added a comment - Integrated (23, 24 & master), thanks!
          Hide
          Dan Poltawski added a comment -

          I would say its good to specifically target a squid proxy fix like this.

          Eloy, did you happen to get the headers for any of the public proxies? I think its worth creating an issue for that.

          Show
          Dan Poltawski added a comment - I would say its good to specifically target a squid proxy fix like this. Eloy, did you happen to get the headers for any of the public proxies? I think its worth creating an issue for that.
          Hide
          Adrian Greeve added a comment -

          Tested on the 2.3, 2.4 and master integration branches.
          I replicated the error and then applied the fix.
          No problems found.
          Test passed.

          Show
          Adrian Greeve added a comment - Tested on the 2.3, 2.4 and master integration branches. I replicated the error and then applied the fix. No problems found. Test passed.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          This is valid for unlimited entries to the, soon to be unveiled, Moodle Codebase Gardens. It includes free access to all facilities.

          Personal and non-transferable to all assignees, reviewers and testers in this issue. Valid until switching to Blackboard (100000€ penalization will be applied).

          Thanks, closing as fixed!

          Show
          Eloy Lafuente (stronk7) added a comment - This is valid for unlimited entries to the, soon to be unveiled, Moodle Codebase Gardens. It includes free access to all facilities. Personal and non-transferable to all assignees, reviewers and testers in this issue. Valid until switching to Blackboard (100000€ penalization will be applied). Thanks, closing as fixed!

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: