Moodle
  1. Moodle
  2. MDL-23296

Provide search function for Server Files area in File Picker

    Details

    • Testing Instructions:
      Hide
      1. Upload many many files (thousands) in different courses, modules, blog posts, everywhere...
        • Different in names, and content
      2. Prepare a few user account which have different set of permissions, meaning that they can only access some of the files
      3. As each user, login and navigate to a course
      4. Find your way to an editor in the course and browse the "Server files" repository
      5. Use the search and make sure...
        • ... you can find all the files you have access to (except your private files)
        • ... more results are loaded automatically as you scroll
        • ... search results appear rather quickly, i.e. they don't crash the server
        • ... search query is case insensitive
        • ... you cannot find files that you're not supposed to (belonging to another course for instance)
      6. Try to find weak spots in the code
      Show
      Upload many many files (thousands) in different courses, modules, blog posts, everywhere... Different in names, and content Prepare a few user account which have different set of permissions, meaning that they can only access some of the files As each user, login and navigate to a course Find your way to an editor in the course and browse the "Server files" repository Use the search and make sure ... ... you can find all the files you have access to (except your private files) ... more results are loaded automatically as you scroll ... search results appear rather quickly, i.e. they don't crash the server ... search query is case insensitive ... you cannot find files that you're not supposed to (belonging to another course for instance) Try to find weak spots in the code
    • Affected Branches:
      MOODLE_20_STABLE, MOODLE_26_STABLE, MOODLE_27_STABLE, MOODLE_28_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-23296-master
    • Sprint:
      Team ';drop tables Sprint 4, Team '; drop tables Sprint 5
    • Issue size:
      Medium

      Description

      At the moment, the only way to re-use a file which I've used in another course (assuming I'm using Server Files and not an external repository) is to browse through Category>Course>Section>Activity/Resource>Files and hope I can remember where I put it. If its something recent AND i was the one who added it then it should be in my recent files, but if not then its unweildy to find a file from another course which I'd like to use.

      One way to get around this would be to have a search facility in the File Picker which would let me search any files to which I have access.

      And yes, I know Moodle isn't a file repository

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            Martin Dougiamas added a comment -

            This would be very CPU intensive but I think it's worth looking at.

            Show
            Martin Dougiamas added a comment - This would be very CPU intensive but I think it's worth looking at.
            Hide
            gavin henrick added a comment -

            What about just changing how the "Server files" browsing works?

            Category>Course>Section>Activity/Resource>File is very precise but having to remember the topic and resource a file was added to, is not feasible for me anyhow

            Would it not just be possible to have Category>Course - lists all files or at worst Category>Course>Section>file list

            Thoughts?

            Show
            gavin henrick added a comment - What about just changing how the "Server files" browsing works? Category>Course>Section>Activity/Resource>File is very precise but having to remember the topic and resource a file was added to, is not feasible for me anyhow Would it not just be possible to have Category>Course - lists all files or at worst Category>Course>Section>file list Thoughts?
            Hide
            Martin Dougiamas added a comment -

            The problem is that we have files all over the place, such as:

            • assignment submissions
            • forum attachments
            • embedded media in labels
            • resources

            etc

            Surely we can't just smoosh all that together in the browsing interface ...

            Show
            Martin Dougiamas added a comment - The problem is that we have files all over the place, such as: assignment submissions forum attachments embedded media in labels resources etc Surely we can't just smoosh all that together in the browsing interface ...
            Hide
            gavin henrick added a comment -

            Good point.

            Currently too precise, my suggestion is too general, whats a usable middle group?

            No clue on solution.

            Show
            gavin henrick added a comment - Good point. Currently too precise, my suggestion is too general, whats a usable middle group? No clue on solution.
            Hide
            Jon Witts added a comment -

            Would a search facility at this level be the first stage to be being able to link in Dublin Core Meta tag data to the uploaded resources? I think I would be really useful to be able to search by tags on the files, as well as file names... I appreciate that this would add load to the servers; but is that justifiable by the added functionality it would bring?

            Show
            Jon Witts added a comment - Would a search facility at this level be the first stage to be being able to link in Dublin Core Meta tag data to the uploaded resources? I think I would be really useful to be able to search by tags on the files, as well as file names... I appreciate that this would add load to the servers; but is that justifiable by the added functionality it would bring?
            Hide
            Elena Ivanova added a comment -

            I think this discussion is related: search, possible re-org, visibility of this area, some confusing breadcrumbs: http://moodle.org/mod/forum/discuss.php?d=154880

            Show
            Elena Ivanova added a comment - I think this discussion is related: search, possible re-org, visibility of this area, some confusing breadcrumbs: http://moodle.org/mod/forum/discuss.php?d=154880
            Hide
            Howard Miller added a comment -

            Just to make some noise. There's way too many places where files can be and ordinary users shouldn't have to be bothered. They just want to find the file. A lost of available files (regardless of where they come from) and a decent search is surely the answer. They don't want to know about 'Recent files', 'Site files', 'Private files'.....

            Show
            Howard Miller added a comment - Just to make some noise. There's way too many places where files can be and ordinary users shouldn't have to be bothered. They just want to find the file. A lost of available files (regardless of where they come from) and a decent search is surely the answer. They don't want to know about 'Recent files', 'Site files', 'Private files'.....
            Hide
            Michael Blake added a comment -

            This issue is affecting MP clients. Please give it priority.

            Show
            Michael Blake added a comment - This issue is affecting MP clients. Please give it priority.
            Hide
            Lael... added a comment -

            I have no idea how the back end of the files works - but this seems incredibly important for encouraging people to re-use / link to files. If it is too difficult people will just re-upload the file. (side question - if someone uploads an identical file with a different name - does the database end up storing one or two files?)

            I actually think Gavin's suggestion is a very good one - show all the files in a course, in alphabetical order. I would suggest adding filters as checkboxes to determine what is shown. By default I would imagine assignment submissions, forum attachments would be off, and resources, assignment attachments (ie: files linked in the editor), labels etc would be on. Extra useful if there was a filter for file types. I have no idea how difficult this would be to implement, but I imagine usability to be high.

            Show
            Lael... added a comment - I have no idea how the back end of the files works - but this seems incredibly important for encouraging people to re-use / link to files. If it is too difficult people will just re-upload the file. (side question - if someone uploads an identical file with a different name - does the database end up storing one or two files?) I actually think Gavin's suggestion is a very good one - show all the files in a course, in alphabetical order. I would suggest adding filters as checkboxes to determine what is shown. By default I would imagine assignment submissions, forum attachments would be off, and resources, assignment attachments (ie: files linked in the editor), labels etc would be on. Extra useful if there was a filter for file types. I have no idea how difficult this would be to implement, but I imagine usability to be high.
            Hide
            Martin Dougiamas added a comment -

            For your side question: if anyone uploads a file again then it gets stored internally just once.

            Show
            Martin Dougiamas added a comment - For your side question: if anyone uploads a file again then it gets stored internally just once.
            Hide
            Dongsheng Cai added a comment -

            This issue was assigned to me automatically, however I will not be able to work on this issue in the immediate future. In order to create a truer sense of the state of this issue and to allow other developers to have chance to become involved, I am removing myself as the assignee of this issue.
            For more information, see http://docs.moodle.org/dev/Changes_to_issue_assignment

            Show
            Dongsheng Cai added a comment - This issue was assigned to me automatically, however I will not be able to work on this issue in the immediate future. In order to create a truer sense of the state of this issue and to allow other developers to have chance to become involved, I am removing myself as the assignee of this issue. For more information, see http://docs.moodle.org/dev/Changes_to_issue_assignment
            Hide
            Grant Harbor added a comment -

            Could we not have a simple Search by File Name, which only searches the "Current Course" by default, or searches the folder currently being displayed? (Limited to course and not Site Level search?)

            Currently a number of my staff are continuing to use "Legacy Course Files" because, as one staff member said, "It's too hard to find my files in the site files".

            Thoughts?

            Show
            Grant Harbor added a comment - Could we not have a simple Search by File Name, which only searches the "Current Course" by default, or searches the folder currently being displayed? (Limited to course and not Site Level search?) Currently a number of my staff are continuing to use "Legacy Course Files" because, as one staff member said, "It's too hard to find my files in the site files". Thoughts?
            Hide
            Julian Ridden added a comment -

            Also another comment in line with Michael Blake to see if we can ad some priority to this issue. Numerous clients are asking for this feature. MootAU also saw me hit up with several requests. Thanks MarkD for reporting the issue.

            Any chance we could at least have this assigned to someone at HQ so it does not just fade into the ether.

            Show
            Julian Ridden added a comment - Also another comment in line with Michael Blake to see if we can ad some priority to this issue. Numerous clients are asking for this feature. MootAU also saw me hit up with several requests. Thanks MarkD for reporting the issue. Any chance we could at least have this assigned to someone at HQ so it does not just fade into the ether.
            Hide
            Martin Dougiamas added a comment -

            We don't "assign things to people at HQ" like that ... it's already in the BACKEND queue with some priority dues to votes and partner labels.

            Show
            Martin Dougiamas added a comment - We don't "assign things to people at HQ" like that ... it's already in the BACKEND queue with some priority dues to votes and partner labels.
            Hide
            Frédéric Massart added a comment -

            Just noting that a proof of concept was developed as part of the investigation which took place in MDL-48908: https://github.com/FMCorz/moodle/compare/7357af2...MDL-48908-master

            Show
            Frédéric Massart added a comment - Just noting that a proof of concept was developed as part of the investigation which took place in MDL-48908 : https://github.com/FMCorz/moodle/compare/7357af2...MDL-48908-master
            Hide
            Mark Nelson added a comment -

            Fred's solution is quicker than my solution as it involves paging. I will write some testing instructions and put it up for peer review.

            Show
            Mark Nelson added a comment - Fred's solution is quicker than my solution as it involves paging. I will write some testing instructions and put it up for peer review.
            Hide
            Mark Nelson added a comment -

            FYI my change before I change the diff on this issue - https://github.com/markn86/moodle/compare/master...MDL-23296_master.

            Show
            Mark Nelson added a comment - FYI my change before I change the diff on this issue - https://github.com/markn86/moodle/compare/master...MDL-23296_master .
            Hide
            Mark Nelson added a comment -

            Fred has agreed to take this issue and push it forward.

            Show
            Mark Nelson added a comment - Fred has agreed to take this issue and push it forward.
            Hide
            Frédéric Massart added a comment -

            Pushing for peer review. In short we do a raw query on the database, then manually check if each of the results are visible to the user. To reduce the load the queries are done in chunk, with a maximum number of loops (arbitrary values for now). The chunking is used in conjunction with pagination.

            The downside is that if there are too many results and you don't have access to the n (current 3000) first results, your file will not be found. We could potentially display a different message to the user but that requires changes in the filepicker API so I'd rather do that as a separate issue.

            Thanks,
            Fred

            Show
            Frédéric Massart added a comment - Pushing for peer review. In short we do a raw query on the database, then manually check if each of the results are visible to the user. To reduce the load the queries are done in chunk, with a maximum number of loops (arbitrary values for now). The chunking is used in conjunction with pagination. The downside is that if there are too many results and you don't have access to the n (current 3000) first results, your file will not be found. We could potentially display a different message to the user but that requires changes in the filepicker API so I'd rather do that as a separate issue. Thanks, Fred
            Hide
            CiBoT added a comment -

            Code verified against automated checks.

            Checked MDL-23296 using repository: git://github.com/FMCorz/moodle.git

            More information about this report

            Show
            CiBoT added a comment - Code verified against automated checks. Checked MDL-23296 using repository: git://github.com/FMCorz/moodle.git master (0 errors / 0 warnings) [branch: MDL-23296-master | CI Job ] More information about this report
            Hide
            Dave Cooper added a comment -

            Hey Fred,

            I have reviewed the code and can't seem to poke any holes in it. In order to understand what it is doing a bit better I did pull in your code and ran through the tests you wrote up (not with a large dataset though) and everything seems pretty good/responsive.

            Pushing for integration review.

            Show
            Dave Cooper added a comment - Hey Fred, I have reviewed the code and can't seem to poke any holes in it. In order to understand what it is doing a bit better I did pull in your code and ran through the tests you wrote up (not with a large dataset though) and everything seems pretty good/responsive. Pushing for integration review.
            Hide
            Eloy Lafuente (stronk7) added a comment -

            The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

            TIA and ciao

            Show
            Eloy Lafuente (stronk7) added a comment - The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week. TIA and ciao
            Hide
            CiBoT added a comment -

            Moving this issue to current integration cycle, will be reviewed soon. Thanks for the hard work!

            Show
            CiBoT added a comment - Moving this issue to current integration cycle, will be reviewed soon. Thanks for the hard work!
            Hide
            CiBoT added a comment -

            Fails against automated checks.

            Checked MDL-23296 using repository: git://github.com/FMCorz/moodle.git

            More information about this report

            Show
            CiBoT added a comment - Fails against automated checks. Checked MDL-23296 using repository: git://github.com/FMCorz/moodle.git master [branch: MDL-23296-master | CI Job ] Error: Unable to fetch information from MDL-23296 -master branch at git://github.com/FMCorz/moodle.git. More information about this report
            Hide
            David Monllaó added a comment -

            Hi Fred, many thanks, this looks a good solution for a technically complex issue. A couple of questions before integrating it:

            • search_server_files $from is set to 0 and $limit is set to 20, https://github.com/FMCorz/moodle/compare/dba564c...MDL-23296-master#diff-6c0a8861eef371f78dde4be1467a8a1fR1981 I can't figure out why 20; you are setting seach() $limit to 100 by default and search_server_files count is being called with from and limit values to null and null; also these null values are later passed to get_recorset_sql which expects int values (I guess these nulls are converted to int=0 at some point because it works as expected, I haven't followed the code)
            • About the search_server_files count outside the $page <= 1 if/else; when $page > 1, are you calling it again in case new files have been added since last time it run? I may be missing something, but if the count value differs then we should "restart" the search, as we order through file name; if that is not the case looks like we could save 1 db query storing that count value in the session.
            Show
            David Monllaó added a comment - Hi Fred, many thanks, this looks a good solution for a technically complex issue. A couple of questions before integrating it: search_server_files $from is set to 0 and $limit is set to 20, https://github.com/FMCorz/moodle/compare/dba564c...MDL-23296-master#diff-6c0a8861eef371f78dde4be1467a8a1fR1981 I can't figure out why 20; you are setting seach() $limit to 100 by default and search_server_files count is being called with from and limit values to null and null; also these null values are later passed to get_recorset_sql which expects int values (I guess these nulls are converted to int=0 at some point because it works as expected, I haven't followed the code) About the search_server_files count outside the $page <= 1 if/else; when $page > 1, are you calling it again in case new files have been added since last time it run? I may be missing something, but if the count value differs then we should "restart" the search, as we order through file name; if that is not the case looks like we could save 1 db query storing that count value in the session.

              People

              • Votes:
                60 Vote for this issue
                Watchers:
                43 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Agile