Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-80891

AI placement - Text Editor (image and text)

XMLWordPrintable

    • MDL-80891-main
    • Hide

      At this moment, this branch also contains base implementation and OpenAI provider. So no need to do any other setup.

      Testing 1 - Provider

      1. Login as admin.
      2. Disable the OpenAI API Provider.
      3. Verify that you will not see any AI buttons in the Tiny editor.
      4. Enable the OpenAI API Provider.
      5. Click on the Setting link.
      6. Disable the Generate text and Generate image actions.
      7. Verify that you will not see any AI buttons in the Tiny editor.
      8. Enable the Generate text action.
      9. Verify that you will see a single AI button called AI generate text in the Tiny editor.
      10. Enable the Generate image action and disable the Generate text action.
      11. Verify that you will see a single AI button called AI generate image in the Tiny editor.
      12. Enable both Generate image action and the Generate text action.
      13. Verify that you will see a Generate AI content dropdown button.
      14. Click on that button
      15. Verify that you will see 2 items called AI generate text and AI generate image.

      Testing 2 - Placement

      1. Login as admin.
      2. Disable the TinyMCE Editor Placement.
      3. Verify that you will not see any AI buttons in the Tiny editor.
      4. Enable the TinyMCE Editor Placement.
      5. Click on the Setting link.
      6. Disable the Generate text and Generate image actions.
      7. Verify that you will not see any AI buttons in the Tiny editor.
      8. Enable the Generate text action.
      9. Verify that you will see a single AI button called AI generate text in the Tiny editor.
      10. Enable the Generate image action and disable the Generate text action.
      11. Verify that you will see a single AI button called AI generate image in the Tiny editor.
      12. Enable both Generate image action and the Generate text action.
      13. Verify that you will see a Generate AI content dropdown button.
      14. Click on that button
      15. Verify that you will see 2 items called AI generate text and AI generate image.

      Testing 3 - Error

      1. Login as admin.
      2. Provide wrong information for API key of OpenAI provider.
      3. Try to Generate text or image by TinyMCE placement.
      4. Verify that you will see the correctponing error.

      Testing 4

      1. Login as admin.
      2. Provide the correct information for API key of OpenAI provider.
      3. Try to Generate text or image by TinyMCE placement.
      4. Verify that you can generate text or image normally.
      Show
      At this moment, this branch also contains base implementation and OpenAI provider. So no need to do any other setup. Testing 1 - Provider Login as admin. Disable the OpenAI API Provider. Verify that you will not see any AI buttons in the Tiny editor. Enable the OpenAI API Provider. Click on the Setting link. Disable the Generate text and Generate image actions. Verify that you will not see any AI buttons in the Tiny editor. Enable the Generate text action. Verify that you will see a single AI button called AI generate text in the Tiny editor. Enable the Generate image action and disable the Generate text action. Verify that you will see a single AI button called AI generate image in the Tiny editor. Enable both Generate image action and the Generate text action. Verify that you will see a Generate AI content dropdown button. Click on that button Verify that you will see 2 items called AI generate text and AI generate image. Testing 2 - Placement Login as admin. Disable the TinyMCE Editor Placement. Verify that you will not see any AI buttons in the Tiny editor. Enable the TinyMCE Editor Placement. Click on the Setting link. Disable the Generate text and Generate image actions. Verify that you will not see any AI buttons in the Tiny editor. Enable the Generate text action. Verify that you will see a single AI button called AI generate text in the Tiny editor. Enable the Generate image action and disable the Generate text action. Verify that you will see a single AI button called AI generate image in the Tiny editor. Enable both Generate image action and the Generate text action. Verify that you will see a Generate AI content dropdown button. Click on that button Verify that you will see 2 items called AI generate text and AI generate image. Testing 3 - Error Login as admin. Provide wrong information for API key of OpenAI provider. Try to Generate text or image by TinyMCE placement. Verify that you will see the correctponing error. Testing 4 Login as admin. Provide the correct information for API key of OpenAI provider. Try to Generate text or image by TinyMCE placement. Verify that you can generate text or image normally.
    • 10
    • Team Hedgehog 2024 Review 2, Team Hedgehog 2024 Sprint 3.1

      Provide a way for users to interact with an AI action to generate an image or text based on a description and LMS contextual information, from within the TinyMCE editor interface and allow this image to be inserted into existing content. Both images and text will be implemented together as they are different actions within the same placement (so there is a lot of reusable code).

      Inline with our AI Principles we will need to provide notification to users that they are using an AI enhanced service, and provide them a way to understand what this means. 

      Base code for this issue is available here, and further changes should be based on these commits:  **
      https://github.com/moodle/moodle/compare/main...mattporritt:moodle:main_MDL-80891
       
      Please arrange a handover session with matt.porritt@moodle.com prior to commencing any work on this issue.
       
      Below details outline image and text information, and the todo lists are tasks not yet completed in the provided branch.

      Image action

      Image metadata should indicate that the image was generated (or altered) using AI.

      Images TODO: 

      • Action selection:
        • Currently the action selection displayed in the toolbar and Insert menu of TinyMCE do not respect the enabled action settings for this Placement, only enabled actions should be available
        • Related to the above only actions that have a provider that can provide the action should be available
        • Related to the above only users with the required capability for that action in this placement should be able to use the action. The capabilities already exist, they are just not being respected. 
        • We should follow the same pattern as what was implemented in MDL-81212 for how to display the actions.
      • Error handling:
        • There is currently error handling if the API returns an error, but it is general
        • We should add a specific message for when the error is due to a rate limit 
        • We should add a specific message for when the error is due to a content moderation or AI guard rail issue. 
        • There is currently a “bug” where after an error with generating an image and then the next generation succeeds, the error class isn’t removed from the preview window.
      • Handoff to image management
        • After the image is generated and it is “handed off” to the regular LMS Image insertion workflow, there is currently a JS error being thrown.
        • Use the user description that made the image to prefill the alt tag field for the image
        • Use the image dimensions to prefill the image size attributes
      • Watermark
        • Add a watermark to the generated image to say it was generated by AI. (Depending on timing this could be a follow up issue).

      Text action

      This covers the initial implementation which will be basic text generation and insertion. There will be a follow up issue to add greater configuration options later.

      generated text content should be tagged (indicated in markup etc.) that it was generated with the assistance of AI, to allow future identification of AI generated content.

      Technical Spike Plugin: https://github.com/mattporritt/moodle-tiny_ai

      Text TODO: 

      • Action selection:
        • Currently the action selection displayed in the toolbar and Insert menu of TinyMCE do not respect the enabled action settings for this Placement, only enabled actions should be available
        • Related to the above only actions that have a provider that can provide the action should be available
        • Related to the above only users with the required capability for that action in this placement should be able to use the action. The capabilities already exist, they are just not being respected. 
        • We should follow the same pattern as what was implemented in MDL-81212 for how to display the actions.
      • Error handling:
        • There is currently error handling if the API returns an error, but it is general
        • We should add a specific message for when the error is due to a rate limit 
        • We should add a specific message for when the error is due to a content moderation or AI guard rail issue. 
        • There is currently a “bug” where after an error with generating an image and then the next generation succeeds, the error class isn’t removed from the preview window.

        1. Generate image with Ai.png
          Generate image with Ai.png
          26 kB
        2. Generate image with Ai-result.png
          Generate image with Ai-result.png
          308 kB
        3. Generate text with Ai.png
          Generate text with Ai.png
          15 kB
        4. Generate text with Ai-result.png
          Generate text with Ai-result.png
          78 kB
        5. Image_generation_Proto2.mp4
          3.19 MB
        6. MDL-80891_cursor_position.png
          MDL-80891_cursor_position.png
          29 kB
        7. MDL-80891_disabledbutton.png
          MDL-80891_disabledbutton.png
          20 kB
        8. MDL-80891_focus_groupedbutton.png
          MDL-80891_focus_groupedbutton.png
          20 kB
        9. MDL-80891_responsive_image.png
          MDL-80891_responsive_image.png
          258 kB
        10. MDL-80891_responsive_text.png
          MDL-80891_responsive_text.png
          21 kB
        11. MDL-80891_texttoolarge.png
          MDL-80891_texttoolarge.png
          12 kB
        12. mobile_image.png
          mobile_image.png
          175 kB
        13. mobile_text.png
          mobile_text.png
          142 kB
        14. Text_generation_Proto2.mp4
          2.96 MB

            huongn@moodle.com Huong Nguyen
            matt.porritt@moodle.com Matt Porritt
            Meirza Meirza
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 week, 4 days, 1 hour, 32 minutes
                1w 4d 1h 32m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.