Moodle
  1. Moodle
  2. MDL-20804

SOAP server: generate detailled object structure in WSDL (this issue was initially: "Create a JAVA compatible SOAP WSDL (SOAP server) + JAVA client")

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0, 2.1.1, 2.2
    • Fix Version/s: 2.2
    • Component/s: Web Services
    • Labels:
    • Rank:
      772

      Description

      SOAP works with a PHP client, however it needs to be tested with a Java client. It's more likely it doesn't work. We need to generate a more detailled WSDL for Java/.Net.
      Have a look to the Zend framework, it seems that library functions exist to help you to create the WSDL. You can use some of the discovery class functions.

      Finally create a Java client and write a Moodledocs document about how to create a Java client with an IDE client generator.

        Issue Links

          Activity

          Hide
          Jérôme Mouneyrac added a comment - - edited

          I assign to nobody cause it is not an emergency yet. Any participation is welcome.

          Show
          Jérôme Mouneyrac added a comment - - edited I assign to nobody cause it is not an emergency yet. Any participation is welcome.
          Hide
          Jérôme Mouneyrac added a comment -

          Some comments from Martin for later:

          • Also the guys here at CPIT are experienced with SOAP and PHP and Java
          • they had an idea about your Java SOAP issue
          • there is a apaprently a setting in php.ini for PHP that changes the SOAP behaviour to make it compatible for Java
          • ie changes array/object behaviour for some cases
          • would be worth hunting that down for documentation
          Show
          Jérôme Mouneyrac added a comment - Some comments from Martin for later: Also the guys here at CPIT are experienced with SOAP and PHP and Java they had an idea about your Java SOAP issue there is a apaprently a setting in php.ini for PHP that changes the SOAP behaviour to make it compatible for Java ie changes array/object behaviour for some cases would be worth hunting that down for documentation
          Hide
          Luis Herrera added a comment -
          Show
          Luis Herrera added a comment - You need this for autocreate wsdl http://framework.zend.com/manual/en/zend.soap.autodiscovery.html
          Hide
          Jérôme Mouneyrac added a comment -

          Hi Luis,
          we already use it. However the autodiscovered WSDL is only generated from the PHPdoc. The PHPdoc doesn't contain our full description array. So JAVA/.Net have some problems with it.
          Note that this comment concerned Zend 1.9 and I'm not an expert with JAVA/SOAP. It seems to me that the solution would be to use the library functions that Zend uses for their Autodiscover class, in order to generate a fully detailled WSDL.

          Show
          Jérôme Mouneyrac added a comment - Hi Luis, we already use it. However the autodiscovered WSDL is only generated from the PHPdoc. The PHPdoc doesn't contain our full description array. So JAVA/.Net have some problems with it. Note that this comment concerned Zend 1.9 and I'm not an expert with JAVA/SOAP. It seems to me that the solution would be to use the library functions that Zend uses for their Autodiscover class, in order to generate a fully detailled WSDL.
          Hide
          Ashley Holman added a comment -

          Voted +1. WSDL will be useful for not only java but also PHP itself (PHP-SOAP supports WSDL for automatically binding functions), python, perl etc.

          Makes it a lot easier to work with SOAP when there's WSDL available.

          Show
          Ashley Holman added a comment - Voted +1. WSDL will be useful for not only java but also PHP itself (PHP-SOAP supports WSDL for automatically binding functions), python, perl etc. Makes it a lot easier to work with SOAP when there's WSDL available.
          Hide
          Jérôme Mouneyrac added a comment - - edited

          I edited the issue title and description, it was not clear that we already generate a WSDL. I just could not make it work with a Java client.

          Show
          Jérôme Mouneyrac added a comment - - edited I edited the issue title and description, it was not clear that we already generate a WSDL. I just could not make it work with a Java client.
          Hide
          Martin Dougiamas added a comment -

          I'm trying to track down the guys from CPIT who I was talking to, hopefully they pop up here.

          Otherwise, did anyone try some googling to find the solution I mentioned?

          Show
          Martin Dougiamas added a comment - I'm trying to track down the guys from CPIT who I was talking to, hopefully they pop up here. Otherwise, did anyone try some googling to find the solution I mentioned?
          Hide
          Martin Dougiamas added a comment -

          OK, it seems that the problem is something else, I had misunderstood.

          The problem is that the Zend autodiscovery method, in which our PHP structure description is converted to phpdoc and then parsed by Zend, is not capable of capturing our complex structures in enough detail, and so the resulting WSDL is not enough for Java (PHP on the other hand is more lazy and works fine).

          So we either need:

          • some custom code to convert our structures to a more detailed WSDL that keeps Java happy, or
          • some information about making Java more tolerant like PHP
          Show
          Martin Dougiamas added a comment - OK, it seems that the problem is something else, I had misunderstood. The problem is that the Zend autodiscovery method, in which our PHP structure description is converted to phpdoc and then parsed by Zend, is not capable of capturing our complex structures in enough detail, and so the resulting WSDL is not enough for Java (PHP on the other hand is more lazy and works fine). So we either need: some custom code to convert our structures to a more detailed WSDL that keeps Java happy, or some information about making Java more tolerant like PHP
          Hide
          Martin Dougiamas added a comment -

          Some info from Kris at CPIT:

          Looking at your issue seems to be in two parts incorrect/incomplete automatic generation of WSDL and soap encoding of array elements.

          The first I can't help you with too much as I mostly use Handcrafted WSDL on PHP 5 SoapServer class. The reason behind this is the ability to version the interface. Having a dynamically built WSDL would mean that when the code changes the interface changes not desirable as the WSDL is your contract to external consumers/providers and having such a dynamic interface can pose unforeseen issues especially in compiled Java/Objective-c(IPad/IPhone) clients.

          The second point is easy. The PHP setting in question that will bite you is in the handling of Array Complex type either maxOccurrences="unbounded" or soapenc:ArrayType when there is only one element in the array PHP will encode it as an element rather than an array of one. To get round this construct the SoapServer like so:

          $server = new SoapServer($wsdl,array('features' => SOAP_SINGLE_ELEMENT_ARRAYS) );

          Show
          Martin Dougiamas added a comment - Some info from Kris at CPIT: Looking at your issue seems to be in two parts incorrect/incomplete automatic generation of WSDL and soap encoding of array elements. The first I can't help you with too much as I mostly use Handcrafted WSDL on PHP 5 SoapServer class. The reason behind this is the ability to version the interface. Having a dynamically built WSDL would mean that when the code changes the interface changes not desirable as the WSDL is your contract to external consumers/providers and having such a dynamic interface can pose unforeseen issues especially in compiled Java/Objective-c(IPad/IPhone) clients. The second point is easy. The PHP setting in question that will bite you is in the handling of Array Complex type either maxOccurrences="unbounded" or soapenc:ArrayType when there is only one element in the array PHP will encode it as an element rather than an array of one. To get round this construct the SoapServer like so: $server = new SoapServer($wsdl,array('features' => SOAP_SINGLE_ELEMENT_ARRAYS) );
          Hide
          Martin Dougiamas added a comment -

          Bumping this for now. I think we might need a PHP+Java expert to est this in a real situation.

          Show
          Martin Dougiamas added a comment - Bumping this for now. I think we might need a PHP+Java expert to est this in a real situation.
          Hide
          Tim Hunt added a comment -

          Please can we increase the priority of this? It is a pain in the bum for us at the OU. Our IT department likes to call SOAP from .Net, which needs proper WSDL.

          Show
          Tim Hunt added a comment - Please can we increase the priority of this? It is a pain in the bum for us at the OU. Our IT department likes to call SOAP from .Net, which needs proper WSDL.
          Hide
          Tim Hunt added a comment -

          setComplexTypeStrategy in lib/zend/Zend/Soap/AutoDiscover.php looks like a promising plugin point. Can we hook into that and get it to parse our arrays with detailed struct info?

          Alternatively, can we parse our detailed arguments and return information, and build an accurate Zend_Soap_Wsdl ourselves?

          Show
          Tim Hunt added a comment - setComplexTypeStrategy in lib/zend/Zend/Soap/AutoDiscover.php looks like a promising plugin point. Can we hook into that and get it to parse our arrays with detailed struct info? Alternatively, can we parse our detailed arguments and return information, and build an accurate Zend_Soap_Wsdl ourselves?
          Hide
          Troy Williams added a comment -

          Hi,

          The last few weeks I have been working a project to get grade and log data out of Moodle using soap web services. The last few days have progressed well. I have everything using Moodle web service APIs etc and all works well using testclient in Moodle. The problem was is I needed/wanted to use a Document/literal wrapped WSDL. I have finally got it sorted I think.. code is still a little rough, will be doing some testing next week at work but at this stage looks good under testing with soapUI. I will post code up when at work next week if anyone interested?

          Anyways here brief details of what I did.

          My services are sitting in Moodle as local module. Copied soap/server.php and soap/locallib.php to the local folder. I have script wsdl.php that accepts wstoken and read my wsdl.xml file and sets up targetnamespace with the wstoken. I override service_class_method_body() to build methods for the virtual class. I then load soap server $this->zend_server = new $this->zend_class(htmlentities($wsdlurl)); under init_zend_server(). It seems to work.

          Regards

          Troy

          Show
          Troy Williams added a comment - Hi, The last few weeks I have been working a project to get grade and log data out of Moodle using soap web services. The last few days have progressed well. I have everything using Moodle web service APIs etc and all works well using testclient in Moodle. The problem was is I needed/wanted to use a Document/literal wrapped WSDL. I have finally got it sorted I think.. code is still a little rough, will be doing some testing next week at work but at this stage looks good under testing with soapUI. I will post code up when at work next week if anyone interested? Anyways here brief details of what I did. My services are sitting in Moodle as local module. Copied soap/server.php and soap/locallib.php to the local folder. I have script wsdl.php that accepts wstoken and read my wsdl.xml file and sets up targetnamespace with the wstoken. I override service_class_method_body() to build methods for the virtual class. I then load soap server $this->zend_server = new $this->zend_class(htmlentities($wsdlurl)); under init_zend_server(). It seems to work. Regards Troy
          Hide
          Tim Hunt added a comment -

          Yay! I think I have cracked it. At least these changes are good enough for my web service that returns an object with 4 fields.

          Jerome (or anyone else) please review this and see what you think.

          Show
          Tim Hunt added a comment - Yay! I think I have cracked it. At least these changes are good enough for my web service that returns an object with 4 fields. Jerome (or anyone else) please review this and see what you think.
          Hide
          Jérôme Mouneyrac added a comment -

          Does it pass the web service unit tests (xmlrpc + soap enabled) in webservice/simpletest/ (you need to set to true protocols and functions in the file; only test on a dev site that you don't care much about the db) ?
          When you said you cracked it, do you mean it generates a wsdl compatible with java/.net client generator? If yes that is just really cool!

          I'll try when I'm back anyway, thanks.

          Show
          Jérôme Mouneyrac added a comment - Does it pass the web service unit tests (xmlrpc + soap enabled) in webservice/simpletest/ (you need to set to true protocols and functions in the file; only test on a dev site that you don't care much about the db) ? When you said you cracked it, do you mean it generates a wsdl compatible with java/.net client generator? If yes that is just really cool! I'll try when I'm back anyway, thanks.
          Hide
          Jérôme Mouneyrac added a comment - - edited

          Hi Tim,
          I reviewed it:

          • all current Moodle web service unit tests are still working (soap + xmlrpc)
          • the patch generates classes with a full detailed phpdoc (recursive)
          • I don't think it could break anything, at worst the soap server only

          Thanks for the code reduction improvement too.

          I need to test the recursivity, but that seems very good. From what you are saying I understand once you have the full details, Zend SOAP server do generate a JAVA compatible?

          Can you give me a JAVA client if you have one simple? I'll try netbeans client generator by tomorrow anyway.

          You are going to make many people happy

          Show
          Jérôme Mouneyrac added a comment - - edited Hi Tim, I reviewed it: all current Moodle web service unit tests are still working (soap + xmlrpc) the patch generates classes with a full detailed phpdoc (recursive) I don't think it could break anything, at worst the soap server only Thanks for the code reduction improvement too. I need to test the recursivity, but that seems very good. From what you are saying I understand once you have the full details, Zend SOAP server do generate a JAVA compatible? Can you give me a JAVA client if you have one simple? I'll try netbeans client generator by tomorrow anyway. You are going to make many people happy
          Hide
          Tim Hunt added a comment -

          I don't have a good Java client available, but fortunately these people do: http://www.soapui.org/ (I know it looks commercial, but they do have a free download available.)

          SOAPUI worked for me with a non-recursive function (just returning an object with 4 fields that were strings or ints.) I did not try a more complex case.

          However, even after my changes, people in the other half our IT department failed to connect to the web service using .Net, however, they only tried once then went on holiday, so the will try again later.

          However, I agree with your analysis, this won't break anything other than SOAP, and it will probably only make SOAP work better, not worse, so probably OK to integrate this for now, even if there is more work to do.

          Show
          Tim Hunt added a comment - I don't have a good Java client available, but fortunately these people do: http://www.soapui.org/ (I know it looks commercial, but they do have a free download available.) SOAPUI worked for me with a non-recursive function (just returning an object with 4 fields that were strings or ints.) I did not try a more complex case. However, even after my changes, people in the other half our IT department failed to connect to the web service using .Net, however, they only tried once then went on holiday, so the will try again later. However, I agree with your analysis, this won't break anything other than SOAP, and it will probably only make SOAP work better, not worse, so probably OK to integrate this for now, even if there is more work to do.
          Hide
          Jérôme Mouneyrac added a comment -

          I'm not going to send the patch for integration, I'm not an expert of SOAP and there are two things I'm not sure about:

          1- when the param type is an array, then nothing is defined in the WSDL.

          We currently generate (patch or not patch, same issue):

          <xsd:complexType name="webservices_struct_class_000000">
                          <xsd:all>
                              <xsd:element name="parents" type="soap-enc:Array"/>
                              <xsd:element name="files" type="soap-enc:Array"/>
                          </xsd:all>
                      </xsd:complexType>
          

          I guess we still need to transform the type="soap-enc:Array" in "complexType", and we need to fully describe the complexType like Tim did in his patch for single_structure (i.e. we need to do it for multiple_structure)

          2- the patch is not compatible yet with the JAX-WS generator like Netbeans SOAP client generator. However it is most likely a problem related to the Zend SOAP server config. I noticed there is a related issue in the Zend Jira tracker: http://framework.zend.com/issues/browse/ZF-6077

          Show
          Jérôme Mouneyrac added a comment - I'm not going to send the patch for integration, I'm not an expert of SOAP and there are two things I'm not sure about: 1- when the param type is an array, then nothing is defined in the WSDL. We currently generate (patch or not patch, same issue): <xsd:complexType name= "webservices_struct_class_000000" > <xsd:all> <xsd:element name= "parents" type= "soap-enc:Array" /> <xsd:element name= "files" type= "soap-enc:Array" /> </xsd:all> </xsd:complexType> I guess we still need to transform the type="soap-enc:Array" in "complexType", and we need to fully describe the complexType like Tim did in his patch for single_structure (i.e. we need to do it for multiple_structure) 2- the patch is not compatible yet with the JAX-WS generator like Netbeans SOAP client generator. However it is most likely a problem related to the Zend SOAP server config. I noticed there is a related issue in the Zend Jira tracker: http://framework.zend.com/issues/browse/ZF-6077
          Hide
          Tim Hunt added a comment -

          Assigning back to Jerome. Since he spurned my efforts, he can take this forwards

          Show
          Tim Hunt added a comment - Assigning back to Jerome. Since he spurned my efforts, he can take this forwards
          Hide
          Jérôme Mouneyrac added a comment -

          Thanks Tim, I was surprised by your JAVA server patch and very happy

          My concern being that the arrays are not fully detailled in the generated phpdoc (this code part is not changed in the patch) and so there are not fully detailled in the WSDL. If we end up to find out that we can not generate a fully detailled WSDL for arrays, we would need to go back into the direction to generate the WSDL from parsing the Moodle web service description. And so removing your previous code. This doesn't seem like a quick task and it is not a Blocker/Critical issue (yet).

          To people who voted: is Tim's patch good enough for your Java needs or you can not create a JAVA client with the patch?

          Show
          Jérôme Mouneyrac added a comment - Thanks Tim, I was surprised by your JAVA server patch and very happy My concern being that the arrays are not fully detailled in the generated phpdoc (this code part is not changed in the patch) and so there are not fully detailled in the WSDL. If we end up to find out that we can not generate a fully detailled WSDL for arrays, we would need to go back into the direction to generate the WSDL from parsing the Moodle web service description. And so removing your previous code. This doesn't seem like a quick task and it is not a Blocker/Critical issue (yet). To people who voted: is Tim's patch good enough for your Java needs or you can not create a JAVA client with the patch?
          Hide
          Michael de Raadt added a comment - - edited

          Yes, please comment people.

          We're wondering if the current suggested implementation is sufficient for your needs or if we should continue developing this for a future release.

          Show
          Michael de Raadt added a comment - - edited Yes, please comment people. We're wondering if the current suggested implementation is sufficient for your needs or if we should continue developing this for a future release.
          Hide
          Tim Hunt added a comment -

          Surprised to see this re-classified as a new feature. Moodle 2.0 claimed to support SOAP, but it does not actually work. That is: it is buggy.

          Also, this is a blocker for the OU, and if I recall, at least on other person in the web services forum.

          Oh well. We have a work-around, which is to copy the code in my patch into a local plugin. Just a pity that others in the community cannot benefit from it.

          Show
          Tim Hunt added a comment - Surprised to see this re-classified as a new feature. Moodle 2.0 claimed to support SOAP, but it does not actually work. That is: it is buggy. Also, this is a blocker for the OU, and if I recall, at least on other person in the web services forum. Oh well. We have a work-around, which is to copy the code in my patch into a local plugin. Just a pity that others in the community cannot benefit from it.
          Hide
          Martin Dougiamas added a comment -

          Can this just be submitted for intergation into master? 2.2

          Can't be worse than what we had.

          Show
          Martin Dougiamas added a comment - Can this just be submitted for intergation into master? 2.2 Can't be worse than what we had.
          Hide
          Jérôme Mouneyrac added a comment -

          submitted for integration in master

          Show
          Jérôme Mouneyrac added a comment - submitted for integration in master
          Hide
          Jérôme Mouneyrac added a comment - - edited

          Once integrated we need to create a new issue about having a WSDL generating fully detailled arrays + JAVA client.

          Show
          Jérôme Mouneyrac added a comment - - edited Once integrated we need to create a new issue about having a WSDL generating fully detailled arrays + JAVA client.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Horrible eval, nasty SOAP, but if it helps... integrated. Please check it does not break others.

          Thanks and ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Horrible eval, nasty SOAP, but if it helps... integrated. Please check it does not break others. Thanks and ciao
          Hide
          Jérôme Mouneyrac added a comment -

          Just a note to help the testing team, I tested this: I checked that the unit tests were running fine with soap and xmlrpc, I read the entire code, and also I checked the generated phpdoc. You can pass it and blame me if it doesn't work

          Show
          Jérôme Mouneyrac added a comment - Just a note to help the testing team, I tested this: I checked that the unit tests were running fine with soap and xmlrpc, I read the entire code, and also I checked the generated phpdoc. You can pass it and blame me if it doesn't work
          Hide
          Rajesh Taneja added a comment -

          Jerome has tested this, and it seems it's not easy to test it. So blindly passing it.
          Thanks Jerome

          Show
          Rajesh Taneja added a comment - Jerome has tested this, and it seems it's not easy to test it. So blindly passing it. Thanks Jerome
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Such an effort has get its prize. Universal Award to the very-best Moodle developer in the family. Thanks!

          Closing. Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Such an effort has get its prize. Universal Award to the very-best Moodle developer in the family. Thanks! Closing. Ciao
          Hide
          Jérôme Mouneyrac added a comment -

          I created two new issues now that this one is closed:
          MDL-28988
          MDL-28989

          Show
          Jérôme Mouneyrac added a comment - I created two new issues now that this one is closed: MDL-28988 MDL-28989

            People

            • Votes:
              8 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: