finesse gadgets.io.makerequest PUT/POST data with utf-8 charset

[michalkaska99]

Hallo,

I have developed a simple gadget into finesse 11 on UCCX, which use UCCX admin api to manage skill assignment on resources.

I used gadget.io.makerequest and successfully made a GET request to UCCX and successfully received data in XML format for all resources with all information.

When I tryed to do PUT request with POST_DATA where I changed some elements (mainly skil assignment) I was success only if all of the element value contained in the element values following chars [a-z, A-Z, 0-9, !@#$%^&*]. It seem all chars from lower ASCII table.

Unfortunately, some elements, like firstname and lastname contains chars from extended ascii. We are CZECH so we have czech letters like ěščřžýáíé in the names and when i try to PUT resource, where name contains such Czech letters, the request is processed, but czech letters are exchanged by for example \302\304 on others. Only letters from lower ascii are correctly transfered.

For example <lastaname>Káňa</lastname> element are transfered as <lastname>K\303\203\302\241\303\205\302\210a</lastname>.

When CCX receives this XML, answer, that lastname cannot be modified.

This problem occurs when I do PUT. When I do GET request to CCX admin api, I receive XML data correctly with Czech letters, but when I generate PUT, in Firefox console I see correct string with czech letters, but in packet capture I see the incorect formating

I am attaching the packet capture, where is visible the PUT Request.

I played with "charset" in content-type, but without any effect.

I also tryed to play with escape codes in the string, but this function did not converted only chars from lower ASCII.

Can you please help me, what should be wrong in my PUT request with gadget.io.makeRequest ?

Full XML is here:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><resource><self>http://10.164.248.90/adminapi/resource/179034</self><userID>179034</userID><firstName>Martin</firstName><lastName>Káňa</lastName><extension>954292406</extension><alias/><resourceGroup name="HD Brno"><refURL>http://10.164.248.90/adminapi/resourceGroup/2</refURL></resourceGroup><skillMap><skillCompetency><competencelevel>5</competencelevel><skillNameUriPair name="H_EXT"><refURL>http://10.164.248.90/adminapi/skill/3</refURL></skillNameUriPair></skillCompetency><skillCompetency><competencelevel>5</competencelevel><skillNameUriPair name="H_OST"><refURL>http://10.164.248.90/adminapi/skill/20</refURL></skillNameUriPair></skillCompetency><skillCompetency><competencelevel>5</competencelevel><skillNameUriPair name="H_P84"><refURL>http://10.164.248.90/adminapi/skill/22</refURL></skillNameUriPair></skillCompetency><skillCompetency><competencelevel>5</competencelevel><skillNameUriPair name="H_POL"><refURL>http://10.164.248.90/adminapi/skill/24</refURL></skillNameUriPair></skillCompetency><skillCompetency><competencelevel>5</competencelevel><skillNameUriPair name="H_VIP"><refURL>http://10.164.248.90/adminapi/skill/25</refURL></skillNameUriPair></skillCompetency></skillMap><autoAvailable>true</autoAvailable><type>1</type><team name="Brno"><refURL>http://10.164.248.90/adminapi/team/2</refURL></team><primarySupervisorOf/><secondarySupervisorOf/></resource>

[michalkaska99]

One hint, it seems that problem is with UTF-8 char length. All 2-bytes chars are posted perfectly and all 4-byte UTF-8 chars are wrong.

Any idea is appriciated.

Michal

[dekwan]

Hi Michal,

Sorry for the delayed response. I was trying to figure this one out for myself as I was able to reproduce the problem. Unfortunately I don't have an answer for you yet and am still working on it. I would like to note that gadgets.io.makeRequest is from Shindig itself. It is possible that the text needs to be encoded or escaped somehow and we just have to figure it out. When using a client such as Postman to make the request, the REST request goes through successfully. Hopefully I have some good news for you soon, if you have not already figured it out yourself.

Thanx,

Denise

[michalkaska99]

Hallo,

thank you for your interest. I played with escape /unescape without effect, but maybe you will be more lucky.

I made the functional workaround using ajax for HTTP PUT and here is not problem with UTF-8:

xmlPut: function (url,document){

       $.ajax({

          type: 'PUT',   

          url: url,   

          headers: {   

                "Authorization": "Basic xxxxxxxxxxxx",

                "Content-Type": "application/xml"

         },

         dataType: 'xml',   

         data: document,   

         success: function(xhr){

              clientLogs.log ("xmlPut(): success ");

              updateResourceSuccess(xhr);

         },

        error: function (xhr,textStatus,error){   

           clientLogs.log ("xmlPut(): error " + xhr.responseText);

           clientLogs.log ("xmlPut(): error " + textStatus);

           clientLogs.log ("xmlPut(): error " + error);

        }

    });

},

But as far I searched, this is not ideal for finesse.

[dekwan]

Hi,

I got a little bit farther, but not quite there yet (although promising). I used these two pages as references to understand the different encodings/escapes for á and ň:

http://graphemica.com/%C3%A1

http://graphemica.com/%C5%88

What I noticed is that under the Source Code section, for á, JavaScript and JSON is \u00E1 and for ň, it is \u00148. According to the CCX Config API spec, it accepts content type of both application/xml as well as application/json. So, I switched to JSON and had last name as "lastName": "K\u00E1\u0148a", and it went through successfully! It isn't quite there yet because now you have to figure out how to get á to \u00E1 and ň to \u00148, but I found many online tools (Unicode code converter) that do it. So, my assumption is that there is an algorithm out there to do it.

From all of my trial and error, it is obvious that the gadgets.io.makeRequest is doing some sort of encoding/escaping. Unfortunately it is doing it incorrectly. Since gadgets.io.makeRequest is not Finesse code, there isn't much Finesse can do about it. Also, it appears that when using application/xml format, CCX does not decode anything before processing it so manually typing in the octal sequence /303/241/305/210 does not work either.

I am not sure how you feel about using JSON instead of XML, but it seems like that is the most promising thing out there if you want to continue to use gadgets.io.makeRequest. I agree that using ajax is not ideal, but you might want to evaluate how often this request will be made. It might not be so bad. There might be other javascript HTTP PUT request libraries out there too.

Thanx,

Denise

[dekwan · Accepted solution]

I managed to figure out the value for the lastName with application\xml format that is successful. You need to use the Decimal HTML Entity so it will look like <lastName>K&#225;&#328;a</lastName>. Again, I haven't figured out how to get á to &#225; but I am sure there are libraries out there to do it.

[michalkaska99]

Hallo,

Great Job!!! I replicated your way with HTML entity, and it really works.

Thank you!

[dekwan]

Hi,

Awesome! I'm glad that it worked!

Thanx,

Denise