Home ›

Escaping examples and the worst test data

Christian Harms's picture
by Christian Harms on Mon, 12/14/2009 - 21:18
in

After the fine and long article about escaping from Matthias here some examples for special characters in a simple web application. This article should be only an inspiration, I will describe some code samples with python/javascript and explain why [<"@%'&_\?/:;,>কী €] is the ultimate input to test input in web applications.

The demo application offers a simple form with name and message field for an one-line guest book.

The examples will start with three lines. First the input data (which can testet in the demo app), second the escaped values in the transport data and a hint, why the escaping is needed.

Escaping in url parameters

To send values via urls use a simple formular and your browser will do all for you. For testing use these special values are &, ?, =, # and the space.

  1. Input: [Barnes&Noble] says [TOP #10 question?]
  2. used transport url is:
    1. ... resource?who=Barnes+%26+Noble&what=TOP+%2310+question%3F

  3. If you forget to escaping the "#" the url parameter ended and the url fragment part starts.

parameters in a url

A simple html formular for sending with the GET-method:

  1. <form id="form" method="GET">
  2.   Name: <input id="who" name="who" size="10"> says <input name="what"/>
  3.   <input type="submit" value="Send as formular with GET-parameter"/>
  4. </form>

Because the browser sends the values your dont need to do something special.

Unescaping the url parameter in python

Dont try to parse the parameter directly from the ready build url. In the python webapp.RequestHandler has the right helper function:

  1. class MessageExampleApp(webapp.RequestHandler):
  2.     def get(self):
  3.         data = [self.request.get("who", ""), self.request.get("what", "") ]
  4.     #get
  5. #MessageExampleApp

special characters in JSON

If you use for sending (ajax-request) the data in the json format the special characters will be control characters and the ".

  1. Input:[Barnes & Noble] says [Peter cry "Yahoo"]
  2. the generated json code is:
    1. {"who":"Barnes & Noble","what":"Peter cry \"Yahoo\""}
  3. If you forget to escape the " the json could be incorrect and can not parsed.

send json with javascript & jQuery

To generate the complete data as a json string, do not simply adding the values. I use the json-lib from json.org, because the actual jQuery lib dont offers the stringify-function. To send the data simply use the $.ajax() function.

  1. <script type="text/javascript" src="//pyunitedcoders.appspot.com/json2.js"></script>
  2. <script type="text/javascript">
  3.   var data = {who: $('[name=who]')[0].value, what:$('[name=what]')[0].value};
  4.   $.ajax({ type: "POST",
  5.            data : JSON.stringify(data),
  6.            contentType : "application/json; charset=utf-8",
  7.            url: "...", dataType:"json", success: insertData });
  8. })
  9. </script>

insert json data into the document

Getting data in json format from a web service is a common task (and jQuery will callback the function insertData with ready parsed data). But if you insert the data in the document there is the next escaping pitfall. I used jQuery for the response and node.text() instead node.html() for inserting the data - the text()-function will escape problematic characters.

  1. function insertData(res) {
  2.     $("#msg").text(res.who + ' says "' + res.what + '"');
  3. };

Escaping in html

After getting data from a database and inserting in html, you should take care of html injection (special characters are <, > and &)!

  1. Input: [Barnes & Noble] says [<TOP10>]
  2. after including in the result page the html code is
    1. Barnes &amp; Noble says "&lt;TOP 10&gt;"

  3. If you forget to escape the characters < and > the message will become an html tag and (mostly) invisible!

escape the data in a django template

How to solve the problem? I used the escape filter in the basic django template.

  1.   <h2>Show the last 10 messages</h2>
  2.   <ul id="msg">
  3.   {% for msg in msg_list %}
  4.     <li>({{ msg.when|escape }}) {{ msg.who|escape }} says "{{ msg.what|escape }}"</li>
  5.   {% endfor %}
  6.   </ul>

the perfect test data

Use this string in every webform for testing: [<"@%'&_\?/:;,>কী €] ! Here is a (never complete) list of special characters for some common used components:

the embedding format the special characters
HTML < , > , &
JSON "
SQL in mySql Strings: ", ', Wildcards: %, _
rfc 1738 for URL-parameter ;, /, ?, :, ", @, =, & and space

Put all characters together and spice it with some special utf-8 characters. I used a random character from unicode.org and the special euro sign. If you find problems try the following developer tools to inspect your requests:

If these tools wont help, try more logging in your application or use a transparent proxy (for example Charles Web Debugging Proxy for windows).

Comments

Christian Harms's picture
And some other service dont can handle the data ...
Christian Harms - Mon, 12/14/2009 - 23:00

Ok, my test data can inserted in reddit, but not in dzone and ycombinator ;-)