Home ›

TOP 7 Exceptions in my google app engine described

Christian Harms's picture
by Christian Harms on Sat, 11/21/2009 - 19:40
in

Back to the basics: The google app engine for python developers provides an environment for web applications, which fullfills many needs:

  • a scalable storage (including big table) with 1Gb storage quota, 12GB of incoming and 116 outgoing GB traffic
  • the URLfetch API for access to web services like the google crawler
  • scaled memcached (10 GB incoming and 50 GB of outgoing traffic)
  • Google login integration
  • django-compatible web development

First, the app engine dashboard.

The web application is available via the app engine point in my google "my account" and offers 10 slots for 2nd-level domains under *. appspot.com. On the overview page a chart is displayed, which offered a selector with values-pairs such "response time per request" (1) for example. The usage of important quota values follows in a smaller box (2) below. Since the daily available quotas for small web applications are very high, there is usually nothing important (if it does, you should update your account under the billing point). The detailed quota page shows all API calls in real time for the last 24 hours.

Directly below the diagram of the home page there is the list of "Current Load" with paths (3) which in the last 24 hours produced the greatest cpu load. To the right are shown the mostly happed errors per path (4).

The error-log

Like the Apache web server error.log all entries are displayed in an ascending list (filtered by error level) in the dashboard. For application logging use the python logging module. If you carry on an official and a developer version you have to select the small select box in the upper corner (5) for the non default application log (online at http://#version#latest.#subdomain#.appspot.com).

look into the datastore

Even the google big table can be accessed directly from the dashboard. You can view them by select-boxes or GQL (a slimmed-SQL dialect). All change in the web application will appear in real time. Whether it delays under load or are large amounts of data, I will review in the future.

Previous general fault cases

In addition to real programming errors (and then displayed directly to your visitors eg, HTTP status code 500), some interesting mistakes in nature are listed. If the google app engine has a common problem, you can visit the status page of google app engine .

catching python exceptions in the app engine application

A general disadvantage of python compared to strictly declarative languages such as java there are not a defined set of exception per modules/function with syntax checking. In python is the agreement that the general failures of one module are caught with module.Error. Special cases can be read in the documentation of the module.

I found the following errors and will try to give some hints to solve the problem.

Error while fetch: ApplicationError: 5

This error occurs after ~ 5 seconds without results on the fetchURL call and could be very common. Its called "DEADLINE_EXCEEDED"

  1. from google.appengine.api import urlfetch
  2. data = urlfetch.fetch(url_with_long_response_time)

This can be intercepted with a simple try-catch. Slow sources should be mitigated with a memcached pre-call

  1. from google.appengine.api import memcache
  2. from google.appengine.api import urlfetch
  3. def fetchUrl(url):
  4.     data = memcache.get(url)
  5.     if !data:
  6.         try:
  7.             data = urlfetch.fetch(url)
  8.             memcache.set(url, data)
  9.         except urlfetch.Error:
  10.             data = ""
  11.     return data
  12. #fetchUrl

Error while fetch: ApplicationError: 2

Wow - the next well named error which is defined in "google/appengine/api/urlfetch_service_pb.py" as "FETCH_ERROR". This could happen with the following reasons:

  • missing host
  • MAX_REDIRECTS is exceeded
  • the protocol of the redirected URL is bad or missing.
  • normal fetching exception like httplib.error, socket.error, IOError

The other ApplicationError numbers of urlfetch are:

  • 1: INVALID_URL
  • 3: UNSPECIFIED_ERROR
  • 4: RESPONSE_TOO_LARGE

And remind the allowed http limitations. Ports in live apps could be 80, 443, 4443, 8080 - 8089, 8188, 8444, 8990 and allowed http methods are GET, POST, HEAD, PUT and DELETE. Other will raise an UNSPECIFIED_ERROR (id=3).

The ResponseTooLargeError will raised if the response (or request) larger than 1 megabyte. It can be catched and you are happy. Or try the option allow_truncated=True in the fetch-function for truncating the response.

Timeout or Error: An error occurred for the API request datastore ...

Also in the runtime environment for the python google app engine some components are not available. This applies particularly to the datastore. There are google app engine datastore error handling is needed.

  1. from google.appengine.ext import db
  2. try:
  3.     for stat in db.GqlQuery("select * from ..."):
  4.         pass
  5. except db.Error, exStr:
  6.     logging.error("Error while reading from db:% s"% exStr)

CapabilityDisabledError

That is a strange error and should result in an general error page in the application. Google offers more information in the python traceback:

Datastore writes are temporarily unavailable. 
Please see http://code.google.com/status/appengine for more information.
Traceback (most recent call last):
...
...
...
CapabilityDisabledError: Datastore writes are temporarily unavailable. 
Please see http://code.google.com/status/appengine for more information.

ZeroDivisionError: float division

If you have nothing (or an error) picked up by urlfetch, this error would be possible. The length of the amount of unclaimed items and calculation with the number 0 can provoke the ZeroDivisonError and this is a simple programming error!

UnicodeDecodeError: 'utf8' codec ...

I have collected some data by urlfetch, but the encoding of the data was ignored. That is a typical application problem. If you do not define the encoding of the incoming data from urlfetch, for example, this errors occurs. Not directly - at least by saving such data in the datastore.

No handlers matched this URL.

Some one try to call a not declared resource of your application. If all your components are correct linking it would be possible a script kiddie having some fun and can be ignored. If the url should be part of your application edit your app.yaml:

  1. handlers:
  2. - url: /geo_data.js
  3.   script: geo_data.py
  4.  
  5. - url: /index.html
  6.   static_files: geo_demo.html
  7.   upload: geo_demo.html

ValueError: Unterminated string starting at ...

This looks like a syntax error which should not happen in a live application. After checking the module the reason is simple. I got JSON data from a web resource and parsing it directly with simplejson! This can be caught like this:

  1. from simplejson import decoder
  2. try:
  3.     result = decoder.JSONDecoder().decode(self.request.get(url))
  4. catch ValueError:
  5.     logging.warn("Error while parsing the json data of %s." % url)
  6.     result= None

Conclusion

official exceptions documentation

That was just a random collection of the common mistakes that you can fix quickly and easily. Here is a link list of the defined exceptions for the mainly used google python modules:

  • google datastore exceptions defined in the google.appengine.api.db module
  • memcached comes from danga, but google build a new python implementation: google.appengine.api.memcache module. By calling the client-methods there exists only TypeError and ValueError. For serialization the protocollbuffer module is used which can raise an ProtocolBuffer.ProtocolBufferDecodeError.
  • the fetchurl api has only the self-explained InvalidURLError, DownloadError and ResponseTooLargeError defined in the google.appengine.api.urlfetch package.
  • Exceptions for sending mails are defined in the google.appengine.api.mail package.
  • to manipulate images take note the exception defined in google.appengine.api.images package.

other resources

Comments

FFD's picture
App Engine
FFD (not verified) - Mon, 12/14/2009 - 11:55

I have been working with the Google App Engine for Java for the last weeks, on a similar project (feed/playlist/directory proxy). I love it so much that I hate returning to any self-managed hosting ever again.

One weak point was that your app gets unloaded after being quiet for sometime, which is especially ugly if you're in "demo mode": it takes some time to restart (thus, will fail with timeout in your browser demo). As a workaround, one may write a cron-job calling the app regularly.

Second, the database layer is just bad, at least in Java. Two standard APIs are offered, JPA and JDO, very similar to each other, both providing an object-relational view of an essentially non-transactional, non-relational database. The result is very poor and often unpredictable. I'd recommend to go with the non-standard low level API (you won't be able to write portable code with the high-level APIS).

Apart from these, the overall experience is fantastic.