IPv6 request crashed my google app engine application

by Christian Harms on Mon, 03/29/2010 - 21:34

My IP geo location script cache requests coming from the same ip range (subnet mask 255.255.255.0) named as class-c ip address (last number cutted). This value is the reference for look up in memcache. But yesterday I got an exception while calculating the ip from the request object.

def get(self):
    ipStr = self.request.remote_addr
    classC = reduce(lambda a, b: int(a)*256+int(b), ipStr.split(".")[:3])
    logging.info("request from classC ip address:: %d" % classC)
    ...

This request crashed (TypeError: int argument required) by using an IPv6 address (found in the google app engine logs):

03-18 05:00PM 37.126 /geo_data.js 500 35ms 38cpu_ms 0kb Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-PT; rv:1.9.2) Gecko/20100115 Firefox/3.6,gzip(gfe)
2a01:e35:2f20:f770:6c54:xxx:67fb:df8 - - [18/Mar/2010:17:00:37 -0700] "GET /geo_data.js HTTP/1.1" 500 450 "http://www.xxx.com.br/spl19/index.php?refid=cm_av_ri" "Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-PT; rv:1.9.2) Gecko/20100115 Firefox/3.6,gzip(gfe)"

The typical log line looks like this:

03-19 02:12PM 23.047 /geo_data.js 200 411ms 272cpu_ms 175api_cpu_ms 0kb Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5,gzip(gfe)
189.55.xxx.160 - - [19/Mar/2010:14:12:23 -0700] "GET /geo_data.js HTTP/1.1" 200 770 "http://www.xxx.com.br/spl17/index.php?refid=gv_av_ge" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5,gzip(gfe)"

(A part of the complete ip address is changed)

Since March 8, 2010 App Engine joins the Google over IPv6 Program and the time is over to parse ip address with the typical ip address regex.

If you want to calculate a number from a IPv6 address try this short functional python code:

def ipToInt(ipStr):
    if ipStr.find(".")!=-1:
        return reduce(lambda a, b: int(a)*256+int(b), ipStr.split("."))
    else:
        return reduce(lambda a, b: a*65536+b, [int(x or "0",16) for x in ipStr.split(":")])

And I should check if the ip-to-geo-location provider offers his mapping with IPv6 adresses:

try it with ipInfoDb: Invalid IP address or domain name.
try it with geoPlugin.net: Invalid IP address
try it with hostIp: Location: Private block address.

*ups* - no one supports the IPv6 !

Comments

Your new function isn't going
Anonymous (not verified) - Tue, 03/30/2010 - 09:04

Your new function isn't going to work either. If you run ipToInt("2001:db8::1"), it says:

ValueError: invalid literal for int() with base 16: ''

ipv6 part can be empty.
Christian Harms - Tue, 03/30/2010 - 22:42

Funny - a 0-part can be keep empty, but the simple 'or "0"' will help. Thanx for the tip.

That still doesn't work,
Anonymous (not verified) - Wed, 03/31/2010 - 01:50

That still doesn't work, because you'll get a different result for different lengths of compressed zeros.

The moral of the story is, IPv6 addresses are too complicated to parse with a one-line hack. Use something like ipaddr-py, as suggested in the blog post you linked.

There's also socket.inet_pton(), but that's unfortunately not part of App Engine's stripped-down socket library.

zero compression looks crazy
Christian Harms - Wed, 03/31/2010 - 20:26

OK, thanx for the tip - after getting the right keywords, ipv6 zero compressing looks funny. But there should be a follow up for a one line ipv6 algorithm ;-)

oh no :-(
sebs (not verified) - Tue, 03/30/2010 - 22:32

oh no :-(

geoPlugin now provides limited support for IPv6
geoPlugin (not verified) - Tue, 04/13/2010 - 11:53

Thanks for your post which was sent to us by one of our users.

Based on this, geoPlugin has been modified to support IPv6 addresses.

However, due to the limited database functionality for IPv6 addresses, geolocation is limited to the country database and not the more detailed city database.

Hope this helps with your programs!

Tested on United-coders.com server iP (82.165.204.178) with the various IPv6-valid addresses:

http://geoplugin.net/json.gp?ip=2002:52A5:CCB2:0:0:0:0:0
http://geoplugin.net/json.gp?ip=2002:52A5:CCB2::
http://geoplugin.net/json.gp?ip=0:0:0:0:0:FFFF:82.165.204.178
http://geoplugin.net/json.gp?ip=::FFFF:82.165.204.178
http://geoplugin.net/json.gp?ip=::FFFF:52A5:CCB2
http://geoplugin.net/json.gp?ip=0:0:0:0:0:0:82.165.204.178
http://geoplugin.net/json.gp?ip=::82.165.204.178
http://geoplugin.net/json.gp?ip=::52A5:CCB2

Thanx for the reply
Christian Harms - Wed, 04/14/2010 - 22:55

Wow - the next direct answer from the location provider. That's a good news to see ipv6 is (in some cases) supported and can direct used with the script by our user.