IP address regex example - not in java

by Christian Harms on Thu, 09/17/2009 - 23:45

Finding an IP address in text or string with python is a simpler task than in java. Only the regex is not shorter than in the java regex example!

First an example with python: build a RegExp-Object for faster matching and than loop over the result iterator.

import re
logText =  'asdfesgewg 215.2.125.32 alkejo 234 oij8982jldkja.lkjwech . 24.33.125.234 kadfjeladfjeladkj'
bytePattern = "([01]?\d\d?|2[0-4]\d|25[0-5])"
regObj = re.compile("\.".join([bytePattern]*4))
for match in regObj.finditer(logText):
    print match.group()

A regex like /\d+\.\d+\.\d+\.\d+/ wont work, because there match "999.999.111.000" too. But for the usage in python - that is it! Using a regular expression is more native in python than in java. Or in javascript or in perl or asp.net...

And how to find it with JavaScript?

It looks like the small python example. Build the RegExp-Object for faster matching and a loop for finding all.

var logText = 'asdfesgewg 215.2.125.32 alkejo 234 oij8982jldkja.lkjwech . 24.33.125.234 kadfjeladfjeladkj';
var regObj = new RegExp("(?:(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])");
while (var result = regObj.exec(logText)) {
   alert("Matched: `" + result[0]);
}

For a more detailed example have a look at experts-exchange.com.

the native playground : perl

$txt = "asdfesgewg 215.2.125.32 alkejo 234 oij8982jldkja.lkjwech . 24.33.125.234 kadfjeladfjeladkj";
while( $txt=~/(([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))/g) {
  print $1."\n";
}

Some regex basics can be found at trap17.com and more regex examples on perl cookbook.

a more complex script: IP - log file statistic

But only finding an IP address in some irregular text is not a common use-case. An apache logfile is well formatted and the IP part can be found directly.

Do a simply split every line and the first part should be the IP address. To check if the first element match the IP pattern, use a ready function (for the example from the socket module). As addition and much more complexer example I will try to find the TOP10 IPs with the request count from the logfile.

import re, socket
hits = {}
 
#212.174.187.49 - - [13/Jul/2009:01:06:38 +0200] "GET /index.html HTTP/1.1" 400 335 - "-" "-" "-"
try:
    fp = file("all.log")
    for line in fp:
        elements = re.split("\s+", line)
        try:
            socket.inet_aton(elements[0])
            hits[elements[0]] += 1
        except KeyError:
            hits[elements[0]] = 1
        except socket.error:
            pass # no ip in the starting logline
finally:
    fp.close()
 
#Sorting the IPs with the hit-count
ipKeys = hits.keys()
ipKeys.sort(lambda a,b: hits[b]-hits[a])
for ip in ipKeys[:10]:
    print "%10dx %s" % (hits[ip], ip)

The result looks like this and runs 0.7 sec for 10.000 lines logfile (on Intel Atom N270).

      1406x 10.72.199.111
      1291x 10.214.141.196
       937x 10.43.81.243
       569x 10.43.205.83
       302x 10.235.116.128
       260x 10.121.239.210
       164x 10.145.232.155
       125x 10.106.120.225
       113x 10.93.210.194
       104x 10.174.153.69

The first block of the real addresses has been replaced with the number 10 for anonymity.

And the same IP script as command line with perl

The python script could made shorter and more ugly. Finding all IPs, sorting it, counting and print the TOP10 IPs.

perl -wlne 'print $1 if /(([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))/' all.log |sort|uniq -c | sort -n -r|head -n 10

The perl/shell example needs only 0.2 seconds for the same logfile.

conclusion

I like to use regex and python for getting some statistics, but the usage of some command line unix tool is useful too!

Comments

Same regex shortening available in perl and JS
Anonymous - Fri, 09/18/2009 - 23:03

The technique you use in the python code works just as well in the perl or javascript code. For example:

Also changed the "print $1 if /.../" to "print $1 while /.../g" to better match the earlier examples (in the case that multiple IP addresses appear on the same line).

Thanx, the improved example
Christian Harms - Mon, 09/21/2009 - 22:00

Thanx, the improved example for perl - it works for other languages - too!

There is only one OBVIOUS way... ;)
jh (not verified) - Thu, 10/08/2009 - 09:07

>>> bytePattern = "foo"
>>> [bytePattern for x in range(4)]
['foo', 'foo', 'foo', 'foo']
>>> [bytePattern] * 4
['foo', 'foo', 'foo', 'foo']

list * number
Christian Harms - Fri, 10/16/2009 - 15:47

Thanx - that's shorter and I edit the example.

And BTW, your regex strings
jh (not verified) - Thu, 10/08/2009 - 09:12

And BTW, your regex strings should be raw strings, i.e. r"...". Else you walk on thin ice regarding which characters are consumed as an escape sequence and which are not.

And finally, you can avoid
jh (not verified) - Thu, 10/08/2009 - 09:16

And finally, you can avoid the KeyError mumbo-jumbo by using a defaultdict:

>>> from collections import defaultdict
>>> h = defaultdict(int)
>>> h["1.2.3.4"] += 1
>>> h["1.2.3.4"] += 1
>>> h
defaultdict(, {'1.2.3.4': 2})

what about 666.666.666.666 ?
Anonymous (not verified) - Sat, 10/31/2009 - 05:34

666.666.666.666 is correct Ip address with your pattern in python?

No - the regex example
Christian Harms - Mon, 11/02/2009 - 22:05

No - the regex example matches only on correct IPs. And the python example use the socket module to check the correct ips. Try this out:

>>> import socket
>>> socket.inet_aton("666.666.666.666")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.error: illegal IP address string passed to inet_aton

if you'd like to parse URLs
ildar (not verified) - Thu, 11/12/2009 - 14:33

if you'd like to parse URLs (with hostnames or IPs) in JavaScript you can try this by the following link.
http://with-love-from-siberia.blogspot.com/2009/07/url-parsing-in-javasc...

Such a scripting of find out
green tea (not verified) - Thu, 11/12/2009 - 16:24

Such a scripting of find out IP address can be done in Java. It is very easy. You can use the protocol of IP address like ARP and RARP directly.

Pingback
IPv6 request crashed my google app engine application | unit (not verified) - Mon, 03/29/2010 - 21:46

[...] Since March 8, 2010 App Engine joins the Google over IPv6 Program and the time is over to parse ip address with the typical ip address regex. [...]

Could you please explain how this works.
Sum (not verified) - Mon, 05/23/2011 - 17:31

Firstly, thank you for putting up this IP address regex example in Python. Could you please explain how you are able to identify IP addresses using
"
bytePattern = "([01]?\d\d?|2[0-4]\d|25[0-5])"
regObj = re.compile("\.".join([bytePattern]*4))
"

How do these lines of code work?

Kind regards
Sum

explaining the regex
Christian (not verified) - Sat, 05/28/2011 - 18:41

the max. IPv4 number is 255.255.255.255.

First line describe the possible pattern for 255, the "|" means "or"

* match "[01]?\d\d?" - all numbers from 0 to 199
* or match "2[0-4]\d" - all numbers from 200-249
* or match "25[0-5]" - all numbers from 250-255

Second line only multiply the pattern, because a IPv4 adress has 4 identical parts.

Thank you for clearing that
Sum (not verified) - Sun, 05/29/2011 - 16:25

Thank you for clearing that up, I now understand how it is able to identify IP addresses.

import re logText =
kalai (not verified) - Wed, 07/27/2011 - 04:25

import re
logText = 'asdfesgewg 515.2.125.32 alkejo 234 oij8982jldkja.lkjwech .24.33.125.234 kadfjeladfjeladkj'
bytePattern = "([01]?\d\d?|2[0-4]\d|25[0-5])"
regObj = re.compile("\.".join([bytePattern]*4))
for match in regObj.finditer(logText):
print match.group()

I changed ip address 215.2.125.32 to 515.2.125.32, output produed is

15.2.125.32
24.33.125.234

How to correct it

not complete regex
Christian Harms - Sat, 07/30/2011 - 07:37

You have to extend the regex because it's matching correct IPs without check, if the character before or after the match is a number. Add an "\D" before and after the regexp:

bytePattern = "([01]?\d\d?|2[0-4]\d|25[0-5])"
regObj = re.compile("\D" + ("\.".join([bytePattern]*4)) + "\D")

ok thanks
kalai (not verified) - Sat, 07/30/2011 - 23:18

ok thanks