Long IP Address

Most of the time, people will think of an IP adress as a String. Especially in Java, most of the time, developers will deal with IP address either with URL or String class. However, representing an IP adress with String has several disadvantages. First a String usually takes more memory comparing to the "same" value int or long, and it will be difficult to compare with other IP addresses with a String. And more importantly, it will not be possible for people to easily determine whether an IP address is in the range of another two IP addresses.

Since IP addresses (IPv4) are composed by four integers ranging from 0 - 255, it will be obviously easy to convert an String IP address to an numerical form which can also uniquely represent the IP address. That's how Long IP address emerges.

A simple method to convert the String IP address to a Long IP address (A.B.C.D) would be:

256*256*256*A + 256*256*B + 256*C + D

Using Long IP address will be helpful in certain scenarios, one is when you dealing with IP-Location mapping in Google App Engine. A very popular data called GeoIP created by MaxMind is heavily used in a lot of different projects. However, when parsing the IP addresses, what they have done in the Java library is first transform the IP String into an InetAddress, and then using getAddress() to get its byte[], and finally get the Long value. There will be no problem when you using this library in other platforms. But in Google App Engine, things will got stuck because of the InetAddress is on the black-list of GAE, which means you will not be able to play with this class. The workaround here would be using the converting method above, you can directly get the Long value which is what they have calculated all the way along.

There might be some other places Long IP addresses is useful, especially when dealing with range query of IP addresses.

Using GAE python to bulk load CSV data into Java datastore

The official document is here: http://code.google.com/appengine/docs/python/tools/uploadingdata.html, more details will be covered here with Java applications.

1. Using any Windows environment to download Python SDK 2.5.X, preferably 2.5.4 since it is the last stable version with Windows Installer. Avoid to download 2.6.X and 3.X.X because GAE doesn't officially support these.

2. Download Google App Engine SDK for Python. Current version is 1.3.3. You may download GAE launcher which is only available in Windows.

3. Create a new project, naming it uploaddata (or whatever you like), add an app.yaml file

application: XXX
version: 1
runtime: python
api_version: 1

handlers:
-url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin

Add above code to the app.yaml file. Use the correct application name and version, do not change the script of the handler.

4. Generate a python class, which can mapping the datastore table into a class. An example is:

(Student.py)
from google.appengine.ext import db

class Student(db.Model):
studentId = db.StringProperty()
name = db.StringProperty()
address = db.StringProperty()
......

5. Create a data loader file used by the handler. Here is another example:

(loader.py)
import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader
import Student

class StudentLoader(bulkloader.Loader):
def __init__(self):
bulkloader.Loader.__init__(self, 'Student',
[('studentId', str), ('name', str), ('address', str)])
loaders = [StudentLoader]

Pay attention to those columns may contain characters in French Accent or Asian languages, use proper unicode to convert.

4. With command line, using the following command to upload data. With previous example, a sample command would look like:

appcfg.py upload_data --config_file=loader.py --filename=data.csv --kind=Student uploaddata

App Engine SDK 1.3.3 released

Although the Eclipse plug-in has not updated yet, I have tried to download the zip file. This is only a minor updated version with limited features. One of the most important thing worth to pay attention is the SQLite support for Python. Despite of the official word "Note that this feature does not add SQL support to the App Engine SDK or service", it is a great advantage for developers to harness the benefits of SQLite. Just wondering when will they expand the support to Java field. It seems Java is always a step backward compared to the development team of Python.

New Warning Message on Google App Engine

I just noticed a pretty new warning appears in the log of App Engine:

"This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application."

Previously, there is no such thing highlighted the reason why sometimes a simple operation might take longer CPU hours than expected. Now with this handy warning, things are much clear and you don't have to worry about any internal coding issues with your project.

However, if Google can solve this issue, it will be much better. Currently, in order to avoid such re-loading time waste, I have to reduce my cron job frequency to once per minute. Yep, when it runs in 2/min, App Engine will complain every time, and the overall CPU cost is even higher than the more frequently job. Ironically, Ridiculously, huh?

I know some guys are working this issue now, hopefully it can be fixed in the near future.