App Engine SDK 1.3.3 released

Although the Eclipse plug-in has not updated yet, I have tried to download the zip file. This is only a minor updated version with limited features. One of the most important thing worth to pay attention is the SQLite support for Python. Despite of the official word "Note that this feature does not add SQL support to the App Engine SDK or service", it is a great advantage for developers to harness the benefits of SQLite. Just wondering when will they expand the support to Java field. It seems Java is always a step backward compared to the development team of Python.

New Warning Message on Google App Engine

I just noticed a pretty new warning appears in the log of App Engine:

"This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application."

Previously, there is no such thing highlighted the reason why sometimes a simple operation might take longer CPU hours than expected. Now with this handy warning, things are much clear and you don't have to worry about any internal coding issues with your project.

However, if Google can solve this issue, it will be much better. Currently, in order to avoid such re-loading time waste, I have to reduce my cron job frequency to once per minute. Yep, when it runs in 2/min, App Engine will complain every time, and the overall CPU cost is even higher than the more frequently job. Ironically, Ridiculously, huh?

I know some guys are working this issue now, hopefully it can be fixed in the near future.

Install spring 3.0 plug-in in Eclipse 3.5

Actually this should be a pretty old topic, since I have done the same thing more than 5 times. However, without note down all the details, each time I have to google to solve some issue. Every time I just wondered why in the first place I keep this details down, now finally I decide to write all this down in case next time I have to google again and again.

So what you need to do is simply use the install new software tool in the Help menu. The address of spring plug-in is:http://springide.org/updatesite .And actually all you need is pretty clearly written on the web site of http://springide.org/project/wiki/SpringideInstall.

However, there might be several issues you will encounter. In my situation, you will always get a complaint about the missing component of "package org.eclipse.ajdt.internal.ui.lazystart 0.0.0". This package belongs to AJDT (which stands for AspectJ Development Tools) could be downloaded through the site of http://download.eclipse.org/tools/ajdt/35/update, and relevant info is here: http://www.eclipse.org/ajdt/downloads/.

After that try the spring plugin once again, in my case no complain any more and you should be able to get dirty with Spring finally.

PS: I have tried this method both in Ubuntu 9.10 and Mac OS Snow Leopard. Haven't try on Windows so please forgive me if it fails with Microsoft.

Warning in JDO lazy fetch with App Engine

If you consistently meets the warning "org.datanucleus.store.appengine.MetaDataValidator warn: Meta-data warning for ****: The datastore does not support joins and therefore cannot honor requests to place related objects in the default fetch group. The field will be fetched lazily on first access. You can modify this warning by setting the datanucleus.appengine.ignorableMetaDataBehavior property in your config. A value of NONE will silence the warning. A value of ERROR will turn the warning into an exception." like this, please add the following line into your jdoconfig.xml:

<property name="datanucleus.appengine.ignorableMetaDataBehavior" value="NONE" />

Although it is quite obvious, but it's the problem where you put it might confuse many people.

Using memcache in GAE/J

Actually memcache is not a brand new concept, it has been utilized in many large scale projects. The most famous memcache event will be the Whale in Twitter's front page. To understand the background knowledge of memcache, please Google it. In one sentence, memcache is a way to cache those frequently used data internal or external to reduce the cost of database query and remote invocation. This is a very classical way in dealing with all kinds of database, however since most of the codes are written in low level , most of the developers are out of touch. Here, with the help of memcache, to transplant this idea into the application level, which will definitely be a great boost to large scale applications.

Google App Engine supports memcache for a long time, which should be one of its born advantage. Here I'd like to briefly introduce how to use memcache in GAE/J. Sorry for those who are interested in Python.

First, let's construct a scenario. There is a university system in GAE/J which stores information for around 10k students. Such a system includes all kinds of information for students to use, such as enrollment, study blackboard, course selection and so on. As a result, there will be a huge demand on the database query. However, such kind of demand always falls in two parts, which part of the students who really like the system and would like to log in everyday, another part students who can be considered as "lazy" seldom care about this. As a result, to improve the efficiency of the system. to cache those frequently used information will be a good help. Here we assume there is a table called Student, no matter what kind of action to happen, there is always the need to query the data in Student table. Let's see how memcache to store the Student information.

First we generate a JDO POJO class to store Student information. As an example, the fields in the class are pretty simple.

[java]@PersistenceCapable(identityType = IdentityType.APPLICATION)

@Inheritance(customStrategy = “complete-table”)

public class Student implements Serializable{

@PrimaryKey

@Persistent

private String uuid;

@Persistent

private String name;

@Persistent

private String email;

@Persistent

private String address;

public Student(){

this.uuid = UUID.randomUUID().toString();

//setter&getter

}[/java]

Then we need to construct a Cache class which will in charge of the operations in Cache layer.

[java]

public class QueryCache {

private static final Logger log = Logger.getLogger(QueryCache.class

.getName());

private static QueryCache instance;

private Cache cache;

private QueryCache(){

try{

CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory();

cache = cacheFactory.createCache(Collections.emptyMap());

}catch(CacheException e){

log.severe(”Error in creating the cache”);

}

}

public static synchronized QueryCache getInstance(){

if(instance==null){

instance = new QueryCache();

}

return instance;

}

public void putInCache(String address, String student){

cache.put(address, student);

}

public String findInCache(String address){

if(cache.containsKey(address)){

return (String)cache.get(address);

}else{

return null;

}

}

}

[/java]

Inside this class, we generate a new Cache instance under the Singleton pattern. A map resides in this class. At the same time, we define two methods, one to put the student information into the cache and another to get the information out of the cache.

Finally, we construct a servlet to query the student information.

public class QueryServlet extends HttpServlet{
private static final Logger log = Logger.getLogger(QueryServlet.class.getName());
@Override
protected void doGet(HttpServletRequest req, HttpServletResposne resp) throws ServletException, IOException{
log.info(”Now start……”);
QueryCache cache = QueryCache.getInstance();
String studentC = cache.findInCache(”Address7694″);
if(studentC!=null){
resp.getWriter().write(”Found the item in cache!”);
}else{
resp.getWriter().write(”No hit in cache!”);
PersistenceManager pm = PMF.get().getPersistenceManager();
Query query = pm.newQuery(Student.class);
query.setFilter(”address==’Address7694′”);
List students = List query.execute();
if(students.iterator().hasNext()){
log.info(”Found one:”+student.toString());
resp.getWriter().write(”Found one:”+student.toString());
cache.putInCache(”Address7694″, student.toString());
}else{
log.info(”None found!”);
resp.getWriter().write(”None Found!”);
}
}
}

This is a very simple example to briefly show how to use memcache in GAE/J. However, a lot more things need to think about in reality such as where to use memcache, how to set the expire time of each cache, etc.

Export a jar file in Eclipse project

If you would like to export your eclipse project into a jar file, maybe the first thing comes out of your mind is to build with Ant. But here I'd like to recommend a very handy tool called FatJar, which is a very helpful plug-in for eclipse to package your project.

Really simple, just install the plug-in from http://kurucz-grafika.de/fatjar. Then export as a fat jar will generate the jar file you need.

More details can be found in its tutorial, just Google it.

A piece of Java code to split large file

Working with Azure recently, sometimes when you are trying to upload large files into the Azure storage service, you cannot simply push it. For files larger than 64MB, you have to split the file into small trunks, and upload each of them as a blob list. It's not hard to understand the concept however split large files may not seem to be easy for junior developers.

Here is a simple code to split large files into small pieces which may be helpful if tbis is what you want to achieve:

class FileSplit {

private File f;

private FileInputStream fis;

private String path;

private String fileName;

int count;

public FileSplit(File f) {

this.f = f;

fileName = f.getName();

count = 0;

path = f.getParent();

}

public int split() {

try {

log.info("Start to split files");

fis = new FileInputStream(f);

byte buf[] = new byte[4 * 1000 * 1000];

int num = 0;

while ((num = fis.read(buf)) != -1) {

if (createSplitFile(buf, 0, num) == -1) {

return 0;

}

count++;

log.info("Finished one piece");

}

log.info("All finished");

} catch (Exception e) {

log.severe(e.getMessage());

} finally {

if (fis != null) {

try {

fis.close();

} catch (Exception e) {

log.severe(e.getMessage());

}

}

}

return count;

}

private int createSplitFile(byte buf[], int zero, int num) {

FileOutputStream fosTemp = null;

try {

fosTemp = new FileOutputStream(path + "/" + count + ".tmppt");

fosTemp.write(buf, zero, num);

fosTemp.flush();

} catch (Exception e) {

return -1;

} finally {

try {

fosTemp.close();

} catch (Exception e) {

log.severe(e.getMessage());

}

}

return 1;

}

}