Integrating Amazon S3 and CloudFront for video streaming

Amazon is a key player in the competition of cloud computing, and the service it provides can always satisfy our requirements if you can just dig a little bit deeper. Here, I'd like to show how to integrate the S3 and CloudFront service to provide video streaming service.

First, let's clarify some basic concepts of S3 and CloudFront. Amazon S3 is a storage provider which can store all kinds of data. But in order to deliver the content more rapidly for users globally, CloudFront uses its edge locations to obviously improve the responsive time. When people try to fetch a content via CloudFront, the server will automatically routes the user to the most closest location which host the replica of the data. However, CloudFront doesn't provide storage service, which is complemented by S3. That's why they always appear in the same place.

1. Distribute S3 bucket to CloudFront location

OK, so our first step is to link the S3 and CloudFront to allow distribution. It can be done with S3 Fox plugin of FireFox, or by programming. But we need to clarify another issue before we step forward. There are also two kinds of distribution existed in CloudFront, static file distribution and streaming distribution. Currently, the latest S3 Fox only supports the static distribution, which can be simply  done by right clicking the bucket and 'manage the distribution'. In order to streaming distribution our bucket, we kind of have the only choice to code. Here is a short snippet of Java code to implement this function by using Jets3t library.

StreamingDistribution newStreamingDistribution = null;

try {

newStreamingDistribution = cloudFrontService.createStreamingDistribution(bucket.getName(), ""+ System.currentTimeMillis(),

null, "Test streaming distribution", true );

} catch (CloudFrontServiceException e1) {

log.error(e1.getMessage());

}

log.info("New Streaming Distribution: " + newStreamingDistribution);

StreamingDistributionConfig streamingDistributionConfig;

try {

streamingDistributionConfig = cloudFrontService.getStreamingDistributionConfig(newStreamingDistribution.getId());

log.info("Streaming Distribution Config: "+ streamingDistributionConfig);

} catch (CloudFrontServiceException e) {

log.error(e.getMessage());

}

2. After we enable the streaming distribution of the bucket, we will get a distribution URL. This is the base URL address that we will use throughout the whole process. Now we can use any methods to upload a multimedia file into the bucket we just created. Next we will use Flowplayer rtmp plugin to display the file into a browser.

Download the latest version of FlowPlayer as well as its rtmp plugin, and write a follow html page:

<html>

<head><title>Video</title><script src="flowplayer/flowplayer-3.1.4.min.js"></script>

</head>

<body>

<a class="rtmp" href="50f9307fbcdcdcaae65c4bc58857ca19-LOW" style="display:block;width:640px;height:360px;"></a>

<script type="text/javascript">$f("a.rtmp", "flowplayer/flowplayer-3.1.5.swf",

{clip:{provider: 'rtmp',autoPlay: true,},plugins: {rtmp: {url: 'flowplayer/flowplayer.rtmp-3.1.3.swf',netConnectionUrl: 'rtmp://s240vvr18v7md1.cloudfront.net/cfx/st'}}});</script>

</body></html>

Several places need to notice:

1) you must have the flowplayer.js, flowplayer.swf, and flowplayer.rtmp.swf ready to use and with the right path.

2) In the anchor tag, the href attribute is the file path/name of which you would like to play. Say for example, your file is XXX.mp3, and with the full path of http://AAA.s3.amazonaws.com/XXX.mp3, then you should place XXX in the href attribute. DON'T ADD THE EXTENSION!!!.

3) In the JavaScript section, the autoPlay indicate whether to run the file automatically, the netConnectionUrl must be set according to the distribution URL you retrieved from part 1. Remember, you must prefix "rtmp://" before the url and append "cfx/st" to the URL, and also you must not ignore the single quote around the whole URL.

Now, you can have your streaming video playing in your browser! Easy and Sweat!

GAE/J deployment transaction conflict error 409

Today when I tried to deploy my application onto the App Engine, a weird error prompt  up:

Unable to update app: Error posting to URL: https://appengine.google.com/api/appversion/create?app_id=metacdn&version=1&
409 Conflict
Another transaction by user featheast.lee is already in progress for this app and major version. That user can undo the transaction with appcfg.py's "rollback" command.
See the deployment console for more details

I haven't seen error before, and have no clue how it comes out. After several google page, I found a way to solve this problem.

1. Open your terminal and get into the directory of your project. ( For windows users, please do it in CMD accordingly).

2. Execute the sh script of appcfg.sh under with the parameter of rollback and war. You should prefix the path of your appcfg.sh, which is usually under the directory of eclipse plugin. ( Windows guys, continue to use cmd to replace the sh script)

3. Now after successfully rollback the deployment, you are free to do anything you want to now.

I guess the same story should be applied to Python as well, just simply use the appcfg.py instead of the script and things should work out.

PS: I got a ZipExeption during the rollback process with a warning that Could not find API version from .svn. Though it still solves my deploy issue.

HTTP and HTTPS setup a Restlet environment

Restlet is a handy tool for people to setup an environment running RESTful web service. In order to secure some endpoint resource, sometimes it will be required to use HTTPS for communication, and Restlet, no doubt, supports both of the two ways.

1) HTTP

It's pretty easy to setup the HTTP environment, all you need to do is create a new component, and register the HTTP protocol into the component's server and things will work out as expected.

Component component = new Component();
component.getServers().add(Protocol.HTTP, 8183);
component.getDefaultHost().attach(new XXXApplication());

Notice, the 8183 is the port number you have to provide.

2) HTTPS

Unlike HTTP, in HTTPS mode, you need provide three more things: keystore, keystorePassword, and keyPassword.

For those who are not familiar with keystore: A Java container of keys and certificates is called a keystore. There are two usages for keystores: as a keystore and as a truststore. The keystore contains the material of the local entity, that is the private key and certificate that will be used to connect to the remote entity. Its counterpart, the truststore, contains the certificates that should be used to check the authenticity of the remote entity's certificates.

The steps to construct a keystore is detailed on the page: http://wiki.restlet.org/docs_2.0/13-restlet/27-restlet/46-restlet/213-restlet.html. Basically speaking, you need to use a SSL tool to generate keys first, then self-signed the certification. After all these steps finished,  following the code list below and you will run the HTTPS server successfully.

Component component = new Component();
Server server = component.getServers().add(Protocol.HTTPS, 8183);
server.getContext().getParameters.add("keystorePath", keystorePath);
server.getContext().getParameters.add("keystorePassword", keystorePassword);
server.getContext().getParameters.add("keyPassword', keyPassword);
component.start();

As the same theory, if you need to run any client side code as well under the same project, simply add the client's support protocol to the component as:

component.getClients().add(Protocol.HTTPS);

Using GAE python to bulk load CSV data into Java datastore

The official document is here: http://code.google.com/appengine/docs/python/tools/uploadingdata.html, more details will be covered here with Java applications.

1. Using any Windows environment to download Python SDK 2.5.X, preferably 2.5.4 since it is the last stable version with Windows Installer. Avoid to download 2.6.X and 3.X.X because GAE doesn't officially support these.

2. Download Google App Engine SDK for Python. Current version is 1.3.3. You may download GAE launcher which is only available in Windows.

3. Create a new project, naming it uploaddata (or whatever you like), add an app.yaml file

application: XXX
version: 1
runtime: python
api_version: 1

handlers:
-url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin

Add above code to the app.yaml file. Use the correct application name and version, do not change the script of the handler.

4. Generate a python class, which can mapping the datastore table into a class. An example is:

(Student.py)
from google.appengine.ext import db

class Student(db.Model):
studentId = db.StringProperty()
name = db.StringProperty()
address = db.StringProperty()
......

5. Create a data loader file used by the handler. Here is another example:

(loader.py)
import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader
import Student

class StudentLoader(bulkloader.Loader):
def __init__(self):
bulkloader.Loader.__init__(self, 'Student',
[('studentId', str), ('name', str), ('address', str)])
loaders = [StudentLoader]

Pay attention to those columns may contain characters in French Accent or Asian languages, use proper unicode to convert.

4. With command line, using the following command to upload data. With previous example, a sample command would look like:

appcfg.py upload_data --config_file=loader.py --filename=data.csv --kind=Student uploaddata

App Engine SDK 1.3.3 released

Although the Eclipse plug-in has not updated yet, I have tried to download the zip file. This is only a minor updated version with limited features. One of the most important thing worth to pay attention is the SQLite support for Python. Despite of the official word "Note that this feature does not add SQL support to the App Engine SDK or service", it is a great advantage for developers to harness the benefits of SQLite. Just wondering when will they expand the support to Java field. It seems Java is always a step backward compared to the development team of Python.

New Warning Message on Google App Engine

I just noticed a pretty new warning appears in the log of App Engine:

"This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application."

Previously, there is no such thing highlighted the reason why sometimes a simple operation might take longer CPU hours than expected. Now with this handy warning, things are much clear and you don't have to worry about any internal coding issues with your project.

However, if Google can solve this issue, it will be much better. Currently, in order to avoid such re-loading time waste, I have to reduce my cron job frequency to once per minute. Yep, when it runs in 2/min, App Engine will complain every time, and the overall CPU cost is even higher than the more frequently job. Ironically, Ridiculously, huh?

I know some guys are working this issue now, hopefully it can be fixed in the near future.

Install spring 3.0 plug-in in Eclipse 3.5

Actually this should be a pretty old topic, since I have done the same thing more than 5 times. However, without note down all the details, each time I have to google to solve some issue. Every time I just wondered why in the first place I keep this details down, now finally I decide to write all this down in case next time I have to google again and again.

So what you need to do is simply use the install new software tool in the Help menu. The address of spring plug-in is:http://springide.org/updatesite .And actually all you need is pretty clearly written on the web site of http://springide.org/project/wiki/SpringideInstall.

However, there might be several issues you will encounter. In my situation, you will always get a complaint about the missing component of "package org.eclipse.ajdt.internal.ui.lazystart 0.0.0". This package belongs to AJDT (which stands for AspectJ Development Tools) could be downloaded through the site of http://download.eclipse.org/tools/ajdt/35/update, and relevant info is here: http://www.eclipse.org/ajdt/downloads/.

After that try the spring plugin once again, in my case no complain any more and you should be able to get dirty with Spring finally.

PS: I have tried this method both in Ubuntu 9.10 and Mac OS Snow Leopard. Haven't try on Windows so please forgive me if it fails with Microsoft.