tmpreaper and ctime, atime, and mtime in Ubuntu

There is a package in Ubuntu can be used to clean directories with those files older than a certain period of time. Before we get into that, let's first clarify three terms related to times in Ubuntu: ctime, atime and mtime.

ctime is the creation time of the file. Say I created this file at Wed Jun 16, 9:45:15, 2010, then this time spot will be exactly the ctime of this file.

atime is the access time of the file. Displaying the file contents or executing the file script will update the atime.

mtime is the modify(ication) time which will be updated when the actual content of the file is modified.

Back to the tmpreaper command, since it is not default installed into the Ubuntu, you have to sudo apt-get install to get the latest version of tmpreaper.

It is a simple command tmpreaper TIME-FORMAT DIRS to invoke the function to do the clean job for you.

TIME-FORMAT is a parameter that specifies the duration of the file which has not been accessed. By default, the time here is about atime. So even if you modify the content in a later stage but does not access the file, the file might still be deleted. Of course, you can enforce the command to run in terms of mtime which you have to append --mtime to the original command.

While the DIRS is the directory you would like to invoke this function, such as /tmp. Never try to do such a thing on the root directory or you may encounter a disaster.

If you have to manually run the command every time, then there is no sense to use this. While the power strengthens with combining another tool CronTab.

CronTab is used to create cron job to run specific script in a period of time. In order to run the cron job, all you need to do is write a script which include the command we have talked previously, then edit the configuration file of CronTab, then the scripts will run as you required in the background.

To edit the configuration file, simply run sudo crontab -e, add an entry in to the file. The format of the file is m h dom mon dow command, the first five sections are divided by space, and you can use asterisk to specify anytime like a wildcard.
Fox example, * * * * * /XXX.bash will run every minute. More usage can be seen from the documentations.

Handling Azure Large File Upload

In Azure storage, files smaller than 64MB can be directly stored as a single blob into the storage. However, when you want to store a file larger than 64MB, things will become a little bit complicated. The way to accomplish this task is to use the block list service.

Block, unlike blob, is a small unit of file which can be aggregated as a list to form a large file, with each of the small chunk to have a limit of 4MB. For example, say if you have a 100MB file which you want to store into Azure, you have to manually split the files into at least 25 pieces, and then using the put block & put block list operation to upload all the 25 items. More details are listed below:

1) Split large files: this can be done in various ways, via existing tools or write your simple code. Pay attention to write down those file names and make them in the sequence you split them.

2) Put Block: Each of the pieces created last step is called a block, and the second will upload each block one by one into the storage via Put Block operation. The basic process is no difference with other methods, however, one thing need to pay attention is the blockid is a required parameter and all blockids of the blocks must be the same size. In our example, you can have a Base64 blockid with arbitrary length less than 64, but you have to enforce all of the 25 items to have the same length. If not, a 400 exception, or The specified blob or block content is invalid error message will be returned.

3) Put Block List: The last but not the least step is to notify the server that all pieces are uploaded and now it's your job to combine them altogether.

After the three steps, you will be able to upload any size files into the Azure storage.

How to tell the version of your Mac OS

Step 1: Click the apple icon in the upper left corner of the screen

Step 2: Choose About this Mac

Step 3: The label right below the Apple indicates the version of this Mac OS, e.g. Version 10.6.2

Step 4: Right to the Processor label is the version of your CPU, from which we can tell the bits of the CPU

Here is a table to clarify the relationships:

Processor Name	32- or 64-bit
Intel Core Solo	32 bit
Intel Core Duo	32 bit
Intel Core 2 Duo	64 bit
Intel Quad-Core Xeon	64 bit
Dual-Core Intel Xeon	64 bit
Quad-Core Intel Xeon	64 bit

Ref : http://support.apple.com/kb/ht3696

Redirect vs Forward

In the context of web programming, a lot of ambiguity existed between Redirect and Forward. Here is a short summary of the differences between these two terms.

1) Forward can only be direct to an internal page, however Redirect can be used both to an internal page and external page.

2) Forward is much faster than Redirect

3) With Forward the browser is unaware of what happens, and the URL address remains the original link, while Redirect will initiate a new request that the browser will update its address to the new link.

4) As a result of the browser awareness, the refresh function will be failed in the Forward since the URL has not changed, but the Redirect will be all the same.

5) Still with the same reason, with those operations have side-effect, say update the status of database, a Redirect should be used to avoid the refresh Forward to generate any duplicate operations.

Ubuntu Path Setting

/etc/profile is loaded once on login for every user
/etc/bash.bashrc is loaded every time every user opens a terminal
~/.bashrc is loaded every time a single user opens a terminal
~/.profile is loaded once when a single user logs on

Integrating Amazon S3 and CloudFront for video streaming

Amazon is a key player in the competition of cloud computing, and the service it provides can always satisfy our requirements if you can just dig a little bit deeper. Here, I'd like to show how to integrate the S3 and CloudFront service to provide video streaming service.

First, let's clarify some basic concepts of S3 and CloudFront. Amazon S3 is a storage provider which can store all kinds of data. But in order to deliver the content more rapidly for users globally, CloudFront uses its edge locations to obviously improve the responsive time. When people try to fetch a content via CloudFront, the server will automatically routes the user to the most closest location which host the replica of the data. However, CloudFront doesn't provide storage service, which is complemented by S3. That's why they always appear in the same place.

1. Distribute S3 bucket to CloudFront location

OK, so our first step is to link the S3 and CloudFront to allow distribution. It can be done with S3 Fox plugin of FireFox, or by programming. But we need to clarify another issue before we step forward. There are also two kinds of distribution existed in CloudFront, static file distribution and streaming distribution. Currently, the latest S3 Fox only supports the static distribution, which can be simply  done by right clicking the bucket and 'manage the distribution'. In order to streaming distribution our bucket, we kind of have the only choice to code. Here is a short snippet of Java code to implement this function by using Jets3t library.

StreamingDistribution newStreamingDistribution = null;

try {

newStreamingDistribution = cloudFrontService.createStreamingDistribution(bucket.getName(), ""+ System.currentTimeMillis(),

null, "Test streaming distribution", true );

} catch (CloudFrontServiceException e1) {

log.error(e1.getMessage());

}

log.info("New Streaming Distribution: " + newStreamingDistribution);

StreamingDistributionConfig streamingDistributionConfig;

try {

streamingDistributionConfig = cloudFrontService.getStreamingDistributionConfig(newStreamingDistribution.getId());

log.info("Streaming Distribution Config: "+ streamingDistributionConfig);

} catch (CloudFrontServiceException e) {

log.error(e.getMessage());

}

2. After we enable the streaming distribution of the bucket, we will get a distribution URL. This is the base URL address that we will use throughout the whole process. Now we can use any methods to upload a multimedia file into the bucket we just created. Next we will use Flowplayer rtmp plugin to display the file into a browser.

Download the latest version of FlowPlayer as well as its rtmp plugin, and write a follow html page:

<html>

<head><title>Video</title><script src="flowplayer/flowplayer-3.1.4.min.js"></script>

</head>

<body>

<a class="rtmp" href="50f9307fbcdcdcaae65c4bc58857ca19-LOW" style="display:block;width:640px;height:360px;"></a>

<script type="text/javascript">$f("a.rtmp", "flowplayer/flowplayer-3.1.5.swf",

{clip:{provider: 'rtmp',autoPlay: true,},plugins: {rtmp: {url: 'flowplayer/flowplayer.rtmp-3.1.3.swf',netConnectionUrl: 'rtmp://s240vvr18v7md1.cloudfront.net/cfx/st'}}});</script>

</body></html>

Several places need to notice:

1) you must have the flowplayer.js, flowplayer.swf, and flowplayer.rtmp.swf ready to use and with the right path.

2) In the anchor tag, the href attribute is the file path/name of which you would like to play. Say for example, your file is XXX.mp3, and with the full path of http://AAA.s3.amazonaws.com/XXX.mp3, then you should place XXX in the href attribute. DON'T ADD THE EXTENSION!!!.

3) In the JavaScript section, the autoPlay indicate whether to run the file automatically, the netConnectionUrl must be set according to the distribution URL you retrieved from part 1. Remember, you must prefix "rtmp://" before the url and append "cfx/st" to the URL, and also you must not ignore the single quote around the whole URL.

Now, you can have your streaming video playing in your browser! Easy and Sweat!

GAE/J deployment transaction conflict error 409

Today when I tried to deploy my application onto the App Engine, a weird error prompt  up:

Unable to update app: Error posting to URL: https://appengine.google.com/api/appversion/create?app_id=metacdn&version=1&
409 Conflict
Another transaction by user featheast.lee is already in progress for this app and major version. That user can undo the transaction with appcfg.py's "rollback" command.
See the deployment console for more details

I haven't seen error before, and have no clue how it comes out. After several google page, I found a way to solve this problem.

1. Open your terminal and get into the directory of your project. ( For windows users, please do it in CMD accordingly).

2. Execute the sh script of appcfg.sh under with the parameter of rollback and war. You should prefix the path of your appcfg.sh, which is usually under the directory of eclipse plugin. ( Windows guys, continue to use cmd to replace the sh script)

3. Now after successfully rollback the deployment, you are free to do anything you want to now.

I guess the same story should be applied to Python as well, just simply use the appcfg.py instead of the script and things should work out.

PS: I got a ZipExeption during the rollback process with a warning that Could not find API version from .svn. Though it still solves my deploy issue.