transcoding video formats to flv format

you can do this using ffmpeg. you can see the other post (http://khawarblog.blogspot.co.uk/2014/01/video-thumbnails-creating-montage.html) to know how you can install ffmpeg with LGPL license configuration. Once everything is ready, you can use the following command to transcode. For example, you have a test.avi file that you want to convert to flv format.

ffmpeg -i ./test.avi -ar 22050 -ab 32 -f flv avi.flv

It will generate a avi.flv output. However, there will be no sound in this. In order to tackle this issue, you have to reconfigure the ffmpeg with --enable-libmp3lame flag and repeat the installation process.

./configure --disable-gpl --enable-libmp3lame

make

make install

once it is done, you can use the above ffmpeg command and this time audio will work :)

sql tips

# for quick debugging

instead of connecting first to the mysql server and then see the result of your query. We can do it in one command using the 'e' flag.

mysql -uUSER -pPASSWORD -D DATABASE -e "select * from your_table;"

you can also remove the default table formatting provided by mysql.

mysql -uUSER -pPASSWORD -D DATABASE -s -e "select * from your_table;"

#return limited rows from a result

MySQL provides a keyword 'limit' to achieve this task. Lets say you want to retrieve first 10 records from a query.

select * from your_table limit 10;

you can also specify the 'start' and 'end' of this result. Let's say you want to retrieve 10 records after first 10 records. This is a useful feature especially for pagination.

select * from your_table limit 11, 20;

http://www.petefreitag.com/item/59.cfm

#for oracle

Oracle provides a keyword 'rownum' that behaves exactly same as 'limit' in mysql.

select name, price

from items

where rownum > 5 and

rownum < 11;

http://www.adp-gmbh.ch/ora/sql/analytical/dense_rank_vs_rank_vs_row_number.html

create video thumbnails

To create video thumbnails, if you search on google, you might get plenty of free softwares. Howover, to use them in a program may be cumbersome job because of GUI issues. For the command line tools, I figured out ffmpeg is the best option available. But there is one problem with it. It has two licenses GPL and LGPL. with GPL, you can not have a software for commercial purpose. However, with LGPL, you can develop a commercial software but in this license, you have to compile ffmpeg yourself and some of the featuers are reduced. Another opton is to use gstreamer-ffmpeg as gstreamer comes with LGPL license. However, it is not clear how things will work.

On ubuntu, it is easy to install a LGPL ffmpeg and a simple command can create thumbnail for you.

Installation of ffmpeg with LGPL license.

cd dev_area

git clone git://git.videolan.org/ffmpeg

cd ffmpeg

./configure --disable-gpl

make

make install

when you do the ./configure --disable-gpl, you should see at the end 'LGPL license'.  Once installed, you can start creating thumbnails of your video.

Creating thumbnails of a video

for example, the following shell script create thumbnails from a given video file at different times.

for i in 4 8 12 16 20 24; do

ffmpeg -itsoffset -$i -i /localscratch/dev_area/test.avi -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 test$i.jpg

done

Other options

VLC can also be used to create video thumbnails. But I could not make it work on ubuntu machine. However, if you are interested to explore it further, please have a look at the following URL. I am planning to use its snapshot options to see if they work.

http://wiki.videolan.org/How_to_create_thumbnails

mplayer can also do this, but not tested.

http://forum.xbmc.org/showthread.php?t=30023

using expect tool in shell script

Often times we run into problems with servers and there is a need to access multiple of them (if working in a cluster environment). Mostly, SSH is used to login into these systems and run the appropriate commands for debugging or maintenance purposes. However, SSH provides PKI-based or password-based authentication. with Password-based setup, it is very annoying to provide your password each time. Same situation occurs with other services such as FTP, telnet, rlogin, passwd etc. which needs user interaction for providing password to these commands. This makes it hard to write a automated script that requires no such user-interaction and speed-up the process. For this purpose, expect tool (http://expect.sourceforge.net/) is quite handy. For SSH only, I have tried sshpass tool and it also works with SSH. The following section provides details about using the expect tool.

Use expect on Ubuntu or other linux distros

In order to user it, first check if you have expect command available. in case you don’t have it, first install it on ubuntu using sudo apt-get install expect. For other linux distro, use appropriate package manager such as yum to install it.

Once it is installed, use the following script.

#!/usr/bin/expect

spawn ssh khawar@IP_ADDRESS/HOSTNAME
expect "*?assword:*"

send - "<PASSWORD>"
send — “\r"
send -- "\r"
send -- "ls\r"
expect eof

replace <password> with your remote machine password and this will let you login into the remote machine. This example will allow you to login into the given host and see its entires because we are sending ‘ls’ command.

you can use scp command here for file transfer. the script will be straight forward.

#!/usr/bin/expect

spawn scp /home/khawar/boto_s3_test.py khawar@IP_ADDRESS:~/
expect "*?assword:*"

send - "<PASSWORD>"
send — “\r"
expect eof

for sftp use the following script.

#!/usr/bin/expect

spawn -nottycopy -nottyinit sftp khawar@IP_ADDRESS
expect "*?assword:*"

send - "<PASSWORD>"
send — “\r"
expect “sftp>"
send -- "put  \r"
expect "sftp>"
send -- "quit\r"

expect eof

Using pexpect (python library) to pass password for password-based SSH authentication

In earlier post, I have talked about the expect tool. see http://khawarblog.blogspot.co.uk/2014/01/using-expect-tool.html. This post provides information about using a python library for the same purpose.

Python provides a library pexpect (https://pexpect.readthedocs.org/en/latest/) that behaves the similar as the expect tool. Since python is a high level language as compared with shell scripting, there is more power available to a user/developer to use it for various complex scenarios. Following is a sample using pexpect. This code launches a SCP command with password and transfers a local file to a remote server.

import pexpect
import sys

"""
username = will be your user
host = hostname or ip address of the remote machine
password = is password for the username on remote machine
"""
user_host="username@host"
user_pass="password"

filename="file path"
child = pexpect.spawn("scp  %s %s:~/" % (filename, user_host) )
#this will print the received output on stdout
child.logfile_read = sys.stdout
#the pseudo terminal waits for a line that ends with password: which is normally the case in ssh
child.expect(".*ssword: $")
#this will send user password to the terminal
child.sendline(user_pass)
#here it waits for EOF output
child.expect(pexpect.EOF)

print '\ndone'

CUnit test - Installation and sample Test

I tried installing CUnit with default prefix and settings in Ubuntu11.04, but I always got error during compilation. Major error was "Fatal error: CUnit/Basic.h: No such file or directory". After doing a bit of googling, I stumble upon to the following URL with the instructions given below.
http://ubuntuforums.org/archive/index.php/t-1727067.html

Step 1: Download CUnit package from Source Forge.
Step 2: Find where you saved the package on your system, and uncompress it:
tar xjf CUnit-2.1-2-src.tar.bz2

Step 3: Go into the CUnit-2.1.2 directory.
cd CUnit-2.1.2

Step 4: Run the following sequence of commands:
mkdir -p $HOME/local

./configure --prefix=$HOME/local
make clean
make
make install

Step 5: Goto the directory $HOME/local. Verify that you have the following sub-directories: doc, include, lib, and share. Each of these should have the products you seek.

Step 6. To build an application using CUnit, something like this is used:

gcc -Wall -I$HOME/local/include MyProg.cpp -L$HOME/local/lib -lcunit -o myprog

OR

g++ -Wall -I$HOME/local/include MyProg.cpp -L$HOME/local/lib -lcunit -o myprog

Note: In your source code, include CUnit header files such as:

#include "CUnit/CUnit.h";
#include "CUnit/Basic.h";

==How to run the program

Since this library is not part of the system lib path, we need to include it each time into system variables or pass them as flag (as we did in gcc/g++ command above).

lets include this path into system variable.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/local/lib

Now run the program

./myprog

==== Now with System wide installation and Testing====

After practicing above sequence and commands for compilation and execution, I tried the same logic with system wide installation which was giving me errors early on. In Ubuntu, CUnit's head files are located at /usr/local/include and its libs are available at /usr/lib. So the compilation command would be something like this.

gcc -Wall -I/usr/local/include MyProgram.cpp -L/usr/lib -lcunit -o myprogram

To run, simply try ./myprogram and it should work. in case if it doesn't, set the LD_LIBRARY_PATH as we did above.
first make sure /usr/lib is not part of the LD_LIBRARY_PATH by printing this variable by 'echo $LD_LIBRARY_PATH'
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib

==Mac OS X (10.6)
The same system-wide installation and execution instructions work for mac os x 10.6.

Apache WebServer2 with Tomcat 7

On Mac OS X (10.6)

download Tomcat 7 from official site.

preinstalled and configured apache httpd server on Mac.

you have to download and installed mod_jk apache connector.

A mix of the following two articles enable me to use Apache webserver to server usual contents and tomcat to serve the jsp or servlet.

http://macdevcenter.com/pub/a/mac/2002/08/20/tomcat_integration.html?page=1

mod_jk installation instrcution from the this link

http://www.bartbusschots.ie/blog/?p=1347

though the first link can be used for the entire working, however the instructions for compiling and installing mod_jk are complex.

After doing all this, I got the following error message in mod_jk.log

[Thu Jun 09 13:40:27 2011] [53894:140735072558240] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1216): (ajp13) can't receive the response header message from tomcat, tomcat (127.0.0.1:8080) has forced a connection close for socket 20

[Thu Jun 09 13:40:27 2011] [53894:140735072558240] [error] ajp_get_reply::jk_ajp_common.c (2058): (ajp13) Tomcat is down or refused connection. No response has been sent to the client (yet)

[Thu Jun 09 13:40:27 2011] [53894:140735072558240] [info] ajp_service::jk_ajp_common.c (2543): (ajp13) sending request to tomcat failed (recoverable), (attempt=2)

[Thu Jun 09 13:40:27 2011] [53894:140735072558240] [error] ajp_service::jk_ajp_common.c (2562): (ajp13) connecting to tomcat failed.

[Thu Jun 09 13:40:27 2011] [53894:140735072558240] [info] jk_handler::mod_jk.c (2627): Service error=0 for worker=ajp13

After googling and hit a try, I came to know that the port mentioned in worker.properties and AJP connector settings in server.xml should be same. After chaning the port in worker.properties to 8009 (default in AJP/1.3 connector), things were working like charm.