Shell Utilities

basename
Strips the path information from a file name, printing only the file name. The construction basename $0 lets the script know its name, that is, the name it was invoked by. This can be used for "usage" messages if, for example a script is called with missing arguments:
echo "Usage: `basename $0` arg1 arg2 ... argn"

split, csplit
These are utilities for splitting a file into smaller chunks. Their usual use is for splitting up large files in order to back them up on floppies or preparatory to e-mailing or uploading them.

The csplit command splits a file according to context, the split occuring where patterns are matched.

sum, cksum, md5sum, sha1sum
These are utilities for generating checksums. A checksum is a number mathematically calculated from the contents of a file, for the purpose of checking its integrity. A script might refer to a list of checksums for security purposes, such as ensuring that the contents of key system files have not been altered or corrupted. For security applications, use the md5sum (message digest 5 checksum) command, or better yet, the newer sha1sum (Secure Hash Algorithm).

NOTE :
There have been reports that the 128-bit md5sum can be cracked, so the more secure 160-bit sha1sum is a welcome new addition to the checksum toolkit.

uuencode
This utility encodes binary files (images, sound files, compressed files, etc.) into ASCII characters, making them suitable for transmission in the body of an e-mail message or in a newsgroup posting.
This is especially useful where MIME (multimedia) encoding is not available.

uudecode
This reverses the encoding, decoding uuencoded files back into the original binaries.

crypt
At one time, this was the standard UNIX file encryption utility.

mktemp
Create a temporary file with a "unique" filename. When invoked from the command line without additional arguments, it creates a zero-length file in the /tmp directory.
bash$ mktemp
/tmp/tmp.zzsvql3154

PREFIX=filename
tempfile=`mktemp $PREFIX.XXXXXX`

make
Utility for building and compiling binary packages. This can also be used for any set of operations that is triggered by incremental changes in source files.

The make command checks a Makefile, a list of file dependencies and operations to be carried out.

install
Special purpose file copying command, similar to cp, but capable of setting permissions and attributes of the copied files.


finger
Retrieve information about users on a network. Optionally, this command can display a user's ~/.plan, ~/.project, and ~/.forward files, if present.

run-parts
The run-parts command executes all the scripts in a target directory, sequentially in ASCII-sorted filename order. Of course, the scripts need to have execute permission.
The cron daemon invokes run-parts to run the scripts in the /etc/cron.* directories.

Anacron
Anacron can be used to execute commands periodically, with a frequency specified in days. Unlike cron(8), it does not assume that the machine is running continuously. Hence, it can be used on machines that aren’t running 24 hours a day, to control daily, weekly, and monthly jobs that are usually controlled by cron.

When executed, Anacron reads a list of jobs from a configuration file, normally /etc/anacrontab (see anacrontab(5)). This file contains the list of jobs that Anacron controls. Each job entry specifies a period in days, a delay in minutes, a unique job identifier, and a shell command.

For each job, Anacron checks whether this job has been executed in the last n days, where n is the period specified for that job. If not, Anacron runs the job’s shell command, after waiting for the number of minutes specified as the delay parameter.

After the command exits, Anacron records the date in a special timestamp file for that job, so it can know when to execute it again. Only the date is used for the time calculations. The hour is not used.

When there are no more jobs to be run, Anacron exits.

yes
In its default behavior the yes command feeds a continuous string of the character y followed by a line feed to stdout. A control-c terminates the run. A different output string may be specified, as in yes different string, which would continually output different string to stdout.

tee
This is a redirection operator, but with a difference. Like the plumber's tee, it permits "siponing off" to a file the output of a command or commands within a pipe, but without affecting the result. This is useful for printing an ongoing process to a file or paper, perhaps to keep track of it for debugging purposes.

dd
This is the somewhat obscure and much feared data duplicator command. Originally a utility for exchanging data on magnetic tapes.
Some basic options to dd are:
◊ if=INFILE INFILE is the source file.
◊ of=OUTFILE OUTFILE is target file, that will have the data written to it.
◊ bs=BLOCKSIZE
◊ count=BLOCKS Copy only this many blocks of data.

Ex.
file_subscript=copy
dd if=$0 of=$0.$file_subscript 2>/dev/null

mcookie
This command generates a "magic cookie," a 128-bit (32-character) pseudorandom hexadecimal number, normally used as an authorization "signature" by the X server. This also available for use in a script as a "quick 'n dirty" random number.
random000=$(mcookie)
Of course, a script could use md5 for the same purpose. The mcookie command gives yet another way to generate a "unique" filename.

Shell Complex Commands - Part II

grep -
To force grep to show the filename when searching only one target file, simply give /dev/null as the second file.
grep Linux osinfo.txt /dev/null

egrep -- extended grep --
Is the same as grep -E. This uses a somewhat different, extended set of Regular Expressions, which can make the search a bit more flexible. It also allows the boolean | (or) operator.
For example,
bash $ egrep 'matches|Matches' file.txt
Line 1 matches.
Line 3 Matches.
Line 4 contains matches, but also Matches

look
The command look works like grep, but does a lookup on a "dictionary," a sorted word list. By default, look searches for a match in /usr/dict/words, but a different dictionary file may be specified.
Example,

#!/bin/bash
# lookup: Does a dictionary lookup on each word in a data file.
file=words.data # Data file from which to read words to test.
echo
while [ "$word" != end ] # Last word in data file.
do # ^^^
read word # From data file, because of redirection at end of loop.
look $word > /dev/null # Don't want to display lines in dictionary file.
lookup=$? # Exit status of 'look' command.
if [ "$lookup" -eq 0 ]
then
echo "\"$word\" is valid."
else
echo "\"$word\" is invalid."
fi
done <"$file" # Redirects stdin to $file, so "reads" come from there.
echo

Shell Complex Commands - Part I

find -

-exec COMMAND \;

Carries out COMMAND on each file that find matches. The command sequence terminates with ; (the ";" is escaped to make certain the shell passes it to find literally, without interpreting it as a special character).

If COMMAND contains {}, then find substitutes the full path name of the selected file for "{}".

- find ~/ -name 'core*' -exec rm {} \;
# Removes all core dump files from user's home directory.

find ~/ -maxdepth 1 -name '*.sh' -exec ls -al {} \;
-rw-r--r-- 1 testuser dummy 1168 2008-07-24 21:08 /home/testuser/log_cleanup.sh
-rw-r--r-- 1 testuser dummy 17 2008-07-30 18:45 /home/testuser/test1.sh

xargs
A filter for feeding arguments to a command, and also a tool for assembling the commands themselves. It breaks a data stream into small enough chunks for filters and commands to process. Consider it as a powerful replacement for backquotes. In situations where command substitution fails with a too many arguments error, substituting xargs often works.

Normally, xargs reads from stdin or from a pipe, but it can also be given the output of a file.

The default command for xargs is echo. This means that input piped to xargs may have linefeeds and other whitespace characters stripped out.

every file in current directory, one at a time, prompting before each operation.
ls | xargs -p -l gzip gzips

Grep for linux in mails. find ~/mail/ -type f | xargs grep "Linux

Copying files in current directory to another,

#!/bin/bash
# copydir.sh
# Copy (verbose) all files in current directory ($PWD) to directory specified on command line.

E_NOARGS=65
if [ -z "$1" ] # Exit if no argument given.
then
echo "Usage: `basename $0` directory-to-copy-to"
exit $E_NOARGS
fi
ls . | xargs -i -t cp ./{} $1
# ^^ ^^ ^^
# -t is "verbose" (output command line to stderr) option.
# -i is "replace strings" option.
# {} is a placeholder for output text.
# This is similar to the use of a curly bracket pair in "find."
#
# List the files in current directory (ls .),
#+ pass the output of "ls" as arguments to "xargs" (-i -t options),
#+ then copy (cp) these arguments ({}) to new directory ($1).
#
# The net result is the exact equivalent of
#+ cp * $1
#+ unless any of the filenames has embedded "whitespace" characters.

Xargs EXIT STATUS
xargs exits with the following status:
0 if it succeeds
123 if any invocation of the command exited with status 1-125
124 if the command exited with status 255
125 if the command is killed by a signal
126 if the command cannot be run
127 if the command is not found
1 if some other error occurred.

find /tmp -name core -type f -print | xargs /bin/rm -f
Find files named core in or below the directory /tmp and delete them. Note that this will work incorrectly if there are any filenames containing newlines or spaces.

find /tmp -name core -type f -print0 | xargs -0 /bin/rm -f
Find files named core in or below the directory /tmp and delete them, processing filenames in such a way that file or directory names containing spaces or newlines are correctly handled.

ps aux | grep "/sbin/klogd" | grep -v grep | awk '{ print $2}' | xargs -I kill {}
This will kill process klogd. Useful when you want to kill process by there name.

Shell Commands

source, . (dot command)

         This command, when invoked from the command line, executes a script. Within a script, a source file-name loads the file file-name. Sourcing a file (dot-command) imports code into the script, appending to the script (same effect as the #include directive in a C program).

Ex. cat data-file

variable=11
variable1=111
variable2=222

# cat source.sh 
#!/bin/bash
# import from data-file.
. data-file
echo "$variable"
echo "$variable1"

exec
  This shell builtin replaces the current process with a specified command. Normally when the shell encounters a command, it forks off a child process to actually execute the command. Using the exec builtin, the shell does not fork, and the command exec'ed replaces the shell. When used in a script, therefore, it forces an exit from the script when the exec'ed command terminates. 
Ex. 

#!/bin/bash
exec echo "Exiting \"$0\"." # Exit from script here.
# ----------------------------------
# The following lines never execute.
echo "This echo will never echo."
exit 99

wait
  Suspend script execution until all jobs running in background have terminated, or until the job number or process ID specified as an option terminates. Returns the exit status of waited-for command.

You may use the wait command to prevent a script from exiting before a background job finishes executing (this would create a dreaded orphan process).