Scripting

Shell Scripting from Scratch

In this series, we are going to learn Linux Shell Scripting. Shell scripting is a very important skill that Unix/Linux administrators can never go without. Shell Scripting means writing scripts (programs) for the UNIX/Linux Shell.

Shell Scripting- Automation Laboratories

So, let me tell you what Linux Shell is.

What is Shell?

A shell in a Linux operating system takes input from you in the form of commands, processes it, and then gives an output. It is the interface through which a user works on the programs, commands, and scripts. A shell is accessed by a terminal which runs it.

What are shell scripts?

A Shell Script is a collection of Linux commands to be executed in sequence. The commands are written and stored in an ordinary text file. When the script is to be executed, the shell reads the file contents, interprets and executes them as if they were typed on the shell command prompt.

Why Shell Scripts?

As being set of Linux commands, a shell script can be used to achieve one or more tasks. For example, a script could be written to start a database, shutdown a database, start backup, check whether a specific service is up, and if not, it could start it. So many other tasks can be done using shell scripts. In general, when a specific task is frequently needed, it is wise to write a script that can do it for you. This is much better than having to remember, and type the commands every time the task is needed. Writing shell scripts half-automates the administration tasks, saves the administrator’s time, and minimizes typos effects.

Shell scripts combined with cron jobs:

Crontab job is a Unix/Linux tool that helps system administrators schedule jobs to run at specific times on periodic basis.

Together, crontab jobs and shell scripts can fully-automate many administration and monitoring tasks. For instance, the weekly backup that run every Saturday at midnight, the purging of old log files that takes place on the first of every month, and the system monitoring utility that checks the CPU, memory, file systems, paging space utilizations every 10 minutes, and notifies the admin by email, all considered typical usages for the shell scripts combined with crontab job automation.

What Types of Shell are Available?

  • The Bourne Shell, which is the default shell in many UNIX operating systems.
  • The Bourne Again Shell (Bash). This is the de facto standard shell for almost all Linux Distributions, like Red Hat, CentOS, Fedora, Ubuntu, etc.
  • The Korn Shell (ksh).

In this series, we will use the Bash shell.

Preparing your tools:

Before writing shell scripts (which are collections of Linux commands), we must be first aware of how to use the Linux shell, and how to type and execute commands in and the most important is to have rich knowledge of Linux commands.

Basics of Linux:

Linux Directory Structure Diagram: 

  • /bin : All the executable binary programs (file) required during booting, repairing, files required to run into single-user-mode, and other important, basic commands viz., catdudftarrpmwc, history, etc.
  • /boot : Holds important files during boot-up process, including Linux Kernel.
  • /dev : Contains device files for all the hardware devices on the machine e.g., cdrom, cpu, etc
  • /etc : Contains Application’s configuration files, startup, shutdown, start, stop script for every individual program.
  • /home : Home directory of the users. Every time a new user is created, a directory in the name of user is created within home directory which contains other directories like Desktop, Downloads, Documents, etc.
  • /lib : The Lib directory contains kernel modules and shared library images required to boot the system and run commands in root file system.
  • /lost+found : This Directory is installed during installation of Linux, useful for recovering files which may be broken due to unexpected shut-down.
  • /media : Temporary mount directory is created for removable devices viz., media/cdrom.
  • /mnt : Temporary mount directory for mounting file system.
  • /opt : Optional is abbreviated as opt. Contains third party application software. Viz., Java, etc.
  • /proc : A virtual and pseudo file-system which contains information about running process with a particularProcess-id aka pid.
  • /root : This is the home directory of root user and should never be confused with ‘/‘
  • /run : This directory is the only clean solution for early-runtime-dir problem.
  • /sbin : Contains binary executable programs, required by System Administrator, for Maintenance. Viz.,iptablesfdiskifconfig, swapon, reboot, etc.
  • /srv : Service is abbreviated as ‘srv‘. This directory contains server specific and service related files.
  • /sys : Modern Linux distributions include a /sys directory as a virtual filesystem, which stores and allows modification of the devices connected to the system.
  • /tmp :System’s Temporary Directory, Accessible by users and root. Stores temporary files for user andsystem, till next boot.
  • /usr : Contains executable binaries, documentation, source code, libraries for second level program.
  • /var : Stands for variable. The contents of this file are expected to grow. This directory contains log, lock,spool, mail and temp files.

 Exploring Important file, their location and their Usability:

Linux is a complex system which requires a more complex and efficient way to start, stop, maintain and reboota system unlike Windows. There is a well-defined configuration files, binaries, man pages, info files, etc. for every process in Linux. 

  • /boot/vmlinuz : The Linux Kernel file.
  • /dev/hda : Device file for the first IDE HDD (Hard Disk Drive)
  • /dev/hdc : Device file for the IDE Cdrom, commonly
  • /dev/null : A pseudo device, that don’t exist. Sometime garbage output is redirected to /dev/null, so that it gets lost, forever.
  • /etc/bashrc : Contains system defaults and aliases used by bash shell.
  • /etc/crontab : A shell scriptto run specified commands on a predefined time Interval.
  • /etc/exports : Information of the file system available on network.
  • /etc/fstab : Information of Disk Drive and their mount point.
  • /etc/group : Information of Security Group.
  • /etc/grub.conf : grub bootloader configuration file.
  • /etc/init.d : Service startup Script.
  • /etc/lilo.conf : lilo bootloader configuration file.
  • /etc/hosts : Information of Ip addresses and corresponding host names.
  • /etc/hosts.allow : List of hosts allowed to access services on the local machine.
  • /etc/host.deny : List of hosts denied to access services on the local machine.
  • /etc/inittab : INIT process and their interaction at various run level.
  • /etc/issue : Allows to edit the pre-login message.
  • /etc/modules.conf : Configuration files for system modules.
  • /etc/motd : motdstands for Message Of The Day, The Message users gets upon login.
  • /etc/mtab : Currently mounted blocks information.
  • /etc/passwd : Contains password of system users in a shadow file, a security implementation.
  • /etc/printcap : Printer Information
  • /etc/profile : Bash shell defaults
  • /etc/profile.d : Application script, executed after login.
  • /etc/rc.d : Information about run level specific script.
  • /etc/rc.d/init.d : Run Level Initialisation Script.
  • /etc/resolv.conf : Domain Name Servers (DNS) being used by System.
  • /etc/securetty : Terminal List, where root login is possible.
  • /etc/skel : Script that populates new user home directory.
  • /etc/termcap : An ASCII file that defines the behaviour of Terminal, console and printers.
  • /etc/X11 : Configuration files of X-window System.
  • /usr/bin : Normal user executable commands.
  • /usr/bin/X11 : Binaries of X windows System.
  • /usr/include : Contains include files used by ‘c‘ program.
  • /usr/share : Shared directories of man files, info files, etc.
  • /usr/lib : Library files which are required during program compilation.
  • /usr/sbin : Commands for Super User, for System Administration.
  • /proc/cpuinfo : CPU Information
  • /proc/filesystems : File-system Information being used currently.
  • /proc/interrupts : Information about the current interrupts being utilised currently.
  • /proc/ioports : Contains all the Input/Output addresses used by devices on the server.
  • /proc/meminfo : Memory Usages Information.
  • /proc/modules : Currently using kernel module.
  • /proc/mount : Mounted File-system Information.
  • /proc/stat : Detailed Statistics of the current System.
  • /proc/swaps : Swap File Information.
  • /version : Linux Version Information.
  • /var/log/lastlog : log of last boot process.
  • /var/log/messages : log of messages produced by syslog daemon at boot.
  • /var/log/wtmp : list login time and duration of each user on the system currently.

Basic Commands:

cdChange the current directory. With no arguments “cd” changes to the users home directory.
chmodchmod<specification><filename> – Effect: Change the file permissions.
Ex: chmod 751 myfileEffect: change the file permission to rwx for owner, re for group
Ex: chmod go=+r myfileEffect: Add read permission for the owner and the group
character meanings u-user, g-group, o-other, + add permission, – remove, r-read, w-write,x-exe
Ex: chmod a +rwxmyfileEffect: Allow all users to read, write or execute myfile
Ex: chmod go -r myfileEffect: Remove read permission from the group and others
chmod +s myfile – Setuid bit on the file which allows the program to run with user or group privileges of the file.
chmod {a,u,g,o}{+,-}{r,w,x} (filenames) – The syntax of the chmod command.
chownchown<owner1><filename> Effect: Change ownership of a file to owner1.
“chown -R owner:owner /filename”
cpcp<source><destination> Copy a file from one location to another.
findEx: find $Home –name readme Print search for readme starting at home and output full path.
How to find files quickly using the find command:
Ex: find ~ -name report3 –print
“~” = Search starting at the home directory and proceed through all its subdirectories
“-name report3” = Search for a file named report3
“-print” = Output the full path to that file
lnMake links between files.
“ln -sf /usr/jdk/instances/jdk1.7.0_25/bin/java /usr/bin/”
locateFile locating program that uses the slocate database.
“locate filename”
lsList files. Option -a, lists all, see man page “man ls”
Ex: “ls Docum Projects/Linux” – The contents of the directories Docum and Projects/Linux are listed.
To list the contents of every subdirectory using the ls command:
1. Change to your home directory.
2. Type: ls -R
mkdirMake a directory.
“mkdir /name”
mvMove or rename a file. Syntax: mv <source><destination> Ex: mv filename directoryname/newfilename
pwdPrint or list the working directory with full path (present working directory).
rmEx: “rm .*” – Effect: Delete system files (Remove files) –i is interactive option.
rmdirrmdir<directory> – Remove a directory. The directory must be empty.
stat(1u)Used to print out inode information on a file.
touchChange file timestamps to the current time. Make the file if it doesn’t exist.
whatisSearch the whatis database for complete words.
wherisLocate the binary, source and man page files for a command.
whichShow full path of commands where given commands reside.

 

edEditor
emacsFull screen editor.
gitviewA hexadecimal or ASC file viewer.
headhead linuxdoc.txt – Look at the first 10 lines of linuxdoc.txt.
jedEditor
joeEditor
lessq-mandatory to exit, Used to view files.
moreb-back q-quit h-help, Used to view files.
picoSimple text editor.
tailtail linuxdoc.txt – Look at the last 10 lines of linuxdoc.txt.
viEditor with a command mode and text mode. Starts in command mode.

File compression, backing up and restoring:

arCreate modify and extract from archives.
bunzip2Newer file decompression program.
bzcatDecompress files to stdout.
bzip2Newer file compression program.
bzip2recoverRecovers data from damaged bzip2 files.
compressCompress data.
cpioCan store files on tapes. to/from archives.
dumpReads the filesystem directly.
gunzipunzip <file> – unzip a gz file.
gzexeCompress executable files in place.
gzipgzip<file> – zip a file to a gz file.
mtControl magnetic tape drive operation.
tarCan store files on tapes.
Usage: tar cvf<destination><files/directories> – Archive copy groups of files
Ex: tar /dev/fdo temp Effect: Copy temp to drive A:
uncompressExpand data.
unzipunzip <file> – unzip a zip file. Files ending in “.gz” or “.zip” are compressed.
zcatUsed to restore compressed files.
zcmpCompare compressed files.
zdiffCompare compressed files.
zforceForce a .gz extension on all gzip files.
zgrepSearch possibly compressed files for a regular expression.
zmoreFile filter for crt viewing of compressed text.
znewRecompress .z files to .gz files.
zipzip <file> – make a zip file.

Extra control and piping for files and other outputs:

basenameStrip directory and suffix information from filenames.
catEx: cat < filename — Effect: put keyboard input into the file. CTRL-D to exit (end).
cmpCompare two files.
colrmRemove columns from a file.
columnColumnate lists.
commEx: comm file1 file2 — Effect compare the contents of file1 and file2 produces 3 columns of output. Lines in the first file, lines in second file, lines in both files.
csplitSplit a file into sections determined by context lines.
cutRemove sections from each line of files.
diffShow the differences between files. Ex: diff file1 file2
diff3Find differences between 3 files.
dirnameStrip the non-directory suffix from a filename.
echoDisplay a line of text.
egrepSimilar to grep -E, compatible with UNIX egrep.
expandConvert tabs to spaces.
exprEvaluate expressions.
FALSEDo nothing. Exit with a status indicating failure.
fgrepSame as grep -F.
foldWrap each input line to fit in specified width.
joinJoin lines of two files in a common field.
grepgrep pattern filename.
Ex: grep ” R ” — Effect: Search for R with a space on each side
Ex: ls –a |grep R — Effect: List all files with an R in them or their info listing.
hexdumpasc, decimal, hex, octal dump.
lognamePrint user’s login name.
lookDisplay lines beginning with a given string.
mkfifoCreate named pipes with the given names.
nlWrite each file to standard output with line numbers added.
odDump files in octal and other formats.
patchApply a diff file to an original.
pasteCombines from 2 or more files. Ex: paste file1 file 2
printfPrint and format data.
revReverses lines in a file.
scriptMake a typescript of a terminal session.
sdiffFind differences between 2 files and merge interactively.
sedA stream editor. Used to perform transformations on an input stream.
sleepDelay for a specified amount ot time.
sortSort a file alphabetically.
splitSplit a file into pieces.
stringsPrint the strings of printable characters in files.
tacConcatenate and print files in reverse.
teeRead from standard input and write to standard output and files.
trTranslate or delete characters.
uniqRemove duplicate lines from a sorted file.
wcCount lines, words, characters in a file. Ex: wc filename.

Bash Scripts:

Bash scripts are primarily written as a text file of commands that can be executed. They can end in the .sh extension or no extension at all. They can be executed with the Bash command preceding-it— bashmyscript.sh — or by having the file mode set as executable and placing the path to Bash’s executable at the beginning as such:

Let us understand the steps in creating a Shell Script

  • Create a file using a vi editor(or any other editor).  Name  script file with extension .sh
  • Start the script with #! /bin/sh
  • Write some code.
  • Save the script file as sh
  • For executing the script type bash sh

“#!” is an operator called shebang which directs the script to the interpreter location. So, if we use“#! /bin/sh” the script gets directed to the bourne-shell

Let’s create a small script for better understanding:

#!/bin/bash

Then you can execute that script by giving the path and file name ./myscript.sh

In Bash, comments are created with the pound symbol. For the sake of this post, I will be writing # Output: inline in cases where it will be more helpful.

value="Hello World!"
echo value      ### Output: value
echo $value     ### Output: Hello World!
echo ${value}  ### Output: Hello World!
echo "value"    ### Output: value
echo "$value"   ### Output: Hello World!
echo "${value}" ### Output: Hello World!

 As you can see above, even though the variable value was passed, it wasn’t interpreted as a variable unless we preceded it with the dollar sign. Also notice that there are no spaces in value=”Hello World!”. This is very important in Bash as many things won’t work if the spaces aren’t as they need to be.

Conditional checks in Bash:

 value=0
 if [ $value == 0 ]; then
  echo "Zero"
else
  echo "Not Zero"
fi
#### Output: Zero

 

The if syntax is a bit strange but not too difficult to learn. Note the space inside of each outer edge square bracket for the if condition. This is important or the script won’t work. In Bash, the semicolon is the equivalent of a new-line for your code. You could also place then on the next line instead of using a semicolon.

For a second condition statement, you can use elif for else if. You will still need to follow the same ; then pattern for that.

Writing Functions in Bash is very simple:

cow() {
echo Moo
}
cow   ###### Output: Moo

 In bash, parameters passed from the console or to a function take a dollar number format, where the number indicates which position of the parameter it is.

cow_eat() {
  echo Cow chews $1
}
 cow_eat grass  ######## Output: Cow chews grass

 Bash loads a .bashrc file from your home directory for all its defaults. You can place as many functions as you want in there and they will be available on the command line at any time as if they’re a program on your system.

What are shell variables?

Variables store data in the form of characters and numbers. Similarly, Shell variables are used to store information and they can by the shell only

For example, the following creates a shell variable and then prints it:

variable ="Hello"
echo $variable
#!/bin/sh
echo "what is your name?"
read name
echo "How do you do, $name?"
read remark
echo "I am $remark too!"

get prepared for more stuffs like this in the next blog………..

 
Divya Gupta on EmailDivya Gupta on Facebook
Divya Gupta

Leave a Reply

Your email address will not be published. Required fields are marked *