[Linux for Newbies]

Linux for Newbies, Part 15:
Shell Concepts

by Gene Wilburn

(The Computer Paper, Oct 2000. Copyright © Wilburn Communications Ltd. All rights reserved)

This article is translated into the Serbo-Croatian language by WHG Team

The Linux toolkit, with its rich collection of GNU and Unix utilities, provides a rewarding computing environment for programmers and end users alike. The glue that binds these utilities together is the shell, or command-line interface (CLI). Although X programs and graphical environments such as KDE and GNOME are now a common, and welcome, part of Linux, you may notice that Unix veterans always keep a few xterm windows open so they can access the command line. The Unix shell is an integral part of Unix (hence Linux) philosophy. There are literally thirty years' worth of sophisticated, refined utilities awaiting your command.

Something that will quickly propel you from newbie to intermediate status is learning how to harness these utilities, incorporating them into shell scripts. But before you can write scripts, you need some concepts of what the Linux shell is and what it can do. Let's begin with a brief overview of Unix shells.

The Shell Game

The shell is an essential layer of the Unix operating system, providing an interpreter that reads input from the command line; sets up pipes, redirection and background processing; and sets up programs for execution. If you've used the command-line interpreter of DOS or Windows, you already have some idea of what a shell does. Think of the Unix shell as DOS on steroids.

Unix offers several shells to choose from. The earliest and best-known Unix shell, written by Stephen Bourne for V7 of AT&T Unix, is called the Bourne shell (sh). To this day, the Bourne shell retains its role as the common denominator of most shell scripting. At the University of California at Berkeley, Bill Joy, the author of vi, created the C shell (csh) as part of 2BSD Unix. The C shell introduced many innovations, including command-line history and a programming environment similar to the C programming language. Unfortunately, the syntax of C shell is not compatible with Bourne shell syntax. Even so, the C shell has remained popular for years and is readily available for Linux systems--tcsh (an enhanced but totally compatible csh) being the most widely distributed version.

Next came the Korn Shell (ksh), created by David Korn at AT&T Labs in the mid-1980's. The Korn Shell maintained a high level of Bourne Shell compatibility while adding some slick new features and a nicer command-line history than the C Shell. The Korn Shell is still widely used in commercial Unix systems such as IBM's AIX, but, until recently, licensing restrictions have prevented the Korn shell from being widely deployed in the open-source community. There is a public-domain version of the Korn shell called pdksh available for Linux systems.

The default shell of Linux is the Free Software Foundation's GNU Bash (Bourne-Again SHell). As the punny name suggests, Bash evolved from the Bourne Shell. It incorporates many of the best features of C and Korn, as well as introducing a number of its own enhancements. Bash is highly Bourne-shell and Korn-shell compatible, meaning that if you pick up an older Unix book on shell scripting, most of the examples will work without change.

Some Shell Basics

When you first log in to a Linux system, the OS automatically starts up the default shell for your system, which in turn initializes a number of shell variables and file descriptors. The shell variables include PATH, HOME, HOSTTYPE, OSTYPE, PS1, PS2, and several others. If you would like to see which variables are initialized in your session, type "set | more" at the command line.

The standard command-line prompt in Linux for a normal user account using Bourne, Bash or Korn shell is the dollar sign ($) while the root prompt is a hash mark (#). By contrast the traditional C shell prompt is a percent sign (%). A quick glance at the command line prompt will usually let you know if you're issuing a command as a normal user or as root. The command-line prompt for Red Hat systems includes additional information, as seen in this prompt on my Red Hat Linux 6.2 portable:

This prompt has three Bash-shell variables set: user name ("genew"); host name ("755cx"); and current working directory ("newbies") minus the full path. This format is configured system wide in the file /etc/bashrc:

In the primary command-line prompt string (PS1), the \u indicates user, \h is host and \W is current directory. If you prefer a different command-line prompt, you can change your individual profile by adjusting .bashrc in your home directory or adjusting the /etc/bashrc file to set the string globally for all users. Some system admininstrators prefer to include the entire path, which you can achieve by changing \W to \w (lower case). I personally find that the full pathname gets tiresome once you get deep into a file system so I prefer the Red Hat default. The actual default for Bash without any tweaking is:

Which produces the very spare:

Some distributions of Linux give you only the default Bash prompt string, assuming you will adjust it to taste. That's fine if you already know how, but having an intelligently configured prompt, as per Red Hat, is a nice touch for beginning users. You'll find other metacharacters that can be used for the command-line prompt by typing "man bash." Remember the refrain: "man is your friend." To see which prompting symbols are available in Bash, type "/PROMPTING" while reading the man page and it will lead you to a list of all the metacharacters. Here is a partial listing:

You can use these metacharacters in your prompt string to customize the command-line prompt to your liking.

Shell Features

The Bash shell incorporates a number of features that you can use on an ad-hoc basis from the command line, or write into a script if the commands are something you want to repeat periodically. Let's review a few of the common features:

Redirection. When you log in to Linux, the shell sets up three file descriptors and assigns numbers to them:

  1. stdin (standard in). Your keyboard: file descriptor 0
  2. stdout (standard out). Your display: file descriptor 1
  3. stderr (standard error). Your display: file descriptor 2

You can redirect these I/O (input/output) streams with the following operators: >; >>; and <. The > operator will redirect into a file, overwriting any file of the same name. The >> operator appends to an existing file, if the file exists, or creates a new file if not. The < operator redirects input from a file to some process, rather than using keyboard input.

Here's an example: you want to know where all the .txt files are on your system. You can start out looking for them with the find command by typing:

at the command line. You may want to save the results into a file that you examine at leisure. To do this you simply redirect the above, which will normally go to stdout (your terminal) into a file instead, e.g.:

The Bash shell in this example provides us with two special features: redirection into a file called textfiles and "tilde expansion", a feature that uses the tilde character, "~", as shorthand meaning "my home directory." Fully expanded, on my system, the file will be written to /home/genew/tmp/textfiles.

Perhaps after examining the results you wish it had also included .doc files. To add these to the file, you can use the append operator:

In the course of running this command, especially as a non-root user, you probably received a bunch of error lines. These errors occur when find attempts to read files in directories for which you have no access privilege, e.g.:

We can easily bypass this annoyance. Remember that stderr is set as file descriptor 2 by the shell. Although it is initially assigned to your display screen, it can be redirected elsewhere. One of the time-honoured places to redirect stuff you're not interested in is /dev/null, the old "bit bucket in the sky." To send something to /dev/null is to send it to oblivion. Here's how to see all the .txt files on your system without any distracting error messages:

Notice the "2>" construct. It redirects all the stuff for filehandle 2 (standard error) to /dev/null.

Pipelines. Another major feature the shell provides is pipelining using the "|" pipe, or vertical bar, character. A pipeline spawns a new process that takes the output of the previous process (stdout) as the input to a new process (stdin). Programs or utilities that can be plugged into a pipeline are frequently called filter programs--most classic Unix utilities are written specifically to be used as filter programs.

One of the things that makes Unix (hence Linux) so attractive is that the possibilities for combining pipelines, filters and redirection are limited only by your imagination. By using pre-written, time-tested utilities, you can create custom programs that take very little time to assemble, compared to writing programs from scratch in C or Java. The shell provides conditional logic, branching, and looping structures for creating sophisticated custom scripts.

Shell Examples

Let's take a cursory look at how tools can be combined. In the following example we extract user names and their home directories from the /etc/password file, then sort them alphabetically and store them in a file called users:

The sequence here is that cat pipes the contents of the password file to awk which parses the file and pipes field 1 (login name) and field 6 (home directory) separated by a tab ("\t"), to sort which alphabetizes the list then redirects it to a file. The backslash "\" at the end of a line of Bash prompts allows you to carry commands over several lines for easier readability. The awk program, a predecessor to Perl, is very adept at parsing fields from a line of text.

Let's close with a simple shell script. Many of you have your Linux system connected to the Internet 24 hrs/day via a high-speed cable or DSL modem connection. Here's a script that will adjust your system clock to an atomic time standard every day at 2am, then adjust your system's hardware clock to match the updated system clock. Log in as root and using vi or your favourite editor, create a file in /root called timesync with the following lines:

Make the file executable by typing "chmod +x timesync". The Linux utility for running scheduled programs is called cron and it is accessed with the crontab (cron table) command. Type "crontab -e" to add the following lines to crontab (using vi commands):

Your computer will now have its system clock adjusted to an atomic time standard once every 24 hours at 2am. Forget the TV--you can set your wristwatch by your Linux system.

Next month we'll take a further look at some shell scripting examples.

Gene Wilburn (gene@wilburn.ca) is a Toronto-based IT specialist, musician and writer who operates a small farm of Linux servers.