Part Workbook 4.

System
Initialization and Services
 Table of Contents
1. Boot Sequence Overview ................................................................................................ 4
Discussion .............................................................................................................. 4
Introduction ..................................................................................................... 4
BIOS ............................................................................................................. 4
POST System Check ................................................................................ 4
Device Initialization .................................................................................. 5
Boot Device Selection ............................................................................... 5
Handing Over the Reigns ........................................................................... 5
Questions ................................................................................................................ 5
2. The GRUB Bootloader ................................................................................................... 8
Discussion .............................................................................................................. 8
What a Bootloader Does .................................................................................... 8
What is an Initial Ramdisk? ....................................................................... 8
The GRUB Bootloader .............................................................................. 9
Bootloader Design ............................................................................................ 9
Stage 1 ................................................................................................... 9
Stage 2 ................................................................................................... 9
GRUB Design ......................................................................................... 9
GRUB Configuration ........................................................................................ 9
How GRUB refers to files ........................................................................ 10
GRUB Global Configuration ..................................................................... 12
GRUB Menuitem Configuration ................................................................ 12
Using GRUB ................................................................................................. 13
Secure Mode ......................................................................................... 13
Basic Mode ........................................................................................... 13
Edit Mode ............................................................................................. 14
Append Mode ........................................................................................ 14
Command Mode ..................................................................................... 14
Installing Grub ............................................................................................... 14
Managing GRUB Passwords ............................................................................. 15
GRUB BIOS Related Limitations ...................................................................... 15
The Linux Kernel ........................................................................................... 16
The Kernel Command Line: Boot Parameters ............................................... 16
Examples .............................................................................................................. 17
Referencing a New Kernel ............................................................................... 17
Migrating a GRUB Configuration File ................................................................ 18
Online Exercises .................................................................................................... 19
Specification .................................................................................................. 19
Deliverables ................................................................................................... 20
Questions .............................................................................................................. 20
3. /sbin/init and Run Levels ............................................................................................... 23
Discussion ............................................................................................................. 23
Upstart configuration files ................................................................................ 23
/sbin/init and Upstart ....................................................................................... 23
Runlevels ...................................................................................................... 24
Configuration for System Initialization and Runlevel Management ........................... 25
Configuration for External Events ...................................................................... 25
Gettys ........................................................................................................... 26
Managing Init ................................................................................................ 27
Observing Runlevels with runlevel ............................................................. 27
Changing Runlevels with init or telinit ........................................................ 27

rha130-6.1-0 2 Copyright 2011, Red Hat Inc.

 System Initialization and Services

Shutting Down the Machine with shutdown ................................................. 27

The /etc/rc.d/rc.sysinit Script ............................................................................. 28
Examples .............................................................................................................. 31
Running a getty on ttyS1 ................................................................................. 31
Customizing CTRL ALT DELETE for a Laptop ................................................... 32
Online Exercises .................................................................................................... 32
Specification .................................................................................................. 32
Deliverables ................................................................................................... 32
Cleaning Up .......................................................................................... 33
Questions .............................................................................................................. 33
4. Red Hat Enterprise Linux Service Scripts ......................................................................... 36
Discussion ............................................................................................................. 36
The /etc/rc.d Directory ..................................................................................... 36
The /etc/rc.d/init.d Directory and Service Scripts ................................................... 37
Services and the service Command .................................................................... 39
Service Runlevel Configuration and chkconfig ..................................................... 41
Local Startup Configuration .............................................................................. 43
Summary ...................................................................................................... 43
Online Exercises .................................................................................................... 44
Specification .................................................................................................. 44
Deliverables ................................................................................................... 44
Clean Up ...................................................................................................... 44
Questions .............................................................................................................. 44
5. Troubleshooting the Boot Sequence ................................................................................. 47
Discussion ............................................................................................................. 47
The Services Graphical Application ................................................................... 47
Managing Virtual Consoles .............................................................................. 47
Observing the Boot Sequence ........................................................................... 48
Modifying the Boot Sequence with the Kernel Command Line ................................ 50
Booting into Runlevel 1 ........................................................................... 51
Bypassing /sbin/init ................................................................................. 51
Online Exercises .................................................................................................... 52
Specification .................................................................................................. 52
Deliverables ................................................................................................... 53
Questions .............................................................................................................. 53

rha130-6.1-0 3 Copyright 2011, Red Hat Inc.

Chapter 1. Boot Sequence Overview
Key Concepts
• Upon booting, Red Hat Enterprise Linux Initialization occurs in four stages: BIOS, a bootloader, the
kernel, and /sbin/init.

• BIOS settings allow users to specify a sequence of devices which should be considered bootable.

• Once BIOS has selected a boot device, it transfers control to whatever executable is found in the device's
Master Boot Record.

Discussion
Introduction
From the time that the power is switched on, until users are able to log in to a fully functioning machine,
the system passes through four distinct stages of initialization.

1. BIOS: The minimal operating system which is located in the ROM of the system's motherboard.

2. Bootloader: The minimal application which is loaded from a disk's MBR, used to load and pass control
to the Linux Kernel.

3. Kernel: The Linux kernel initializes internal structures and device drives, provides an initial filesystem,
and starts off the first process.

4. /sbin/init: The /sbin/init process performs most of the hard work (relevant to system administration)
to initialize the machine.

This Workbook will discuss the various stages in detail. This lesson introduces the overall structure, and
relevant BIOS settings. The second lesson addresses the GRUB bootloader, and the role of the Linux
kernel at bootup. The last two lessons discuss the role of the initial process, /sbin/init, and Red Hat
Enterprise Linux service configuration.

BIOS
Every computer, when first powered on, starts a minimal operating system called BIOS (for Basic Input/
Output System). A system's BIOS is a permanent feature of its mother board. 1 Most BIOS's provide an
interactive utility, which can be started by pressing a particular key sequence during startup, such as the
DELETE key, or F10. The particular key sequence differs from system to system, and is displayed as
part of the startup process.

When started, a system's BIOS generally performs the following functions.

POST System Check

The BIOS performs an integrity check of the system, assuring that core components, such as a CPU and
memory, are present and functioning. The test is often referred to as the POST, or Power On Self Test.
1
Most modern BIOS's are implemented on an EPROM chip (Erasable Programmable Read Only Memory), and can be replaced, through a technique
often referred to as "flashing" a system's BIOS.

rha130-6.1-0 4 Copyright 2011, Red Hat Inc.

 Boot Sequence Overview

Device Initialization
A system's BIOS initializes core device drivers for the keyboard, video, serial ports, and others, assigning
appropriate system resources such as IRQ lines and I/O Ports. Modern BIOS's also implement the Plug n'
Play protocol, which is used to probe PCI devices for supported resource settings, and then assign each
PCI device a non-conflicting configuration.

Many operating systems can also perform similar Plug n' Play configuration, and most BIOS's provide a
configurable setting, such as Plug n' Play Aware OS? [Y/n], which can cause BIOS to defer
configuration until the operating system is loaded. Although Linux does provide Plug n' Play support, it
is generally suggested to let BIOS perform Plug n' Play configuration (for example, by answering "No"
to the above BIOS setting).

Another BIOS setting indicates whether or not the system (specifcally Intel CPUs) have the Virtualization
Technology enabled. Many systems ship with this turned off and the system administrator must change
the system setting to "enabled" before loading the hypervisor. Often a cold boot (power off completely
then power on) is required after changing this setting.

Boot Device Selection

After initializing devices, the BIOS searches for an appropriate boot device. The boot device is generally
selected from the possible options.

• Floppy Drive

• CD-ROM

• USB removable Hard Disk

• IDE or SCSI Hard Disk

• PXE compliant Network Card

A PXE (Preboot Execution Environment) compliant network card generally implements the DHCP (or
BOOTP) IP configuration protocol and the TFTP file transfer protocol natively, which allows the network
card to discover and download a kernel image from the network. Such a network card is the exception
instead of the rule, and PXE configuration is beyond the scope of our current discussion.

The choice of the appropriate boot device is configurable through BIOS's interactive applications. Usually,
BIOS allows a "pecking order" to be established, first looking for a bootable floppy, for example, and
if a bootable floppy is not present, looking for a bootable CD-ROM, and finally looking for a bootable
harddisk.

Handing Over the Reigns

Once the appropriate boot disk is selected, BIOS loads the disk's first block (512 bytes) into memory,
and passes execution to this region. The first block of any bootable disk must contain a small executable,
referred to as a bootloader. We continue with the topic of bootloaders in the next lesson.

Questions
1. Which of the following controls the first part of the boot sequence?

a. BIOS

rha130-6.1-0 5 Copyright 2011, Red Hat Inc.

 Boot Sequence Overview

b. the Linux kernel

c. /sbin/init

d. a bootloader

2. Which of the following controls the last part of the boot sequence?

a. the Linux kernel

b. a bootloader

c. BIOS

d. /sbin/init

3. BIOS usually passes control to which of the following?

a. the Linux kernel

b. The /sbin/init process

c. a bootloader

d. The /bin/login process

4. A bootloader usually passes control to which of the following?

a. the Linux kernel

b. BIOS

c. /sbin/init

d. /sbin/login

5. Which is the first process started by the Linux kernel?

a. /sbin/init

b. BIOS

c. A bootloader

d. /sbin/login

6. Which of the following is responsible for initially mounting the root partition?

a. the Linux kernel

b. BIOS

c. a bootloader

d. /sbin/init

7. Which of the following is responsible for determining the appropriate boot device?

a. a bootloader

rha130-6.1-0 6 Copyright 2011, Red Hat Inc.

 Boot Sequence Overview

b. BIOS

c. the Linux kernel

d. /sbin/init

8. Which of the following is usually found in a disk's Master Boot Record?

a. a bootloader

b. BIOS

c. the Linux kernel

d. the /sbin/init process

9. Which of the following is found in the ROM of a system's motherboard?

a. the Linux kernel

b. a bootloader

c. BIOS

d. the /sbin/init process

10. What is the name of the protocol used to assign non-conflicting resource configurations to devices?

a. Plug-'n-Pray

b. Plug-'n-Play

c. Plug-'n-Play-'n-Play-'n-Play

d. None of the above

rha130-6.1-0 7 Copyright 2011, Red Hat Inc.

Chapter 2. The GRUB Bootloader
Key Concepts

• In general, bootloaders in Linux have three responsibilities:

1. Compose an appropriate kernel command line (which references the correct root partition).

2. Load an appropriate initial ram disk (initrd).

3. Load and pass control to the appropriate Linux kernel.

• Bootloaders are usually designed in two stages, with the first stage small enough to reside in a Master
Boot Record.

• GRUB is the recommended bootloader for Red Hat Enterprise Linux.

• GRUB uses the /boot/grub/grub.conf configuration file.

• Upon startup, GRUB can be used to interactively edit its configuration and boot to the new specification.

• Upon startup, the Linux kernel mounts the root partition read-only, and starts the first process, usually
/sbin/init.

Discussion
What a Bootloader Does
Upon booting, BIOS passes control to a small executable referred to as a bootloader. Usually, the
bootloader performs the following three tasks.

1. Compose a kernel command line.

2. Load an initial ramdisk (initrd), if necessary.

3. Load and pass control to the Linux kernel

What is an Initial Ramdisk?

Sometimes, Red Hat chooses to implement portions of the kernel which are needed to bootstrap the
filesystem as kernel modules. This choice naturally leads to a problem. From where does the kernel load
its kernel modules, if they are needed in the boot process before the filesystem is created? The answer is an
initial ramdisk, which is a filesystem image which contains the necessary modules. The bootloader loads
the ramdisk image into memory for the kernel, from where the kernel can trivially mount the filesystem
and load any necessary modules.

The details of initial ramdisk images, more commonly called initrds, will be discussed in a later Workbook.
For now, just appreciate that in Red Hat Enterprise Linux, kernels almost always have associated initrd's,
traditionally the initrd's were conventionally called /boot/initrd-kernelversion.img, and it is
the bootloader's responsibility to load them. More recently the images are called /boot/initramfs-
kernelversion.img for initial ram filesystems. They are still loaded by the bootloader.

rha130-6.1-0 8 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

The GRUB Bootloader

Red Hat Enterprise Linux uses GRUB (the Grand Unified Bootloader) as its bootloader. The GRUB
bootloader is a GNU [http://www.gnu.org] sponsored project, and as the name implies, is designed to
operate with multiple operating systems (and filesystems).

Bootloader Design
As mentioned in the previous Lesson, BIOS generally passes control to a bootloader by loading and then
executing the first block of a disk. This forces bootloaders to generally implement a two stage design.

Stage 1
The first stage of a boot loader is generally small, designed to fit into a 512 byte Master Boot Record (in
fact, share the MBR with a partition table!) or the bootsector of a partition. 1 The sole task of the first
stage bootloader is to locate, load, and pass control to the second stage bootloader. Because the first stage
bootloader lives in the MBR or a bootsector, it is generally not an observable file in the filesystem.

Stage 2
The second stage of a bootloader is generally the bootloader proper, which runs some form of (often
interactive) application at startup, and is able to read configuration information about default settings. The
second stage bootloader is generally an identifiable binary file in the filesystem.

GRUB Design
The various components of the GRUB bootloader traditionally inhabit the /boot/grub directory.
[root@station root]# ls /boot/grub/
device.map grub.conf minix_stage1_5 stage2
e2fs_stage1_5 iso9660_stage1_5 reiserfs_stage1_5 ufs2_stage1_5
fat_stage1_5 jfs_stage1_5 splash.xpm.gz vstafs_stage1_5
ffs_stage1_5 menu.lst stage1 xfs_stage1_5

The stage two bootloader is fairly easy to identify: it's called stage2. The file stage1 is a copy of the
stage1 bootloader which is installed to a MBR or bootsector (notice its suspicious size in the following
transcript).
[root@station root]# ls -l /boot/grub/stage1
-rw-r--r-- 1 root root 512 Apr 1 2003 /boot/grub/stage1

The various files which end 1_5 are filesystem specific "stage one and a halves", which are used to locate
and read stage2, grub.conf, and other files from various filesystems.

Generally, the only "user serviceable" part of GRUB is its configuration file, /boot/grub/
grub.conf.

GRUB Configuration
GRUB uses an ASCII text configuration file, /boot/grub/grub.conf, which is usually hand edited
with a text editor. Because the file is read by the BIOS drivers, before the Linux kernel has loaded, it
resides in the /boot partition. Red Hat Enterprise Linux provides the symbolic link /etc/grub.conf
so the file can be edited from a more natural location.
1
Just as the first block of a disk is reserved as the Master Boot Record, most filesystems (and other partition structures such as swap space) reserve
the first block of a partition, referred to as the bootsector of the partition. (A notable exception is the FAT filesystem).

rha130-6.1-0 9 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

[root@station root]# cat /etc/grub.conf

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
password --md5 $2$MrXSq0xS$0LXqkBTfpywyD3TVCao3d0
title Red Hat Enterprise Linux Server (2.6.32-131.4.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-131.4.1.el6.x86_64 ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rhgb quiet
initrd /initramfs-2.6.32-131.4.1.el6.x86_64.img

The file is structured with a leading "global" section, which consists of all lines before the first line starting
with the keyword title, and one or more menuitem definitions, which each start with the keyword
title. Whitespace within a line is insignificant, though customarily menuitem definition stanzas are
indented with a TAB. Each relevant line must start with a recognized keyword, and the remainder of the
keyword's parameters must all occur on the same line. (Users of text editors that automatically wrap lines,
beware!)

Generally, the /boot/grub/grub.conf file is edited directly, and changes take effect the next time
grub runs (i.e., at the next system boot).

How GRUB refers to files

The syntax of the GRUB configuration file is pleasantly straightforward, with the exception of the syntax
which GRUB uses to refer to files, which follows.
(drive,partnum)filepath

The components within parentheses are used to identify a particular disk partition. The filepath
identifies a file within the filesystem contained by the specified partition. Interpreting each component
involves subtleties outlined in the table below.

Table 2.1. How GRUB Refers to Files

Component Role
drive GRUB does not refer to drives as hda or sdc or fd0, as does the Linux kernel.
Instead, GRUB refers to drives using the following names.

• hd0: "BIOS DRIVE #1"

• hd1: "BIOS DRIVE #2"

• fd0: "FLOPPY DRIVE #1"

Every BIOS maintains a concept of its first drive. Often it is the primary master
IDE drive, sometimes it is another IDE drive, sometimes it is a SCSI drive. By
changing BIOS settings, the identity of the "first drive" may change.
partnum The partnum is an integer used to identify the partition on a drive.
Unfortunately, GRUB counts partitions starting with 0, while the Linux kernel

rha130-6.1-0 10 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

Component Role
counts partitions starting with 1. Therefore, what GRUB calls partition number
3, The Linux kernel would call partition number 4.
filepath The filepath is the absolute path of the file, relative to the root directory of
the filesystem. GRUB is referring to files before filesystems are mounted into
a single directory tree, so GRUB has no concept of mount points. This point
is emphasized in the NOTICE placed by the Anaconda installer in the sample
configuration file displayed above.

To illustrate, we examine the following line from the configuration file, which specifies the background
("splash screen") image for the GRUB application.
splashimage=(hd0,0)/grub/splash.xpm.gz

Interpreting the file name involves the following three steps.

1. Determine the Drive: Because we know our system booted from the primary SATA drive, we will
assume that hd0, or "BIOS Drive #1", refers to it. What GRUB is calling hd0, the Linux kernel would
call /dev/sda.

2. Determine the Partition: Now that the drive is known, we locate the partition on the drive using the
partnum, remembering that we need to add 1 to compensate for GRUB's different starting point. The
0 above therefore specifies the first partition on the drive, /dev/sda1.

3. Determine the Mount Point: Because GRUB references files relative to the filesystem's root directory,
we need to determine where the partition /dev/sda1 is mounted in our directory tree. We use the
df command to examine current mounts.
[root@station root]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 8159420 5299244 2445700 69% /
/dev/sda1 69973 25972 40388 40% /boot
/dev/sdb5 30241928 25947992 2757724 91% /home
/dev/sdb7 28130956 10578516 16123460 40% /var
none 370024 0 370024 0% /dev/shm
/dev/fd0 1430 1380 51 97% /mnt/floppy

Because the partition /dev/sda1 is mounted to /boot, the file /grub/splash.xpm.gz relative
to the filesystem's root directory is the file /boot/grub/splash.xpm.gz relative to the root of
the directory tree.

After determining the partition, and where the partition is mounted, we can translate the "GRUB speak"
file (hd0,0)/grub/splash.xpm.gz into "Linux speak" as /boot/grub/splash.xpm.gz.

In order to reinforce the process of translating GRUB file references into Linux file references, the
following table lists files using both. Remember that the mapping relies on the partitioning structure
identified in the output of the df command seen above, and would be different on disks which are
partitioned differently.

Table 2.2. GRUB to Linux Filename Translations

What GRUB calls... ... Linux would call...
(hd0,0)/grub/splash.xpm.gz /boot/grub/splash.xpm.gz
(hd0,2)/etc/passwd /etc/passwd
(hd1,6)/log/dmesg /var/log/dmesg

rha130-6.1-0 11 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

What GRUB calls... ... Linux would call...

(hd1,4)/elvis/civgame+0220.sav.gz /home/elvis/civgame+0220.sav.gz
(fd0,0)/syslinux.cfg /mnt/floppy/syslinux.cfg
(hd0,0)/ /boot/
vmlinuz-2.6.32-131.4.1.el6.x86_64 vmlinuz-2.6.32-131.4.1.el6.x86_64
(hd0,0)/ /boot/
initramfs-2.6.32-131.4.1.el6.x86_64.img
initramfs-2.6.32-131.4.1.el6.x86_64.img

Why must GRUB be so different? Recall that GRUB is running before the Linux kernel is loaded, and
can be considered a minimal operating system on its own. As GRUB is meant to be the "Grand Unified"
loader, and not just a Linux loader, its syntax has no special relationship to that of the Linux kernel.

GRUB Global Configuration

With the hard part out of our way, we return to the global section of the GRUB configuration file, that is,
all lines that appear before a line that starts title.
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
password --md5 $2$MrXSq0xS$0LXqkBTfpywyD3TVCao3d0
hiddenmenu

The following entries are often found within the global section.

Table 2.3. GRUB Global Configuration Syntax

Keyword Purpose
default=N Define the default menu entry, if the user does not choose one. Menu
entries are counted starting from 0.
timeout=N Specify the amount of time (in seconds) the user has to choose a menu
entry, before the default menu entry is used.
splashimage=file Use file as a background image for the menu.
password plaintext Specify a password for the GRUB application using plaintext.
password --md5 ciphertext Specify a password for the GRUB application using MD5 based
encryption.
hiddenmenu Do not display the GRUB menu at bootup.

Use password, not passwd

A common mistake within the GRUB configuration file is to use the incorrect passwd keyword,
instead of password.

GRUB Menuitem Configuration

When booting, GRUB usually presents the user with a menu of possible configuration choices. The choices
are configured with stanzas in the grub.conf configuration file which start with the keyword title,
followed by text which is used to label the stanza, as exemplified below.

title Red Hat Enterprise Linux Server (2.6.32-131.4.1.el6.x86_64)

root (hd0,0)
kernel /vmlinuz-2.6.32-131.4.1.el6.x86_64 ro root=UUID=09ec85c1-d948-40

rha130-6.1-0 12 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

ee-a288-6875ee3a5ee2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SY

SFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet
initrd /initramfs-2.6.32-131.4.1.el6.x86_64.img

The following syntax is found in menuitems used to boot Linux kernels. Unlike the global section, items
must occur in the specified order.

Table 2.4. GRUB Menuitem Configuration Syntax

Keyword Purpose
root partition Mount and implicitly use the specified partition on all
subsequent file references.
kernel file commandline... Load the specified file as the kernel image, and pass any
remaining arguments to the kernel as the kernel commandline.
initrd file Load the specified file as an initial ramdisk for the kernel.

Notice how simply the design of a menuitem stanza reflects the necessary parameters. The kernel
configuration line specifies the kernel image to load and its associated command line, while the initrd
configuration line specifies an initial ramdisk to load.

Using GRUB
Usually, grub is properly installed by the Anaconda installer, and will be used whenever the system is
booted. When GRUB runs, it first displays simple text announcing its startup, and then presents the user
with a graphical menu, and in the default Red Hat Enterprise Linux setting, a timer counting down from 5.

For our purposes, we will refer to GRUB as running in one of five modes:

• Secure Mode: This mode presents a menu of boot selections to the user.

• Basic Mode: The Basic mode resembles the Secure mode, but rather than simply selecting a menu item,
the user can choose to enter into of the following modes.

• Edit Mode: This mode allows the user to perform edits on an existing configuration.

• Append Mode: Akin to edit mode, this mode allows the user to edit the kernel's command line.

• Command Mode: In this mode, the user operates within an interactive grub shell.

Each of these modes are discussed in more detail below.

Secure Mode
Secure mode is the default mode if a password was specified in the grub.conf configuration file. Grub
presents the user with a list of menu choices, whose titles are a direct reflection of the title lines
found in the grub.conf configuration file. By selecting a menu item, the system boots to the specified
configuration.

If the correct password can be specified, a user can jump from Secure mode to Basic mode.

Basic Mode
The Basic mode is the default mode if no password is supplied in the grub.conf configuration file.
Users may choose an appropriate menu item, or may use any of the following keys to enter into one of
the following modes.

rha130-6.1-0 13 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

Table 2.5. Commands for Shifting Modes in GRUB.

Key Action
e enter Edit mode for the selected menuitem
a enter Append mode for the selected menuitem
c enter Command mode

Edit Mode
When entering edit mode, grub displays lines from the relevant grub.conf stanza in a primitive text
editor. The editor allows the user to add a new line, delete an existing line, or edit an existing line with
arbitrary edits.

Once the user has edited the stanza to their liking, they may use b to boot using the new configuration, or
ESCAPE to abandon all changes and return to Basic Mode.

In any event, changes to the configuration made using grub's edit mode apply to that boot only. The grub
application does not allow users to permanently edit files.

Append Mode
Append mode can be thought of as a reduction of edit mode, where only the arguments which compose the
kernel command line can be edited. While less versatile (a new kernel or initrd file could not be specified,
for example), the simpler append mode can be used for the majority of cases when a user would want to
override the default grub configuration. As in Edit mode, changes apply to that boot only.

Command Mode
In Command mode, the existing grub.conf file is ignored, and the user operates within an interactive
grub shell. (Here, the term shell is being used in the general sense. Don't expect to find all of the capabilities
found in the bash shell!) Knowledgeable users can use the shell to manually specify a kernel and initrd,
and boot the system, or use the shell to examine the filesystem.

The grub shell can be exited with the ESCAPE key, which returns the user to Basic mode.

Installing Grub
When people refer to installing a bootloader, they are generally referring to transferring the first stage
bootloader to the appropriate Master Boot Record or bootsector. System administrators should seldom
need to reinstall the first stage GRUB bootloader, as Anaconda installs it upon system installation, and
other than edits to the grub.conf configuration file, the installation should not need to be modified for
routine use. 2

If for some reason GRUB does need to be installed, however, the grub-install command can be used to
simply perform the operation. The command is called with a single argument, the name of the drive whose
Master Boot Record should be used for the installation.
[root@station root]# grub-install /dev/sda
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

2
Users familiar with the LILO bootloader will recognize this as a dramatic and welcome change. LILO, unlike GRUB, cannot read its configuration
file directly, so first stage LILO must be reinstalled with new low level configuration information every time LILO's configuration file is edited.

rha130-6.1-0 14 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

# this device map was generated by anaconda

(fd0) /dev/fd0
(hd0) /dev/sda

Upon installation, grub-install displays the current interpretation of "BIOS Drive #1" (or whichever drives
were referenced). Generally, this information is correct. If it does not look appropriate, you might need to
edit the file /boot/grub/device.map and try again.

Managing GRUB Passwords

As will be seen in a later Lesson, the ability to modify the kernel commandline at bootup provides great
flexibility, but also introduces security risks. For example, by supplying the right arguments, the system can
be booted into a maintenance shell from which the root account's password can be changed. Accordingly,
it is suggested that a password by supplied in the grub.conf configuration file.

While the configuration file supports plaintext passwords, as in the following, storing plaintext passwords
on a filesystem is never a good idea.
password redhat

Instead, the grub-md5-crypt command can be used to generate encrypted passwords, which can then be
pasted into the configuration file, as in the following.
[root@station root]# grub-md5-crypt
Password: redhat
Retype password: redhat
$1$NaJ7E0$7UPMvSbjbq48W9ip6CEH61

The following line could then be included in the grub.conf configuration file.
password --md5 $1$NaJ7E0$7UPMvSbjbq48W9ip6CEH61

This line would set the GRUB password to redhat, but not leave the password exposed in the configuration
file in plaintext.

GRUB BIOS Related Limitations

When the GRUB application is running, the Linux kernel is not yet in place. Accordingly, GRUB uses
BIOS supplied device drivers to access disks. Some older BIOS drivers have two notable limitations.

1. Some BIOS's cannot access beyond 1024 cylinders.

2. Many BIOS's are only aware of primary partitions.

The first limitation is the motivation for the /boot partition. By design, the /boot partition should
contain all elements of the system which are needed before the Linux kernel is loaded: the kernel image
itself, any initrd's, and the bootloader. The partition is generally small, and occurs as early as possible on
the drive, so that it fits entirely within the first 1024 disk cylinders.

This being said, most modern BIOS's provide extensions which overcome this limitation.

Interestingly, a Linux system can generally operate quite well without ever mounting the /boot partition.
(Remember, only aspects of the system needed before the kernel is started are found within it). In practice,
this is not a good idea, as there is the occasional file which is referenced from the running system (such as
the System.map, or the grub.conf configuration file, if the administrator wants to make edits).

The second limitation only affects where GRUB may be installed if another bootloader is being used in
the Master Boot Record. Namely, only the bootsectors of primary partitions are candidates for installing

rha130-6.1-0 15 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

first stage GRUB. This is not considered a standard configuration, and the details of the configuration are
beyond the scope of this course.

In closing, we reemphasize that both of these limitations are BIOS related limitations, not Linux
limitations. Once Linux kernel is running, neither of these limitations are relevant.

The Linux Kernel

The previous lesson presented Linux initialization as occurring in four distinct phases. This lesson
primarily discusses the boot loader; the next two lessons discuss /sbin/init and related processes.
Squeezed in between them, providing little more than a transition, is this short section describing the role
of the kernel itself it bootstrapping the system.

Although much can be said about kernel initialization from an operating systems design standpoint, from
a system administration standpoint, there is little to say. But what there is to say is important. Essentially,
a system administrator can view the kernel as performing three steps.

1. Replace all BIOS device drivers with Linux device drivers.

2. Mount the root partition read-only.

3. Start the first process.

And that's it. The kernel does remarkably little to initialize the system, relying on the first process to do
all of the hard work.

There are some repercussions associated with each step.

1. Because Linux replaces all BIOS device drivers, the BIOS limitations mentioned above are not relevant
once the kernel has been started.

2. However, if the root partition uses advanced partitioning types such as Software RAID or Logical
Volume Manager (LVM), then the /boot files must be on a separate partition. The files in /boot
must be accessed before the kernel loads the modules and software required to read the software RAID
or LVM partitions.

3. Because the kernel must mount the root partition, it must be told which partition is the right one to
mount. This is usually accomplished with the root= kernel command line parameter, which may either
refer to a specific partition (such as root=/dev/sda5) or, like the mount command, may refer to
the root partition UUID or ext2 filesystem label (using a fairly bizarre root=LABEL=/ syntax). Why
does the kernel mount it read-only, instead of read-write? The answer comes in a later lesson.

4. Because the kernel starts the first process directly, the correct root partition must be specified. The
name of the first process is hardwired into the kernel: /sbin/init. The kernel will mount as the
root partition whichever partition you tell it, but will only be able to find /sbin/init if you tell it
the right one.

The Kernel Command Line: Boot Parameters

Just as the processes have a command line, the kernel itself has a kernel command line. The command line
parameters for the current kernel can always be examined by reading /proc/cmdline.
[root@station ~]# cat /proc/cmdline
ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF

Any tokens specified after the kernel's filename in the grub.conf configuration file are passed to the
kernel, which handles them as follows.

rha130-6.1-0 16 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

1. First, the kernel checks to see if the token is a built in kernel parameter. If so, the kernel applies the
parameter. Some of the more common kernel parameters are documented in the bootparam(7) man
page.

2. Secondly, if the kernel doesn't recognize the token, and the token has the form name=value, the
kernel passes the value to the first process as the environment variable name. (The first process is
almost invariably /sbin/init).

3. Lastly, anything that the kernel doesn't recognize, and that doesn't have the form name=value, the
kernel passes through to the first process as a command line argument.

Some of the kernel command line arguments most commonly encountered in Red Hat Enterprise Linux
are tabled below.

Table 2.6. Common Kernel Boot Parameters

Parameter Effect
root=device Use device as the root partition (see above). This parameter is almost always
required.
ro Mount the root partition read-only (see above). This parameter is almost always
required.
quiet Do not display kernel messages at startup. Although enabled by default in Red Hat
Enterprise Linux, many administrators choose to remove it to see the reassuring
stream of kernel messages.
rhgb This parameter, which stands for "Red Hat Graphical Boot", causes an X server
to start early in the bootup process. The parameter can easily be removed if this
complicates or slows the bootup process.
rd_NO_LUKS This parameter indicates that the root device is not a LUKS encrypted drive. Similar
entries may exist to indicate if the root device uses LVM, Software RAID, or Device
Mapper

Examples
Referencing a New Kernel
A system administrator has compiled a custom kernel and created an associated initial ramdisk, and placed
the in the /boot directory as the files vmlinuz-2.6.32-custom and initramfs-2.6.32-
custom.img, respectively. He would now like to update his grub.conf file so that he can boot to the
new kernel. His current file looks like the following.
[root@station root]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
password --md5 $2$MrXSq0xS$0LXqkBTfpywyD3TVCao3d0

rha130-6.1-0 17 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

title Red Hat Enterprise Linux (2.6.32-131.0.15.el6.x86_64)

root (hd0,0)
kernel /vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rd_NO_LUK
initrd /initramfs-2.6.32-131.0.15.el6.x86_64.img

The administrator opens the file in a text editor, and first duplicates the existing "known good" stanza.

title Red Hat Enterprise Linux (2.6.32-131.0.15.el6.x86_64)

root (hd0,0)
kernel /vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rd_NO_LUK
initrd /initramfs-2.6.32-131.0.15.el6.x86_64.img
title Red Hat Enterprise Linux (2.6.32-131.0.15.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rd_NO_LUK
initrd /initramfs-2.6.32-131.0.15.el6.x86_64.img

Next, the administrator edits the second stanza, changing the title, kernel, and initrd references, as
appropriate.

title Red Hat Enterprise Linux (2.6.32-131.0.15.el6.x86_64)

root (hd0,0)
kernel /vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rd_NO_LUK
initrd /initramfs-2.6.32-131.0.15.el6.x86_64.img
title Custom (2.6.32-custom)
root (hd0,0)
kernel /vmlinuz-2.6.32-custom ro root=UUID=09ec85c1-d948-40ee-a288-6875ee3a5ee2 rd_NO_LUKS rd_NO_LVM r
initrd /initramfs-custom.img

Now the administrator saves his edits and reboots. Upon rebooting, he can choose his new custom kernel
at the GRUB menu. If something goes wrong (such as a typo in the grub.conf configuration file), he
can always just reboot back to the "known good" entry.

Migrating a GRUB Configuration File

An administrator is porting a grub.conf configuration file from one system to another. Here is the
contents of the file.
[root@station root]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
password --md5 $2$MrXSq0xS$0LXqkBTfpywyD3TVCao3d0
title Red Hat Enterprise Linux Server (2.6.18-8.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-8.el5 ro root=LABEL=/ vga=0x317
initrd /initrd-2.6.18-8.el5.img

As the comments imply, on the original system, the /boot directory was partitioned off. On the current
system, however, there is only a single root partition: the device /dev/sda2. Because the /boot
directory is now part of the root partition, the administrator needs to make the following adjustments.

1. The specification of the root partition needs to be updated to refer to the device /dev/sda2.

rha130-6.1-0 18 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

2. The filepaths for both the kernel and the initial ramdisk need to be updated to reflect the fact that the
/boot directory is now part of the root filesystem.

Because, on the new machine, the original configuration will not work, she decides to edit the stanza "in
place". The edited stanza is listed below.
title Red Hat Enterprise Linux Server (2.6.18-8.el5)
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-8.el5 ro root=LABEL=/ vga=0x317
initrd /boot/initrd-2.6.18-8.el5.img

Online Exercises
Lab Exercise
Objective: Gain experience configuring the GRUB bootloader.

Estimated Time: 30 mins.

Specification
You are going to copy (not move!) your existing kernel and initrd to a new location, and modify your
grub.conf appropriately to boot the newly created kernel.

1. Create the directory /kernel.

2. Use the command uname -r to report your current kernel version. In the following, use this for
kernelversion.

3. Copy the files /boot/vmlinuz-kernelversion and /boot/initramfs-

kernelversion.img to the newly created /kernel directory. Make sure to copy the file starting
vmlinuz, not vmlinux.

4. As a precaution, make a backup copy of your /boot/grub/grub.conf file, called /boot/grub/

grub.conf.prelab4.2. Open your /boot/grub/grub.conf file, and make the following
edits.

a. Duplicate the menuitem stanza relevant to your current kernel, and set the title of the newly created
stanza to the single word lab4.2. Do not modify your original stanza in any way.

b. In your newly created menuitem stanza, update the references to the kernel and initrd files, so that
they refer to the newly created files /kernel/vmlinuz-kernelversion and /kernel/
initramfs-kernelversion.img. You probably will have to update the reference to the root
partition as well.

c. To the kernel command line, add the argument panic=42. (You may consult the bootparam(7) man
page for more information, if you are curious).

d. Set the bootloader password to redhat, using a plaintext password. (If the bootloader already had a
password, merely comment out the line specifying the original password).

Save your edits.

5. Reboot your machine. At the GRUB menu, enter the bootloader's password (redhat). Open your
newly added stanza (lab4.2) for editing. Within the editor, append the argument rha=fun to the kernel
commandline, leaving all other arguments in place, and boot your machine.

rha130-6.1-0 19 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

6. Once booted, log in using your Red Hat Academy account, and grade the exercise.

Note
On some machines, the machine may not be able to reboot using your new configuration, due to
the 1024 BIOS cylinder limit. If this is the case, reboot your machine using the original stanza
(you didn't modify it, did you?). Repeat the Lab, but instead of copying your kernel and initrd
to the directory /lab4.2, copy them to the directory /boot/lab4.2 instead, and edit your
lab4.2 stanza in the grub.conf file accordingly.

Deliverables
1.
1. A proc filesystem file /proc/cmdline which reports the arguments panic=42 and rha=fun.

2. A /boot/grub/grub.conf file, which contains a stanza for a menuitem titled lab4.2. The
menuitem should reference the appropriate new kernel and initrd location, and specify a kernel
command line argument panic=42 but not a kernel commandline argument rha=fun.

Questions
1. Which of the following files is GRUB's configuration file?

a. /boot/grub/grub.conf

b. /boot/grub.conf

c. /etc/sysconfig/grub

d. /etc/grub/grub.conf

e. None of the above

2. What would be the conventional name of the initial ramdisk for the 2.6.32-8.el6 kernel?

a. /boot/initramfs-2.6.32-8.el6.img

b. /initramfs-2.6.32-8.el6

c. /lib/initramfs-2.6.32-8.el6.img

d. /var/initramfs-2.6.32-8.el6

e. None of the above

3. What command is used to install GRUB?

a. grub

b. init-grub

c. grub-install

d. mkgrub

e. None of the above

rha130-6.1-0 20 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

4. In GRUB's configuration file, lines beginning with which keyword define the user visible menu
labels?

a. label

b. title

c. menuitem

d. choice

e. None of the above

5. When booting, which file is executed as GRUB's stage 1?

a. /boot/grub/grub.conf

b. /boot/grub/stage1

c. /sbin/gstage1

d. /sbin/grub

e. The question is inappropriate, as GRUB's stage 1 lives in a Master Boot Record or bootsector.

Use the following transcript to answer the next 4 questions.

[root@station root]# cat /etc/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
password --md5 $2$MrXSq0xS$0LXqkBTfpywyD3TVCao3d0
title Red Hat Enterprise Linux Server (2.6.18-8.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-8.el5 ro root=LABEL=/ rhgb quiet
initrd /initrd-2.6.18-8.el5.img

[root@station root]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 5131108 4606964 263492 95% /
/dev/sda1 124427 9508 108495 9% /boot
none 127616 0 127616 0% /dev/shm

6. How would GRUB refer to the file /etc/passwd?

a. (hda,1)/etc/passwd

b. (hd0,3)/etc/passwd

c. (hda,3)/etc/passwd

d. (hd0,2)/etc/passwd

e. None of the above

rha130-6.1-0 21 Copyright 2011, Red Hat Inc.

 The GRUB Bootloader

7. How would GRUB refer to the file /boot/grub/grub.conf?

a. (hd0,1)/grub/grub.conf

b. (hda,0)/boot/grub/grub.conf

c. (hda,1)/grub/grub.conf

d. (hd0,0)/boot/grub/grub.conf

e. None of the above

8. Where is the initial ramdisk referenced in the above configuration file located?

a. /initrd-2.6.18-8.el5.img

b. /hd0/initrd-2.6.18-8.el5.img

c. /proc/initrd-2.6.18-8.el5.img

d. /boot/initrd-2.6.18-8.el5.img

e. None of the above

9. When booting, approximately how long does the user have to select a menu item?

a. 10 minutes

b. 1 second

c. 10 seconds

d. The answer depends on the speed of the CPU.

e. None of the above

10. After editing the GRUB configuration file, which command must be run to implement the changes?

a. grub

b. grub-install

c. grubby

d. mkgrub

e. None of the above

rha130-6.1-0 22 Copyright 2011, Red Hat Inc.

Chapter 3. /sbin/init and Run Levels
Key Concepts
• The first process started by the kernel is /sbin/init

• The init command runs as long as the system runs, and is in charge of starting and stopping other
processes.

• The init command organizes the state of running processes using a concept called runlevels.

• The init command uses the /etc/sysconfig/init configuration file and scripts located in the /
etc/init/ directory to start jobs when various events happen on the system.

• The init command uses the /etc/inittab configuration file to determine which runlevel enter on
startup.

• The init and telinit commands can be used to change runlevels

• The shutdown, reboot, halt, and poweroff commands can be used to halt or reboot the machine.

Discussion
Upstart configuration files
As noted in the comments of the /etc/inittab most of the startup configuration has moved to files
in the /etc/init directory.
[elvis@station ~]$ ls /etc/init
control-alt-delete.conf rc.conf serial.conf
init-system-dbus.conf rcS.conf splash-manager.conf
kexec-disable.conf rcS-sulogin.conf start-ttys.conf
plymouth-shutdown.conf readahead-collector.conf tty.conf
prefdm.conf readahead.conf
quit-plymouth.conf readahead-disable-services.conf

In addition to these scripts, init will also use the values of variables stored in the /etc/sysconfig/
init.

/sbin/init and Upstart

The name of the first process to start is hardwired into the Linux kernel: /sbin/init. This command
is executed directly by the kernel on system startup, and therefore always has a process Id of 1. The init
program is designed to remain running as long as the system is running, and is the last process to die.
To help ensure that the process does not accidentally terminate, process Id 1 is even protected from the
otherwise lethal SIGKILL signal (signal number 9).

The init process spends its life starting, stopping, and monitoring other processes. The first thing init does
is read its configuration file, the /etc/inittab file. In the past this file contained instructions for all the
remaining startup steps. As a key configuration file, a mangled file coule leave your system unbootable.

Beginning in Red Hat Enterprise Linux 6, an event-based replacement for /sbin/init called upstart was
introduced to handle the starting, stopping, monitoring, and restarting of services at boot time and while

rha130-6.1-0 23 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

the system is running. For now the upstart feature is configured to be backward compatable and with
the exception replacing most of the /etc/inittab with a set of files in the /etc/init/ directory,
continues to use the older System V startup scripts described below.

The init program is responsible for completing the boot process by starting all other non-kernel system
processes. With Upstart, init start "jobs" when various "events" happen, such as when the system boots,
we enter a runlevel, or another init job starts or stops. These jobs are stored as scripts in the /etc/init/
directory.

Runlevels
We begin by looking at the traditional startup configuration file, /etc/inittab. The comments at the
top of the file list the new upstart configuration files. Following that are comments defining the runlevels
and finally, the only remaining configuration line.
#
# inittab is only used by upstart for the default runlevel.
#
# ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.
#
# System initialization is started by /etc/init/rcS.conf
#
# Individual runlevels are started by /etc/init/rc.conf
#
# Ctrl-Alt-Delete is handled by /etc/init/control-alt-delete.conf
#
# Terminal gettys are handled by /etc/init/tty.conf and /etc/init/serial.conf,
# with configuration in /etc/sysconfig/init.
#
# For information on how to write upstart event handlers, or how
# upstart works, see init(5), init(8), and initctl(8).
#
# Default runlevel. The runlevels used are:
# 0 - halt (Do NOT set initdefault to this)
# 1 - Single user mode
# 2 - Multiuser, without NFS (The same as 3, if you do not have networking)
# 3 - Full multiuser mode
# 4 - unused
# 5 - X11
# 6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:

From the comments in the middle of the file, we find that init organizes a Linux machine using a concept
called runlevels. Runlevels are traditionally named using the integers 0 through 6. In many versions of
Unix, the integers imply a sequence (i.e., to boot into runlevel 5, you must pass through runlevels 1 through
4). This is not the case in Linux; the runlevels could just as easily have been called "orange", "blue", and
"green".

Unix traditionally associates the following configurations to the various runlevels, and Linux adheres to
the Unix convention.

Table 3.1. Run Level Conventions

RunlevelUse Comments
0 Halt A utility runlevel, used to halt the machine.
1 Single User Mode A maintenance runlevel, traditionally referred to as single user mode.
When in runlevel 1, only root may log into the machine, and must log
in at the console. Network interfaces are generally not active, and most

rha130-6.1-0 24 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

RunlevelUse Comments
(if not all) daemons are shut down. This runlevel is designed to let
the administrator perform maintenance operations, such as mounting and
unmounting partitions, without unexpected interference.
2 No NFS In traditional Unix, runlevel 2 was multiuser mode without networking
(recall that in its early days, Unix was essentially a terminal server). Most
modern Unixes use runlevel 2 to somehow qualify the extent of networking.
In Red Hat Enterprise Linux, runlevel 2 implies multiuser mode with
networking, but without any NFS related filesystems.
3 Full Multiuser Runlevel 3 is a full operating mode, with complete networking and multiple
users, but no graphical interface (X server) is started. On many machines
used as network servers, this is the default runlevel.
4 Local Config Runlevel 4 traditionally has no implications, and is left for sites to use for
implementing local policy.
5 Multiuser with X Runlevel 5 has all of the implications of runlevel 3, with the addition of
maintaining a persistent X server.
6 reboot A utility runlevel, used to reboot the machine.

The last line in the above listing establishes the default runlevel for the machine, if none is specified. For
most machines, the default runlevel would either be runlevel 3 or runlevel 5.

Configuration for System Initialization and Runlevel

Management
The first of the /etc/init scripts run at boot time is the /etc/init/rcS.conf which in turn does
the following:

• The script /etc/rc.d/rc.sysinit is executed. The script will be examined in detail below, where
we discover it is the key initialization script that performs many system initializations.

• Next the /etc/inittab is examined to find the runlevel. If no runlevel is listed, a default reunlevel
is set. The rcS.conf also checks to see if a runlevel was passed to the kernel at boot time. A runlevel
passed to the kernel would override the runlevel found in the /etc/inittab

• Sends an event trigger to init telling it to enter that runlevel.

Upon each change of runlevel (including booting into the initial default runlevel), the script /etc/init/
rc.conf is processed. In this script, /etc/rc.d/rc is executed, with the appropriate runlevel passed
as its lone argument. This script performs all of the hard work when a system transitions from one runlevel
to another, and is the topic of the next Lesson.

Configuration for External Events

Other files in the /etc/init directory associate commands to various external events, such as a user
keying CTRL+ALT+DELETE at the console.

[elvis@station ~]$ cat /etc/init/control-alt-delete.conf

# control-alt-delete - emergency keypress handling
#
# This task is run whenever the Control-Alt-Delete key combination is

rha130-6.1-0 25 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

# pressed. Usually used to shut down the machine.

start on control-alt-delete

exec /sbin/shutdown -r now "Control-Alt-Delete pressed"

Notice that the final line of this script issues a command to reboot the system. The shutdown command
is discussed in more detail below.

Gettys
[elvis@station ~]$ cat /etc/init/tty.conf
# tty - getty
#
# This service maintains a getty on the specified device.

stop on runlevel [S016]

respawn
instance $TTY
exec /sbin/mingetty $TTY

Another traditional use of the init process is the launching, monitoring, and relaunching of gettys. In Linux
(and Unix), a getty (named for "get tty") is a process which monitors a serial line. If it ever hears "noise"
on the line (such as someone hitting the RETURN key a few times), it writes a login banner to the line,
and waits for a username in return. When a username is received, it execs the login command to continue
the login process. When a user logs off, the login process (which was the getty process) dies, and init
respawns a new getty to take its place.

While the above script is responsibly for starting, monitoring, and respawning the gettys, the list of
active consoles is controlled by the /etc/init/start-tty.conf which in turn reads the /etc/
sysconfig/init file.
[elvis@station ~]$ cat /etc/sysconfig/init
...
# What ttys should gettys be started on?
ACTIVE_CONSOLES=/dev/tty[1-6]
...

A getty should be launched to monitor every serial line which users can log in on. The above variable
results in one login (gdm or mingetty) for each of the first 6 virtual consoles. Why does Linux usually have
six virtual consoles that users can log in on? Because init spawns six gettys. If an administrator wanted to
provide more or less virtual consoles, the /etc/sysconfig/init can be modified.

Similarly, if a terminal were attached to the serial port of a machine, users would not be able to use the
terminal until a getty was attached to the serial line. If the serial port is the primary console for the kernel,
then the /etc/init/serial.conf file, would do the trick for the specified serial port. If your serial
console is not the primary console, or you want a getty on serial even if it's not the console, create your
own event by copying /etc/init/tty.conf, and changing the getty line in that file.

Lastly, the event which primarily distinguishes runlevel 3 from runlevel 5 is controlled by the /etc/
init/prefdm.conf. In runlevel 5, init consistently respawns the script /etc/X11/prefdm, which
starts the "preferred display manager", i.e., the X server which allows a user to log into the machine using
the graphical interface.
[elvis@station ~]$ cat /etc/init/prefdm.conf
# prefdm - preferred display manager
#
# Starts gdm/xdm/etc by preference

rha130-6.1-0 26 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

start on stopped rc RUNLEVEL=5

stop on starting rc RUNLEVEL=[!5]

console output
respawn
respawn limit 10 120
exec /etc/X11/prefdm -nodaemon

If you want the display manager to run in a different, or multiple, runlevels, the "start on" and "stop on"
lines can be modified. For example, to have the display manager start in both runlevel 4 and 5, use the
following lines:

start on stopped rc RUNLEVEL=[45]

stop on starting rc RUNLEVEL=[!45]

Managing Init
Observing Runlevels with runlevel
The runlevel command can be used to observe the current runlevel of the machine.
[root@station root]# runlevel
N 5

The output is not quite as simple as would be imagined, as two values are given. The latter is the current
runlevel, the first is the previous runlevel, or “N” for a newly booted machine. The output above is from
a machine freshly booted into runlevel 5.

Changing Runlevels with init or telinit

Init can be asked to change runlevels by invoking it with the desired runlevel as its single argument.
[root@station root]# init 3

Changing runlevels tends to be violent. Processes die; users get kicked off of the machine. On multiuser
machines, administrators try to schedule changes in runlevel (such as dropping into single user mode for
maintenance) so that users doesn't get caught by surprise.

In some versions of Unix, the init process is not designed to be called reentrantly, and a second telinit
command is used to change runlevels. In Linux, either init or telinit can be used to change runlevels.

Shutting Down the Machine with shutdown

True of any multitasking operating system, there are good and bad ways to shutdown a Linux machine.
Bad ways include yanking the power cord out from under it. We here discuss some good ways.

Fundamentally, a Linux machine is shut down by switching to runlevel 0 (for reasons we'll discover in the
next Lesson). Therefore, the init command can be used to start an instant shutdown of a machine.
[root@station root]# init 0

Generally, administrators prefer to use a command called shutdown, or one of its variants. The shutdown
command has two advantages. First, the shutdown can be scheduled to occur at a certain time or after a
specified delay. Secondly, an announcement is broadcast to the terminals of all currently logged on users
announcing the shutdown, and therefore giving them a chance to close up any work and log off.

rha130-6.1-0 27 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

The shutdown command uses the following syntax.

/sbin/shutdown [-hprc] time [warning-message]

Table 3.2. Shutdown Command Lines Switches

Switch Effect
-h Halt the machine
-p After halting, poweroff the machine.
-r Reboot the machine
-c Cancel a pending shutdown

The shutdown command is generally called with either the -h, -p, or -r command line switch, to qualify
that the machine should halt, poweroff, or reboot, respectively. If none of the three are specified, shutdown
takes the machine to runlevel 1. The -c command line switch may be used to cancel a pending shutdown.

The mandatory time argument can be specified one of three ways.

Table 3.3. Specifying Times for the shutdown Command

Syntax Interpretation
hh:mm An absolute time when the shutdown should occur, in hours and minutes.
+n A relative time when the shutdown should occur, n minutes from now.
now A synonym for +0.

A few commonly used shortcuts for the shutdown command exist as well, which are tabled below.

Table 3.4. Shortcuts for the shutdown Command

Shortcut Equivalent Command
halt shutdown -h now
poweroff shutdown -p now
reboot shutdown -r now

The /etc/rc.d/rc.sysinit Script

Upon booting, the kernel starts the /sbin/init process. The init process recognizes the startup event
and reads the /etc/init/rcS.conf script, where it encounter the following lines.

start on startup
stop on runlevel
task
console output
exec /etc/rc.d/rc.sysinit

Translating, these lines tells init "on startup, but only on startup, execute the script /etc/rc.d/
rc.sysinit and place the output on the console". The task line indicates this is a run once event
rather than a service to monitor and respawn. We now want to look at this startup script in detail. The
rc.sysinit script is a complicated bash shell script, which performs a lot of duties. We will not be
able to address every detail of the script, but will comment on some of the important or interesting points.

rha130-6.1-0 28 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

Do not feel that you need to remember the details of any or all of what is presented. In fact, one could be
excused for treating the bash code excerpts as many would physics equations, and just skip over them.
However, while reading the comments, one should compose a mental checklist of the various actions that
the script implements every time the system boots. The curious might also compose a second checklist of
bash shell programming tricks to learn more about.

Before we begin, let's recall the state of the machine. The kernel booted, mounted the root partition read-
only, and started init, which in turn started this script. When this script runs, how many processes are
running on the machine? Two. What does the filesystem look like? Only the root partition is mounted,
and it is mounted read-only.
#!/bin/bash
#
# /etc/rc.d/rc.sysinit - run once at boot time
#

...

if [ -f /etc/sysconfig/network ]; then
. /etc/sysconfig/network
fi
if [ -z "$HOSTNAME" -o "$HOSTNAME" = "(none)" ]; then
HOSTNAME=localhost
fi

if [ ! -e /proc/mounts ]; then
mount -n -t proc /proc /proc
mount -n -t sysfs /sys /sys >/dev/null 2>&1
fi

...

The script begins by examining /etc/sysconfig/network to determine a hostname for the system.
Next, early in the startup process, the proc filesystem and sysfs virtual filesystems are mounted to /proc
and /sys, respectively.

...
# Check SELinux status
SELINUX_STATE=
if [ -e "/selinux/enforce" ] && [ "$(cat /proc/self/attr/current)" != "kernel" ]; then
...
disable_selinux() {
...
relabel_selinux() {
...

Next a number of SELinux related checks are performed to enable or disable SELinux or initiate a
relabeling of the system.
...
# Print a text banner.
echo -en $"\t\tWelcome to "
read -r system_release < /etc/system-release
if [[ "$system_release" == *"Red Hat"* ]]; then
[ "$BOOTUP" = "color" ] && echo -en "\\033[0;31m"
echo -en "Red Hat"
[ "$BOOTUP" = "color" ] && echo -en "\\033[0;39m"
PRODUCT=$(sed "s/Red Hat \(.*\) release.*/\1/" /etc/system-release)
echo " $PRODUCT"
elif [[ "$system_release" == *Fedora* ]]; then
[ "$BOOTUP" = "color" ] && echo -en "\\033[0;34m"
echo -en "Fedora"

rha130-6.1-0 29 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

[ "$BOOTUP" = "color" ] && echo -en "\\033[0;39m"

PRODUCT=$(sed "s/Fedora \(.*\) \?release.*/\1/" /etc/system-release)
echo " $PRODUCT"
else
PRODUCT=$(sed "s/ release.*//g" /etc/system-release)
echo "$PRODUCT"
fi

Next, the script spends a good deal of effort examining the /etc/redhat-release file to decide what
to call itself in the welcome message it displays.

We now skip over a long section, as the script attempts to detect a large number of special disk
related configurations, such as software RAID arrays, LVM (Logical Volume Management) Volumes,
encrypted partitions, "stateless" (predominantly read only) systems, and other such complexities. For
simply configured systems, the next relevant line found below.
...
if [ -z "$fastboot" -a "$READONLY" != "yes" ]; then
STRING=$"Checking filesystems"
echo $STRING
fsck -T -t noopts=_netdev -A $fsckoptions
rc=$?

if [ "$rc" -eq "0" ]; then

success "$STRING"
echo
elif [ "$rc" -eq "1" ]; then
passed "$STRING"
echo
elif [ "$rc" -eq "2" -o "$rc" -eq "3" ]; then
echo $"Unmounting file systems"
umount -a
mount -n -o remount,ro /
echo $"Automatic reboot in progress."
reboot -f
fi
...

About 4 lines into this is the important line: fsck -T -t noopts=_netdev -A $fsckoptions,
or, stripping it down to its essence, fsck -A: perform a file system check on all partitions. We now
understand why the kernel mounted the root partition read-only. A read-only mounted filesystem can be
fscked and repaired, a read-write mounted filesystem cannot. The kernel's read-only mount allows the root
filesystem to be checked, along with all of the other (still unmounted) filesystems defined in the /etc/
fstab file. If problems occur, the user is dropped into a rescue shell.
...
# Remount the root filesystem read-write.
update_boot_stage RCmountfs
if remount_needed ; then
action $"Remounting root filesystem in read-write mode: " mount -n -o remount,rw /
fi

...
# Mount all other filesystems (except for NFS and /proc, which is already
# mounted). Contrary to standard usage,
# filesystems are NOT unmounted in single user mode.
if [ "$READONLY" != "yes" ] ; then
action $"Mounting local filesystems: " mount -a -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2 -O no_n
else
action $"Mounting local filesystems: " mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2 -O no
fi

Now that the fsck has been performed, the root partition can be remounted read-write with mount -
o remount,rw /, and, after a little cleaning of the root filesystem that we skipped over, mount -

rha130-6.1-0 30 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

a -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs -O no_netdev is used to mount all other (not

network related) partitions.

At this point, the entire filesystem should be in place. Let's remind ourselves what processes are running
on the machine: init, this script, and whatever processes this script has started. In other words, not many.
...
# Clean out /.
rm -f /fastboot /fsckoptions /forcefsck /.autofsck /forcequotacheck /halt \
/poweroff /.suspended &> /dev/null

...

# Clean up /var.
rm -rf /var/lock/cvs/* /var/run/screen/*
find /var/lock /var/run ! -type d -exec rm -f {} \;
rm -f /var/lib/rpm/__db* &> /dev/null
rm -f /var/gdm/.gdmfifo &> /dev/null
...

All of these lines, and other neighboring lines we have omitted, are purging the filesystem of various PID
files left by daemons and lock files and sockets left by various applications from the previous boot.
...
# Now that we have all of our basic modules loaded and the kernel going,
# let's dump the syslog ring somewhere so we can find it later
[ -f /var/log/dmesg ] && mv -f /var/log/dmesg /var/log/dmesg.old
dmesg -s 131072 > /var/log/dmesg
...

Recall that the file /var/log/dmesg contains a dump of kernel messages recorded soon after the most
recent boot. Here is where the file is created.

Summarizing, We find that the rc.sysinit script does a lot (we omitted most of it), but here is the
essence. As the script starts, the filesystem consists of only the root partition, mounted read-only. As it
exits, the filesystem is fully checked, mounted, and cleaned, ready for use.

The boot process now continues with runlevel specific configuration, which is the topic of the next lesson.

Examples
Running a getty on ttyS1
A system in the elvis's datacenter normally does not have a monitor or keyboard attached. If direct console
access is required he will need to connect his PDA to the second serial port using a serial cable. To prepare
for this, elvis needs to create a new upstart event which will start a getty on the device /dev/ttyS1.

Looking at the various files in the /etc/init directory, elvis finds samples of the lines he needs for
his new file. He takes the "start on" line from the start-ttys.conf file and the "stop on" line from
the tty.conf file. He also copies and modifies the "exec" line from the serial.conf file. The final
file look like the following:

# This service maintains a getty on /dev/ttyS1.

start on stopped rc RUNLEVEL=[2345]

stop on starting runlevel [016]

respawn
exec /sbin/agetty /dev/ttyS1 115200 vt100-nav

rha130-6.1-0 31 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

Customizing CTRL ALT DELETE for a Laptop

A laptop owner realized that they seldom felt the need to reboot his laptop, but instead he was often
powering it off so that he could pack it up and take it with him. He edited his /etc/init/control-
alt-delete.conf file to customize the behavior of the CTRL+ALT+DELETE key sequence. The
excerpt below shows the original line, commented out, with the modified line below.

#exec /sbin/shutdown -r now "Control-Alt-Delete pressed"

exec /sbin/shutdown -h now "Control-Alt-Delete pressed"

Online Exercises
Lab Exercise
Objective: Manage the init process with the /etc/inittab file.

Estimated Time: 20 mins.

Specification
1. As a precaution, create a /root/prelab4 subdirectory and make a backup of your /etc/inittab
and /etc/init files, by copying them to the new prelab4 directory. Also backup the /etc/
sysconfig/init file.

2. Make the following changes to your files.

a. In the /etc/sysconfig/init file, remove virtual consoles five and six from the active
consoles.

b. Modify the /etc/init/control-alt-delete.conf and duplicate the line which executes

the shutdown command. Comment out the original line, and edit the new copy so that, instead of
rebooting the machine, the CTRL+ALT+DELETE key sequence echo's the message Three finger
salute disabled. (This change can only be observed from a virtual console, as the X server blocks
the CTRL+ALT+DELETE key sequence.) NOTE: In order to see the output on the console either
redirect the output to /dev/tty0 or insert the line console output before the exec line.

c. Modify the /etc/init/prefdm.conf file so that the graphical display manager is respawned
in both runlevel 4 and runlevel 5.

d. In the /etc/inittab file, change your default runlevel to runlevel 3 (if it is not already).

3. When finished, reboot your machine. Upon rebooting, your machine should enter runlevel 3.

4. Log in from the first virtual console. Use the init command to manually change from runlevel 3 to
runlevel 4.

5. In runlevel 4, the graphical display manager (login screen) should be started by init. Log in using your
Red Hat Academy account, and grade this exercise.

Deliverables
1.
1. No processes (particularly mingetty's) should be attached to terminals /dev/tty5 or /dev/
tty6.

rha130-6.1-0 32 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

2. The runlevel command should reveal the current runlevel as runlevel 4, and the previous
runlevel as runlevel 3.

3. The /etc/inittab file should set the default runlevel to 3.

4. The upstart event handler should respawn the preferred display manager for both runlevels 4
and 5.

5. From within a virtual console, the CTRL+ALT+DELETE key sequence should echo the
phrase Three finger salute disabled, but not reboot the machine.

Cleaning Up
After you have completed this exercise, restore your original files by moving the files in /root/
prelab4 back into place, and either manually switch to runlevel 5, or reboot your machine.

Questions
1. Which of the following would be a reasonable default runlevel for a rack mounted network server?

a. 6

b. 3

c. 0

d. 1

e. None of the above

2. Which runlevel is traditionally associated with shutting down a machine?

a. 5

b. 6

c. 0

d. 1

e. None of the above

3. Which of the following command lines or actions could be used to reboot the machine?

a. reboot

b. init 6

c. shutdown -r now

d. Entering CTRL+ALT+DELETE from the console.

e. All of the above

Use the following information to answer the next question.

rha130-6.1-0 33 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

[elvis@station ~]$ grep ^id /etc/inittab

id:3:initdefault:
[elvis@station ~]$ grep ^ACTIVE /etc/sysconfig/init
ACTIVE_CONSOLES=/dev/tty[1-6]
[elvis@station ~]$ grep exec /etc/init/control-alt-delete.conf
exec /sbin/shutdown -h now "Control-Alt-Delete pressed"

4. If a machine were booted without anyuser intervention, how many mingetty processes would be
started directly by the init process?

a. 0

b. 1

c. 5

d. 6

e. None of the above

5. Upon pressing the CTRL-ALT-DEL key sequence, what will happen?

a. Nothing, the key sequence will be ignored.

b. The system will reboot.

c. The system will gracefully shutdown.

d. The system will enter a single user configuration

e. The system will crash as if the power plug had been pulled.

6. Once a machine has booted directly into runlevel 5, which of the following processes would not
have been started directly by init?

a. /etc/rc.d/rc

b. /sbin/mingetty

c. /etc/rc.d/rc.sysinit

d. /etc/X11/prefdm

e. All of the above processes would have been started by init

7. You decide that you would like to have two additional virtual consoles present login prompts. What
file should be edited to add the additional mingetty processes?

a. /etc/inittab

b. /etc/init/vc.conf

c. /etc/sysconfig/virtual

d. /etc/sysconfig/init

e. None of the above

8. Which of the following command lines would schedule a shutdown of the machine for 2 hours
after it was executed?

rha130-6.1-0 34 Copyright 2011, Red Hat Inc.

 /sbin/init and Run Levels

a. shutdown -h +120

b. halt +2

c. shutdown +120

d. shutdown -r +2

e. None of the above

You find a Red Hat Enterprise Linux machine with the following line in its /etc/inittab file.
id:6:initdefault:

9. What would be the most appropriate name for the machine?

a. slide

b. brick

c. yo-yo

d. molasses

10. Which of the following commands could be used to perform an immediate shutdown of the
machine?

a. poweroff

b. halt

c. shutdown -h now

d. init 0

e. All of the above

rha130-6.1-0 35 Copyright 2011, Red Hat Inc.

Chapter 4. Red Hat Enterprise Linux
Service Scripts
Key Concepts

• Red Hat Enterprise Linux service scripts are found in the /etc/rc.d/init.d directory.

• Services may be started or stopped using the service command.

• The runlevel default configuration for services can be configured with the chkconfig command.

Discussion
The /etc/rc.d Directory
As we continue our narrative of The Booting Linux Machine begun in previous Lessons, the kernel has
begun the init process, which triggered the upstart event using the /etc/init/rcS.conf, which has
in turn run the script /etc/rc.d/rc.sysinit. Upon exiting this script, the machine's filesystem has
been assembled.

At the end of the rcS.conf configuration, upstart issues a telinit to enter a specific runlevel. This in
turn triggers an event to process the /etc/init/rc.conf file.

[root@station ~]# cat /etc/init/rc.conf

# rc - System V runlevel compatibility
#
# This task runs the old sysv-rc runlevel scripts. It
# is usually started by the telinit compatibility wrapper.

start on runlevel [0123456]

stop on runlevel [!$RUNLEVEL]

task

export RUNLEVEL
console output
exec /etc/rc.d/rc $RUNLEVEL

Paying close attention to the last line, upstart calls the /etc/rc.d/rc script with the appropriate
runlevel as an argument.

We won't examine the script rc directly, but we will discuss what it does. First, we examine the layout
of the /etc/rc.d directory.
[root@station rc.d]$ ls -F /etc/rc.d/
init.d/ rc0.d/ rc2.d/ rc4.d/ rc6.d/ rc.sysinit*
rc* rc1.d/ rc3.d/ rc5.d/ rc.local*

Taking a quick inventory, we see three script scripts, several directories apparently named after runlevels,
and a directory init.d. At the risk of getting ahead of ourselves, we present a summary of the contents
of the directory.

rha130-6.1-0 36 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

Table 4.1. Contents of the /etc/rc.d Directory

File Type Purpose
rc.sysinit script system initialization at bootup
rc.local script local customizations to system initialization at bootup
rc script drives changes in runlevel
init.d directory contains Red Hat Enterprise Linux service scripts
rcn.d directory contains symbolic links which implement service state configuration
for runlevel n.

Examining one of the runlevel specific directories, we find the following.

[root@station rc.d]$ ls -F /etc/rc.d/rc3.d/
K01certmonger@ K75quota_nld@ S11portreserve@ S26haldaemon@
K01smartd@ K80kdump@ S12rsyslog@ S26udev-post@
K02oddjobd@ K80sssd@ S13cpuspeed@ S28autofs@
K10psacct@ K84wpa_supplicant@ S13irqbalance@ S50bluetooth@
K10saslauthd@ K85ebtables@ S13iscsi@ S55sshd@
K15httpd@ K86cgred@ S13rpcbind@ S80postfix@
...
K69rpcsvcgssd@ S08ip6tables@ S24rpcidmapd@ S99libvirt-guests@
K73ypbind@ S08iptables@ S25cups@ S99local@
K74ntpd@ S10network@ S25netfs@
K75ntpdate@ S11auditd@ S26acpid@

As the decoration provided by the -F command line switch (or the light blue color of the entries on a color
terminal) implies, the directory is full not of regular files, but meticulously named symbolic links. If we
ask to resolve the symbolic links, we find the following.
[root@station rc.d]$ ls -lF /etc/rc.d/rc3.d/
...
lrwxrwxrwx. 1 root root 20 Aug 1 14:10 K87multipathd -> ../init.d/multipathd*
lrwxrwxrwx. 1 root root 21 Jul 5 14:20 K87restorecond -> ../init.d/restorecond*
lrwxrwxrwx. 1 root root 15 Jul 5 14:20 K89rdisc -> ../init.d/rdisc*
lrwxrwxrwx. 1 root root 19 Jul 5 14:31 K95firstboot -> ../init.d/firstboot*
lrwxrwxrwx. 1 root root 17 Jul 5 14:23 S01sysstat -> ../init.d/sysstat*
lrwxrwxrwx. 1 root root 22 Jul 5 14:23 S02lvm2-monitor -> ../init.d/lvm2-monitor*
lrwxrwxrwx. 1 root root 18 Aug 5 08:56 S05cgconfig -> ../init.d/cgconfig*
lrwxrwxrwx. 1 root root 16 Aug 1 14:10 S07iscsid -> ../init.d/iscsid*
lrwxrwxrwx. 1 root root 19 Jul 5 14:17 S08ip6tables -> ../init.d/ip6tables*
lrwxrwxrwx. 1 root root 18 Jul 5 14:16 S08iptables -> ../init.d/iptables*
lrwxrwxrwx. 1 root root 17 Jul 5 14:20 S10network -> ../init.d/network*
...

Each of the symbolic links seems to refer to a script in the /etc/rc.d/init.d directory. Taking a
quick look at the other runlevel specific directories, we find the same thing.

We seem to have discovered an important directory: /etc/rc.d/init.d.

The /etc/rc.d/init.d Directory and Service Scripts

The /etc/rc.d/init.d directory plays a special role in Red Hat Enterprise Linux, and should be
part of the working vocabulary of any Red Hat Enterprise Linux administrator. The directory exclusively
contains scripts which all share a similar interface.
[root@station init.d]$ cd /etc/rc.d/init.d/
[root@station init.d]$ ls
abrtd firstboot lldpad oddjobd saslauthd
acpid functions lvm2-monitor portreserve single
atd haldaemon matahari-host postfix smartd

rha130-6.1-0 37 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

...
bluetooth iptables messagebus rdisc sshd
certmonger irqbalance multipathd restorecond sssd
cgconfig iscsi netconsole rhnsd sysstat
cgred iscsid netfs rhsmcertd udev-post
cpuspeed kdump network rpcbind vncserver
...
[root@station init.d]$ ./sshd
Usage: ./sshd {start|stop|restart|reload|force-reload|condrestart|try-restart|status}

Each script expects to be called with a single argument, most notably start, stop, restart, reload,
or status. These scripts are referred to as service scripts, and most directly manipulate a daemon that
implements a particular service.

As an example, the /etc/rc.d/init.d/sshd service script manages the /usr/sbin/sshd

daemon, which is the server for the Secure Shell network service.

If we want to know if a service is running, we ask for its status.

[root@station init.d]$ ./sshd status
openssh-daemon (pid 1820) is running...

We find that the daemon is up and running. Of course, we could have looked directly, with the ps command.
[root@station init.d]$ ps aux | grep sshd
root 1820 0.0 0.0 63944 304 ? Ss 10:05 0:00 /usr/sbin/sshd
root 23614 0.0 0.0 103240 856 pts/2 S+ 11:56 0:00 grep sshd

If we want to stop the service, we ask the service script to stop it.
[root@station init.d]$ ./sshd stop
Stopping sshd: [ OK ]
[root@station init.d]$ ./sshd status
openssh-daemon is stopped
[root@station init.d]$ ps aux | grep sshd
root 23713 0.0 0.0 103240 860 pts/2 S+ 11:58 0:00 grep sshd

As the script told us, and as we confirmed twice through two different techniques, the sshd daemon is
dead. It should not be hard to determine how to start the daemon back up.
[root@station init.d]$ ./sshd start
Starting sshd: [ OK ]
[root@station init.d]$ ./sshd status
openssh-daemon (pid 23843) is running...

The sshd daemon is again running, naturally with a different process id. What does a restart do?
[root@station init.d]$ ./sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
[root@station init.d]$ ./sshd status
openssh-daemon (pid 23953) is running...

A stop, followed by a start. What does a reload do? In order to more directly illustrate, we reload
the crond daemon instead of sshd.
[root@station init.d]$ ./crond status
crond (pid 5999) is running...
[root@station init.d]$ ./crond reload
Reloading cron daemon configuration: [ OK ]
[root@station init.d]$ ./crond status
crond (pid 5999) is running...

Unlike a restart, a reload is a single action, and the same process is often running after a reload
as was before (notice the identical crond process id's). Many Linux (and Unix) daemons override signal

rha130-6.1-0 38 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

number 1 (SIGHUP), and use it to as a sign to reread and implement changes made to their configuration
files. In other words, you can reinitialize many daemons by delivering a signal number one to them.
Oldtimers refer to this as rehupping a daemon. While a restart literally stops and then starts a daemon,
a reload generally rehups it.

One caveat about running service scripts from within the /etc/rc.d/init.d directory, however. In
the following, observe the difference in the last two commands.
[root@station root]$ cd /etc/rc.d/init.d/
[root@station init.d]$ ./sshd status
sshd (pid 5864) is running...
[root@station init.d]$ sshd status
Extra argument status.

Why did the apparently same command generate two different outputs? Which file was actually executed
in the latter case? Because the directory "." is not included in a user's PATH, the latter case did not execute
the local sshd service script, but instead the /usr/sbin/sshd command.

Red Hat Enterprise Linux service scripts provide an easy and convenient mechanism for managing
daemons. After a while, /etc/rc.d/init.d/... tends to roll off of the fingers of a Red Hat
Enterprise Linux administrator. Want to start the web server? /etc/rc.d/init.d/httpd start.
Want the sshd daemon to re-read its configuration file? /etc/rc.d/init.d/sshd reload. Want
to kill the ftp daemon? You get the idea.

However, there is yet an easier way...

Services and the service Command

Because administrators are so often running scripts from the /etc/rc.d/init.d directory, a shortcut
command has been created, called service. The service command expects as its first argument the name
of a Red Hat Enterprise Linux service, and as a second argument a single word command such as start,
stop, etc.
service service_name command

What is a Red Hat Enterprise Linux service? Any component of the system which is managed by a service
script in the /etc/rc.d/init.d directory. The following two command lines are almost exactly
equivalent.
service service_name start

/etc/rc.d/init.d/service_name start

Most services have daemons associated with them, but not all. For example, there is a service called
network.
[root@station root]$ service network restart
Shutting down interface eth0: [ OK ]
Shutting down interface eth1: [ OK ]
Shutting down loopback interface: [ OK ]
Disabling IPv4 packet forwarding: [ OK ]
Setting network parameters: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]
Bringing up interface eth1: [ OK ]

As the name implies, the system's networking components are stopped and restarted. As will be discussed
in a later lesson, however, there is not a "network" daemon. Instead, the network service essentially just
changes the state of the kernel.

rha130-6.1-0 39 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

Similarly, some services manage more than one daemon collectively, as is the case for the syslog and nfs
services.
[root@station root]$ service nfs status
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
[root@station root]$ service syslog status
syslogd (pid 4423) is running...
klogd (pid 4427) is running...

What defines a service is not a particular daemon, but the presence of a service script in the /etc/rc.d/
init.d directory. Many services manage a daemon, some do not, while some manage more than one
daemon collectively.

System administration of services is now even easier. Want to start the web server? service httpd
start. Want to know if the Secure Shell service is running? service sshd status.

Even though the service command makes life easy, administrators should still know about the /etc/
rc.d/init.d directory and the role it plays. For example, what if an administrator wanted to start the
SAMBA server?
[root@station root]$ service smbd start
smbd: unrecognized service

A couple things could have gone wrong.

1. The software that implements the SAMBA server might not have been installed on the system.

2. The administrator might have gotten the name of the service wrong.

In either case, a quick look in the /etc/rc.d/init.d directory will identify the problem.
[root@station root]# ls /etc/rc.d/init.d/s*
/etc/rc.d/init.d/saslauthd /etc/rc.d/init.d/smartd
/etc/rc.d/init.d/sendmail /etc/rc.d/init.d/smb
/etc/rc.d/init.d/setroubleshoot /etc/rc.d/init.d/sshd
/etc/rc.d/init.d/single /etc/rc.d/init.d/syslog

In this case, the administrator realizes she had simply misnamed the service.
[root@station root]# service smb start
Starting SMB services: [ OK ]
Starting NMB services: [ OK ]

(As a hint, service scripts that manage a single daemon are generally named after the daemon, and include
the trailing “d”. Service scripts that manage multiple daemons (as is the case for smb) generally do not
include the trailing “d”.)

The following table summarizes the commands that should be supported by all service scripts. Some
service scripts support additional commands which are specific to that service.

Table 4.2. Service Script Commands

Command Effect
start Start the service.
stop Stop the service.
status Generate a service appropriate status message. At the least, the status message
states if the service is running, and the process id of any associated daemons.
Often, the status message includes service specific information as well.

rha130-6.1-0 40 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

Command Effect
restart First stop and then start the service.
reload Reinitialize the service in without restarting, if appropriate.
condrestart If the service is already running, restart it, otherwise, do nothing. (This
command is useful in automated scripting.)

Service Runlevel Configuration and chkconfig

Although our service command side trek was important, it did lead us away from our discussion of the
Red Hat Enterprise Linux boot process, to which we would now like to return. When we left, we were
exploring the /etc/rc.d directory.
[root@station root]# ls -F /etc/rc.d/
init.d/ rc0.d/ rc2.d/ rc4.d/ rc6.d/ rc.sysinit*
rc* rc1.d/ rc3.d/ rc5.d/ rc.local*

In particular, we had observed that the various runlevel specific subdirectories contained meticulously
named symbolic links, which referred to service scripts in the /etc/rc.d/init.d directory.
[root@station root]# ls -F /etc/rc.d/rc3.d/
K01certmonger@ K75quota_nld@ S11portreserve@ S26haldaemon@
K01smartd@ K80kdump@ S12rsyslog@ S26udev-post@
K02oddjobd@ K80sssd@ S13cpuspeed@ S28autofs@
K10psacct@ K84wpa_supplicant@ S13irqbalance@ S50bluetooth@
K10saslauthd@ K85ebtables@ S13iscsi@ S55sshd@
K15httpd@ K86cgred@ S13rpcbind@ S80postfix@
...

The name of each symbolic link has the following form.

Lddservice_name

Here, L is either the letter “K” or the letter “S”, dd is a two digit number, and service_name is the
name of the service script to which the symbolic link resolves.

We are now in a position to describe exactly what the script /etc/rc.d/rc does. Recall, whenever
the system switches runlevels (including entering a runlevel on bootup), init calls the rc script, which
performs the following.

1. Except for when first booting, the rc script examines the runlevel specific directory for the runlevel that
is being entered. Any link which starts “K” (for “Kill”), rc will check to see if that service is running.
If it is, the associated service script is called with the stop argument.

2. The rc script again examines the runlevel specific directory of the runlevel which is being entered. Any
link which starts “S” (for “Start”), rc will check to see if that service is running. If it is not, the
associated service script is called with the start argument.

As a result, once init has completed switching runlevels, the state of all services should reflect the state
of the symbolic links in the runlevel specific directory. All services associated with “K” links should be
stopped, and all services associated with “S” links should be started.

What purpose do the numbers serve? They simply provide an ordering to the services as they are started or
stopped. (It doesn't do much good to start the web server (httpd) before the network has been configured
(network)).

By rearranging the links within runlevel specific directories, administrators can change the default
configuration of their machine. For example, the samba service is currently disabled by default in runlevel

rha130-6.1-0 41 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

3. An administrator could cause the service to be started by default in runlevel 3 by somewhat arbitrarily
choosing a start number, and moving the symbolic link.
[root@station root]# cd /etc/rc.d/rc3.d/
[root@station rc3.d]# mv K35smb S95smb

Such mangling of links "by hand" would get the job done, but there is a more elegant way, the chkconfig
command. The chkconfig command exists to help administrators configure runlevel configuration for
services.

When called with --list, the chkconfig command will display all known services, and their current state
for all seven runlevels.
[root@station rc3.d]# chkconfig --list
NetworkManager 0:off 1:off 2:on 3:on 4:on 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:off 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
avahi-daemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
bluetooth 0:off 1:off 2:off 3:on 4:on 5:on 6:off
certmonger 0:off 1:off 2:off 3:off 4:off 5:off 6:off
cgconfig 0:off 1:off 2:on 3:on 4:on 5:on 6:off
...

When called with --list and an argument, only the configuration of the named service is displayed.
[root@station rc3.d]# chkconfig --list smb
smb 0:off 1:off 2:off 3:off 4:off 5:off 6:off

When called with two arguments, the first argument should be the name of a service, and the second
argument either on, off, or reset. For runlevels 3, 4, and 5, chkconfig will update the symbolic links
appropriately.

In the following, the administrator first examines the state of the relevant link in the /etc/rc.d/rc3.d
directory. Then, using chkconfig, the administrator enables the smb service for runlevels 3, 4, and 5, and
reexamines the state of the link.
[root@station rc3.d]# pwd
/etc/rc.d/rc3.d
[root@station rc3.d]# ls *smb*
K35smb
[root@station rc3.d]# chkconfig smb on
[root@station rc3.d]# chkconfig --list smb
smb 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@station rc3.d]# ls *smb*
S91smb

The chkconfig command handled the "swinging" of the symbolic link for the administrator. Likewise, the
smb service is again disabled in the following sequence.
[root@station rc3.d]# chkconfig smb off
[root@station rc3.d]# chkconfig --list smb
smb 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@station rc3.d]# ls *smb*
K35smb

Every service also comes with its own concept of what its default state should be. The network service,
for example, is "by default" enabled for runlevels 2, 3, 4, and 5. In contrast, most network services, such
a the httpd or smb services, are "by default" off for all runlevels. The reset command can be used to
restore the runlevel states for the service to the "default" settings.

rha130-6.1-0 42 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

[root@station rc3.d]# chkconfig smb reset

[root@station rc3.d]# chkconfig --list smb
smb 0:off 1:off 2:off 3:off 4:off 5:off 6:off

Looking Further
The chkconfig command will also allow administrators to configure the states of specific
runlevels. In practice, the default runlevel of a machine is almost always 3 or 5, so the simple on
and off commands will suffice. Consult the chkconfig(8) man page for more information. Also,
note that chkconfig has some knowledge built into it. For example, it knows that the appropriate
start number for the smb service is 91, and the kill number is 35. Again, the man page provides
more details.

Local Startup Configuration

Our discussion of the Red Hat Enterprise Linux bootup process is almost complete. Our last topic is local
customization of the start up process, and specifically the script /etc/rc.d/rc.local. By listing all
instances of the word local from the runlevel specific directories, we can deduce the role of the rc.local
script.
[root@station rc.d]# ls -l /etc/rc.d/rc?.d/*local*
lrwxrwxrwx 1 root root 11 Apr 1 2003 /etc/rc.d/rc2.d/S99local -> ../rc.local
lrwxrwxrwx 1 root root 11 Apr 1 2003 /etc/rc.d/rc3.d/S99local -> ../rc.local
lrwxrwxrwx 1 root root 11 Apr 1 2003 /etc/rc.d/rc4.d/S99local -> ../rc.local
lrwxrwxrwx 1 root root 11 Apr 1 2003 /etc/rc.d/rc5.d/S99local -> ../rc.local

When entering a runlevel upon bootup, the rc.local script is executed. By default, the script is empty. Any
local customization may be added to this script. Any configuration added to rc.local will execute after
every other service has been started. By adjusting the start number of the S99local symbolic link, this
behavior can be modified appropriately.

Summary
In closing, the following advice is offered. Don't let the depth of the detail presented in this lesson obscure
the simplicity of Red Hat Enterprise Linux service management. Arguably the most important system
administration skills covered in this entire workbook are embodied in the following two commands:
service and chkconfig.

Use the service command to directly manipulate a service, as in the following commands.
[root@station rc.d]# service vsftpd start
Starting vsftpd for vsftpd: [ OK ]
[root@station rc.d]# service vsftpd stop
Shutting down vsftpd: [ OK ]
[root@station rc.d]# service vsftpd status
vsftpd is stopped

Use the chkconfig to set the default state of a service upon booting, as in the following commands.
[root@station rc.d]# chkconfig vsftpd on
[root@station rc.d]# chkconfig vsftpd off
[root@station rc.d]# chkconfig --list vsftpd
vsftpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off

Often, upon installing software for a new service, the two are used in combination to start the service
immediately, and by default upon subsequent reboots.
[root@station rc.d]# service smb start
Starting SMB services: [ OK ]

rha130-6.1-0 43 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

Starting NMB services: [ OK ]

[root@station rc.d]# chkconfig smb on

Online Exercises
Lab Exercise
Objective: Effectively administer Red Hat Enterprise Linux services.

Estimated Time: 15 mins.

Specification
1. Assume that you anticipate no future changes to the hardware on your machine, and that you would like
to decrease your machine's startup time. Change the runlevel configuration for the bluetooth service so
that it is disabled for all runlevels.

2. Immediately disable the rpcbind service (but do not change its runlevel defaults).

3. Manually disable the rhnsd service for runlevel 4, by renaming the appropriate symbolic link
K01rhnsd.

Deliverables
1.
1. A runlevel configuration which disables the bluetooth service for all runlevels.

2. A rpcbind service which is currently not running.

3. A runlevel configuration which disables the rhnsd service for runlevel 4 with a "stop number"
of 01.

Clean Up
Restore the bluetooth, rpcbind, and rhnsd services to their default configuration.

Questions
1. Which of the following is not a standard command for Red Hat Enterprise Linux service scripts?

a. start

b. kill

c. restart

d. status

e. All of the above are standard commands.

2. What runlevel is traditionally referred to as single user mode?

a. 0

rha130-6.1-0 44 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

b. 5

c. 2

d. 6

e. None of the above

Use the following transcript to answer the next 2 questions.

[root@station root]# ls -F /etc/rc.d/rc?.d/*gpm*
/etc/rc.d/rc0.d/K15gpm@ /etc/rc.d/rc3.d/S85gpm@ /etc/rc.d/rc6.d/K15gpm@
/etc/rc.d/rc1.d/K15gpm@ /etc/rc.d/rc4.d/S85gpm@
/etc/rc.d/rc2.d/K15gpm@ /etc/rc.d/rc5.d/S85gpm@

3. Which of the following best describes how the gpm service would respond when changing from
runlevel 2 to runlevel 5?

a. The gpm service, which should not have been running, will continue to not run.

b. The gpm service, which should been running, will be stopped.

c. The gpm service, which should not have been running, will be started.

d. The gpm service, which should have been running, will continue to run.

4. Which of the following best describes how the gpm service would respond when changing from
runlevel 5 to runlevel 6?

a. The gpm service, which should have been running, will be stopped.

b. The gpm service, which should not have been running, will continue to not run.

c. The gpm service, which should have been running, will continue to run.

d. The gpm service, which should not been running, will be started.

5. What directory contains Red Hat Enterprise Linux service scripts?

a. /etc/sysconfig

b. /sbin/init.d

c. /etc/rc.d/rc6.d

d. /var/state/run

e. None of the above

6. Which of the following command lines would immediately start the newly installed httpd service?

a. /sbin/httpd start

b. service httpd start

c. chkconfig httpd on

d. service httpd on

e. None of the above

rha130-6.1-0 45 Copyright 2011, Red Hat Inc.

 Red Hat Enterprise
Linux Service Scripts

7. Which of the following command lines would configure the newly installed httpd service to start
automatically upon reboots?

a. /etc/rc.d/init.d/httpd on

b. /sbin/httpd on

c. chkconfig httpd on

d. chkconfig httpd start

e. None of the above

8. Which of the following command lines would list the runlevel configuration for all services?

a. service --all

b. service --list

c. chkconfig --list

d. chkconfig --all

e. None of the above

9. Which of the following command lines is equivalent to the command line service smb
status?

a. /sbin/smbd status

b. /etc/rc.d/smb/status

c. /etc/rc.d/rc3.d/smb/status

d. /etc/rc.d/init.d/smb status

e. None of the above

10. Which of the following commands would reinitialize the crond daemon, but not kill it and start
a new one?

a. chkconfig crond restart

b. service crond restart

c. chkconfig crond reload

d. service crond reload

e. None of the above

rha130-6.1-0 46 Copyright 2011, Red Hat Inc.

Chapter 5. Troubleshooting the Boot
Sequence
Key Concepts
• By observing the stream of boot messages, administrators can often determine the nature of a machine's
misconfiguration.

• By appending a 1 to the kernel command line, a machine can be booted into a state where the filesystem
is assembled, but no services are started.

• By appending a init=/bin/sh to the kernel command line, a machine can be booted into a state
where there is an interactive shell, but no configuration files have been referenced.

Discussion
The Services Graphical Application
In the previous Lesson, the services and chkconfig commands were introduced as the preferred technique
for managing Red Hat Enterprise Linux services. A graphical utility also exists, which can be accessed
from the System → Administration → Server Settings → Services menu item, or from the command line
as system-config-services.

Figure 5.1. The Graphical Service Configuration Utility

The Start, Stop, and Restart buttons perform the analogous service functions, while the Enable and Disable
buttons are used to set runlevel specific default behavior (similarly to chkconfig). If you want to customize
which runlevels are enabled, use the Customize button. A brief description of the service, and its current
status, are provided in text panes.

Managing Virtual Consoles

Virtual consoles were mentioned in a previous Lesson, so we take this opportunity to collect information
about managing virtual consoles in one place.

rha130-6.1-0 47 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

Because the upstart event handler starts 6 gettys for each of the first 6 virtual consoles, people commonly
speak of Linux "having" 6 virtual consoles. Usually, Linux provides as many as 12 virtual consoles, and
sometimes more. The 6 virtual consoles which do not allow logins may be used for running additional
X servers and displaying the output of other miscellaneous commands. The nth console can be accessed
through the /dev/ttyn device node.

As an example, the following command line would cause the 10th virtual console to continuously display
the last few lines of the /var/log/messages file.
[root@station root]# tail -f /var/log/messages > /dev/tty10 &

Like the first 6 virtual consoles, the higher virtual consoles can be viewed with the ALT+Fn key sequence,
where each function key maps to the corresponding virtual console number n. When switching out of the
X graphical environment, the ALT+Fn key sequence was thought to be too common, so an additional
CTRL needs to be added. If you have trouble remembering this distinction, CTRL+ALT+Fn should
always work.

The following table lists commonly used key sequences for managing virtual consoles.

Table 5.1. Virtual Console Key Sequences

Key Sequence Use
ALT+Fn Switch to virtual console number n.
CTRL+ALT+Fn Switch to virtual console number n from within an
X environment.
ALT+LEFTARROW, ALT+RIGHTARROW Switch to a neighboring console.
SHIFT+PAGEUP, SHIFT+PAGEDOWN Scroll through the virtual console's history. (The
history is lost when the console is switched).

Observing the Boot Sequence

Sometimes, machines can be misconfigured to the point that they have trouble booting. Recognizing where
in the boot process trouble occurs can help diagnose problems. Accordingly, Red Hat Enterprise Linux
system administrators should be able to recognize various phases of the boot process from the stream of
messages admitted upon bootup.

Starting with Red Hat Enterprise Linux 4, determining problems at boottime has gotten a little trickier,
because the kernel boottime parameters quiet and rhgb are specified by default. If your system has trouble
booting, these kernel command line parameters should be removed (either by editing /etc/grub.conf,
or more likely by overriding the kernel command line at bootup using GRUB).

Here, we reproduce the key components of a sample bootup sequence (assuming no quiet and rhdb kernel
boot parameters), and identify some of the key phases. We begin with the GRUB bootloader. If you are
quick, you can catch a message akin to the following as BIOS passes control to GRUB.
GRUB Loading Stage2

Very quickly, however, the message is replaced with the GRUB stage2 application menu. Once a menu
item has been selected, the following is displayed to the screen.
Booting 'Red Hat Enterprise Linux (2.6.18-8.el5.EL)'
root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /vmlinuz-2.6.18-8.el5.EL ro root=LABEL=/ rhgb quiet
[Linux-bzImage, setup=0x1400, size=0x113738]
initrd /initrd-2.6.18-8.el5.EL.img

rha130-6.1-0 48 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

[Linux-initrd @ 0x3cb000, 0x24881 bytes]

Again, you must be quick. Most likely, the only time you will catch this information is if something went
wrong (such as GRUB could not locate the kernel, or could not locate the initial ramdisk), because the
messages are very quickly replaced with startup messages from the kernel as it loads.

Initializing cgroup subsys cpuset

Initializing cgroup subsys cpu
Linux version 2.6.32-131.0.15.el6.x86_64 ([email protected]) (gcc version 4.4.4 20
Command line: ro root=UUID=9b85eb65-e07d-46d7-a1ea-10cd0710a56e rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
...

We cut away the majority of the kernel messages. (The interested can peruse them in the /var/log/
dmesg file.) We pick back up again as the kernel is mounting its initial ramdisk, mounting the root
partition, and starting the init process.
...
EXT4-fs (sda2): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/sda2
dracut: Loading SELinux policy
type=1404 audit(1312827110.157:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
type=1403 audit(1312827111.522:3): policy loaded auid=4294967295 ses=4294967295
dracut:
dracut: Switching root
readahead-collector: starting

At this point, init has started, and the event handler (upstart) launches the rc.sysinit script. The script very
quickly silences kernel messages, and begins a more decorative startup sequence, loading a default font,
displaying a title, and then proceeding to initialize the filesystem.

Welcome to Red Hat Enterprise Linux Server

Starting udev: udev: starting version 147
sd 0:0:0:0: Attached scsi generic sg0 type 0
...
Setting hostname localhost.localdomain: [ OK ]
Setting up Logical Volume Management: No volume groups found [ OK ]
Checking filesystems
Checking all file systems.

Welcome to Red Hat Enterprise Linux server

Press 'I' to enter interactive startup.
Setting clock (localtime): Wed Aug 14 14:32:45 EST 2007 [ OK ]
Starting udev: [ OK ]
Loading default keymap (us): [ OK ]
Setting hostname station.example.com: [ OK ]
Setting up Logical Volume Management: No volume groups found
[ OK ]
Checking filesystems
[/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a /dev/sda2
/dev/sda2: clean, 5808/516096 files, 115894/2060544 blocks
[/sbin/fsck.ext4 (1) -- /boot] fsck.ext4 -a /dev/sda1
/dev/sda1: clean, 38/38456 files, 31561/153600 blocks
[/sbin/fsck.ext4 (1) -- /home] fsck.ext4 -a /dev/sda6
/dev/sda6: clean, 11/512064 files, 69413/2048000 blocks
[ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: EXT4-fs (sda1): mounted filesystem with ordered data mode
EXT4-fs (sda6): mounted filesystem with ordered data mode

rha130-6.1-0 49 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

Enabling local filesystem quotas: [ OK ]

At this point, an important event in the life of the filesystem has occurred: the root partition is now mounted
read-write, and the remaining local partitions have been mounted.
Enabling local filesystem quotas: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
Entering non-interactive startup

At this point, rc.sysinit has finished, and upstart is beginning runlevel specific configuration. The
following lines reflect the various service scripts from the /etc/rc.d/init.d directory starting up,
one at a time.
Entering non-interactive startup
Starting background readahead: [ OK ]
Checking for hardware changes [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]

Now network interfaces should be active, and network related services can be started.
...
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting portmap: [ OK ]
Starting setroubleshootd: [ OK ]
Starting NFS statd: [ OK ]
Starting RPC idmapd: [ OK ]
Starting keytable: [ OK ]
Starting system message bus: [ OK ]
Starting Bluetooth services: [ OK ]
Mounting other filesystems: [ OK ]

What "other filesystems" are left to be mounted? Network related filesystems, such as NFS or SMB
imported shares.
...
Starting up APM daemon: [ OK ]
Starting automount: No Mountpoints Defined [ OK ]
Starting sshd: [ OK ]
Starting xinetd: [ OK ]
...

The startup messages should now disappear, to be replaced by either a text virtual console with a login
prompt (for runlevel 3), or a graphical login display (for runlevel 5).

When all goes well, these messages may be safely ignored. When a machine is misconfigured, however,
carefully observing where in the boot process trouble started occurring can go a long way to helping
diagnose the problem.

Modifying the Boot Sequence with the Kernel Command

Line
In the Lesson on the GRUB bootloader, we mentioned the role of the kernel command line, and how
to modify the command line while booting the machine. Now that a complete picture of the Red Hat
Enterprise Linux boot sequence has been discussed, we return to the topic.

Upon startup, the kernel examines the kernel command line, and responds appropriately to any recognized
arguments. Any arguments that the kernel does not recognize, it passes up to the first process. If the
unrecognized argument has the form name=value, it is passed as an environment variable. Otherwise,
the unrecognized argument is passed as an argument to the first process.

rha130-6.1-0 50 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

Sometimes, systems become misconfigured so that they cannot boot properly. In such situations, the kernel
command line can be used to boot the machine to a reduced level, allowing the system administrator to
troubleshoot problems. The following two scenarios can be used to salvage misconfigured machines in
a majority of cases.

Booting into Runlevel 1

When misconfiguration occurs at late stages of the boot process, such as with network configuration or
user authentication, the system can be booted into a very usable state by bypassing the default runlevel,
and booting directly to single user mode. This is accomplished by merely appending a “1” to the kernel
command line. The kernel, not recognizing the argument “1”, passes the argument on to the init process.
What happens when init is called with an argument of “1”? It boot into runlevel 1.

The init process will start, the rc.sysinit script will run, and upon entering runlevel 1, the user will
be dropped into a root shell. Because the rc.sysinit script executed, the filesystem should be fully
reconstructed. From here, the system can be investigated, and hopefully the problem can be fixed. When
the problem is identified and fixed, the boot process can be continued merely by switching into a higher
runlevel, with init 5, for example.

This trick is particularly helpful for recovering machines for which the root password has been forgotten.

Bypassing /sbin/init
Sometimes, however, problems can occur earlier in the boot process. For example, what if the /etc/
fstab file is mangled? or the /etc/inittab file? or the script /etc/rc.d/rc.sysinit? In
such cases, booting to runlevel 1 is not a solution. The problems are occurring before init even considers
runlevels.

In such situations, the kernel can be asked to bypass /sbin/init, and run an interactive shell as its initial
process. This is accomplished by adding the argument init=/bin/sh to the kernel command line.
The init= boot parameter asks the kernel to run some command other then /sbin/init as the initial
process, and the kernel behaves accordingly.

What happens upon booting? At the point in the kernel startup process where init would start, it doesn't,
and instead the user is dropped into an interactive shell.
sh-3.1#

Recall the state of the filesystem: only the root partition is mounted, and it is mounted read-only. In order
to make changes, the root partition must be manually remounted read-write.
sh-3.1# mount -o remount,rw /
EXT FS on sda2, internal journal
sh-3.1#

At this point, the user should be in a fairly comfortable environment for troubleshooting. Remaining
filesystems can be mounted with mount -a . Once any problems have been identified and fixed, how
does the user exit this minimal environment? Once approach might be to exit.
sh-3.1# exit
exit
Kernel Panic: Attempted to kill init!

At this point, the user's keyboard LED's will be flashing (a characteristic of a crashed Linux kernel), and
otherwise the user is dead in the water. What happened? The interactive /bin/sh process was process
id number 1, and the kernel doesn't like process id number 1 to die.

A more gracefully solution is the following. First, stop any processes (other than your shell). Then
deconstruct your filesystem by reversing the previous process.

rha130-6.1-0 51 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

sh-3.1# umount -a
sh-3.1# mount -o remount,ro /
sh-3.1#

At this point, the system is at the point the kernel would have left it at: only one process is running, and
the root filesystem is mounted read-only. Have your shell become the /init process, by "exec"ing /sbin/
init.
sh-3.1# exec /sbin/init
...

Welcome to Red Hat Enterprise Linux Server

...

Your system should startup cleanly as if nothing had happened.

The Kernel's Search for an Initial Process

If for some reason the kernel cannot find or execute /sbin/init, the kernel next looks for /bin/sh,
and a few other predefined executables. Finally, if the kernel exhausts its list of candidates without finding
a runnable initial process, it dies with the following message.
Kernel panic. No init found. Try passing init= option to kernel.

Online Exercises
Lab Exercise
Objective: Gain competence in overriding the default boot sequence.

Estimated Time: 30 mins.

Specification
1. As a precaution, copy your /etc/fstab file to the file /etc/fstab.prelab4.5. Edit your
original file, and comment out the line defining the mount point for the root partition.

2. Reboot your machine. At the GRUB boot prompt, append the parameter init=/bin/sh to the kernel
command line.

3. Your system should continue to boot, but quickly drop you into a maintenance shell. In the maintenance
shell, perform the following steps.

a. Remount the root partition read-write.

b. Edit the /etc/fstab file, and uncomment the line you commented out at the beginning of the lab.

c. Remount the root partition read-only.

d. Replace your shell with the init process, telling init to bring your machine to runlevel 1, with the
following command line.
init-2.05b# exec /sbin/init 1

4. Your boot process should continue, with init configuring the machine for runlevel 1. Once completed,
init will open another shell, this time with the filesystem fully assembled. From this shell, simply enter
runlevel 5 with the command line init 5.

rha130-6.1-0 52 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

5. At this point, your machine should be fully booted. Log in, and grade your exercise.

Deliverables
1.
1. A /proc/cmdline which includes the argument init=/bin/sh.

2. The runlevel command should report your current runlevel as runlevel 5, and your previous
runlevel as runlevel 1.

Questions
1. What graphical application can be used to configure Red Hat Enterprise Linux services?

a. sysvcfg

b. system-config-chkconfig

c. system-config-services

d. system-config-init

e. None of the above

2. What key sequence can be used to scroll through the history of a virtual console?

a. SHIFT+UPARROW

b. SHIFT+PAGEUP

c. ALT+UPARROW

d. ALT+PAGEUP

e. None of the above

Upon booting, very shortly after selecting the appropriate stanza from the GRUB menu, your system
displays the following message.
Booting 'Red Hat Enterprise Linux (2.6.28-8.el5)'

root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz-2.6.28-8.el5 ro root=LABEL=/

Error 15: File not found

Press any key to continue...

3. Which of the following is the most likely cause of the problem?

a. The /boot partition contains filesystem damage.

b. The /etc/fstab file has a misconfigured entry for the /boot partition.

c. The wrong kernel format has been specified (it should be vmlinux-2.6.28-8.el5).

d. The kernel image has been installed incorrectly (it must reside in the root partition).

rha130-6.1-0 53 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

e. The bootloader has been configured with an incorrect root partition or path to the kernel.

4. Which of the following files will most likely have to be edited to solve the problem?

a. /etc/inittab

b. /etc/rc.d/rc.sysinit

c. /etc/fstab

d. /boot/grub/grub.conf

e. /boot/grub/stage2

Upon booting, you select an entry from from the GRUB menu, and see a series of kernel messages. The
boot sequence ends abruptly at the following point, however.

NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.

RAMDISK: Compressed image found at block 0
Freeing initrd memory: 146k freed
VFS: Mounted root (ext2 filesystem).
Journalled Block Device driver loaded
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
Freeing unused kernel memory: 132k freed
init-2.05b#

You are left at an interactive shell.

5. Which of the following best explains the problem?

a. The GRUB bootloader was configured with the wrong kernel image.

b. For some reason, the kernel could not find or execute its normal first process.

c. The GRUB bootloader could not load the correct initial ram disk.

d. The /etc/inittab file is misconfigured.

e. The /etc/fstab file has a misconfigured entry for the root partition.

6. Which file should be examined and or modified?

a. /etc/inittab

b. /boot/grub/grub.conf

c. /etc/fstab

d. /sbin/init

e. /etc/rc.d/rc.sysinit

Upon booting, you select an entry from from the GRUB menu, and see a series of kernel and init script
messages. The boot sequence ends abruptly at the following point, however.
kjournald starting. Commit interval 5 seconds
Freeing unused kernel memory: 132k freed

rha130-6.1-0 54 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

INIT: version 2.84 booting

Setting default font (latarcyrheb-sun16): [ OK ]

Welcome to Red Hat Enterprise Linux

Press 'I' to enter interactive startup.
Mounting proc filesystem: [ OK ]
Unmounting initrd: [ OK ]
Configuring kernel parameters: [ OK ]
Setting clock (localtime): Fri Nov 14 14:32:45 EST 2003 [ OK ]
Loading default keymap (us): [ OK ]
Setting hostname dhcp63-237.rdu.redhat.com: [ OK ]
Initializing USB controller (usb-uhci): [ OK ]
Mounting USB filesystem: [ OK ]
Initializing USB HID interface: [ OK ]
Initializing USB keyboard: [ OK ]
Initializing USB mouse: [ OK ]
Checking root filesystem
e2fsck 1.32 (09-Nov-2002)
Couldn't find ext2 superblock, trying backup blocks...

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

e2fsck: Bad magic number in super-block while trying to open /dev/hda7

*** An error occurred during the file system check.

*** Dropping you to a shell; the system will reboot
*** when you leave the shell.

Give root password for maintenance

(or type Control-D to continue):

7. What is the most likely problem?

a. The GRUB boot loader specified the incorrect root partition to the kernel.

b. The /etc/fstab file has an incorrect entry referencing the root partition.

c. The /etc/rc.sysinit script has become mangled.

d. The /etc/inittab file is misconfigured, such that the /etc/rc.d/rc.sysinit script

did not execute at the correct time.

e. The GRUB boot loaded did not successfully load the initial ramdisk.

8. Which file would you most likely have to edit to solve this problem?

a. /etc/inittab

b. /etc/rc.d/rc.sysinit

c. /boot/grub/grub.conf

d. /etc/fstab

e. None of the above

Upon booting, you select an entry from from the GRUB menu, and see a series of kernel and init script
messages. The boot sequence is littered with messages such as the following interspersed with failed
service script starts, and eventually the system hangs.

rha130-6.1-0 55 Copyright 2011, Red Hat Inc.

 Troubleshooting the Boot Sequence

Unable to remove file /var/lib/rpm/__db.001: Read-only Filesystem

Unable to remove file /var/lib/rpm/__db.002: Read-only Filesystem
Unable to remove file /var/lib/rpm/__db.003: Read-only Filesystem

9. What is the most likely cause of the problem?

a. The kernel mounted the wrong partition as the root partition.

b. The bootloader could not load the kernel.

c. The /etc/rc.d/rc.sysinit was mangled, and could not execute.

d. The /var partition was mounted read-only.

e. The wrong filesystem was mounted to /var.

10. Which of the following files would most likely need to be edited to fix the problem?

a. /boot/grub/grub.conf

b. /etc/inittab

c. /etc/fstab

d. /etc/rc.d/rc.sysinit

e. /proc/mounts

rha130-6.1-0 56 Copyright 2011, Red Hat Inc.