DevOps Shack

LINUX TROUBLESHOOTING GUIDE

1. System Boot Issues

1.1 Boot Loader Issues (GRUB Issues)

Symptoms:

●​ System does not boot into Linux.

●​ Stuck at GRUB prompt.
●​ Error: grub-rescue>

Causes:

●​ Corrupt or missing GRUB configuration.

●​ Wrong boot device.
●​ Kernel update failure.

Solution:

1.​ Boot using a Live CD/USB.

Mount the root partition:​

sudo mount /dev/sdXn /mnt
Reinstall GRUB:​
sudo grub-install --root-directory=/mnt /dev/sdX
Update GRUB:​
sudo update-grub

1.2 Kernel Panic​

​
​
​

Symptoms:

●​ Black screen with panic messages.

●​ Kernel panic – not syncing error.

Causes:

●​ Incompatible kernel update.

●​ Corrupt boot files.

Solution:

1.​ Reboot and select an older kernel from GRUB.

Check logs using:​

journalctl -xb

2.​ Roll back the kernel update if necessary.

2. Login and User Issues

2.1 Forgotten Root Password

Symptoms:

●​ Cannot log in as root.

Solution:

1.​ Boot into recovery mode.

Remount root partition:​

mount -o remount,rw /​
​
​
​
​
​

Reset password:​
passwd root

2.​ Reboot.

2.2 SSH Connection Refused

Symptoms:

●​ Connection refused when trying to SSH.

Causes:

●​ SSH service not running.

●​ Firewall blocking SSH.

Solution:

Check SSH status:​

sudo systemctl status sshd
Start service if stopped:​
sudo systemctl start sshd
Check firewall rules:​
sudo ufw allow ssh

3. File System and Disk Issues

3.1 Disk Space Full

Symptoms:

●​ No space left on device error.

Solution:​
​
​
​

Check disk usage:​

df -h
Find large files:​
du -ah / | sort -rh | head -20

1.​ Delete unnecessary files.

3.2 Read-Only File System

Symptoms:

●​ Read-only file system error.

Solution:

Remount the filesystem:​

sudo mount -o remount,rw /
Check and repair disk errors:​
sudo fsck -y /dev/sdXn

4. Process and Performance Issues

4.1 High CPU Usage

Symptoms:

●​ System slows down.

●​ High load average in top output.

Solution:

Identify culprit process:​

top
 ​
​

Kill process:​
kill -9 <PID>

5. Networking Issues

5.1 No Internet Connection

Solution:

Check network interfaces:​

ip a
Restart networking:​
sudo systemctl restart NetworkManager

6. Package Management Issues

6.1 Broken Package Dependencies

Symptoms:

●​ Unable to correct problems, you have held broken

packages
●​ Package installation fails

Solution:

Identify broken packages:​

sudo apt --fix-broken install
Clean package cache:​
sudo apt clean && sudo apt autoremove
Reinstall dependencies:​
​
​
​
​
​
sudo apt install -f

6.2 Slow Package Installation

Symptoms:

●​ Installing or updating packages takes too long

Solution:

Update repository lists:​

sudo apt update
Switch to a faster mirror:​
sudo sed -i
's/https:\/\/round-lake.dustinice.workers.dev:443\/http\/archive.ubuntu.com/https:\/\/round-lake.dustinice.workers.dev:443\/http\/mirror.example.com/'
/etc/apt/sources.list

7. Service and Daemon Issues

7.1 Service Not Starting

Symptoms:

●​ Failed to start service-name.service

Solution:

Check service logs:​

sudo journalctl -xe | grep service-name
Restart the service:​
sudo systemctl restart service-name

1.​ Check configuration files for errors.

​

7.2 Port Conflicts

Symptoms:

●​ Service fails to bind to a port

Solution:

Identify the conflicting process:​

sudo netstat -tulnp | grep <port-number>

1.​ Kill the process or reconfigure services.

8. Kernel and Module Issues

8.1 Kernel Update Breaks System

Symptoms:

●​ System does not boot after update

Solution:

1.​ Reboot and select an older kernel in GRUB.

Remove faulty kernel:​

sudo apt remove linux-image-<version>

8.2 Missing Kernel Modules

Symptoms:

●​ modprobe: FATAL: Module not found

Solution:​
​
​

Check available modules:​

lsmod | grep module-name
Load the module manually:​
sudo modprobe module-name

9. Security and Access Control Issues

9.1 Unauthorized Access Attempts

Symptoms:

●​ Unusual login attempts in logs

Solution:

Check logs:​
sudo cat /var/log/auth.log | grep "Failed password"
Block IP using firewall:​
sudo ufw deny from <IP-ADDRESS>

9.2 SELinux Blocking Services

Symptoms:

●​ Service fails due to permission errors

Solution:

Check SELinux logs:​

​
sudo ausearch -m AVC
Temporarily disable SELinux (for debugging only):​
sudo setenforce 0
10. Hardware Issues

10.1 USB Device Not Recognized

Symptoms:

●​ USB device does not show up in lsusb

●​ dmesg shows errors related to USB

Solution:

Check if the USB device is detected:​

​
Lsusb​

Re-scan USB devices:​

​
​
echo "1" | sudo tee /sys/bus/usb/devices/usb1/authorized​

Restart the USB service:​

​
sudo modprobe -r usb_storage && sudo modprobe usb_storage

10.2 Hard Drive Not Detected

Symptoms:

●​ fdisk -l does not show the disk

●​ dmesg shows disk-related errors

Solution:

List available disks:​

​
​
​
​
lsblk
Re-scan the disk:​
​
echo "- - -" | sudo tee /sys/class/scsi_host/host0/scan

11. Time and Date Issues

11.1 System Clock Out of Sync

Symptoms:

●​ Incorrect system time

Solution:

Sync time with NTP:​

​
​
sudo timedatectl set-ntp on

Manually set time:​

​
​
sudo timedatectl set-time "YYYY-MM-DD HH:MM:SS"

12. Firewall and Network Security Issues

12.1 Firewall Blocking Services

Symptoms:

●​ Cannot access services from another system

Solution:​
​
​

List active firewall rules:​

​
sudo ufw status​

Allow required ports:​

​
sudo ufw allow 80/tcp

12.2 Too Many Failed SSH Attempts

Symptoms:

●​ Too many authentication failures in SSH

Solution:

Edit SSH configuration:​

​
sudo nano /etc/ssh/sshd_config​

Increase allowed authentication retries:​

​
MaxAuthTries 10​

Restart SSH:​
​
sudo systemctl restart sshd

13. Virtualization Issues

13.1 Virtual Machine Won't Start​

​
​
​

Symptoms:

●​ Failed to start the virtual machine error

Solution:

Check virtualization support:​

​
egrep -c '(vmx|svm)' /proc/cpuinfo

1.​ Enable virtualization in BIOS.

14. Miscellaneous Issues

14.1 Command Not Found

Symptoms:

●​ command not found error

Solution:

Check if the package is installed:​

​
dpkg -l | grep package-name​

Install missing package:​

​
sudo apt install package-name

15. Logs and Monitoring Issues​

​
​
​

15.1 Log Files Not Updating

Symptoms:

●​ /var/log/syslog or /var/log/messages is empty or not updating.

Solution:

Restart the logging service:​

​
sudo systemctl restart rsyslog

Check disk space:​

​
df -h

Verify SELinux settings:​

​
sudo ausearch -m AVC

15.2 How to Monitor System Logs in Real-Time

Solution:

Use tail to follow log updates:​

​
tail -f /var/log/syslog
Use journalctl for systemd logs:​
​
journalctl -f
 ​

16. SSH Issues

16.1 SSH Connection Timeout

Symptoms:

●​ SSH session freezes or disconnects after inactivity.

Solution:

Edit SSH config:​

​
​
sudo nano /etc/ssh/sshd_config

Set ClientAliveInterval and ClientAliveCountMax:​

plaintext​
​
ClientAliveInterval 60
ClientAliveCountMax 5

Restart SSH service:​

​
​
sudo systemctl restart sshd

16.2 Cannot SSH as Root

Symptoms:​
​
​

●​ Permission denied when connecting as root.

Solution:

Enable root login in SSH config:​

​
sudo nano /etc/ssh/sshd_config
Set:​
plaintext​
​
PermitRootLogin yes
Restart SSH:​
​
sudo systemctl restart sshd

17. Printer and Device Issues

17.1 Printer Not Working

Symptoms:

●​ Printer not detected.

Solution:

Restart the CUPS service:​

​
​
sudo systemctl restart cups​

Check printer connections:​

​
lpstat -p​
​

18. User Management Issues

18.1 User Cannot Run sudo Commands

Symptoms:

●​ user is not in the sudoers file error.

Solution:

Add user to the sudo group:​

​
sudo usermod -aG sudo username

18.2 User Cannot Write to a Directory

Symptoms:

●​ Permission denied error.

Solution:

Change ownership:​
​
sudo chown username:groupname /path/to/directory

Set proper permissions:​

​
​
sudo chmod 755 /path/to/directory

19. Software and Application Issues​

​
​
​

19.1 Application Crashes Unexpectedly

Solution:

Check application logs:​

​
​
journalctl -xe | grep application-name

1.​ Run application in debug mode if available.

19.2 Web Server Not Working

Symptoms:

●​ 502 Bad Gateway or 503 Service Unavailable.

Solution:

Restart the web server:​

​
sudo systemctl restart apache2 # For Apache
sudo systemctl restart nginx # For Nginx

20. Advanced Security Issues

20.1 Detecting Unauthorized Root Access

Solution:

Check recent root logins:​

​
sudo last -n 10
 ​

Check for unauthorized sudo commands:​

​
cat /var/log/auth.log | grep sudo

20.2 Blocking Brute-Force Attacks

Solution:

Install Fail2Ban:​
​
sudo apt install fail2ban​

Enable SSH protection:​

​
​
sudo systemctl enable fail2ban

21. Disk and Storage Issues

21.1 Unable to Mount a Disk

Symptoms:

●​ mount: unknown filesystem type error

●​ Disk not visible in lsblk or fdisk -l

Solution:

Identify the disk:​

​
lsblk
Check filesystem type:​
​
​
​
​
​
sudo blkid /dev/sdX​

Manually mount the disk:​

​
​
sudo mount /dev/sdX /mnt

If the filesystem is corrupted, repair it:​

​
​
sudo fsck -y /dev/sdX

21.2 LVM Volume Not Found

Symptoms:

●​ lvdisplay does not show expected volumes

Solution:

Scan for missing volumes:​

​
​
sudo vgscan --mknodes
sudo lvscan​

Activate volume group:​

​
​
sudo vgchange -ay
 ​
​

Mount the logical volume:​

​
​
sudo mount /dev/mapper/vgname-lvname /mnt

22. Kernel and Driver Issues

22.1 Missing Graphics Drivers

Symptoms:

●​ GUI does not load

●​ NVIDIA driver not found error

Solution:

Identify the graphics card:​

​
lspci | grep -i vga

Install the correct driver:​

​
​
sudo ubuntu-drivers autoinstall

Reboot the system:​

​
​
sudo reboot
22.2 Sound Not Working

Symptoms:

●​ No audio output
●​ No sound device found error

Solution:

Check audio devices:​

​
​
aplay -l

Restart PulseAudio:​
​
​
pulseaudio --kill && pulseaudio --start

Unmute sound:​
​
​
amixer set Master unmute

23. Network Troubleshooting

23.1 Wi-Fi Not Connecting

Symptoms:​
​
​

●​ No Wi-Fi adapter found error

Solution:

List available network interfaces:​

​
​
nmcli device

Restart NetworkManager:​
​
​
sudo systemctl restart NetworkManager

If the Wi-Fi adapter is missing, install drivers:​

​
​
sudo apt install firmware-iwlwifi

23.2 IP Address Not Assigned

Symptoms:

●​ ip a does not show an IP address

Solution:

Restart DHCP client:​

​
sudo dhclient -r && sudo dhclient​
​
​

Manually assign an IP address:​

​
​
sudo ip addr add 192.168.1.100/24 dev eth0

24. Performance Optimization

24.1 System Running Slow

Symptoms:

●​ High load average

●​ Applications freezing

Solution:

Check system load:​

​
​
top

Kill high CPU usage processes:​

​
​
kill -9 <PID>

Clear cached memory:​

​
​
sudo sync && sudo sysctl -w vm.drop_caches=3
24.2 High Memory Usage

Symptoms:

●​ Out of memory errors

Solution:

Check memory usage:​

​
​
free -h

Enable swap:​
​
​
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

25. Backup and Recovery

25.1 Restoring Deleted Files

Symptoms:

●​ Important files deleted accidentally

Solution:

Recover files from the trash:​

​
​
​
​
ls ~/.local/share/Trash/files/

If permanently deleted, use extundelete:​

​
​
sudo extundelete /dev/sdX --restore-all

25.2 Creating a Full System Backup

Solution:

Use rsync for backup:​

​
​
sudo rsync -av --progress /home/ /backup/home/

Restore files when needed:​

​
​
sudo rsync -av /backup/home/ /home/

26. Advanced Security

26.1 Encrypting a Directory

Solution:

Install ecryptfs-utils:​
​
​
​
​
​
sudo apt install ecryptfs-utils

Encrypt a directory:​
​
​
ecryptfs-mount-private

26.2 Setting Up Automatic Security Updates

Solution:

Install unattended-upgrades:​
​
​
sudo apt install unattended-upgrades

Enable automatic updates:​

​
​
sudo dpkg-reconfigure unattended-upgrades

27. Miscellaneous Troubleshooting

27.1 How to Find Which Process is Using a Port

Solution:

Use netstat:​
​
​
​
​
sudo netstat -tulnp | grep :<port>

Use lsof:​
​
​
sudo lsof -i :<port>

Kill the process if needed:​

​
​
sudo kill -9 <PID>

27.2 How to Check System Uptime

Solution:

Display uptime:​
​
​
uptime

View uptime in human-readable format:​

​
​
who -b

28. Debugging and Log Analysis​

​
​

28.1 Checking Kernel Logs for Errors

Solution:

View logs in real-time:​

​
​
dmesg -w

Filter errors:​
​
​
dmesg | grep -i error

28.2 Finding Recently Modified Files

Solution:

Search for files modified in the last 24 hours:​

​
​
find / -type f -mtime -1

29. Essential Linux Commands

29.1 Finding a File by Name

Solution:

Use find:​
​
​
​
​
​
find / -name "filename"

29.2 Checking Disk Usage per Directory

Solution:

Use du to find large directories:​

​
​
du -ah / | sort -rh | head -10

30. Common Errors and Fixes

30.1 Fixing command not found for Common Commands

Solution:

Refresh the PATH:​

​
​
export PATH=$PATH:/usr/local/bin​

Reinstall the package:​

​
​
sudo apt install package-name

30.2 Fixing permission denied When Running a Script

Solution:​
​
​
​

Make the script executable:​

​
​
chmod +x script.sh

Run the script with sudo if needed:​

​
​
sudo ./script.sh

31. Advanced File System Issues

31.1 Filesystem Goes into Read-Only Mode

Symptoms:

●​ Unable to write files

●​ dmesg shows EXT4-fs error

Solution:

Check disk errors:​

​
​
dmesg | tail -50

Remount the filesystem:​

​
sudo mount -o remount,rw /
 ​

Run filesystem check:​

​
​
sudo fsck -y /dev/sdX

31.2 Recovering Deleted Files from EXT4 Filesystem

Solution:

Unmount the affected partition:​

​
​
sudo umount /dev/sdX

Use extundelete:​
​
​
sudo extundelete /dev/sdX --restore-all

32. Advanced Networking Issues

32.1 Identifying Slow Network Connections

Symptoms:

●​ Websites load slowly

●​ High latency on SSH

Solution:​
​
​

Test network speed:​

​
​
speedtest-cli

Identify packet loss:​

​
​
ping -c 10 google.com

Check for dropped packets:​

​
​
sudo netstat -s | grep "dropped"

32.2 Troubleshooting DNS Issues

Symptoms:

●​ ping google.com fails, but ping 8.8.8.8 works

Solution:

Check DNS settings:​

​
​
cat /etc/resolv.conf
 ​
​

Change to Google DNS:​

​
​
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

33. Docker and Container Issues

33.1 Docker Container Won't Start

Symptoms:

●​ docker run fails with an error

Solution:

Check container logs:​

​
​
docker logs container_id

Restart Docker:​
​
​
sudo systemctl restart docker

Prune unused containers:​

​
docker system prune -a​
​

33.2 Docker Container Can't Access the Internet

Solution:

Restart the Docker network:​

​
​
sudo systemctl restart docker

Add network to the container:​

​
​
docker network connect bridge container_id

34. Kernel Panic and Boot Issues

34.1 Kernel Panic After Update

Symptoms:

●​ System does not boot after an update

Solution:

1.​ Boot into an older kernel from GRUB.

Reinstall the latest kernel:​

​
sudo apt install --reinstall linux-image-$(uname -r)

​
​
​

34.2 System Stuck at GRUB Rescue Mode

Solution:

Identify the boot partition:​

​
​
ls

Load the correct partition:​

​
​
set prefix=(hd0,gpt2)/boot/grub
set root=(hd0,gpt2)
insmod normal
normal

35. High Availability and Load Balancing Issues

35.1 Load Balancer Not Distributing Traffic Evenly

Solution:

Check Nginx load balancer configuration:​

​
​
sudo nano /etc/nginx/nginx.conf​

Verify upstream servers:​

​
​
​
​
sudo systemctl restart nginx

36. Virtualization and Cloud Issues

36.1 VM is Running Slowly

Solution:

Enable CPU virtualization:​

​
​
egrep -c '(vmx|svm)' /proc/cpuinfo

1.​ Increase allocated RAM and CPU in the hypervisor.

36.2 Cloud Instance Fails to Boot

Solution:

1.​ Check the instance console logs.

2.​ Attach the disk to another instance and repair the OS files.

37. SELinux and AppArmor Issues

37.1 Application Blocked by SELinux

Symptoms:

●​ Permission denied errors even with correct permissions

Solution:

Check SELinux logs:​

​
​
​
​
​
sudo ausearch -m AVC

Set the correct SELinux policy:​

​
​
sudo semanage fcontext -a -t httpd_sys_content_t
"/var/www/html(/.*)?"
sudo restorecon -Rv /var/www/html

37.2 AppArmor Blocking a Service

Solution:

Check AppArmor logs:​

​
​
sudo journalctl -xe | grep apparmor

Set AppArmor profile to complain mode:​

​
​
sudo aa-complain /usr/sbin/nginx

38. System Recovery and Rescue Mode

38.1 How to Boot into Single User Mode

Solution:​
​
​

1.​ Edit the GRUB boot entry.

2.​ Add single or init=/bin/ to the kernel parameters.

38.2 Resetting Root Password from Recovery Mode

Solution:

1.​ Boot into recovery mode.

Mount the root filesystem:​

​
​
mount -o remount,rw /

Reset password:​
​
​
passwd root

39. Advanced Performance Tuning

39.1 Identifying Bottlenecks with iostat

Solution:

Install sysstat:​
​
​
sudo apt install sysstat​

Check disk I/O performance:​

​
​
​
​
​
iostat -x 1 5

39.2 Reducing Swap Usage

Solution:

Check current swap usage:​

​
​
swapon -s

Lower the swappiness value:​

​
​
sudo sysctl vm.swappiness=10

40. Debugging and Profiling Applications

40.1 Debugging a Segmentation Fault

Solution:

Run the application with gdb:​

​
​
gdb ./application
run
​

40.2 Profiling CPU Usage with perf

Solution:

Install perf:​
​
​
sudo apt install linux-tools-common

1.​ Profile CPU usage:​

​
​
sudo perf top