# Introduction Welcome to Linuxmaster's docs. # LPIC-2 # 200: Capacity Planning ## 200.1 Measure and Troubleshoot Resource Usage (weight: 6) {% hint style="success" %} Candidates should be able to measure hardware resource and network bandwidth, identify and troubleshoot resource problems. **Key Knowledge Areas:** * Measure CPU usage. * Measure memory usage. * Measure disk I/O. * Measure network I/O. * Measure firewalling and routing throughput. * Map client bandwidth usage. * Match / correlate system symptoms with likely problems. * Estimate throughput and identify bottlenecks in a system including networking. **The following is a partial list of the used files, terms and utilities:** * iostat * iotop * vmstat * netstat * ss * iptraf * pstree, ps * w * lsof * top * htop * uptime * sar * swap * processes blocked on I/O * blocks in * blocks out {% endhint %} ### Measuring System Resource Usage * `iostat`: Report on CPU statistics and input/output statistics for devices and partitions. Provided by the `sysstat` package. By default, the command generates two reports: CPU and device utilization. ```bash iostat [OPTIONS] [DEVICE] [INTERVAL [COUNT]] ``` | Option | Purpose | | ------ | ----------------------------------------- | | `-c` | Display CPU utilization | | `-d` | Display device utilization report | | `-h` | Display in human readable format | | `-m` | Display stats in megabytes per second | | `-N` | Display the registered device mapper name | | `-t` | Print the time for each report | | `-x` | Display extended statistics | | `-y` | Omit the first report | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# iostat Linux 3.10.0-1160.76.1.el7.x86_64 (linuxmaster) 09/15/2022 _x86_64_(1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 9.47 0.00 7.00 0.98 0.00 82.55 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 60.79 1802.02 23505.63 210728 2748748 [root@linuxmaster ~]# iostat -c Linux 3.10.0-1160.76.1.el7.x86_64 (linuxmaster) 09/15/2022 _x86_64_(1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 2.92 0.00 2.29 0.30 0.00 94.49 [root@linuxmaster ~]# iostat -c -y 1 4 Linux 3.10.0-1160.76.1.el7.x86_64 (linuxmaster) 09/15/2022 _x86_64_(1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.00 0.00 0.00 0.00 0.00 99.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 ``` {% endcode %} * `sar`: Collect, report, or save system activity information. Provided by the sysstat command. sar creates a binary file for each day of the month with all the information collected (e.g., sa01), as well as a daily summary (e.g.,sar01). These files are overwritten the following month. The sysstat service must be started for sar to start collecting data and the duration of collection can be changed by modifying `/etc/cron.d/sysstat`. ```bash sar [OPTIONS] [INTERVAL [COUNT ]] ``` | Option | Purpose | | ------ | -------------------------------------------- | | `-B` | Report paging statistics | | `-b` | Report I/O and transfer rate statistics | | `-d` | Report activity for each block device | | `-f` | Extract records from a filename | | `-n` | Report network statistics based on a keyword | | `-o` | Save statistics to a binary file | | `-r` | Report memory utilization statistics | | `-u` | Report CPU utilization | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# systemctl status sysstat ● sysstat.service - Resets System Activity Logs Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor preset: enabled) Active: active (exited) since Thu 2022-09-15 11:34:53 UTC; 4min 16s ago Process: 1524 ExecStart=/usr/lib64/sa/sa1 --boot (code=exited, status=0/SUCCESS) Main PID: 1524 (code=exited, status=0/SUCCESS) Sep 15 11:34:53 linuxmaster systemd[1]: Starting Resets System Activity Logs... Sep 15 11:34:53 linuxmaster systemd[1]: Started Resets System Activity Logs. [root@linuxmaster ~]# sar Linux 3.10.0-1160.76.1.el7.x86_64 (linuxmaster) 09/15/2022 _x86_64_ (1 CPU) 11:34:53 AM LINUX RESTART 11:40:01 AM CPU %user %nice %system %iowait %steal %idle 11:50:01 AM all 0.21 0.00 0.20 0.01 0.00 99.58 12:00:01 PM all 0.43 0.00 0.32 0.03 0.00 99.23 12:10:01 PM all 0.20 0.00 0.17 0.02 0.00 99.62 12:20:01 PM all 0.37 0.00 0.24 0.02 0.00 99.37 Average: all 0.30 0.00 0.23 0.02 0.00 99.45 [root@linuxmaster ~]# sar -y 1 5 Linux 3.10.0-1160.76.1.el7.x86_64 (linuxmaster) 09/15/2022 _x86_64_ (1 CPU) 11:39:27 AM TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s 11:39:28 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 11:39:28 AM TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s 11:39:29 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 11:39:29 AM TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s 11:39:30 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 11:39:30 AM TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s 11:39:31 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 11:39:31 AM TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s 11:39:32 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 Average: TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 [root@linuxmaster ~]# sar -d Linux 3.10.0-1160.76.1.el7.x86_64 (linuxmaster) 09/15/2022 _x86_64_ (1 CPU) 11:34:53 AM LINUX RESTART 11:40:01 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 11:50:01 AM dev8-0 0.61 0.08 10.49 17.36 0.00 0.57 0.28 0.02 12:00:01 PM dev8-0 1.01 2.30 24.28 26.24 0.00 0.85 0.30 0.03 12:10:01 PM dev8-0 0.59 0.29 10.25 17.95 0.00 0.44 0.31 0.02 12:20:01 PM dev8-0 1.04 1.41 21.38 21.97 0.00 2.10 0.30 0.03 Average: dev8-0 0.81 1.02 16.60 21.71 0.00 1.12 0.30 0.02 ``` {% endcode %} ```bash [root@linuxmaster sa]# cat /etc/cron.d/sysstat # Run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib64/sa/sa1 1 1 # 0 * * * * root /usr/lib64/sa/sa1 600 6 & # Generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A ``` * `free`: Display the amount of free and used memory in the system (default is Kb). The information displayed by free is pulled from the /proc/meminfo file. Swap is a substitute for physical memory that uses the hard disk. The /proc/sys/vm/swappiness file determines how aggressively a system will swap. ```bash free [OPTIONS] ``` | Option | Purpose | | ------ | ----------------------------------------------- | | `-m` | Display memory in megabytes | | `-g` | Display memory in gigabytes | | `-h` | Display output in human-readable format | | `-w` | Switch to wide mode | | `-s` | Continually display output in seconds specified | | `-c` | Set a count for the -s option | ```bash [root@linuxmaster ~]# free total used free shared buff/cache available Mem: 1832812 96900 1379484 8684 356428 1585340 Swap: 0 0 0 [root@linuxmaster ~]# free -h total used free shared buff/cache available Mem: 1.7G 94M 1.3G 8.5M 348M 1.5G Swap: 0B 0B 0B [root@linuxmaster sa]# free -h -s 1 -c 3 total used free shared buff/cache available Mem: 1.7G 96M 1.3G 8.5M 348M 1.5G Swap: 0B 0B 0B total used free shared buff/cache available Mem: 1.7G 96M 1.3G 8.5M 348M 1.5G Swap: 0B 0B 0B total used free shared buff/cache available Mem: 1.7G 96M 1.3G 8.5M 348M 1.5G Swap: 0B 0B 0B ``` * `vmstat`: Reports virtual memory statistics as well as statistics on paging and block I/O. By default, vmstat prints a summary of usage statistics since the last reboot. vmstat can run at an interval with a count or continuously. The first report in the interval is the summary report. ```bash vmstat [OPTIONS] [DELAY [COUNT]] ``` | Option | Purpose | | ------ | ------------------------------------ | | `-a` | Display active and inactive memory | | `-s` | Display a table of memory statistics | | `-d` | Report disk statistics | | `-D` | Report summary statistics on disks | | `-t` | Append a timestamp to each line | | `-w` | Wide output mode | ```bash [root@linuxmaster ~]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 1379508 13376 343080 0 0 735 467 132 200 4 3 93 0 0 [root@linuxmaster ~]# vmstat 1 4 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 1378676 13728 343292 0 0 226 146 54 82 1 1 98 0 0 0 0 0 1378676 13728 343292 0 0 0 0 17 28 0 0 100 0 0 0 0 0 1378676 13728 343292 0 0 0 0 17 29 0 0 100 0 0 0 0 0 1378676 13728 343292 0 0 0 0 11 21 0 0 100 0 0 [root@linuxmaster ~]# vmstat -d 1 4 disk- ------------reads------------ ------------writes----------- -----IO------ total merged sectors ms total merged sectors ms cur sec sr0 0 0 0 0 0 0 0 0 0 0 sda 5820 8 422073 3096 1989 2803 5547568 9038 0 2 sr0 0 0 0 0 0 0 0 0 0 0 sda 5820 8 422073 3096 1991 2806 5547608 9040 0 2 sr0 0 0 0 0 0 0 0 0 0 0 sda 5820 8 422073 3096 1991 2806 5547608 9040 0 2 sr0 0 0 0 0 0 0 0 0 0 0 sda 5820 8 422073 3096 2003 2808 5547720 9052 0 2 [root@linuxmaster ~]# vmstat -s 1832812 K total memory 99164 K used memory 268444 K active memory 110020 K inactive memory 1376568 K free memory 13768 K buffer memory 343312 K swap cache 0 K total swap 0 K used swap 0 K free swap 1239 non-nice user cpu ticks 0 nice user cpu ticks 986 system cpu ticks 96054 idle cpu ticks 125 IO-wait cpu ticks 0 IRQ cpu ticks 13 softirq cpu ticks 0 stolen cpu ticks 211036 pages paged in 136768 pages paged out 0 pages swapped in 0 pages swapped out 51272 interrupts 78430 CPU context switches 1663241496 boot time 1637 forks ``` ### Viewing Open Files and Running Processes * `lsof`: Lists open files on the system. By default, the command will display all open files belonging to all active processes. When multiple options are used, it will display open files that meet any of the criteria (ORing), but the -a option can be used to AND the selections (ANDing). ```bash lsof [OPTIONS] [NAMES] ``` | Option | Purpos | | ------ | ----------------------------------------------------------------- | | `-a` | Causes selections to be ANDed | | `-c` | List files based on command entered | | `-i` | List files based on a network connection | | `+d` | List files associated with a directory | | `+D` | List files associated with a directory (includes sub directories) | | `-p` | List files based on PID | | `-t` | Display terse output (PIDs only) | | `-u` | List files based on User | | `^` | Used to negate a selection entered | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# lsof | head COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd 1 root cwd DIR 8,1 4096 2 / systemd 1 root rtd DIR 8,1 4096 2 / systemd 1 root txt REG 8,1 1632960 12858 /usr/lib/systemd/systemd systemd 1 root mem REG 8,1 20064 5240 /usr/lib64/libuuid.so.1.3.0 systemd 1 root mem REG 8,1 265576 10126 /usr/lib64/libblkid.so.1.1.0 systemd 1 root mem REG 8,1 90160 5211 /usr/lib64/libz.so.1.2.7 systemd 1 root mem REG 8,1 157440 5229 /usr/lib64/liblzma.so.5.2.2 systemd 1 root mem REG 8,1 23968 5585 /usr/lib64/libcap-ng.so.0.0.0 systemd 1 root mem REG 8,1 19896 5553 /usr/lib64/libattr.so.1.1.0 [root@linuxmaster ~]# lsof -u ^root | head COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME polkitd 540 polkitd cwd DIR 8,1 4096 2 / polkitd 540 polkitd rtd DIR 8,1 4096 2 / polkitd 540 polkitd txt REG 8,1 120432 13264 /usr/lib/polkit-1/polkitd polkitd 540 polkitd mem REG 8,1 61560 4728 /usr/lib64/libnss_files-2.17.so polkitd 540 polkitd mem REG 8,1 68192 5318 /usr/lib64/libbz2.so.1.0.6 polkitd 540 polkitd mem REG 8,1 99944 5540 /usr/lib64/libelf-0.176.so polkitd 540 polkitd mem REG 8,1 19896 5553 /usr/lib64/libattr.so.1.1.0 polkitd 540 polkitd mem REG 8,1 20064 5240 /usr/lib64/libuuid.so.1.3.0 polkitd 540 polkitd mem REG 8,1 265576 10126 /usr/lib64/libblkid.so.1.1.0 [root@linuxmaster ~]# lsof -u root -t 1 2 4 5 6 7 8 9 10 11 ... [root@linuxmaster ~]# lsof -u root -t | wc -l 70 [root@linuxmaster ~]# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME chronyd 546 chrony 5u IPv4 13335 0t0 UDP linuxmaster:323 chronyd 546 chrony 6u IPv6 13336 0t0 UDP linuxmaster:323 dhclient 643 root 6u IPv4 15146 0t0 UDP *:bootpc sshd 1118 root 3u IPv4 16844 0t0 TCP *:ssh (LISTEN) sshd 1118 root 4u IPv6 16846 0t0 TCP *:ssh (LISTEN) sshd 1145 root 3u IPv4 16984 0t0 TCP static.120.221.201.195.clients.your-server.de:ssh->88.232.191.134.dynamic.ttnet.com.tr:58976 (ESTABLISHED) sshd 2029 root 3u IPv4 31059 0t0 TCP static.120.221.201.195.clients.your-server.de:ssh->157.230.183.86:52592 (ESTABLISHED) sshd 2030 sshd 3u IPv4 31059 0t0 TCP static.120.221.201.195.clients.your-server.de:ssh->157.230.183.86:52592 (ESTABLISHED) [root@linuxmaster ~]# lsof -i tcp COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1118 root 3u IPv4 16844 0t0 TCP *:ssh (LISTEN) sshd 1118 root 4u IPv6 16846 0t0 TCP *:ssh (LISTEN) sshd 1145 root 3u IPv4 16984 0t0 TCP static.120.221.201.195.clients.your-server.de:ssh->88.232.191.134.dynamic.ttnet.com.tr:58976 (ESTABLISHED) [root@linuxmaster ~]# lsof -c sshd | head COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1118 root cwd DIR 8,1 4096 2 / sshd 1118 root rtd DIR 8,1 4096 2 / sshd 1118 root txt REG 8,1 852888 14974 /usr/sbin/sshd sshd 1118 root mem REG 8,1 61560 4728 /usr/lib64/libnss_files-2.17.so sshd 1118 root mem REG 8,1 68192 5318 /usr/lib64/libbz2.so.1.0.6 sshd 1118 root mem REG 8,1 99944 5540 /usr/lib64/libelf-0.176.so sshd 1118 root mem REG 8,1 19896 5553 /usr/lib64/libattr.so.1.1.0 sshd 1118 root mem REG 8,1 15688 6404 /usr/lib64/libkeyutils.so.1.5 sshd 1118 root mem REG 8,1 67104 9009 /usr/lib64/libkrb5support.so.0.1 [root@linuxmaster ~]# lsof +d /var/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1122 root 6w REG 8,1 73546 17793 /var/log/messages rsyslogd 1122 root 7w REG 8,1 111655 17794 /var/log/secure rsyslogd 1122 root 8w REG 8,1 1305 6991 /var/log/cron [root@linuxmaster ~]# lsof +D /var/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME auditd 514 root 5w REG 8,1 708911 6714 /var/log/audit/audit.log tuned 855 root 3w REG 8,1 1527 6992 /var/log/tuned/tuned.log rsyslogd 1122 root 6w REG 8,1 73546 17793 /var/log/messages rsyslogd 1122 root 7w REG 8,1 111655 17794 /var/log/secure rsyslogd 1122 root 8w REG 8,1 1305 6991 /var/log/cron [root@linuxmaster ~]# lsof -a -c rsyslogd +d /var/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1122 root 6w REG 8,1 73546 17793 /var/log/messages rsyslogd 1122 root 7w REG 8,1 113779 17794 /var/log/secure rsyslogd 1122 root 8w REG 8,1 1305 6991 /var/log/cron [root@linuxmaster ~]# lsof -c rsyslogd +d /var/log | head COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1122 root cwd DIR 8,1 4096 2 / rsyslogd 1122 root rtd DIR 8,1 4096 2 / rsyslogd 1122 root txt REG 8,1 664008 17789 /usr/sbin/rsyslogd rsyslogd 1122 root mem REG 0,20 8388608 8320 /run/log/journal/ffc3a14c22e24c419f4abca29738ab08/system.journal rsyslogd 1122 root mem REG 8,1 68192 5318 /usr/lib64/libbz2.so.1.0.6 rsyslogd 1122 root mem REG 8,1 99944 5540 /usr/lib64/libelf-0.176.so rsyslogd 1122 root mem REG 8,1 402384 5189 /usr/lib64/libpcre.so.1.2.0 rsyslogd 1122 root mem REG 8,1 19896 5553 /usr/lib64/libattr.so.1.1.0 rsyslogd 1122 root mem REG 8,1 338672 12539 /usr/lib64/libdw-0.176.so ``` {% endcode %} * `ps`: Report a snapshot of the current processes. By default, ps selects all process with the same effective user ID as the current user and the associated terminal. ```bash ps [OPTIONS] ``` `ps` accept several different kinds of options: * UNIX - May be grouped and must be preceded by a dash. * BSD - May be grouped but must not use a dash. * GNU long - Preceded by two dashes. | Option | Purpose | | ------ | ------------------------------ | | `-e` | Select all processes | | `-C` | Select by command name | | `-p` | Select by PID | | `-t` | Select by tty | | `-u` | Select by effective UID | | `-f` | Display in full-format listing | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# ps PID TTY TIME CMD 1332 pts/0 00:00:00 bash 1921 pts/0 00:00:00 ps [root@linuxmaster ~]# ps -ef | head UID PID PPID C STIME TTY TIME CMD root 1 0 0 11:31 ? 00:00:02 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 root 2 0 0 11:31 ? 00:00:00 [kthreadd] root 4 2 0 11:31 ? 00:00:00 [kworker/0:0H] root 5 2 0 11:31 ? 00:00:00 [kworker/u2:0] root 6 2 0 11:31 ? 00:00:00 [ksoftirqd/0] root 7 2 0 11:31 ? 00:00:00 [migration/0] root 8 2 0 11:31 ? 00:00:00 [rcu_bh] root 9 2 0 11:31 ? 00:00:00 [rcu_sched] root 10 2 0 11:31 ? 00:00:00 [lru-add-drain] [root@linuxmaster ~]# ps -fC sshd UID PID PPID C STIME TTY TIME CMD root 1118 1 0 11:31 ? 00:00:00 /usr/sbin/sshd -D root 1145 1118 0 11:31 ? 00:00:00 sshd: root@pts/0 root 2065 1118 0 12:28 ? 00:00:00 sshd: [accepted] root 2087 1118 0 12:29 ? 00:00:00 sshd: unknown [priv] sshd 2088 2087 0 12:29 ? 00:00:00 sshd: unknown [net] [root@linuxmaster ~]# ps -fp 1 UID PID PPID C STIME TTY TIME CMD root 1 0 0 11:31 ? 00:00:02 /usr/lib/systemd/systemd --switched-root --system - ``` {% endcode %} * `pstree`: Display running processes as a visual tree. ```bash pstree [OPTIONS] ``` | Option | Purpose | | ------ | ------------------------------------- | | `-A` | Use ASCII characters to draw the tree | | `-G` | Use VT100 line drawing characters | | `-a` | Show command line arguments | | `-p` | Show PIDs | | `-n` | Sort by PID instead of name | | `-Z` | Show SELinux security contex | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# pstree systemd─┬─NetworkManager─┬─dhclient │ └─2*[{NetworkManager}] ├─2*[agetty] ├─anacron ├─auditd───{auditd} ├─chronyd ├─crond ├─dbus-daemon───{dbus-daemon} ├─lvmetad ├─polkitd───6*[{polkitd}] ├─qemu-ga ├─rsyslogd───2*[{rsyslogd}] ├─sshd───sshd───bash───pstree ├─systemd-journal ├─systemd-logind ├─systemd-udevd └─tuned───4*[{tuned}] [root@linuxmaster ~]# pstree -p systemd(1)─┬─NetworkManager(627)─┬─dhclient(643) │ ├─{NetworkManager}(628) │ └─{NetworkManager}(630) ├─agetty(1126) ├─agetty(1127) ├─anacron(1809) ├─auditd(514)───{auditd}(515) ├─chronyd(546) ├─crond(1125) ├─dbus-daemon(543)───{dbus-daemon}(549) ├─lvmetad(401) ├─polkitd(540)─┬─{polkitd}(548) │ ├─{polkitd}(550) │ ├─{polkitd}(553) │ ├─{polkitd}(554) │ ├─{polkitd}(555) │ └─{polkitd}(558) ├─qemu-ga(538) ├─rsyslogd(1122)─┬─{rsyslogd}(1132) │ └─{rsyslogd}(1136) ├─sshd(1118)───sshd(1145)───bash(1332)───pstree(10018) ├─systemd-journal(381) ├─systemd-logind(551) ├─systemd-udevd(417) └─tuned(855)─┬─{tuned}(1024) ├─{tuned}(1025) ├─{tuned}(1030) └─{tuned}(1033) [root@linuxmaster ~]# pstree -pa | head systemd,1 --switched-root --system --deserialize 21 |-NetworkManager,627 --no-daemon | |-dhclient,643 -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf... | |-{NetworkManager},628 | `-{NetworkManager},630 |-agetty,1126 --keep-baud 115200,38400,9600 ttyS0 vt220 |-agetty,1127 --noclear tty1 linux |-anacron,1809 -s |-auditd,514 | `-{auditd},515 ``` {% endcode %} * `top`: Display a dynamic, real-time view of processes running on the system. The program provides a limited interactive interface for process mainpulation, as well as a much more extensive interface for personal configuration. ```bash top [OPTIONS] ``` | Option | Purpose | | ------ | --------------------------------- | | `d` | Change output delay | | `h` | Show list of options (help) | | `H` | Show thread count | | `m` | Display visual output of memory | | `f` | Open field management window | | `R` | Sort from least to most | | `W` | Write configuration to a file | | `n` | Set number of processes displayed | | `k` | Kill a process | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# top top - 12:19:34 up 47 min, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 71 total, 1 running, 70 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 1832812 total, 1370016 free, 98232 used, 364564 buff/cache KiB Swap: 0 total, 0 free, 0 used. 1582868 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1950 root 20 0 161968 2144 1548 R 0.3 0.1 0:00.02 top 1 root 20 0 128128 6776 4220 S 0.0 0.4 0:02.60 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 4 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 5 root 20 0 0 0 0 S 0.0 0.0 0:00.06 kworker/u2:0 6 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0 7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 ... ``` {% endcode %} ```bash # load average: 0.47, 24.71, 35.31 # 1, 5 and 15 minutes ``` ### Measuring Network Usage and System Availability * `netstat`: Print information on network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. By default, netstat displays a list of open sockets. The netstat command is obsolete and has been replaced by ss. other replacements include: `ip route` for `netstat -r`, `ip -s link` for `netstat -i`, and `ip maddr` for `netstat -g.` ```bash netstat [OPTIONS] [DELAY] ``` | Option | Purpose | | ------- | ------------------------------------------------ | | `-r` | Display kernel routing table | | `-g` | Display multicast group information | | `-i` | Display a table for network interfaces | | `-s` | Display summary statistics for each protocol | | `-n` | Show numerical address (do not resolve hostname) | | `-c` | Display output continuously | | `-p` | Show PID and process name | | `-l` | Show only listening sockets | | `-t` | Display TCP connections | | `-u` | Display UDP connections | | `delay` | Set DELAY in seconds | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# netstat | head Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 53 0 static.120.221.201.:ssh 157.230.183.86:47510 CLOSE_WAIT tcp 0 36 static.120.221.201.:ssh 88.232.191.134.dy:58976 ESTABLISHED Active UNIX domain sockets (w/o servers) Proto RefCnt Flags Type State I-Node Path unix 3 [ ] DGRAM 8213 /run/systemd/notify unix 2 [ ] DGRAM 8215 /run/systemd/cgroups-agent unix 2 [ ] DGRAM 13337 /var/run/chrony/chronyd.sock unix 5 [ ] DGRAM 8235 /run/systemd/journal/socket [root@linuxmaster ~]# netstat -n | head Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 195.201.221.120:22 61.177.172.91:38436 ESTABLISHED tcp 0 36 195.201.221.120:22 88.232.191.134:58976 ESTABLISHED Active UNIX domain sockets (w/o servers) Proto RefCnt Flags Type State I-Node Path unix 3 [ ] DGRAM 8213 /run/systemd/notify unix 2 [ ] DGRAM 8215 /run/systemd/cgroups-agent unix 2 [ ] DGRAM 13337 /var/run/chrony/chronyd.sock unix 5 [ ] DGRAM 8235 /run/systemd/journal/socket [root@linuxmaster ~]# netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface default gateway 0.0.0.0 UG 0 0 0 eth0 gateway 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 static.120.221. 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 [root@linuxmaster ~]# netstat -nl | head Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp6 0 0 :::22 :::* LISTEN udp 0 0 0.0.0.0:68 0.0.0.0:* udp 0 0 127.0.0.1:323 0.0.0.0:* udp6 0 0 ::1:323 :::* Active UNIX domain sockets (only servers) Proto RefCnt Flags Type State I-Node Path unix 2 [ ACC ] SEQPACKET LISTENING 11043 /run/udev/control [root@linuxmaster ~]# netstat -nlp | head Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1118/sshd tcp6 0 0 :::22 :::* LISTEN 1118/sshd udp 0 0 0.0.0.0:68 0.0.0.0:* 643/dhclient udp 0 0 127.0.0.1:323 0.0.0.0:* 546/chronyd udp6 0 0 ::1:323 :::* 546/chronyd Active UNIX domain sockets (only servers) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 2 [ ACC ] SEQPACKET LISTENING 11043 1/systemd /run/udev/control root@linuxmaster ~]# netstat -tlnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1118/sshd tcp6 0 0 :::22 :::* LISTEN 1118/sshd [root@linuxmaster ~]# netstat -ulnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name udp 0 0 0.0.0.0:68 0.0.0.0:* 643/dhclient udp 0 0 127.0.0.1:323 0.0.0.0:* 546/chronyd udp6 0 0 ::1:323 :::* 546/chronyd [root@linuxmaster ~]# netstat -tulnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1118/sshd tcp6 0 0 :::22 :::* LISTEN 1118/sshd udp 0 0 0.0.0.0:68 0.0.0.0:* 643/dhclient udp 0 0 127.0.0.1:323 0.0.0.0:* 546/chronyd udp6 0 0 ::1:323 :::* 546/chronyd ``` {% endcode %} * `w`: Display information about the users currently on the machine, and their processes. The header shows the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes. The following entries are displayed for each user: login name, the tty name, the remote host, login time, idle time, JCPU, PCPU, and the command line of their current process. ```bash w [OPTIONS] USER ``` | Option | Purpose | | ------ | ------------------------------------------ | | `-h` | Don't print the header | | `-s` | Use short format | | `-i` | Display IP addresses rather than hostnames | | `user` | Display information for USER specified | ```bash [root@linuxmaster ~]# w 12:39:35 up 1:07, 1 user, load average: 0.00, 0.01, 0.05 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0 88.232.191.134 11:31 7.00s 0.32s 0.00s w [root@linuxmaster ~]# w -h root pts/0 88.232.191.134 11:31 4.00s 0.37s 0.01s w -h [root@linuxmaster ~]# w -s 12:51:12 up 1:19, 1 user, load average: 0.00, 0.01, 0.05 USER TTY FROM IDLE WHAT root pts/0 88.232.191.134 0.00s w -s ``` * `uptime`: Display how long the system has been running. By default, displays the same information found in the header of the `w` command. Shows the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes. Both `w` and `uptime` get their information from `/var/run/utmp`. ```bash uptime [OPTIONS] ``` | Option | Purpose | | ------ | -------------------------------------------- | | `-p` | Show uptime in pretty format | | `-h` | Display help text | | `-s` | Display uptime in yyyy-mm-dd HH:MM:SS format | | `-V` | Display version information | ```bash [root@linuxmaster ~]# uptime 12:39:56 up 1:08, 1 user, load average: 0.00, 0.01, 0.05 [root@linuxmaster ~]# uptime -p up 1 hour, 20 minutes [root@linuxmaster ~]# uptime -s 2022-09-15 11:31:36 ``` ## 200.2 Predict Future Resource Needs (weight: 2) {% hint style="success" %} Candidates should be able to monitor resource usage to predict future resource needs.\ \\ **Key Knowledge Areas:** * Use monitoring and measurement tools to monitor IT infrastructure usage. * Predict capacity break point of a configuration. * Observe growth rate of capacity usage. * Graph the trend of capacity usage. * Awareness of monitoring solutions such as Icinga2, Nagios, collectd, MRTG and Cacti **The following is a partial list of the used files, terms and utilities:** * diagnose * predict growth * resource exhaustion {% endhint %} ### Monitoring Solutions #### Nagios Nagios is an industry-standard IT monitoring solution. * Provides complete monitoring for Linux operating systems and distributions * Provides two options for monitoring hosts * Using dedicated Nagios agents * Using SNMP for "agentless" monitoring * Provides advanced graphs and visual aids * Provides automated and integrated capacity planning #### Icinga2 Icinga2 is a monitoring solution that reports on various usage statistics by using a primary node, satellites, and agents. * Monitors network resources * Notifies users of outages * Generates performance data * Runs on Linux/Unix and Windows #### MRTG The Multi Router Traffic Grapher is a tool for monitoring routers and other network devices. * Monitors SNMP network devices * Monitors traffic load on network links * Generates HTML pages containing PNG images of live data * Implements a data consolidation algorithm * Provides built-in hooks for using RRDTool #### Cacti A network graphing solution designed to work with RRDTool. * Provides a complete frontend to RRDTool * Uses a MySQL database to store data * Provides a built-in user-based management system * Provides graphs for displaying data * Allows data sources to be defined for data collection #### Collectd Collectd is a service daemon that collects system and applicatkin performance metrics periodically and provides mechanisms to store the values in a variety of ways (e.g. RRD files). * Gathers metrics from various sources * Stores metrics gathered or makes them available over the network * Provides information for performance analysis and capacity planning * Provides over 100 plugins for monitoring and integration with other solutions Installing and Configuration Collectd {% code overflow="wrap" %} ```bash root@linuxmaster:~# apt install collectd apache2 librrd-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libcgi-session-perl root@linuxmaster:~# systemctl restart collectd root@linuxmaster:~# cp -r /usr/share/doc/collectd/examples/collection3/ /var/www/html root@linuxmaster:~# vim /etc/apache2/apache2.conf ## Add the following: AllowOverride None Options +ExectCGI AddHandler cgi-script .cgi .pl Require all granted root@linuxmaster:~# ln -s /etc/apache2/mod-available/cgi.load /etc/apache2/mods-enabled/ root@linuxmaster:~# systemctl restart apache2 ``` {% endcode %} ## Commands and Notes * `iostat`: Report on CPU statistics and input/output statistics for devices and partitions. * `sar`: Collect, report, or save system activity information. * sar creates a binary file for each day of the month with all the information collected (e.g., `sa01`), as well as a daily summary (e.g.,`sar01`). These files are overwritten the following month. The sysstat service must be started for sar to start collecting data and the duration of collection can be changed by modifying `/etc/cron.d/sysstat`. * `free`: Display the amount of free and used memory in the system. * `vmstat`: Reports virtual memory statistics as well as statistics on paging and block I/O. * `lsof`: Lists open files on the system. * `ps`: Report a snapshot of the current processes. * `pstree`: Display running processes as a visual tree. * `top`: Display a dynamic, real-time view of processes running on the system. * `netstat`: Print information on network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. * `w`: Display information about the users currently on the machine, and their processes. * `uptime`: Display how long the system has been running. * **Nagios** using SNMP for "agentless" monitoring. * **Icinga2** originally created as a fork of Nagios. * **MRTG** provides built-in hooks for using RRDTool. * **Cacti** a network graphing solution designed to work with RRDTool. * **Collectd** is a service daemon that collects system and applicatkin performance metrics periodically and provides mechanisms to store the values in a variety of ways (e.g. RRD files). ## Sample Questions **1.** Which of the following provides a complete frontend for RRDTool? * [ ] MRTG * [x] Cacti * [ ] Collectd * [ ] Nagios **2.** What command displays a dynamic view of processes running on the system? * [ ] lsof * [x] top * [ ] pstree * [ ] ps **3.** Which of the following was originally created as a fork of Nagios? * [ ] Collectd * [ ] MRTG * [ ] Cacti * [x] Icinga2 **4.** What file is used to change the duration of collection for sar? * [ ] /var/log/sa * [ ] /etc/sar.conf * [x] /etc/cron.d/sysstat * [ ] /etc/cron.d/sar **5.** What option for lsof lists open files based on network connections? * [ ] -c * [ ] -u * [x] -i * [ ] -p **6.** What commands show how long the system has been running? * [x] w * [ ] date * [x] uptime * [ ] stat **7.** What option for netstat displays the kernel routing table? * [ ] -l * [ ] -n * [ ] -i * [x] -r **8.** Which of the following are aspects of capacity planning? * [x] Collecting long term resource usage data * [x] Observing trends in resource usage * [ ] Implementing competitive salaries based on what the market will bear * [x] Consdiering business Initiatives **9.** What option only generates the device report with the iostat command? * [ ] -c * [ ] -y * [ ] -x * [x] -d **10.** What kinds of options are accepted by the `ps` command? (choose all that apply) * [ ] LSB * [x] UNIX * [x] GNU long * [x] BSD **11.** Which command reports information on memory usage, paging and block input/output? * [ ] free * [ ] memshow * [ ] lsof * [ ] pidof * [x] vmstat **12.** Which program lists information about files and network connections opened by processes? (Specify ONLY the command without any path or parameters.) **Answer:** lsof **13.** Which of the following commands by default provides the PIDs of the processes sorted by which are using the most CPU cycles on the Linux system? * [x] top * [ ] uptime * [ ] ps aux * [ ] vmstat * [ ] freemem **14.** In the following output of the command uptime, the load averages represent the system load averages for what time frames? ```bash 12:10:05 up 18 days, 19:00, 2 users, load average: 0.47, 24.71, 35.31 ``` * [x] 1, 5 and 15 minutes * [ ] 1, 15 and 30 minutes * [ ] 5, 20 and 25 seconds * [ ] 15, 30 and 60 minutes * [ ] 15, 45 and 90 seconds **15.** In order to gather performance data with `sar` over a longer period of time, which command should be run periodically from cron? * [x] sa1 * [ ] sarmon * [ ] sarec * [ ] sadf * [ ] sarcron **16.** In the output of `sar –b`, what does tps stand for? * [ ] Terrabyte per second * [ ] Traffic per second * [x] Transfers per second * [ ] Total data per second * [ ] Terrabit per second **17.** Which of the following monitoring commands allow you to view system memory information? (Choose all that apply.) * [x] free * [ ] mpstat * [x] top * [ ] mtr * [x] vmstat **18.** Which of the following will show CPU usage information 20 times in 2-second intervals? * [ ] uptime 20 2 * [ ] uptime 2 20 * [ ] sar 20 2 * [x] sar 2 20 * [ ] None of the above **19.** Which of the following are considered to be system resource-monitoring solutions? (Choose all that apply) * [x] collectd * [ ] Duplicity * [x] MRTG * [x] Nagios * [ ] All of the above **20.** Which of the following are true about Linux system memory? (Choose all that apply) * [ ] Memory is divided into sheets. * [x] Memory for an idle process can be swapped out to swap space. * [x] Swap can also be called virtual memory. * [ ] Swapping is the memory management scheme, where idle processes swap memory. * [ ] Memory-swapping statistics can be viewed using the psmap utility. **Answer:** B, C. Memory for idle processes can be swapped out from RAM to a special partition called swap space, swap, or virtual memory. Therefore, options B and C are correct. Memory is divided up into chunks called pages (option A), not sheets. Swapping is the memory management technique where idle processes’ memory is swapped out to swap. The processes do not trade memory (option D). Finally, the psmap tool is primarily for mapping processes (option E). The best tools for viewing memory statistics are tools like `free`, `sar`, and `vmstat`. **21.** In the following output from top, which processes contribute to the percentage of time that the CPU spends in the state of wa? ```bash Tasks: 193 total, 1 running, 190 sleeping, 2 stopped, 0 zombie Cpu(s): 0.5%us, 0.3%sy, 0.0%ni, 98.2%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st ``` * [ ] Processes waiting for user interaction. * [ ] Processes that were already closed and are waiting to be launched again. * [ ] Processes that have not been scheduled yet because they haven't been fully loaded into RAM or are in swap. * [x] Processes waiting for IO operations to complete. **22.** In this example output, which descriptions match the purpose of the free, buff and cache columns? (Choose THREE correct answers.) ```bash # vmstat 1 100 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 282120 134108 5797012 0 0 0 2 0 0 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1007 359 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1117 577 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1007 366 0 0 100 0 ``` * [ ] Used swap space * [ ] RAM available for filesystem buffers * [x] Available free RAM * [x] RAM used for buffers * [x] RAM used for filesystem cache **23.** When is historical data of resource usage important? (Select THREE correct answers.) * [x] Predicting when resources will need to be increased. * [ ] Selecting a computer vendor. * [ ] Identifying processes killed during out of memory occurrences. * [x] Diagnosing capacity problems. * [x] Troubleshooting a software problem. **24.** In capacity planning exercises, which tools assist in listing and identifying processes of interest? (Choose TWO correct answers.) * [ ] acpid * [x] lsof * [x] pstree * [ ] telinit **25.** In the below example output, which columns detail the percent of time the CPU spent running non-kernel code and the percent of time the CPU spent running kernel code? (Choose TWO correct answers.) ```bash # vmstat 1 100 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 282120 134108 5797012 0 0 0 2 0 0 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1007 359 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1117 577 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1007 366 0 0 100 0 ``` * [ ] id * [x] us * [ ] wa * [x] sy **26.** In the following output, what percentage of time was the CPU waiting for pending I/O? ```bash # vmstat 1 100 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 282120 134108 5797012 0 0 0 2 0 0 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1007 359 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1117 577 0 0 100 0 0 0 0 282120 134108 5797012 0 0 0 0 1007 366 0 0 100 0 ``` * [x] 0 * [ ] 100 * [ ] 35.9 * [ ] 57.7 * [ ] 36.6 **27.** Which of the following tools are used to measure memory usage? (Choose THREE correct answers.) * [ ] mpstat * [ ] pstree * [x] sar * [x] top * [x] vmstat **28.** Which of the following is a side effect of extensive usage of swap space? * [ ] The root filesystem may become full because swap space is always located on the system root partition. * [x] The overall system performance may degrade because of heavy hard disk use and memory reorganization. * [ ] Since processes always exist completely in either RAM or swap, regular RAM may become unused if the kernel does not move processes back from the swap space to memory. * [ ] The memory may become fragmented and slow down the access to memory pages. However, this can be kept to a minimum by the regular use of memfrag -d. * [ ] Applications need to restart because their virtual memory addresses change to reflect memory relocation to the swap address area. **29.** Which of the following commands will provide the PIDs of the processes sorted by which are using the most CPU cycles on the Linux system? * [x] top * [ ] uptime * [ ] ps aux * [ ] vmstat * [ ] freemem **30.** Which command will report information on memory usage, paging and block input/output? * [x] vmstat * [ ] free * [ ] memshow * [ ] ps * [ ] top **31.** Using the iostat command you want to display more detailed statistics (Extended), but you want the output of this command to be printed on the screen 5 times between 2-second update intervals. Which of the following is correct to accomplish this task? * [ ] iostat -v 5 2 * [ ] iostat -x 5 2 * [x] iostat -x 2 5 * [ ] iostat -v 2 5 **32.** Which of the commands below will return virtual memory statistics information about processes, memory, pagination, block IO, trap and CPU usage? * [x] vmstat * [ ] sar * [ ] pstree * [ ] iostat **33.** Which of the commands below is used to monitor the load of system input / output (I/O) devices. Where is this done by observing the devices are active in relation to their average transfer rates? * [ ] sar * [x] iostat * [ ] vmstat * [ ] pstree **34.** Which of the commands below collects, reports, or saves system activity information? * [x] sar * [ ] iostat * [ ] vmstat * [ ] pstree **35.** Which of the iostat command options can be used to display device statistics information (disks and partitions)? * [ ] iostat -x * [x] iostat -p * [ ] iostat -l * [ ] iostat -v **36.** Which of the iostat options can be used to display a CPU usage report? * [ ] iostat -b * [ ] iostat -x * [ ] iostat -d * [x] iostat -c **37.** Which of the iostat command options can be used to display a device usage report (Disks)? * [ ] iostat -b * [ ] iostat -x * [x] iostat -d * [ ] iostat -c **38.** Which command below displays information about the users currently connected to the machine, their processes and the same statistics as provided by the uptime command? * [ ] top * [x] w * [ ] free * [ ] whoami **39.** When outputting the vmstat command, in the procs field, what do the columns r and b mean? (Select 2 responses). * [ ] The number of executable processes (running and waiting for execution time). * [ ] Shows the percentage of time spent running non-kernel code. * [ ] The number of processes in uninterrupted sleep. * [ ] Show the percentage of time stolen from a virtual machine. **40.** Which of the columns below is related to the system field in the output of the vmstat command that shows the number of contexts per second, including the clock? * [x] cs * [ ] sy * [ ] in * [ ] wa * [ ] us **41.** Which of the columns below is related to the system field in the output of the vmstat command that shows the number of interrupts per second, including the clock? * [ ] cs * [ ] sy * [x] in * [ ] wa * [ ] us **42.** Which of the alternatives below are columns related to the swap field presented in the output of the vmstat command? (Select 2 responses). * [x] si * [ ] bi * [x] so * [ ] bo * [ ] id **43.** Which of the alternatives below are columns related to the io field presented in the output of the vmstat command? (Select 2 responses). * [ ] si * [x] bi * [ ] so * [x] bo * [ ] id **44.** Which of the following, using the vmstat command, will display event counter statistics? * [ ] vmstat -a * [ ] vmstat -d * [x] vmstat -s * [ ] vmstat -p * [ ] vmstat -D **45.** Which of the following, using the vmstat command, will display disk statistics? * [ ] vmstat -a * [x] vmstat -d * [ ] vmstat -s * [ ] vmstat -p * [ ] vmstat -D **46.** Which of the following, using the vmstat command, will display a summary of disk statistics? * [ ] vmstat -a * [ ] vmstat -d * [ ] vmstat -s * [ ] vmstat -p * [x] vmstat -D **47.** Which of the following, using the vmstat command, will display statistics for a specific partition? * [ ] vmstat -a * [ ] vmstat -d * [ ] vmstat -s * [x] vmstat -p * [ ] vmstat -D **48.** Which of the free command options will update the command output every 2 seconds for 5 times on the screen? (Select 2 responses). * [x] free -s 2 -c 5 * [x] free -s 2 --count 5 * [ ] free -s 2 5 * [ ] free -s 2 -r 5 **49.** Which of the free command options will update the command output on the screen every 2 seconds? * [ ] free -n 2 * [ ] free -t 2 * [x] free -s 2 * [ ] N.D.A **50.** Which of the alternatives below will list detailed memory statistics, showing the total RAM + SWAP memory, printing the result in the output in a more readable (- human) form for the user? * [ ] free -vht * [x] free -lth * [ ] free -ghl * [ ] free -x **51.** Which option in the sar command displays paging statistics? * [x] -B * [ ] -b * [ ] -d * [ ] -p * [ ] -r **52.** Which option in the sar command displays a memory usage statistics report? * [ ] -m * [ ] -b * [ ] -d * [ ] -R * [x] -r **53.** Which option of the sar command will report the queue size and load averages? * [ ] -S * [x] -q * [ ] -w * [ ] -y * [ ] -d **54.** Which option of the sar command will report an I/O transfer rate report? * [ ] -S * [ ] -q * [x] -b * [ ] -y * [ ] -d **55.** Which option of the sar command will report swap statistics? (Select 2 responses). * [x] -S * [ ] -q * [x] -W * [ ] -y * [ ] -d **56.** Which option of the sar command will report the activity of each block device? * [ ] -S * [ ] -q * [ ] -b * [ ] -y * [x] -d **57.** Which option in the sar command will report the activity of TTY (Remote Terminals) devices? * [ ] -S * [ ] -q * [ ] -b * [x] -y * [ ] -d **58.** Which of the commands below can be used to view the PID of a program that we have just started? (Select 2 responses). * [ ] iostat * [x] ps * [ ] vmstat * [x] top **59.** What is the result of the `ps ef` command? * [ ] List user processes running as a tree * [ ] List stopped processes. * [x] List all processes running as a tree. * [ ] List all system processes running as a tree. **60.** Which of the commands below will return a list of processes that are running on the current terminal? * [ ] top * [x] ps * [ ] vmstat * [ ] iostat **61.** Which command will list all processes, displaying the names of the users who started them? * [ ] ps a * [ ] ps ag * [x] ps aux * [ ] ps ac **62.** Which command displays details about the processes that are running in real time on the server? * [ ] ps * [x] top * [ ] vmstat * [ ] iostat **63.** Which of the options ("Internal option of the top command") below can be used to change the priority of a process already running? * [ ] -t * [ ] -d * [ ] -b * [x] -r * [ ] -k **64.** Which of the following will set the update time to 2 seconds on standard output using the top command? * [ ] top -n 2 * [x] top -d 2 * [ ] top -i 2 * [ ] N.D.A **65.** Which "top" command will ignore idle processes? * [ ] top -b * [x] top -i * [ ] top -d * [ ] N.D.A **66.** Which command will update the top command's standard output 4 times and then exit the program? * [ ] top -d 4 * [ ] top -i 4 * [x] top -n 4 * [ ] N.D.A **67.** Which top command will only list the processes of the user root? * [ ] top -L root * [x] top -u root * [ ] top -l root * [ ] N.D.A # 201: Linux Kernel ## 201.1 Kernel components (weight: 2) {% hint style="success" %} Candidates should be able to utilise kernel components that are necessary to specific hardware, hardware drivers, system resources and requirements. This objective includes implementing different types of kernel images, understanding stable and longterm kernels and patches, as well as using kernel modules.\ \ **Key Knowledge Areas:** * Kernel 2.6.x, 3.x and 4.x documentation **The following is a partial list of the used files, terms and utilities:** * /usr/src/linux/ * /usr/src/linux/Documentation/ * zImage * bzImage * xz compression {% endhint %} ### The Kernel Source Tree and Compressed Kernel Images * The Kernel Source Tree: The default location for the kernel source tree is in `/usr/src` (for CentOS, this is `/usr/src/kernels`). The kernel documentation is in the Documentation directory within the source tree. The 00-INDEX in the documentation directory gives list of the various files and directories as well as a short description. The two main types of kernel images are `zImage` and `bzImage`. Once compiled, the compressed image is copied to the boot directory and renamed to vmlinuz followed by the kernel version and architecture. * **zImage** was used on older systems and designed to fit within the limited amount of memory that was available (the first 640 KB, AKA low memory). The image was compressed down to 512 KB using gzip compression. * **bzImage** stands for big zImage and takes advantage for the higher memory available (around 1 MB, AKA high memory). This change happened around version 2 of the Linux kernel and by version 2.6 bzip2 was the standard compression used. ```bash [root@linuxmaster ~]# ls /boot/ ... vmlinuz-3.10.0-1160.71.1.el7.x86_64 vmlinuz-3.10.0-1160.76.1.el7.x86_64 ... ``` * `xz` Compression: As of 2013, kernel.org started providing kerel archives in xz compression rather than bzip2 (older archives will remain in bzip2). xz used the LZMA2 compression algorithm and has become a standard beacause of its efficiency and lower compression ratio. ## 201.2 Compiling a Linux kernel (weight: 3) {% hint style="success" %} Candidates should be able to properly configure a kernel to include or disable specific features of the Linux kernel as necessary. This objective includes compiling and recompiling the Linux kernel as needed, updating and noting changes in a new kernel, creating an initrd image and installing new kernels. \ **Key Knowledge Areas:** * /usr/src/linux/ * Kernel Makefiles * Kernel 2.6.x, 3.x and 4.x make targets * Customize the current kernel configuration. * Build a new kernel and appropriate kernel modules. * Install a new kernel and any modules. * Ensure that the boot manager can locate the new kernel and associated files. * Module configuration files * Use DKMS to compile kernel modules. * Awareness of dracut **The following is a partial list of the used files, terms and utilities:** * mkinitrd * mkinitramfs * make * make targets (all, config, xconfig, menuconfig, gconfig, oldconfig, mrproper, zImage, bzImage, modules, modules\_install, rpm-pkg, binrpm-pkg, deb-pkg) * gzip * bzip2 * module tools * /usr/src/linux/.config * /lib/modules/kernel-version/ * depmod * dkms {% endhint %} ### Installing a Linux Kernel and Kernel Modules ```bash # Compile the Linux Kernel make menuconfig make bzImage # Building and Installing Kernel Modules make modules make modules_install # Installing the Kernel and Generating the initramfs cp /usr/src/kernel_tree/arch/x86/boot/bzImage /boot mv bzImage kernel_image mkinitrd initrd_image kernel_version #CentOS mkinitramfs -o initrd_image kernel_version #Ubuntu ``` ### Understanding Make Targets and Associated Utilities `make` will execute the commands in the makefile to one or more target names. If the makefile isn't provided, then the make command will look for the makefiles starting with `gnumakefile`, `makefile`, and `Makefile` (`Makefile` is the suggested usage) | Target | Purpose | | ----------------- | ------------------------------------------------------------ | | `all` | Builds all targets marked with an asterisk | | `config` | Update current config utilizing a line-oriented program | | `xconfig` | Update current config utilizing a ncurses menu-based program | | `menuconfig` | Update current config utilizing a menu-based program | | `gconfig` | Update current config utilizing a GTK+ based frontend | | `oldconfig` | Update current config utilizing a provided `.config` as base | | `mrproper` | Remove all generated files + config + various backup files | | `zImage` | Compressed kernel image (`arch/x86/boot/zImage`) | | `bzImage` | Compressed kernel image (`arch/x86/boot/bzImage`) | | `modules` | Build all modules | | `modules_install` | Install all modules to `INSTALL_MOD_PATH` (default: /) | | `rpm_pkg` | Build both source and binary RPM kernel packages | | `binrpm_pkg` | Build only the binary kernel RPM package | | `dep_pkg` | Build both source and library deb kernel packages | * `depmod`: `depmod` is short for dependency modules and is used to generate a dependency list file, `modules.dep`, and associated map files. `depmod` analyzes the `/lib/modules/kernel_version` directory and creates the dependency file `modules.dep`, as well as a binary hash version called `modules.dep.bin`. Another important file created by `depmod` is `modules.symbols` (and `modules.symbols.bin`). This file contains a dependency list of symbols - uniqe service provided by modules - that can be used by other modules. ```bash [root@linuxmaster ~]# ls -n /path/to/your-kernel-module.ko /lib/modules/`uname -r` [root@linuxmaster ~]# depmod -a ``` * `dkms`: Dynamic Kernel Module Support (DKMS) provides a framework for generating kernel modules whose sources generally reside outside of the kernel source tree. Prepare a module for DKMS: * Download and extract the kernel module * Create or modify the `dkms.conf` file * Copy the kernel module source code to `/usr/src` Add, build, and install with DKMS: ```bash dkms add -m dkms build -m -v dkms install -m -v ``` ## 201.3 Kernel runtime management and troubleshooting (weight: 4) {% hint style="success" %} Candidates should be able to manage and/or query a 2.6.x, 3.x, or 4.x kernel and its loadable modules. Candidates should be able to identify and correct common boot and run time issues. Candidates should understand device detection and management using udev. This objective includes troubleshooting udev rules. \ **Key Knowledge Areas:** * Use command-line utilities to get information about the currently running kernel and kernel modules. * Manually load and unload kernel modules. * Determine when modules can be unloaded. * Determine what parameters a module accepts. * Configure the system to load modules by names other than their file name. * /proc filesystem * Content of /, /boot/ , and /lib/modules/ * Tools and utilities to analyze information about the available hardware * udev rules **The following is a partial list of the used files, terms, and utilities:** * /lib/modules/kernel-version/modules.dep * module configuration files in /etc/ * /proc/sys/kernel/ * /sbin/depmod * /sbin/rmmod * /sbin/modinfo * /bin/dmesg * /sbin/lspci * /usr/bin/lsdev * /sbin/lsmod * /sbin/modprobe * /sbin/insmod * /bin/uname * /usr/bin/lsusb * /etc/sysctl.conf, /etc/sysctl.d/ * /sbin/sysctl * udevmonitor * udevadm monitor * /etc/udev/ {% endhint %} ### Working with Kernel Modules * `uname`: A command line utility that prints basic information about the system. The most common use is to determine the processor architecture, the system hostname, and the kernel version of the system. Without any options, `uname` prints kernel name as if the `-s` option were used. ```bash uname [OPTIONS] ```
OptionDescription
-aPrint all information
-sPrint the kernel name
-nPrint the network node hostname
-rPrint the kernel release
-vPrint the kernel version
-mPrint the machine hardware name
-pPrint the processor type or "unknown"
-iPrint the hardware platform or "unknown"
-oPrint the operation system
* `dmesg`: Print or control the kernel ring buffer. By default, it shows all the messages in the kernel ring buffer. ```bash dmesg [OPTIONS] ```
OptionDescription
-CClear the ring buffer
-cClear the ring buffer after printing the contents
-fRestrict the output to a give list of facilities
-HEnable human-readable format
-kPrint kernel messages
-lRestrict output to the given list of levels
-TPrint human-readable timestamps
-uPrint userspace messages
* `lsmod`: A trivial program that nicely formats the contents of the `/proc/modules` file, showing what kernel modules are currently loaded. * `modinfo`: Shows information about a kernel module. By default, `modinfo` lists each attribute of the module in a `filename: value` format. If the module name is not a filename, then `/lib/modules/kernelversion` is searched. ```bash modinfo -O -F field -k kernel modulename [filename] ```
OptionDescription
-OUse ASCII zero characters to separate field values instead of new lines
-FOnly print field value specified
-kProvide information about a kernel other than the running one
* `insmod`: A simple program to insert a kernel module into the Linux kernel. It is recommended to use `modprobe`, which is a more sophisticated command. ```bash insmod [FILENAME] [MODULE_OPTIONS] ``` * `rmmod`: A simple program to remove a module from the Linux kernel. It is recommended to use `modprobe` with the `-r` option instead. ```bash rmmod [OPTIONS] [MODULE_NAME] ```
OptionDescription
-vUse verbose output
-fForce the removal of a module
-wWait until the module is not in use
-sSend errors to syslog instead of the standard error
* `modprobe`: A program used to add and remove modules from the Linux kernel. `modprobe` intelligently adds and removes modules by considering module dependencies. Configuration added to `/etc/modprobe.d/` or `/etc/modprobe.conf` are considered when the command is executed. ```bash modprobe [OPTIONS] [MODULE_NAME] [MODULE_PARAMERERS] ```
OptionDescription
-vUse verbose output
-COverride the default configuration directory/file
-nPerform a dry run
-rRemove a module
-wBlock new requests to a module and wait until it is available
-fAttempt to force an insert or a remove
-DList the dependencies of a module
-oAttempt to rename a module
#### Module Configuration Files * `/lib/modules/kernel-version/module.dep`: List the dependencies for every module in the directories under `/lib/modules/kernel_version`. This file is used by `modprobe` to know the order in which to load modules. * `/etc/modules` and `/etc/modules-load.d/modules.conf` and /`etc/modules.conf`: Used to configure kernel modules to load at boot. The configuration file(s) should contain a list of kernel module names to load, separated by newlines. Empty lines and lines beginning with `#` or `;` are ignored. * `/etc/modprobe.d` and `/etc/modprobe.conf`: The configuration directory/file for `modprobe`. The format of this configuration is one command per line with ‘\’ at the end of the line. Blank lines and lines beginning with ‘#’ are ignored. **Commands:** * `alias`: Give alternate names for modules * `options`: Add options to the module when it is inserted into the kernel * `install`: run a shell command instead of inserting a module as normal * `remove`: run a shell command when `modprobe -r` is invoked rather than removing a module * `blacklist`: ignore a particular module’s internal aliases ### Understand the /proc Filesystem and the Device Filesystem * The `proc` Filesystem: A virtual filesystem that gets created and mounted during boot time. It presents information about the process and other system-related information. Processes, for which the filesystem is named, are represented by numbered directories that correlate with their PID. System and kernel-level parameters can be changed by manually editing files within the `/proc` directory or by using the `sysctl` utility. These changes are only effective for the current runtime and will not persist through a reboot. ```bash [root@linuxmaster ~]# ls -ld /proc/* | awk ‘{print $9}’ /proc/1 /proc/10 /proc/1719 … /proc/cpuinfo /proc/devices /proc/driver … ``` * `sysctl`: Used to configure kernel parameters at runtime and to display the current values. Available parameters can be found in `/proc/sys/`. `sysctl` settings added to `/etc/sysctl.conf` or `/etc/sysctl.d/` will be applied on system boot. Configuration files added to `/etc/sysctl.d/` should end with “.conf” and be prefixed with a number based on priority. The higher the prefix number the greater the priority. ``` sysctl [options] [variable[=value]] ```
OptionDescription
-aDisplay all value currently available
-wChange a sysctl settings
-pLoad a sysctl settings from a file
-nPrint names without names
-NPrint names without values
* The Device Filesystem: Contains files for every device of which the system is aware and special device files that are used by the system. Some examples of this are `urandom`, `zero`, and `null`. The kernel device manager, udev, continually probe the system to check for connected devices and creates a device file when one is detected. Udev rules end with `.rules` an extension and determine how to identify devices and assign names to them. These rules are located in `/etc/udev/rules.d/` and `/lib/udev/rules.d/`. * `udevadm monitor`: Listen to the kernel `uevents` and events are sent out by a `udev` rule and prints the devpath of the event to the console. This command replaces the now deprecated `udevmonitor` utility ```bash udevadm monitor [options] ```
OptionDescription
-kPrint kernel uevents
-uPrint udev event after rule processing
-pAlso print the properties of the event
### Analyzing Information about Available Hardware * `lspci`: A utility for displaying information about Peripheral Component Interconnect (PCI) buses in the system and the devices connected to them. By default, `lspci` shows a brief list of devices. At more verbose levels, the output is only useful to those who are very familiar with PCI. ```bash lspci [options] ```
OptionDescription
-mDump data in a backward-compatible, machine-readable format
-mmDump data in a machine-readable format
-tDisplay data in a tree-like diagram
-vIncrease verbosity (-vv and -vvv for more)
-kShow kernel drivers handling each device and modules that are capable of handling it
-nShow PCI vendor and devices codes as numbers
* `lsusb`: A utility for displaying information about USB buses in the system and the devices connected to them. ```bash lsusb [options] ```
OptionDescription
-vIncrease verbosity
-sShow devices specified with bus and/or devnum provided
-dShow devices specified with vendor and product ID provided
-tDisplay USB device hierarchy as a tree
* `lsdev`: Displays information about installed hardware. `lsdev` gathers this information from the interrupts, `ioports`, and `dma` files in the `/proc` directory. There are no command line options for this command. ## Commands and Notes ## Sample Questions **1.** What command prints information about and controls the kernel ring buffer? * [ ] insmod * [ ] uname * [x] dmesg * [ ] modprobe **2.** What are the two main types of compressed kernel images? (Choose two.) * [x] zImage * [ ] vmlinuz * [x] bZimage * [ ] initramfs **3.** What is the utility used to generate modules whose source resides outside of the main kernel source tree? * [ ] make modules\_install * [x] dkms * [ ] depmod * [ ] make modules **4.** What command is used to generate a compressed kernel image? * [ ] make menuconfig * [ ] make image * [x] make bzImage * [ ] make kImage **5.** What command is used to configure kernel parameters at runtime? * [ ] modprobe * [ ] proc * [x] sysctl * [ ] make modules **6.** What is the default location for the kernel source tree? * [ ] /etc/src * [ ] /usr/lib/src * [x] /usr/src * [ ] /src **7.** What are accepted names for the makefile? (Choose all that apply.) * [x] makefile * [ ] make\_file * [x] gnumakefile * [x] Makefile **8.** What command intelligently adds and removes modules from the kernel? * [x] modprobe * [ ] modinfo * [ ] modules.dep * [ ] insmod **9.** Which of the following commands are used to generate the initramfs image? (Choose two.) * [x] mkinitramfs * [x] mkinitrd * [ ] make-initrd * [ ] make initramfs **10.** What command is used to monitor kernel uevents? * [ ] modinfo * [x] udevadm monitor * [ ] dmesg * [ ] udevadm-monitor **11.** What option for lspci shows kernel drivers handling each device and modules that are capable of handling it? * [ ] -m * [ ] -t * [ ] -n * [x] -k **12.** What command displays information about installed hardware? * [ ] lsusb * [ ] lspci * [x] lsdev * [ ] uname **13.** Which part of a Linux system manages system memory? * [ ] GNU utilities * [x] kernel * [ ] Graphical desktop * [ ] Application software **14.** What is the area on the hard disk called that is used as memory storage? * [x] Swap Space * [ ] Virtual memory * [ ] Filesystem * [ ] Physical Memory **15.** What method does Linux use to load device drivers into the kernel? * [ ] DLL Files * [ ] Virtual memory * [x] Modules * [ ] Bootloader **16.** Which kernel file type is used to boot the Linux system? * [x] Binary file * [ ] Modules * [ ] Patch file * [ ] Source code **17.** What type of release can you use to upgrade an existing kernel to a newer version? * [x] Patch release * [ ] Modules * [ ] Binary files * [ ] Development release **18.** Which make utility target uses text-based questions to configure the kernel? * [ ] xconfig * [ ] mrproper * [ ] clean * [x] config **19.** Which make utility target uses a graphical interface to configure the kernel? * [ ] config * [ ] mrproper * [x] gconfig * [ ] clean **20.** Which make utility target should you use to remove any old object files from a previous compile? * [ ] gconfig * [x] mrproper * [ ] bzImage * [ ] oldconfig **21.** Which make utility target is commonly used to generate a compressed kernel binary file? * [x] bzImage * [ ] kernel * [ ] config * [ ] clean **22.** Which program should you use to create an initial RAM disk for a Debian-based system? * [ ] mkinitrd * [ ] make bzImage * [x] mkinitramfs * [ ] make initrd **23.** Which command should you use to list all of the installed modules? * [ ] modinfo * [ ] insmod * [ ] rmmod * [x] lsmod **24.** Which command should you use to install a module if you don’t know its filename? * [ ] insmod * [ ] rmmod * [x] modprobe * [ ] lsmod **26.** Which command should you use to display information about USB devices installed on the system? * [x] lsusb * [ ] lspci * [ ] lsmod * [ ] lsdev **27.** Which option in the lsusb command displays detailed information about the USB devices? * [ ] -d * [ ] -s * [ ] -t * [x] -v **28.** Which type of hardware devices can you connect to the Linux system as it’s running? * [ ] Coldplug * [x] Hotplug * [ ] PCI * [ ] PCIe **29.** What program does Linux use to detect hotplug devices and load the appropriate modules? * [ ] lsusb * [x] udevd * [ ] modprobe * [ ] insmod **30.** Where are the udevd configuration settings stored? * [ ] /lib/modules * [ ] /boot * [ ] /etc/udev/rules.d * [x] /etc/udev/udev.conf **31.** Where are the udevd rules stored? * [ ] /etc/udev/udevd.conf * [x] /etc/udev/rules.d * [ ] /boot * [ ] /lib/modules **32.** Where is the dynamic pseudo-directory the kernel creates to peek at kernel settings and performance statistics located? * [ ] /etc/kernel * [ ] /boot/kernel * [x] /proc * [ ] /usr/src/linux **33.** What command allows you to change kernel settings during runtime? * [x] sysctl * [ ] lsmod * [ ] lsdev * [ ] modprobe **34.** Which of the following terms are used to describe 3.x kernel releases? (Choose TWO correct answers.) * [ ] beta * [ ] final * [x] longterm * [ ] prereleased * [x] stable **35.** How can the kernel parameter for the maximum size of the shared memory segment (shmmax) be changed to 2GB (2147483648 Bytes) on a running system? (Choose TWO correct answers.) * [ ] Edit /etc/shmmax and set the parameter to 2147483648. * [ ] sysctl shmmax=2147483648 * [x] sysctl kernel.shmmax=2147483648 * [x] echo 2147483648 > /proc/sys/kernel/shmmax * [ ] export kernel.shmmax=2147483648 **36.** What is the correct parameter to pass to the kernel at boot time to force it to use only one of the available processors? * [x] maxcpus=1 * [ ] usecpus=1 * [ ] smpcpus=1 * [ ] vcpumx=1 **37.** Which commands are used to load modules into the Linux kernel? (Choose TWO correct answers.) * [x] insmod * [ ] loadmod * [ ] kernload * [x] modprobe * [ ] probemod **38.** What is a key difference between a zImage and bzImage kernel image? * [ ] zImage is compressed using gzip, bzImage is compressed using bzip2. * [ ] zImage is for 2.6 series kernels, bzImage is for 3.x series kernels. * [ ] zImage is limited to 64k, bzImage has no such restriction. * [x] zImage gets loaded completely into low memory, bzImage will load into high memory once low memory is full. **39.** Which of the following command sequences can be used to extract files contained in a initramfs file (/boot/initramfs) which is used by the kernel at boot time? * [ ] cp /boot/initramfs /tmp/initramfs.gz; gzip -c /tmp/initramfs.gz; mkdir /tmp/initramfs.dir ; cd\ /tmp/initramfs.dir ; cpio -i < /tmp/initramfs * [x] cp /boot/initramfs /tmp/initramfs.gz; gunzip /tmp/initramfs.gz; mkdir /tmp/initramfs.dir ; cd\ /tmp/initramfs.dir ; cpio -i < /tmp/initramfs * [ ] cp /boot/initramfs /tmp/initramfs.gz; gunzip /tmp/initramfs.gz; mount /tmp/initramfs /mnt/ -o loop -t initramfs * [ ] cp /boot/initramfs /tmp/initramfs.gz; gunzip /tmp/initramfs.gz; mkdir /tmp/initramfs.dir ; cd\ /tmp/initramfs.dir ; cpio -e /tmp/initramfs * [ ] cp /boot/initramfs /tmp/initramfs.gz; gunzip /tmp/initramfs.gz; mount /tmp/initramfs /mnt/ -o loop -t initrd **Description**: 1. `cp /boot/initramfs /tmp/initramfs.gz` - This command copies the initramfs file from the /boot directory to `/tmp` and renames it to `initramfs.gz`. Note that this step assumes the initramfs file is compressed with gzip. If it's not a gzipped file, the `.gz` extension shouldn't be added. 2. `gunzip /tmp/initramfs.gz` - This command decompresses the file initramfs.gz. After decompression, the file will be named `initramfs` in the `/tmp` directory. 3. `mkdir /tmp/initramfs.dir` - This creates a new directory where the contents of the initramfs file will be extracted. 4. `cd /tmp/initramfs.dir` - This changes the current working directory to the newly created directory. 5. `cpio -i < /tmp/initramfs` - This command uses `cpio` to extract the contents of the initramfs file into the current directory (`/tmp/initramfs.dir`). The `-i` option stands for "extract". **40.** On a server running the 3.4.50-11 Linux kernel, which file in the /boot directory contains the kernel configuration parameters? * [ ] config-linux-3.4.50-11 * [x] config-3.4.50-11 * [ ] system-3.4.50-11 * [ ] vmlinuz-3.4.50-11 * [ ] rc.config-3.4.50-11 **Description**: In a Linux system, the file that contains the kernel configuration parameters is typically named with the `config-` prefix followed by the kernel version. This file is located in the `/boot` directory. Based on the options you've provided and the kernel version you mentioned (3.4.50-11), the correct file would be: `config-3.4.50-11` **41.** Which archive format is used to create an initramfs image? * [ ] gzip * [ ] tar * [ ] RAR * [x] cpio * [ ] bzip2 **42.** What information does the file modules.dep provide? * [ ] A list of all modules, compiled or not, that are available to the kernel. * [ ] A list of modules trusted by the installed kernel. * [ ] A list of devices and their module name. * [x] A list of all compiled modules and their dependencies. * [ ] A list of modules the kernel needs to run. **43.** What is the purpose of the command udevadm monitor? * [x] It listens to kernel events produced by a udev rule and print information to the console. * [ ] It monitors the /dev directory for new devices. * [ ] It monitors the udev process and prints performance statistics to the console. * [ ] It communicates with D-Bus to setup new devices. **44.** How is the source code for the main Linux kernel modules distributed? * [x] It is inculded with the Linux kernel source code. * [ ] The kernel modules and downloaded on demand as they are used during compilation. * [ ] The kernel modules have their own release cycle and can be maintained seprately from the Linux kernel source. * [ ] It is provided as a seprate download alongside the Linux kernel source code of the same version. **45.** Which option to the update-rc.d command will cause the removal of all symlinks to /etc/rcX.d/\*test2 even when the script /etc/init.d/test2 still exists? * [ ] -n * [x] -f * [ ] -r * [ ] -d **Description**: The `-f` option stands for "force". When you use `update-rc.d -f test2` remove, it forces the removal of all symbolic links to the `/etc/init.d/test2` script from the `/etc/rcX.d` directories regardless of whether the `/etc/init.d/test2` script still exists. **46.** Which of the following files will be looked for and used by GNU make, if one of them exists, unless a different file is specified on the command line when trying to compile software from source code? (Choose TWO correct answers.) * [x] makefile * [ ] configure * [ ] makefile.in * [ ] config.h.in * [x] Makefile **47.** Before compiling a new kernel , what needs to be done? * [ ] Compile kernel modules. * [x] Configure the kernel options. * [ ] Delete olf kernel sources. * [ ] Change to runlevel 1. **Description**: Configuring the kernel options is a critical step in the kernel compilation process. This involves selecting the necessary modules, features, and settings that the new kernel will support. This configuration can be done using tools like `make menuconfig`, make `xconfig`, or `make gconfig` in the Linux kernel source directory. The configuration process creates a `.config` file, which is used by the `make` command during the compilation process. **48.** That script is included with the source of the kernel to fix a kernel ? * [x] patch * [ ] patch-kernel * [ ] apply-patch * [ ] update-kernel **Description**: In the context of Linux kernel development and maintenance, the `patch` utility is used to apply changes to the kernel source code. These changes are often distributed in the form of "patch files," which contain the differences between one set of files and another. When developers fix bugs or add features, they often do so by creating a patch file that can be applied to the kernel source using the `patch` command. **49.** What command would you use to apply a diff file to an original? * [ ] cat diff -file * [ ] cat diff -file >> kernel * [ ] patch * [x] patch < diff -file * [ ] patch > diff -file **Description**: This command uses the `patch` utility, which is specifically designed for applying diff files. The diff file contains the differences between files and is applied to the original files to update them. The syntax `patch < diff-file` means that the `patch` command takes the contents of the diff file as input. **50.** After configuring a new 2.4 series kernel , all dependencies, such as included files, need to be created. How can this be achieved? * [ ] make dependencies * [ ] make clean * [ ] make mrproper * [x] make dep * [ ] make test **Description**: The `make dep` command was used specifically with older Linux kernel series like 2.4 to create dependencies. This command ensures that all the necessary headers and files are properly linked based on the configuration choices made. **51.** What two archiving formats are used to create an initramfs image? * [x] gzip * [ ] tar * [ ] rar * [x] cpio * [ ] bzip2 **52.** After unpacking the source code for a Linux kernel, what is the first make command that should be executed, which will delete any current configuration and all generated files? This command will ensure that the maintainer does not leave inappropriate files in the kernel file. * [ ] make depend * [ ] make distclean * [ ] make config * [ ] make clean * [x] make mrproper **Description:** The `make mrproper` command is used to thoroughly clean the kernel source tree. It removes not only the compiled binaries and object files (which `make clean` does) but also the configuration files and other generated files that might not be needed for a fresh start. This command ensures that you start with a clean state, which is particularly important when compiling a new kernel or switching between different kernel versions. **53.** What script, in the linux / scripts directory , can be used to add upgrades or updates to the 2.6.xx kernel source code? * [ ] patch * [x] patch-kernel * [ ] kernel-update * [ ] upgrade-kernel * [ ] upgrade **Description:** The `patch-kernel` script is designed to apply a series of patch files to the Linux kernel source. It automates the process of updating or upgrading the kernel source code with new patches, which are often released for bug fixes, security updates, and new features. **54.** You have finished updating and resolving dependencies on some source code. What command should you run before recompiling the code in binary format? * [x] make clean * [ ] make all * [ ] make dep * [ ] make install **Description:** The purpose of `make clean` is to remove all files that were generated during previous builds. This ensures that you start the compilation with a clean slate, which is particularly important after updating source code or dependencies, as old object files or binaries might interfere with the new build process. **55.** What tool can be used to control the behavior of runtime of udev? * [ ] udev * [ ] udevctl * [x] udevadm * [ ] udevconfig * [ ] udevclient **Description**: `udevadm` is the command-line utility provided for interacting with the udev Linux device manager. It allows for managing the runtime behavior of the udev system, including triggering events, querying udev database, monitoring udev events, and controlling the udev daemon. **56.** Which command can be used to view the kernel messages created from the boot time so far? * [ ] initm * [ ] initmessage * [ ] initmsg * [ ] inittab * [x] dmesg **57.** Why mkinitrd was deprecated in favor of mkinitramfs from the series Kermel 2.6? (Select 2 correct answers). * [ ] It is not loaded until later in the boot process * [ ] An easy to use GUI is available in mkinitramfs * [x] mkinitrd depends on devfs * [x] SATA hard drives are not supported **58.** A precompiled module has been moved to `/lib/modules/`, but `modprobe -a ` fails to load it. What needs to be done to use this module? * [ ] make modules\_install must be run * [x] depmod must be run * [ ] The kernel must be re-compiled * [ ] Modules.conf or modprobe.conf must be edited * [ ] The system must be restarted **Description**: The `depmod` command creates a module dependency file (`modules.dep` and other map files) in the `/lib/modules/` directory. This file is used by `modprobe` to automatically load the necessary modules and their dependencies. Running `depmod` ensures that the system is aware of the new module and its dependencies, making it possible for `modprobe` to load the module successfully. **59.** To restore a kernel source to its previous, unpatched version, which of the following commands could be used? * [ ] patch - restore * [ ] patch --remove * [ ] patch -U * [ ] patch -undo * [x] patch -R **Description:** The `-R` option with the `patch` command reverses the patching process. It essentially "undoes" the changes made by a patch. When you apply a patch using `patch`, it makes changes to the source code based on the contents of the patch file. Using `patch -R` with the same patch file will revert those changes, restoring the source code to its previous state. **60.** If the current directory is `/root` and the kernel source is located in `/usr/src/linux`, which of the following commands should be used to apply the `/tmp/foopatch` patch? * [ ] cat /tmp/foopatch | patch -p0 * [ ] cd /usr/src/linux; cat /tmp/foopatch | patch -p0 * [ ] cd /usr/src/linux; cat /tmp/foopatch | patch * [x] cd /usr/src/linux; patch -p1 < /tmp/foopatch * [ ] cd /usr/src/linux; patch -p1 > /tmp/foopatch **61.** A 2.6.9-ac1 kernel would be: * [ ] An alpha kernel. * [x] Some patch for a stable kernel. * [ ] A stable kernel. * [ ] An unstable kernel. * [ ] Some patch for an unstable kernel. **62.** What command should be used to find out which PCI devices can be seen, without consulting the kernel? * [x] lspci -v * [ ] lspci -s * [ ] lspci -x * [ ] lspci -b * [ ] lspci -p **Description**: The `lspci` command lists all PCI devices. The `-v` flag (for "verbose") provides detailed information about each device. This command reads the device information directly from the PCI configuration space, so it does not rely on the kernel's view of the devices. **63.** User Joseph successfully extracted and compiled a program from source code. Installing the Binaries produces errors. What is the most likely reason? * [ ] The source code was compiled for a different CPU. * [ ] The permissions set on the `/usr/bin` directory are wrong. * [x] Binaries requier root privileges to be installed. * [ ] An incorrect prefix was used when configuration the source code. **64.** The structure of the official Linux kernel can be classified as: * [x] Monilithic * [ ] Micro-kernel * [ ] Stable * [ ] Multitasking **Description**: The Linux kernel is known as a monolithic kernel. This means it operates in a single large process running entirely in a single address space. It's distinguished from micro-kernel systems where the kernel is broken down into separate processes known as servers. The terms "stable" and "multitasking" are characteristics or features of an operating system, but they do not describe its structural classification. **65.** What is the correct Linux kernel version numbering format? * [ ] 09.04 * [ ] 2008 * [x] 2.6.31 * [ ] 2008 Server **66.** What do the suffixes added to the kernel version number represent? * [ ] The initials of the name of the author of the kernel. * [x] A specific kernel, changed from the corresponding official kernel * [ ] An unstable version of kernel * [ ] A kernel that is not provided by the official website kernel.org **67.** Where can the official kernel version specific documentation be found? * [ ] In linux newsgroups * [ ] In the source code of the module * [x] In the Documentation directory where the kernel source code was placed. * [ ] In the comments left by the developers. **68.** What commands can be used to generate an Initial Ramdisk? * [ ] mkisofs * [ ] fdisk * [x] mkinitramfs * [x] mkinitrd **69.** Which command can be used to configure the Linux kernel? * [x] make config * [x] make xconfig * [x] make menuconfig * [x] make gconfig **70.** The uname -r command reports: * [ ] The resources available in the running kernel. * [ ] The modules loaded into memory for the running kernel. * [ ] The modules not loaded into memory for the running kernel. * [x] The version of the kernel source code that is running **71.** The correct way to load the ehci\_hcd module and its dependencies is: * [ ] make modules\_install * [ ] make modules\_install ehci\_hcd * [x] modprobe ehci\_hcd * [ ] insmode ehci\_hcd **72.** What type of compression is used in a zImage kernel? * [x] gzip * [ ] bzip2 * [ ] rar * [ ] cpio **73.** What information is not normally provided by the modinfo command? Select everything that applies. * [ ] license * [ ] author * [x] in use * [ ] description * [x] size **74.** After compiling a new kernel, which of the following is a valid filename for the compressed kernel? * [ ] kernel * [ ] image * [x] zimage * [ ] vmlinux **Description**: `vmlinux` is the uncompressed kernel image. **75.** Which of the following commands will show the status of the running kernel? * [x] cat /proc/sys/kernel/threads-max * [ ] cat /dev/kernel/threads-max * [ ] cat /etc/sys/kernel/threads-max * [ ] cat /etc/sys/kernel/threads-max * [ ] cat /proc/sys/kernel/threads/max **76.** What is the conventional name for a Linux kernel initially compiled and bootable on an x86 system? * [ ] /boot/vmlinux-version * [x] /boot/vmlinuz * [ ] /boot/bzImage-version * [ ] /usr/src/linux-version **77.** In which directory can you look for information about the options that can be applied to your computer's Ethernet driver module? * [x] /usr/src/linux/Documentation * [ ] /lib/modules/options * [ ] /usr/share/doc/modules * [ ] /etc/modules.conf **77.** What kind of kernel image was largely abandoned on x86 computers in favor of the bzImage format? * [ ] vmlinux * [x] zImage * [ ] initramfs * [ ] initrd **78.** Which of the following commands, when typed in `/usr/src/linux` after configuring the kernel, compiles the main Linux kernel file and its modules? * [x] make bzImage * [ ] make modules\_install ehci\_hcd * [ ] make xconfig * [ ] make **79.** Which of the following kernel features should you compile in the main kernel file for an x86-64 disk-based installation to simplify system startup? (Select 2 responses). * [x] Drivers for the ATA boot disk controller or SCSI host adapter. * [x] Support for your root file system. * [ ] Drivers for the USB port. * [ ] Framebuffer controllers for the vide card. **80.** You have compiled and installed a new kerne , version 2.6.35.4. You now want to prepare an initial RAM disk. Which of the following commands will do this, depending on your distribution? (Select 2 options). * [x] mkinitrd -o /boot/initrd-2.6.35.4 2.6.35.4 * [ ] mkinitrd /boot/initrd-2.6.35.4 * [x] mkinitramfs -o 2.6.35.4 /boot/initrd-2.6.35.4 * [ ] mkinitramfs /boot/initrd-2.6.35.4 **81.** You downloaded the patch file - 2.6.35.4.bz2. What program will you use as part of the patching operation to handle the compression of this file? * [x] bunzip2 * [ ] gunzip2 * [ ] tar * [ ] cpio **82.** Which of the following commands must you enter to configure a Linux kernel using an interactive text mode tool? * [ ] make xconfig * [x] make menuconfig * [ ] make config * [ ] make textconfig **83.** You have configured and compiled your new kernel, version 2.6.35.4. Now you type make modules\_install. Where can you expect to find the module files? * [ ] /lib/modules/modules-2.6.35.4 * [ ] /usr/src/linux/2.6.35.4 * [x] /lib/modules/2.6.35.4 * [ ] /usr/lib/2.6.35.4 **84.** In which file in the Linux kernel source tree are configuration options stored? * [ ] kernel.conf * [ ] configure * [x] .config * [ ] linux.conf **Description:** The `.config` file in the root of the Linux kernel source tree holds the configuration options for the kernel. This file is generated when you run a kernel configuration tool like `make menuconfig`, `make xconfig`, or `make config`, and it reflects the choices made for various kernel features and modules. The other file names listed are not standard for storing kernel configuration options. **85.** You found a kernel module for your network card, but you don't know what parameters to use. Which command can provide you with this information? * [ ] modeprobe * [ ] insmode * [ ] depmode * [x] modinfo * [ ] cat /proc/modeinfo **86.** Which of the following commands will load a kernel module, along with all necessary dependency modules? * [ ] depmode * [ ] insmode * [x] modeprobe ehci\_hcd * [ ] loadmode **87.** Which command should load the msdos.o module and all its dependencies? * [ ] modeinfo -a msdos * [ ] lsmode -a msdos * [x] modeprobe msdos * [ ] issmode -d msdos **88.** Which command will remove a kernel module? * [x] rmmod * [ ] unmode * [ ] delmod * [ ] modeprobe * [ ] unloadmod **89.** You have just added new modules to the system. Which command should you run to rebuild the modules.dep file? * [ ] depmode -rebuild * [ ] update-dependencies * [x] depmode -a * [ ] insmode -dependencies **90.** Which of the following correctly describes the relationship between depmod and modprobe? * [ ] modproble creats a dependency file for use by depmode. * [x] depmode creats a dependency file for use by modproble. * [ ] They have no relationship. * [ ] They can replace each other. **Description:** `depmod` is used to create a dependency file (typically named `modules.dep`) which lists the dependencies for each module. This file is then used by `modprobe` to understand which modules need to be loaded when a particular module is requested, ensuring that all necessary dependencies are loaded automatically. They have a complementary relationship, but they do not replace each other, as each serves a distinct purpose in the management of kernel modules. **91.** Which option of the depmod command allows you to print a list of all unresolved symbols? * [ ] -l * [ ] -i * [ ] -a * [x] -e **92.** Which of the following commands loads the module file into the kernel and changes any symbols that are defined on the command line? * [ ] depmode * [x] insmode * [ ] modprobe * [ ] setmod **93.** What option of the insmod command can be used to force the module to load even if problems are encountered? * [ ] -F * [x] -f * [ ] -u * [ ] -x **94.** Which option in the rmmod command specifies that all output should be sent to the syslog? * [ ] -a * [ ] -i * [x] -s * [ ] -d **95.** Which utility examines the object module\_file file associated with a kernel module and displays all the information it can collect? * [ ] depmode * [x] modinfo * [ ] modprobe * [ ] insmod **96.** Which modprobe option specifies the loading of all corresponding modules instead of stopping after the first successfully loaded module is loaded? * [ ] -f * [ ] -c * [ ] -e * [x] -a **96.** Which modprobe option allows you to define "autoclean" on loaded modules? * [ ] -e * [ ] -c * [x] -k * [ ] -d **97.** Which parameter of the uname command can be used to display the kernel version? * [ ] -m * [ ] -n * [ ] -s * [ ] -v * [x] -r **98.** Which parameter of the uname command can be used to display the machine type (Hardware / Architecture)? * [ ] -n * [ ] -s * [x] -m * [ ] -r # 202: System Startup ## 202.1 Customising system startup (weight: 3) {% hint style="success" %} Candidates should be able to query and modify the behaviour of system services at various targets / run levels. A thorough understanding of the systemd, SysV Init and the Linux boot process is required. This objective includes interacting with systemd targets and SysV init run levels. **Key Knowledge Areas:** * Systemd * SysV init * Linux Standard Base Specification (LSB) **The following is a partial list of the used files, terms and utilities:** * /usr/lib/systemd/ * /etc/systemd/ * /run/systemd/ * systemctl * systemd-delta * /etc/inittab * /etc/init.d/ * /etc/rc.d/ * chkconfig * update-rc.d * init and telinit {% endhint %} ### The Linux Standard Base (LSB) The goal of the LSB is to develop and promote a set of open standards that will increase compatibility among Linux distributions and enable software applications to run on any compliant system even in binary form. In addition, the LSB will help coordinate efforts to recruit software vendors to port and write products for Linux Operating Systems. ### Understanding Sys-V init #### The init process The init process or init boot is the first process that is called and used to start all other processes. #### /etc/inittab Defines three important items for the init process. * The system’s default runlevel * What processes to start, monitor, and restart if they terminate * What actions to take when the system enters a new runlevel #### Runveles A runlevel is the initialization state that determines which system should be running. | Runlevel | Description | | --------------------------------------- | -------------------------------------------------------------------------------- | | 0 `Halt` | Shuts down the sysem | | 1 `Single-user mod` | Does not configure network interfaces, startat daemons, or allow non-root logins | | 2 `Multi-user mod (Without networking)` | Does not configure network interfaces or export network services | | 3 `Multi-user mod (With networking)` | Start the system normally | | 4 `Undefined` | Not used / User-definable | | 5 `Multi-user (with GUI)` | Start the system normally with appropriate display manager | | 6 `Reboot` | Reboot the system | #### /etc/rc.d/ or /etc/rcN.d/ Once the default runlevel is discovered, scripts in the corresponding rc.d directory (or (rcN.d) are executed to bring the system to the desired states. The scripts are formatted as KNN or SNN where the K determines what is stopped and the S determines what is started. The number (NN) determines the order in which they are stopped or started. #### /etc/init.d/ Contains initialization scripts for all the services on the host which are called by the kill and start scripts in `/etc/rc.d/` or `/etc/rcN.d/`. #### chkconfig Updates and queries runlevel information for system services ```bash chkconfig [options] ```
RunlevelDescription
--levelSpecifies the run levels an operation should pertain to
--addAdds a new service for management by chkconfig
--delRemove a service from chkconfig management
--overrideChange configuration for a service based on /etc/chkconfig.d/
--listList services (or service) for which chkconfig is aware
#### update-rc.d Updates the SysV-init Script links in `/etc/runlevel.d/` (Debian) ```bash update-rc.d [-n] [-f] name remove update-rc.d [-n] name defaults update-rc.d [-n] name disable|enable [S|2|3|4|5] ``` #### init Change the current runlevel ```bash init [options] ... ``` #### telinit Change the current runlevel and allows for additional environment variables. ```bash telinit [options] ... runlevel ``` #### runlevel Display the current and previous runlevel ```bash runlevel [options] ... [UTMP] ``` ```bash [root@linuxmaster ~]# cat /etc/inittab ... id:5:initdefault: # System initialization. si::sysinit:/etc/rc.d/rc.sysinit l0:0:wait:/etc/rc.d/rc 0 l1:1:wait:/etc/rc.d/rc 1 l2:2:wait:/etc/rc.d/rc 2 l3:3:wait:/etc/rc.d/rc 3 l4:4:wait:/etc/rc.d/rc 4 l5:5:wait:/etc/rc.d/rc 5 l6:6:wait:/etc/rc.d/rc 6 ... ``` ### Systemd Boot Process **Systemd vs. SysVinit:** One of the biggest differences between `systemd` and `sysvinit` is the way that they start services. `sysvinit` starts service in a defined order, one at a time, and `systemd` allows for parallel startup of services. This allows `systemd` to start services simultaneously, based on dependencies or when a service is needed. #### Systemd Unit Files A unit file is a plain text init-style file that encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a startup target, a watched file system path, a timer-controlled and supervised by `systemd`, a resource management slice, or a group of externally created processes. Unit files are located in three different directories and operate based on precedence (greatest to least): * `/etc/systemd/`: Unit files created with `systemctl` enable and unit files added for extending service. * `/run/systemd/`: Unit files created at runtime. * `/usr/lib/systemd`: Unit files distributed with installed RPM packages. Unit files contain three main configuration sections: * `[Unit]`: Contains generic options that are independent of the unit type, like the description, the unit’s behavior, and dependencies on other units. * `[Unit_type]`: This heading reflects the unit type and the directives are particular to that kind of unit (type-specific). * `[Install]`: Contains directives used by the `systemctl` `enable` and `disable` commands. ```bash [root@linuxmaster ~]# cat /usr/lib/systemd/system/sshd.service [Unit] Description=OpenSSH server daemon Documentation=man:sshd(8) man:sshd_config(5) After=network.target sshd-keygen.service Wants=sshd-keygen.service [Service] Type=notify EnvironmentFile=/etc/sysconfig/sshd ExecStart=/usr/sbin/sshd -D $OPTIONS ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartSec=42s [Install] WantedBy=multi-user.target ``` #### Systemd Services List installed unit files and their enablement state. e.g. ```bash systemctl list-unit-files ``` List installed service files and their enablement state. ```bash systemctl list-unit-files --type service ``` Reload `systemd` manager configuration. This will rerun all generators, reload all unit files, and recreate the entire dependency tree. ```bash systemctl daemon-reload ``` Find overridden configuration files. The command allows for an optional prefix and suffix, which must be one of the directories containing configuration files: `/etc`, `/run`, or `/usr/lib`. Otherwise, it shows all overriding files. ```bash systemd-delta ``` #### Systemd Targets Targets or target units are used to link a group of units together to describe a desired system state. This can contain services or even other targets which have their own groups of services. In addition to the dependencies mentioned in the unit file, a target can also have a “.wants” directory that links to units that will be started along with the target. ```bash [root@linuxmaster ~]# cat /usr/lib/systemd/system/multi-user.target [Unit] Description=Multi-User System Documentation=man:systemd.special(7) Requires=basic.target Conflicts=rescue.service rescue.target After=basic.target rescue.service rescue.target AllowIsolate=yes ``` Return the default target to boot into. This returns the target unit name `default.target` in aliases (symlinked). ```bash systemctl get-default ``` Set the default target to boot into. This sets (symlinks) the `default.target` alias to the given target unit. ```bash systemctl set-default ``` Start the unit specified on the command line and its dependencies and stop all others. ```bash systemctl isolate ``` ## 202.2 System recovery (weight: 4) {% hint style="success" %} Candidates should be able to properly manipulate a Linux system during both the boot process and during recovery mode. This objective includes using both the init utility and init-related kernel options. Candidates should be able to determine the cause of errors in loading and usage of bootloaders. GRUB version 2 and GRUB Legacy are the bootloaders of interest. Both BIOS and UEFI systems are covered. **Key Knowledge Areas:** * BIOS and UEFI * NVMe booting * GRUB version 2 and Legacy * grub shell * boot loader start and hand off to kernel * kernel loading * hardware initialisation and setup * daemon/service initialisation and setup * Know the different boot loader install locations on a hard disk or removable device. * Overwrite standard boot loader options and using boot loader shells. * Use systemd rescue and emergency modes. **The following is a partial list of the used files, terms and utilities:** * mount * fsck * inittab, telinit and init with SysV init * The contents of /boot/, /boot/grub/ and /boot/efi/ * EFI System Partition (ESP) * GRUB * grub-install * efibootmgr * UEFI shell * initrd, initramfs * Master boot record * systemctl {% endhint %} ### BIOS, UEFI, and NVME Booting #### BIOS (Basic Input/Output System): * Firmware that is stored in a chip on the motherboard * Detects and maps connected devices Perform a power-on-self-test (POST) of the system hardware * Runs in 16-bit processor mode * Support devices using the Master Boot Record (MBR) * Limited to drives of 2.1 TB or less * Initiates the boot sequence from a connected device #### UEFI (Uni Extensible Firmware interface) * Support both the Master Boot Record (MBR) and the GUID Partition Table (GPT) * Support drives greater than 2.1 TB * Runs in 32-bit or 64-bit mode * Support secure boot * Support networking #### UEFI Services * UEFI boot manager: A new method for interfacing between the operating system and the firmware. It is composed of data tables containing platform-related information as well as boot and runtime functions that are available to the operating system and the bootloader. * EFI System Partition (ESP): This is a special GTP partition with configurations and binaries normally located at `/efi` or /`boot/efi`. * UEFI shell: This shell can be used to launch EFI applications (including bootloaders) and gain information about the system or the firmware. * `efibootmgr`: A userspace application used to modify the Extensible Firmware Interface (EFI) Boot Manager. #### NVMe (Non-Volatile Memory Express) * An interface specification for accessing non-volatile memory storage media * Support for NVMe was added in the mainline kernel in version 3.3 * Booting from NVMe devices a motherboard with NVMe support. ### GRUB and GRUB2 Bootloaders #### The Linux Boot Process 1. The BIOS/UEFI: Validates that the hardware is functioning properly and locates a bootable device 2. The Bootloader: Loads the kernel and `initramfs` images 3. The Kernel: Configures and system, mount the root file system and runs the main initialization process 4. Post Kernel/Systemd: Spawns the system and user-level processes #### The GRUB Bootloader * The GRUB Bootloader: The Grand Unified Bootloader (GRUB) takes over from the bios at boot time and then loads the kernel as well as the `initramfs` (or `initrd`) which supports the kernel booting the system. * The `/boot` directory: This directory contains the kernel and `initrd` images as well as the `/grub` directory. * `grub.conf`: This is the main configuration file for GRUB. It allows you to change things like the kernel boot order, menu timeout, menu image, and even supplying a password for editing the kernel arguments at boot. * The GRUB shell: Severe system issues can lead to booting into the GRUB shell. This can also be done manually from the GRUB menu by pressing "c". ```bash [root@linuxmaster ~]# cat /boot/grub/grub.conf default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.18-398.e15) root (hd0,0) kernel /vmliuz-2.6.18.398.e15 ro root=/dev/VolGroup00/LogVol00 initrd /initrd-2.6.18.398.e15.img ``` #### The GRUB2 Bootloader * The GRUB2 Bootloader: GRUB2 is version two of the legacy GRUB bootloader. It brings several improvements over GRUB, including support for UUIDs and a more programmatic approach to generating the GRUB configuration file. * /`etc/grub.d` and `/etc/default/grub`: `/etc/grub.d` contains template files and `/etc/default/grub` contains settings, both of which are used to create the `grub.cfg` configuration file. * `grub2-mkconfig` (CentOS) and `grub-mkconfig` (Ubuntu): These commands are used to generate a new `grub.cfg` by reading the contents of `/etc/grub.d` and `/etc/defaults/grub.` ```bash grub-mkconfig [option] ``` * `grub2-install` (CentOS) and `grub-install` (Ubuntu): Install GRUB on a device. ```bash grub-install [options] [install_device] ``` ```bash [root@linuxmaster ~]# ls -al /etc/grub drwx------. 2 root root 4096 May 20 22:56 . drwxr-xr-x. 104 root root 16384 Aug 19 19:00 .. -rwxr-xr-x. 1 root root 8702 May 20 17:28 00_header -rwxr-xr-x. 1 root root 1043 Mar 22 2019 00_tuned -rwxr-xr-x. 1 root root 232 May 20 17:28 01_users -rwxr-xr-x. 1 root root 10781 May 20 17:28 10_linux -rwxr-xr-x. 1 root root 10275 May 20 17:28 20_linux_xen -rwxr-xr-x. 1 root root 2559 May 20 17:28 20_ppc_terminfo -rwxr-xr-x. 1 root root 11169 May 20 17:28 30_os-prober -rwxr-xr-x. 1 root root 214 May 20 17:28 40_custom -rwxr-xr-x. 1 root root 216 May 20 17:28 41_custom -rw-r--r--. 1 root root 483 May 20 17:28 README ``` ```bash [root@linuxmaster ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="consoleblank=0 systemd.show_status=true elevator=noop no_timer_check console=tty1 console=ttyS0,115200n8" GRUB_DISABLE_RECOVERY="true" ``` ### Recovering Filesystems and Working with Rescue and Emergency Targets * `fsck`: Check and repair a Linux filesystem Check and optionally repairs one or more Linux filesystems. A filesystem can be a device name, a mount point, an ext2 label, or UUID specifier. If no filesystem is given and the -A option is not specified, fsck will default to checking filesystems in /`etc/fstab`. The `fsck` the command is a frontend for the various file system checkers (e.g. `fsck.fstype`) ```bash fsck [options] [filesystem] -- [fs-specific-options] ``` | Option | Purpose | | ------ | --------------------------------------------------- | | `-t` | Specify the type of filesystem to be checked | | `-A` | Check filesystems in /etc/fstab | | `-C` | Display progress bars for the check | | `-N` | Perform a dry run | | `-a` | Automatically repair a filesystem | | `-n` | print errors but do not repair | | `-r` | Interactivity repair filesystems | | `-y` | Always attempt to repair a filesystem automatically | ### Systemd Rescue and Emergency Targets #### rescue.target A special target unit pulls in the base system (including system mounts) and spawns a rescue shell. Isolate to this target in order to administer the system in single-user mode with all file systems mounted but with no services running, except for the most basic. * Attempts to mount all local filesystems and start basic services * Does not activate networking * Does not allow logins * Enter the resuce target by running `systemctl isolate resuce.target` or adding `system.unit=rescue.target` or rescue to the kernel boot parameters #### emergency.target A special target unit starts an emergency shell on the main console. This target does not pull in other services or mounts. It is the most minimal version of starting the system in order to acquire an interactive shell. * Does not mount local filesystems (except `/root`) * Mounts the root filesystem as read-only * Provides a basic shell and starts very basic services * Enter the emergency target by running `systemctl isolate emergency.target` or adding `system.unit=emergency.target` or emergency to the kernel boot parameters ## 202.3 Alternate Bootloaders (weight: 2) {% hint style="success" %} Candidates should be aware of other bootloaders and their major features. \ **Key Knowledge Areas:** * SYSLINUX, ISOLINUX, PXELINUX * Understanding of PXE for both BIOS and UEFI * Awareness of systemd-boot and U-Boot **The following is a partial list of the used files, terms and utilities:** * syslinux * extlinux * isolinux.bin * isolinux.cfg * isohdpfx.bin * efiboot.img * pxelinux.0 * pxelinux.cfg/ * uefi/shim.efi * uefi/grubx64.efi {% endhint %} ### SYSLINUX Project #### SYSLINUX A boot loader for the Linux operating system that runs on an MS-DOS/Windows FAT filesystem. In simplifies first-time installation of Linux and creation of rescue and othe special purpose boot disks. The SYSLINUX configuration files is syslinux.cfg and is searched for in the following order: * `/boot/syslinux/syslinux.cfg` * `/syslinux/syslinux.cfg` * `/syslinux.cfg` Key configuration options: * DEFAUT - Default boot target * LABEL - Name of target * SAY - Text to print when chosen * KERNEL - Name of the kernel file * APPEND - Additional kernel options ```bash syslinux --directory /boot/syslinux/ -- install /dev/sdb1 #Installation Example ``` #### EXTLINUX A SYSLINUX variant that boots from a Linux filesystem. Originally, only supported EXT filesystems. * Supports many filesystem types: EXT, FAT, NTFS, Btrfs, XFS and UFS/FFS * Installer runs on a mounted filesystem rather than a raw devices. Ex: `extlinux --install /mount/point` * The configuration file is `extlinux.conf` and shoud be in the same directory as the EXTLINUX installation * The configuration options are the same as SYSLINUX #### ISOLINUX A SYSLINUX variant that creates a bootable CD-ROM for ISO 9660 filesystems. * All packages and binaries should be places in a directory called `CD_root`. * `isolinux.bin` and `isolinux.cfg` should be placed in a directory called `isolinux` within `CD_root`. * Isohybrid allows ISOLINUX to be installed on image files or block devices (the main binary is `isohdpfx.bin`) * The `mkisofs` command is used to create the bootable ISO. #### PXELINUX Used for booting from a network server that conforms to the Preboot Execution Environment (PXE) specification. * Requires the installation of SYSLINUX on a TFTP server. * On the TFTP server, create the directory `tftpboot`. * Copy `pxelinux.0` (the bootloader) and `idlinux.c32` (the library) to the `tftpboot` directory. * Create a directory called `pxelinux.cfg` within the `tftpboot` directory. * Create a configuration file in the `pxelinux.cfg` for each system by using the client UUID, MAC address, IPv4 address, or lowercase `default` for default configuration. ### PXE Booting for BIOS/UEFI PXE stands for Preboot Execution Environment, and it allows systems to boot over a local area network. * Support for PXE must be present in the network card and the BIOS/UEFI must be configured to allow it. * Once configured, the system will receive an IP address from a DHCP server and the necessary files for booting from a TFTP server. * `shim.efi` is the initial UEFI bootloader used in a secure boot environment. * `grubx64.efi` is the full grub bootloader for EFI systems using PXE booting. * `shim.efi` (if using secure boot) and `grubx64.efi` should be copied to a directory on the TFTP server. ### Systemd-boot A UEFI boot manager that executes configured EFI images. * Operates on the EFI System Partition (ESP) * Operates on the EFI System Partition (ESP) * All configuration files and boot images must reside on the ESP (e.g. kernels, initrds, etc.) * In order to execute a kernel as an EFI image, it myst be built with `CONFIG_EFI_STUB` * Uses a main configuration file to set the timeout and the default boot entry * Each boot entry requires a configuration file that accepts a few basic keywords: title, options, efi, linux, initrd, etc. ### U-boot Das U-boot or the Universal Boot Loader is an open-source primary boot loader for embedded systems. * A highly customizable bootloader, which makes it very popular with bot embedded systems and IoT devices * Can be split into stages to accommodate size constraints * Supports several different filesystems (e.g. ext, FAT, btrf, etc.) ## Sample Questions **1.** Which special target disables logins and networking while attempting to mount local filesystems? * [ ] emergency * [ ] multi-user * [ ] graphical * [x] rescue **2.** What option for fsck automatically attempts to repair a filesystem? * [ ] -A * [ ] -t * [x] -y * [ ] -r **3.** What bootloader has become popular for embedded systems and IoT devices? * [ ] Systemd-boot * [ ] SYSLINUX * [ ] PXELINUX * [x] U-Boot **4.** What directory has the greatest precedence for unit files in Systemd? * [ ] /etc/run/systemd * [ ] /usr/lib/systemd * [ ] /run/systemd * [x] /etc/systemd **5.** What command shows overridden configuration files in Systemd? * [x] systemd-delta * [ ] systemd show-diff * [ ] systemd-reload * [ ] systemctl get-default **6.** What processor mode does the BIOS run? * [ ] 8-bit * [ ] 64-bit * [x] 16-bit * [ ] 32-bit **7.** What commands are used to modify services for runlevels in SysVinit? (Choose two answers.) * [x] chkconfig * [ ] runlevel * [x] update-rc.d * [ ] init **8.** What is the main configuration file for the legacy GRUB bootloader? * [ ] grub.cfg * [ ] default.grub * [ ] /etc/default/grub * [x] grub.conf **9.** What file defines the default runlevel in SysVinit systems? * [ ] update-rc.d * [ ] rc.sysvinit * [ ] init.d * [x] inittab **10.** In what version of the Linux Kernel was NVMe first supported? * [ ] Version 2.6 * [ ] Version 3.4 * [ ] Version 3.3 * [ ] Version 4.3 **11.** Which of the following is a partitioning system supported by UEFI? * [ ] MPT * [x] GPT * [ ] MBR * [ ] PGP **12.** What directory and configuration file are used to build the grub.cfg for GRUB2? (Choose two.) * [ ] /etc/rc.d * [x] /etc/default/grub * [x] /etc/grub.d/ * [ ] /boot/grub/grub.conf **13.** Which of the following is the name of a joint project whose goal is to promote a set of standards to increase compatibility among Linux distributions? * [ ] The Linux Foundation * [x] The Linux Standard Base (LSB) * [ ] The Portable Operating System Interface (POSIX) specification * [ ] The Filesystem Hierarchy Standard (FHS) **14.** What Linux service management and initialization system starts processes in parallel? * [x] Systemd * [ ] systemd-delta * [ ] LinuxInit * [ ] SysVinit **15.** Which of the following are variants of SYSLINUX? (Choose all that apply.) * [ ] NETLINUX * [x] PXELINUX * [x] ISOLINUX * [x] EXTLINUX **16.** What program does the workstation firmware start at boot time? * [x] Bootloader * [ ] The `init` program * [ ] The Windows OS * [ ] The `mount` command * [ ] The `telinit` program **17.** Where does the firmware first look for a Linux bootloader program? * [ ] The `/boot/grub` folder * [x] The Master Boot Record (MBR) * [ ] The `/var/log` folder * [ ] A boot partition * [ ] The `/etc` folder **18.** What Linux command lets you examine the most recent boot messages? * [ ] `fsck` * [ ] `init` * [ ] `mount` * [x] `dmesg` * [ ] `chkconfig` **19.** What folder do most Linux distributions use to store boot logs? * [ ] /etc * [ ] /var/message * [x] /var/log * [ ] /boot * [ ] /proc **20.** Where does the workstation BIOS attempt to find a bootloader program? (Select all that apply.) * [x] An internal hard drive * [x] An external hard drive * [x] A DVD drive * [x] A USB memory stick * [x] A network server **21.** Where is the Master Boot Record located? * [x] The first sector of the first hard drive on the system * [ ] The boot partition of any hard drive on the system * [ ] The last sector of the first hard drive on the system * [ ] Any sector on any hard drive on the system * [ ] The first sector of the second hard drive on the system **22.** Where is the EFI System Partition (ESP) stored on Linux systems? * [ ] `/boot` * [ ] `/etc` * [ ] `/var` * [x] `/boot/efi` * [ ] `/boot/grub` **23.** What file extension do UEFI bootloader files use? * [ ] `.cfg` * [ ] `.uefi` * [ ] `.lst` * [ ] `.conf` * [x] `.efi` **24.** Which was the first bootloader program used in Linux? * [ ] GRUB Legacy * [x] LILO * [ ] GRUB2 * [ ] SYSLINUX * [ ] ISOLINUX **25.** Where are the GRUB Legacy configuration files stored? * [x] `/boot/grub` * [ ] `/boot/efi` * [ ] `/etc` * [ ] `/var` * [ ] `/proc` **26.** Where are GRUB2 configuration files stored? (Select all that apply.) * [ ] `/proc` * [x] `/etc/grub.d` * [x] `/boot/grub` * [ ] `/boot/efi` * [ ] `/var` **27.** What command must you run to generate the GRUB2 grub.cfg configuration file? * [ ] `chkconfig` * [ ] `update-rc.d` * [x] `grub-mkconfig` * [ ] `grub-install` * [ ] `init` **28.** What program does the kernel use to start other programs? * [ ] GRUB2 * [ ] systemctl * [ ] telinit * [x] init * [ ] BIOS **29.** Which configuration file contains the SysV default runlevel? * [ ] `/etc/init.d` * [x] `/etc/inittab` * [ ] `/etc/grub.d` * [ ] `/etc/rc.d` * [ ] `/boot/grub.cfg` **30.** What runlevel is the default for Debian-based systems? * [ ] 0 * [ ] 1 * [ ] 6 * [ ] 5 * [x] 2 **31.** What command would you use to change the current runlevel? (Select all that apply.) * [x] `telinit` * [ ] `chkconfig` * [ ] `update-rc.d` * [x] `init` * [ ] `dmesg` **32.** What command displays the runlevels in which a program will be started? * [x] `chkconfig` * [ ] `init` * [ ] `dmesg` * [ ] `update-rc.d` * [ ] `telinit` **33.** What command do Debian systems use to set the runlevels for programs? * [ ] `chkconfig` * [x] `update-rc.d` * [ ] `init` * [ ] `telinit` * [ ] `dmesg` **34.** What program allows you to fix corrupt hard drive partitions? * [ ] `mount` * [ ] `umount` * [x] `fsck` * [ ] `init` * [ ] `telinit` **35.** Which command allows you to append a partition to the virtual directory on a running Linux system? * [x] `mount` * [ ] `umount` * [ ] `fsck` * [ ] `dmesg` * [ ] `init` **36.** Which of the following init systems comes along with an own UEFI boot loader? * [x] systemd * [ ] SysVinit * [ ] Upstart * [ ] OpenRC * [ ] launchd **37.** What effect does the `–f` option to the `update-rc.d` command have on files in the `/etc/rcX.d/` and `/etc/init.d/` directories? * [ ] It will force the removal of the symlinks in `/etc/rcX.d/` even when the links are read only * [ ] It will remove both the symlinks in `/etc/rcX.d/` and the init script in `/etc/init.d/` * [ ] It will remove the init script in `/etc/init.d/` and the symlinks in `/etc/rcX.d/` and update the package information of installed files * [x] It will force the removal of symlinks in `/etc/rcX.d/` even if the corresponding init script still exists in `/etc/init.d/` **38.** Which command is used to install a GRUB boot loader into the master boot record? * [x] grub-install * [ ] grub-mkconfig * [ ] grub-install-mbr * [ ] grub-glue-mbr * [ ] grub-mbr-setup **39.** Which directory contains system-specific systemd unit files? (Specify the full path to the directory.) **Answer:** `/lib/systemd/system` **40.** A Linux server running systemd booted to `rescue.target` for maintenance. Which commands are used to restore the server to its usual target? (Choose two.) * [ ] `telinit 0` * [x] `systemctl default` * [ ] `sync` * [ ] `systemctl emergency` * [x] `systemctl reboot` # 203: Filesystem and Devices ## 203.1 Operating the Linux filesystem (weight: 4) {% hint style="success" %} Candidates should be able to properly configure and navigate the standard Linux filesystem. This objective includes configuring and mounting various filesystem types. **Key Knowledge Areas:** * The concept of the fstab configuration * Tools and utilities for handling swap partitions and files * Use of UUIDs for identifying and mounting file systems * Understanding of systemd mount units **The following is a partial list of the used files, terms and utilities:** * /etc/fstab * /etc/mtab * /proc/mounts * mount and umount * blkid * sync * swapon * swapoff {% endhint %} ### Mounting Filesystems * `mount`: Mount a filesystem 
mount [option] device mountpoint
| Option | Description | | ------ | ------------------------------------- | | `-a` | Mount all filesystems in `/etc/fstab` | | `-F` | With the -a option, mount in parallel | | `-n` | Mount without writing in `/etc/mtab` | | `-o` | Use the specified mount options | | `-t` | Specify the filesystem type | | `-U` | Mount a partition by UUID | | `-v` | Increase verbosity | * `umount`: Unmount a filesystem 
umount [option] directory|device
| Option | Description | | ------ | -------------------------------------------- | | `-a` | Unmount filesystems described in `/etc/mtab` | | `-t` | Act only on filesystem type specified | | `-f` | Force an unmount | | `-n` | Unmount without writing to `/etc/mtab` | | `-v` | Increase verbosity | * `sync`: Synchronize cached writes to persistent storage 
sync [option] [file] ...
* `blkid`: Command line utility to locate and print block device attributes. It has two main forms of operation: either searching for a device with a specific NAME=value pair or displaying NAME=value pairs for one or more devices. 
blkid -L lable | - U uuid or blkid [option] device
* `/etc/fstab`: Static information about the filesystem {% code overflow="wrap" %} ```bash # A B C D E F UUID=47C4-076C /home ext4 defaults 0 0 # A: Describe a block special device, remote filesystem, filesystem image, or swap file. # B: Describe the mount point or target for the filesystem. # C: Describe the type of filesystem (e.g. ext4, xfs, nfs, etc.) # D: Describe the mount options associated with the filesystem. Basic filesystem independent options include the following: defaults, noauto, user, owner, comment, no fail. # E: Used by dump to determine which filesystems need to be dumped (0 disable). # F: Used by fsck to determine the order in which filesystem checks are done at boot time. the root filesystem should be set to 1 and all other filesystem should be set to 2. The default of 0 disablesthe check. ``` {% endcode %} 
cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=47C4-076C                            /boot/efi      vfat    umask=0077 0 2
UUID=6a6b5a66-f7c0-44c1-8c81-be91ca97b0c9 /              ext4    defaults,noatime 0 1
* `/etc/mtab`: The programs mount and umount traditionally maintained a list of currently mounted filesystems in `/etc/mtba`. For modern systems, `/etc/mtab` is a symlink to `/proc/mounts` (or `/proc/self/mounts`). * `/proc/mounts`: Part of the proc virtual filesystem that presents data from the kernel about the mounted filesystems on the host. Can be a symlink to `proc/self/mounts`. 
cat /proc/mount

proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
/dev/nvme0n1p5 / ext4 rw,noatime 0 0
...
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
### Linux Swap Space Swap space is used as a substitute for physical memory (RAM) when the physical memory is full. This is helpful for systems with small amounts of memory but is not a substitute for adding additional RAM (because of inefficiency is speed). Swap space can be a dedicated partition or file (or both). The `/proc/swaps` file can be viewed to determine the amount of swap in use. * `swapon/swapoff`: Enable or disable devices and files for paging and swapping. 
swapon [options] [specialfile...]
| Option | Description | | -------- | ---------------------------------------------- | | `-a` | Enable all devices marked swap in `/etc/fstab` | | `-o` | Specify swap options | | `-p` | Specify the priority of the device | | `-s` | Display swap usage summary by device | | `--show` | Display a definable table of swap areas | | `-v` | Increase verbosity | * `mkswap`: Set up a Linux swap area. 
mkswap [options] device [size]

cat /proc/swaps

Filename				Type		Size	Used	Priority
/root/swap01                            file		2047996	0	-2
### Systemd Mount Units A systemd mount unit is a unit configuration file that ends in ".mount" and encodes information about a file system mount point controlled and supervised by Systemd. Systemd automatically creates unit files for all the entries in `/etc/fstab`. These are in `/run/systemd/generator/`. Mount units can be manually created, but it is generally advised to add entries to `etc/fstab`. * `cat /run/systemd/generator/mnt-my.mount` ```bash # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) [Mount] What=/dev/disk/by-uuid/baabdf1f-5449-4f48-a8cd-7e188dbefb5d Where=/mnt/my Type=ext4 Options=discard,nofail,defaults ``` ## 203.2 Maintaining a Linux filesystem (weight: 3) {% hint style="success" %} Candidates should be able to properly maintain a Linux filesystem using system utilities. This objective includes manipulating standard filesystems and monitoring SMART devices. **Key Knowledge Areas:** * Tools and utilities to manipulate and ext2, ext3 and ext4 * Tools and utilities to perform basic Btrfs operations, including subvolumes and snapshots * Tools and utilities to manipulate XFS * Awareness of ZFS **The following is a partial list of the used files, terms and utilities:** * mkfs (mkfs.\*) * mkswap * fsck (fsck.\*) * tune2fs, dumpe2fs and debugfs * btrfs, btrfs-convert * xfs\_info, xfs\_check, xfs\_repair, xfsdump and xfsrestore * smartd, smartctl {% endhint %} ### Ext2/3/4 and XFS Filesystems #### ext2 * Developed to overcome some of the limitation of the original ext filesystem. * Max file size ranges from 16GB to 2 TB. * Max filesystem size ranges from 2 TB to 32 TB. * Supports 32,000 subdirectories. * Recommended for flash-based storage becase of the absence of overhead caused by journaling. * Can be directly converted to ext3. #### ext3 * Many of the base features remain the same as ext2. * Introduces a dedicated space for journaling. * Journaling levels: * Journal - Metadata and content are written to the journal. * Ordered (default) - Only metadata is written to the journal. Content is written before metadata is committed. * Writeback - Only metadata is written to the journal. Content may be written before or after metadata is commited. * Supports online filesystem growth. * Introduces HTree indexing for larger directories. #### ext4 * Max file size increase to 16 TB. * Max filesystem increase to 1 Exabyte. * Support a virtually unlimited number of subdirectories. * Introduces journal checksums as well as several other imporovements to reliability and performance. * Support backwards compatibility for ext2 and ext3. #### Utilities for the EXT Filesystem * `mkfs (mkfs.*)`: Build a Linux filesystem. The mkfsfroundend utility deprecated in favor of the filesystem-specific mkfs.utilities (e.g. `mkfs.ext4`, `mkfs.xfs`, etc.) 
mkfs [options] [-t type] [fs-options] device [size]
* `fsck`: Checks and optionally repairs one or more Linux filesystems. A filesystem can be a device name, a mount point, an ext2 label, or a UUID specifier. The `fsck` command is a frontend for the various filesystem checkers (e.g. `fsck.fstype`) 
fsck [options] [filesystem] -- [fs-specific-options]
| Option | Purpose | | ------ | --------------------------------------------------- | | `-t` | Specify the type of filesystem to be checked | | `-A` | Check filesystem in `/etc/fstab` | | `-C` | Display progress bars for the check | | `-N` | Perform a dry run | | `-a` | Automatically repair a filesystem | | `-n` | Print errors but do not repair | | `-r` | Interactively repair filesystems | | `-y` | Always attempt to repair a filesystem automatically | * `tune2fs`: Adjust tunable filesystem parameters on ext2/ext3/ext4 filesystems. 
tune2fs [options] device
| Option | Purpose | | ------ | ------------------------------------------------------------------------------------------------ | | `-l` | List the contents of the filesystem superblock | | `-L` | Set the volume lable for the system | | `-j` | Add an ext3 journal to the filesystem | | `-J` | Override the default ex3 journal parameters | | `-m` | Set the percentage of the filesystem which may only be allocated by privileged processes. | | `-o` | Set or clear the indicated default mount options ('^' is used to clear option) | | `-O` | Set or clear the indicated filesystem features in the filesystem ('^' is used to clear features) | * `dumpe2fs`: Dump ext2/ext3/ext4 filesystem information. Prints the super block and blocks group information for the device specified. The `-h` option is used to only display superblock information. 
dumpe2fs [options] device
* `debugfs`: ext2/ext3/ext4 filesystem debugger. Used to interactively examine and change the state of the filesystem. The `-w` option specifies that the filesystem should be opened in read-write mode. 
debugfs [options] device
#### XFS XFS stands for the extents filesystem, which is a reference to the way blocks are allocated and managed. Extents are one or more contiguous blocks that vary in length. It also implements the concept of allocation groups (AG), which are equally-sized linear regions on the filesystem. Although files and directories can span across multiple AGs, each AG is responsible for managing its own inodes and free space. Allocates blocks using extents, Manages storage through allocation groups (AGs) and Implements journaling Supports large filesystems and individual files (16 exbibvtes and 8 exbibytes respectivelv) * `xfs_info`: Display XFS filesystem geometry information. * `xfs_check`: Check the consistency of an XFS filesystem. This command is deprecated in favor of `xfs_repair -n` * `xfs_repair`: Repair a corrupt or damaged XFS filesystem. Using the `-n` option will perform a dry run. * `xfsdump`: Backs up files and their attributes to a storage media, regular file, or standard output. The `-f` option can be used to specify a device. * `xfsrestore`: Restore a filesystem from dumps produced by the `xfsdump` utility. The `-f` option is used to specify a source of the dump to be restored. ### Btrfs and ZFS Filesystems #### Btrfs Filesystem * The b-tree filesystem, or Btrfs, is a modern copy on write (CoW) filesystem. It introduces advanced features while focusing on fault tolerance, repair, and easy administration. Some of these features include: * Space-efficient packing of small files and indexed directories * Snapshots (Writable and read-only) * Support for software-based RAID * Subvolumes (seprate internal filesystem roots) * Self-healing * Large file size and directory * `btrfs`: The `btrfs` utility is a toobox for managing Btrfs filesystems. 
btrfs <command> [<args>]
| Option | Purpose | | ---------- | ------------------------------------------------------------ | | check | Balance filesystem chunks across a single or several devices | | device | Perform an off-line check on a filesystem | | filesystem | Manae a Btrfs filesystem | | replace | Replace Btrfs devices | | rescue | Try to rescue a damaged filesystem | | restore | Try to restore files from a damaged filesystem | | subvolume | Create/Delete/List/Manage Btrfs subvolumes | 
root@linuxmaster:~# fdisk -l /dev/sdb
Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Volume          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x56dcfbb5

Device     Boot   Start     End Sectors Size Id Type
/dev/sdb1          2048 2099199 2097152   1G 83 Linux
/dev/sdb2       2099200 4196351 2097152   1G 83 Linux

root@linuxmaster:~# mkfs.btrfs /dev/sdb1
btrfs-progs v5.4.1 
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               0384a51c-30da-4c9a-a973-cccbace2210d
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              51.19MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb1

root@linuxmaster:~# mkdir /mnt/btrfs

root@linuxmaster:~# mount /dev/sdb1 /mnt/btrfs/

root@linuxmaster:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdb1       1.0G  3.5M  905M   1% /mnt/btrfs

root@linuxmaster:~# btrfs device add /dev/sdb2 /mnt/btrfs/

root@linuxmaster:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdb1       2.0G  3.5M  1.9G   1% /mnt/btrfs

root@linuxmaster:~# btrfs filesystem show /mnt/btrfs/
Label: none  uuid: 0384a51c-30da-4c9a-a973-cccbace2210d
	Total devices 2 FS bytes used 128.00KiB
	devid    1 size 1.00GiB used 126.38MiB path /dev/sdb1
	devid    2 size 1.00GiB used 0.00B path /dev/sdb2

root@linuxmaster:~# btrfs balance start --full-balance /mnt/btrfs/
Done, had to relocate 3 out of 3 chunks

root@linuxmaster:~# btrfs filesystem show /mnt/btrfs/
Label: none  uuid: 0384a51c-30da-4c9a-a973-cccbace2210d
	Total devices 2 FS bytes used 256.00KiB
	devid    1 size 1.00GiB used 240.00MiB path /dev/sdb1
	devid    2 size 1.00GiB used 448.00MiB path /dev/sdb2

root@linuxmaster:~# btrfs subvolume create /mnt/btrfs/svol1
Create subvolume '/mnt/btrfs/svol1'

root@linuxmaster:~# btrfs subvolume list /mnt/btrfs/
ID 258 gen 25 top level 5 path svol1

root@linuxmaster:~# touch /mnt/btrfs/svol1/test{1..5}.txt

root@linuxmaster:~# ll /mnt/btrfs/svol1/
total 16
drwxr-xr-x 1 root root 90 Sep  9 12:25 ./
drwxr-xr-x 1 root root 10 Sep  9 12:14 ../
-rw-r--r-- 1 root root  0 Sep  9 12:25 test1.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test2.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test3.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test4.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test5.txt

root@linuxmaster:~# mkdir /mnt/svol1

root@linuxmaster:~# mount -t btrfs -o subvol=svol1 /dev/sdb2 /mnt/svol1/

root@linuxmaster:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            949M     0  949M   0% /dev
tmpfs           194M  1.1M  193M   1% /run
/dev/sda1        19G  5.2G   13G  30% /
tmpfs           968M     0  968M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           968M     0  968M   0% /sys/fs/cgroup
/dev/sda15      253M  1.1M  252M   1% /boot/efi
tmpfs           194M     0  194M   0% /run/user/0
/dev/sdb1       2.0G  3.7M  1.6G   1% /mnt/btrfs
/dev/sdb1       2.0G  3.7M  1.6G   1% /mnt/svol1

root@linuxmaster:~# ll /mnt/svol1/
total 4
drwxr-xr-x 1 root root   90 Sep  9 12:25 ./
drwxr-xr-x 5 root root 4096 Sep  9 12:36 ../
-rw-r--r-- 1 root root    0 Sep  9 12:25 test1.txt
-rw-r--r-- 1 root root    0 Sep  9 12:25 test2.txt
-rw-r--r-- 1 root root    0 Sep  9 12:25 test3.txt
-rw-r--r-- 1 root root    0 Sep  9 12:25 test4.txt
-rw-r--r-- 1 root root    0 Sep  9 12:25 test5.txt

root@linuxmaster:~# ll /mnt/btrfs/svol1/
total 16
drwxr-xr-x 1 root root 90 Sep  9 12:25 ./
drwxr-xr-x 1 root root 10 Sep  9 12:14 ../
-rw-r--r-- 1 root root  0 Sep  9 12:25 test1.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test2.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test3.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test4.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test5.txt

root@linuxmaster:~# btrfs subvolume snapshot /mnt/btrfs/svol1/ /mnt/btrfs/snap1
Create a snapshot of '/mnt/btrfs/svol1/' in '/mnt/btrfs/snap1'

root@linuxmaster:~# btrfs subvolume list /mnt/btrfs 
ID 258 gen 27 top level 5 path svol1
ID 259 gen 27 top level 5 path snap1

root@linuxmaster:~# ll /mnt/btrfs/snap1/
total 16
drwxr-xr-x 1 root root 90 Sep  9 12:25 ./
drwxr-xr-x 1 root root 20 Sep  9 12:41 ../
-rw-r--r-- 1 root root  0 Sep  9 12:25 test1.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test2.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test3.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test4.txt
-rw-r--r-- 1 root root  0 Sep  9 12:25 test5.txt
* `btrfs-convert`: Perform an in-place conversion from ext2/3/4 filesystems to Btrfs. 
btrfs-convert [options] <device>

root@linuxmaster:~# lsblk -f | grep sdb3
└─sdb3  ext2         c067a099-27ce-4fb5-84d7-fbe8ec921055

root@linuxmaster:~# mkfs.ext2 -b 4096 /dev/sdb3
mke2fs 1.45.5 (07-Jan-2020)
/dev/sdb3 contains a ext2 file system
	created on Fri Sep  9 12:21:01 2022
Proceed anyway? (y,N) y
Discarding device blocks: done                            
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 6cc56630-86c9-4d54-ba65-43a72b3c5ec5
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

root@linuxmaster:~# btrfs-convert /dev/sdb3 
create btrfs filesystem:
	blocksize: 4096
	nodesize:  16384
	features:  extref, skinny-metadata (default)
	checksum:  crc32c
creating ext2 image file
creating btrfs metadata
copy inodes [o] [     23504/        11]
conversion complete

root@linuxmaster:~# btrfs filesystem show /dev/sdb3
Label: none  uuid: bf9c948e-3575-440c-82e2-65d6b8f2d4fa
	Total devices 1 FS bytes used 17.51MiB
	devid    1 size 1.00GiB used 291.00MiB path /dev/sdb3
#### **ZFS Filesystem** The ZFS filesystem, which originally stood for the zettabyte filesystem, combines the features of a filesystem and a volume manager. Like Btrfs, ZFS is a copy on write (CoW) filesystem. Some features of ZFS include: * Pooled storage * Snapshots (may be mounted as read-only) * Data integrity verification * Self-healing * Support for RAID and RAID-Z * Large file size and directory support ## 203.3 Creating and configuring filesystem options (weight: 2) {% hint style="success" %} Candidates should be able to configure automount filesystems using AutoFS. This objective includes configuring automount for network and device filesystems. Also included is creating filesystems for devices such as CD-ROMs and a basic feature knowledge of encrypted filesystems. **Key Knowledge Areas:** * autofs configuration files * Understanding of automount units * UDF and ISO9660 tools and utilities * Awareness of other CD-ROM filesystems (HFS) * Awareness of CD-ROM filesystem extensions (Joliet, Rock Ridge, El Torito) * Basic feature knowledge of data encryption (dm-crypt / LUKS) **The following is a partial list of the used files, terms and utilities:** * /etc/auto.master * /etc/auto.\[dir] * mkisofs * cryptsetup {% endhint %} ### Using AutoFS to Automatically Mount Filesystems * Indirect mounts: The `auto.master` contains a directory that point to a map file that contians one or more subdirectories (or keys). * Direct mounts: The directory in the `auto.master` is represented as "/-" and points to a map file that contains the individual directory to be mounted. * `/etc/sysconfig/autofs`: Configuration file that sets global options for `autofs` mount. * `/etc/auto.master`: The main configuration file for `autofs`. * `/etc/auto.map_name`: Individual map files for `autofs`. * `cat /etc/automaster`: (Indirect Mount Example) ```bash /mnt/data /etc/auto.data ``` * `cat /etc/auto.data`: (Indirect Mount Example) ```bash backup -fstype=xfs, ro :/dev/nvme1n1p1 current -fstype=ext4 :/dev/nvme1n1p2 ``` * `cat /etc/master`: (Direct Mount Example) ```bash /- /etc/auto.design ``` * `cat /etc/auto.design`: (Direct Mount Example) ```bash /mnt/design -fstype=ext3 :/dev/nvme1n1p3 ``` * `cat /etc/auto.network`: (Network Mounts Example) ```bash key -fstype=nfs ip_addr:/nfs/share key -fstype=cifs,rw,credentials=/etc/creds.txt ://ip_addr/share ``` ### CD-ROM Filesystems * ISO 9660: A filesystem for optical disk media that was published by the International Organization for Standardization. It is a read-only filesystem and has been the de facto standard for CD-ROM disks. * Universal Disk Format (UDF): A vendor neural filesystem that is used for a wide range of media. It was developed to overcome some of the shortcomings of ISO 9660 and is used often for DVDs and other rewritable media. * Hierarchial File System (HFS): A filesystem developed by Apple and the native filesystem used on Macintosh computers. It is also used on read-only media like CD-ROMs. * ISO 9660 Extensions: * Rock Ridge (Interchange Protocol) - Offers extensions for longer file names, POSIX permissions, groups, and user IDs, and support for device files and symbolic links. * Joliet - A Microsoft extension for the ISO 9660 file system format. It allows Unicode characters and supports long file names. * El Torito - Specifies a standard way to make bootable CD-ROMs and DVDs. * `mkisofs` (genisoimage): Create ISO9660/Joliet/HFS filesystem with optional Rock Ridge attributes. ```bash mkisofs [options] [-o filename] pathspec [pathspec...] ``` ### Data Encryption * Linux Unified Key Setup (LUKS): A disk encryption specification in Linux. It uses the device mapper crypt kernel module (`dm-crypt`) which provides transparent encryption for block devices. * `cryptsetup`: Used to setup cryptographic volumes for `dm-crypt` (including LUKS extension). ```bash cryptsetup ``` | Action | Description | | ------------ | -------------------------------------------------- | | `create` | Create a mapping | | `remove` | Remove an existing mapping | | `status` | Report the status for a mapping | | `resize` | Resize an active mapping | | `luksFormat` | Initilize a LUKS partition and set the initial key | | `luksOpen` | Open the LUKS partition and set up a mapping | | `luksClose` | Same as "remove" | * `smartd`: A daemon that monitors the Self-Monitoring, Analysis and Reporting Technology (SMART) system build into many ATA-3 and later ATA, IDE, and SCSI-3 hard drives. * `/etc/smartmontools/smartd.conf` - Define devices and attributes to monitor and mail addresses to notify in case of a failure. * `/etc/sysconfig/smartmontools` - Modify startup arguments for `smartd` * `smartctl`: Control and monitor utility for SMART Disks. ```bash smartctl [options] device ``` | Option | Description | | ------ | ------------------------------------------------------ | | `-i` | Print information about the device | | `-a` | Print SMART information about the device | | `-x` | Print SMART and non-SMART information about the device | | `-s` | Enable or Disable SMART on a device | | `-t` | Execute a test of the device (delay with -C) | ## Sample Questions **1.** What are the two different types of autofs mounts? * [ ] manual * [x] direct * [ ] auto * [x] indirect **2.** What command is used to convert an ext filesystem to Btrfs? * [x] `btrfs-convert` * [ ] `btrfs restore` * [ ] `ext2btrfs` * [ ] `btrfs replace` **3.** What is the main configuration file for autofs? * [x] `auto.master` * [ ] `autofs` * [ ] `auto.map` * [ ] `autofs.conf` **4.** What configuration file contains startup arguments for smartd to be modified? * [ ] `smartd` * [x] `smartmontools` * [ ] `smartctl` * [ ] `smartd.conf` **5.** What are the commands used for enabling or disabling swap space? (Choose two.) * [x] `swapoff` * [x] `swapon` * [ ] `mkswap on` * [ ] `swap --enable` **6.** What special type of RAID level was introduced by the ZFS filesystem? * [ ] RAID-5 * [ ] RAID-3 * [x] RAID-Z * [ ] RAID-F **7.** What command is used to set up cryptographic volumes for dm-crypt? * [ ] `luksFormat` * [ ] `luksOpen` * [ ] `cryptenable` * [x] `cryptsetup` **8.** What option is used to create a journal using tune2fs? * [ ] `-l` * [x] `-j` * [ ] `-J` * [ ] `-m` **9.** What version of the ext filesystem introduced journaling? * [ ] `ext2` * [x] `ext3` * [ ] `ext4j` * [ ] `ext4` **10.** What commands can list the contents of the filesystem super block? (Choose two.) * [ ] `dumpe2fs` * [x] `debug2fs` * [ ] `fsck` * [x] `tune2fs` **11.** What is the mount command option to supply specific mount options? * [ ] `-a` * [x] `-o` * [ ] `-t` * [ ] `-O` **12.** Which of the following is a command used to set up a Linux swap area? * [ ] `swapon` * [ ] `mswapctl` * [x] `mkswap` * [ ] `swapoff` **13.** Which of the following are copy-on-write filesystems? (Choose two.) * [x] btrfs * [ ] udf * [ ] ext2 * [x] zfs **14.** What command writes cached data to persistent storage? * [ ] `dump` * [ ] `blkid` * [ ] `save` * [x] `sync` **15.** What file is read to automatically create Systemd mount units? * [ ] `/etc/systemd/mounts` * [x] `/etc/fstab` * [ ] `/run/systemd/generator` * [ ] `/etc/mtab` **16.** Which ISO 9660 extension is used to make bootable CD-ROMs and DVDs? * [ ] Rock Bridge * [ ] Joliet * [x] El Torito * [ ] HFS **17.** What command is used to display filesystem geometry information for XFS? * [ ] `xfs_restore` * [ ] `xfs_repair` * [ ] `xfs_check` * [x] `xfs_info` **18.** What file traditionally maintained a list of currently mounted filesystems? * [ ] `/proc/mounts` * [ ] `/etc/fstab` * [x] `/etc/mtab` * [ ] `/etc/mountab` **19.** What is the purpose of a system mount unit? * [x] It is used by the command `systemd-mount` and allows users to mount partitions to mount points of their choice * [ ] It is used only to mount network file systems to local mount points. It cannot be used for local media * [ ] It is created by the command `systemd-fstab-generator` to integrate entries from /`etc/fstab` into the system boot process * [ ] It is used by the command mount when using system to mount and unmount file systems # 204: Advanced Storage Device Administration ## **204.1 Configuring RAID (weight: 3)** {% hint style="success" %} Candidates should be able to configure and implement software RAID. This objective includes using and configuring RAID 0, 1 and 5. **Key Knowledge Areas:** * Software RAID configuration files and utilities **The following is a partial list of the used files, terms and utilities:** * mdadm.conf * mdadm * /proc/mdstat * partition type 0xFD {% endhint %} ### **Undrestanding RAID and RAID Levels** **RAID** stands for R**edundant Array of Independent Disks** and is a storage solution that is used to **organize or combine a group of disks into one or more logical units**. * Mirroring * Striping * Parity **RAID 0:** Striping
**RAID 1:** Mirroring
**RAID 5:** Block-Level Stripping with Distributed Parity
### Files and Utilities for Interacting with RAID * `mdadm.conf`: Main configuration file for the management of software raid with the `mdadm` utility * `/proc/mdstat`: Display a snapshot of the kernel's RAID/md stat. * `0xFD` partition type: Type ID number that signifies a RAID partition. * `mdadm`: A utility for creating, managing, and monitoring RAID devices using the md driver in Linux. 
mdadm [mode] <raiddevice> [options] <component-devices>
| Option | Description | | ------ | --------------------------------------- | | `-C` | Create a new array | | `-l` | Set a RAID level | | `-n` | Specify number of devices | | `-x` | Specify number of spare devices | | `-D` | Print details of one or more md devices | | `-a` | Add one or more devices to the array | | `-f` | Mark listed devices as faulty | | `-r` | Remove listed devices | | `-s` | Scan the map file for arrays | | `-v` | Increase verbosity | ### Creating a Software RAID 
[root@linuxmaster ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0 38.2G  0 disk 
└─sda1   8:1    0 38.2G  0 part /
sdb      8:16   0   10G  0 disk 
├─sdb1   8:17   0    1G  0 part 
├─sdb2   8:18   0    1G  0 part 
└─sdb3   8:19   0    1G  0 part 
sdc      8:32   0   10G  0 disk 
├─sdc1   8:33   0    1G  0 part 
└─sdc2   8:34   0    1G  0 part 
sr0     11:0    1 1024M  0 rom  

[root@linuxmaster ~]# mdadm -C /dev/md0 -l raid5 -n 3 /dev/sdb1 /dev/sdb2 /dev/sdb3 -x 2 /dev/sdc1 /dev/sdc2
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

[root@linuxmaster ~]# ll /dev/md0
brw-rw----. 1 root disk 9, 0 Sep 10 10:34 /dev/md0

[root@linuxmaster ~]# mdadm -D /dev/md0 
/dev/md0:
           Version : 1.2
     Creation Time : Sat Sep 10 10:34:50 2022
        Raid Level : raid5
        Array Size : 2093056 (2044.00 MiB 2143.29 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 3
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Sat Sep 10 10:35:29 2022
             State : clean 
    Active Devices : 3
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 2

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : centos-2gb-hel1-1:0
              UUID : 04d4ef0e:16a91fb8:f82fec5f:59832a86
            Events : 18

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       18        1      active sync   /dev/sdb2
       5       8       19        2      active sync   /dev/sdb3

       3       8       33        -      spare   /dev/sdc1
       4       8       34        -      spare   /dev/sdc2

[root@linuxmaster ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb3[5] sdc2[4](S) sdc1[3](S) sdb2[1] sdb1[0]
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>

[root@linuxmaster ~]# mdadm -D -s -v
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=1.2 spares=2 name=linuxmaster:0 UUID=04d4ef0e:16a91fb8:f82fec5f:59832a86
   devices=/dev/sdb1,/dev/sdb2,/dev/sdb3,/dev/sdc1,/dev/sdc2

[root@linuxmaster ~]# mdadm -D -s -v > /etc/mdadm.conf

[root@linuxmaster ~]# cat /etc/mdadm.conf
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=1.2 spares=2 name=linuxmaster:0 UUID=04d4ef0e:16a91fb8:f82fec5f:59832a86
   devices=/dev/sdb1,/dev/sdb2,/dev/sdb3,/dev/sdc1,/dev/sdc2
### Failing Active RAID Devices 
[root@linuxmaster ~]# mkfs.ext4 /dev/md0
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=256 blocks
130816 inodes, 523264 blocks
26163 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

[root@linuxmaster ~]# mkdir /mnt/raid

[root@linuxmaster ~]# mount -t ext4 /dev/md0 /mnt/raid/

[root@linuxmaster ~]# touch /mnt/raid/test{1..5}

[root@linuxmaster ~]# ll /mnt/raid/
total 16
drwx------. 2 root root 16384 Sep 10 11:12 lost+found
-rw-r--r--. 1 root root     0 Sep 10 11:13 test1
-rw-r--r--. 1 root root     0 Sep 10 11:13 test2
-rw-r--r--. 1 root root     0 Sep 10 11:13 test3
-rw-r--r--. 1 root root     0 Sep 10 11:13 test4
-rw-r--r--. 1 root root     0 Sep 10 11:13 test5

[root@linuxmaster ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        871M     0  871M   0% /dev
tmpfs           895M     0  895M   0% /dev/shm
tmpfs           895M  8.7M  887M   1% /run
tmpfs           895M     0  895M   0% /sys/fs/cgroup
/dev/sda1        38G  1.1G   35G   3% /
tmpfs           179M     0  179M   0% /run/user/0
/dev/md0        2.0G  6.0M  1.9G   1% /mnt/raid

[root@linuxmaster ~]# mdadm -f /dev/md0 /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0

[root@linuxmaster ~]# mdadm -D /dev/md0 
/dev/md0:
           Version : 1.2
     Creation Time : Sat Sep 10 10:34:50 2022
        Raid Level : raid5
        Array Size : 2093056 (2044.00 MiB 2143.29 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 3
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Sat Sep 10 11:14:30 2022
             State : clean, degraded, recovering 
    Active Devices : 2
   Working Devices : 4
    Failed Devices : 1
     Spare Devices : 2

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

    Rebuild Status : 30% complete

              Name : centos-2gb-hel1-1:0
              UUID : 04d4ef0e:16a91fb8:f82fec5f:59832a86
            Events : 24

    Number   Major   Minor   RaidDevice State
       4       8       34        0      spare rebuilding   /dev/sdc2
       1       8       18        1      active sync   /dev/sdb2
       5       8       19        2      active sync   /dev/sdb3

       0       8       17        -      faulty   /dev/sdb1
       3       8       33        -      spare   /dev/sdc1

[root@linuxmaster ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb3[5] sdc2[4] sdc1[3](S) sdb2[1] sdb1[0](F)
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [==============>......]  recovery = 73.2% (766720/1046528) finish=0.2min speed=19605K/sec
      
unused devices: <none>

[root@linuxmaster ~]# mdadm -D /dev/md0 
/dev/md0:
           Version : 1.2
     Creation Time : Sat Sep 10 10:34:50 2022
        Raid Level : raid5
        Array Size : 2093056 (2044.00 MiB 2143.29 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 3
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Sat Sep 10 11:15:10 2022
             State : clean 
    Active Devices : 3
   Working Devices : 4
    Failed Devices : 1
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : centos-2gb-hel1-1:0
              UUID : 04d4ef0e:16a91fb8:f82fec5f:59832a86
            Events : 37

    Number   Major   Minor   RaidDevice State
       4       8       34        0      active sync   /dev/sdc2
       1       8       18        1      active sync   /dev/sdb2
       5       8       19        2      active sync   /dev/sdb3

       0       8       17        -      faulty   /dev/sdb1
       3       8       33        -      spare   /dev/sdc1

[root@linuxmaster ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb3[5] sdc2[4] sdc1[3](S) sdb2[1] sdb1[0](F)
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>

[root@linuxmaster ~]# ll /mnt/raid/
total 16
drwx------. 2 root root 16384 Sep 10 11:12 lost+found
-rw-r--r--. 1 root root     0 Sep 10 11:13 test1
-rw-r--r--. 1 root root     0 Sep 10 11:13 test2
-rw-r--r--. 1 root root     0 Sep 10 11:13 test3
-rw-r--r--. 1 root root     0 Sep 10 11:13 test4
-rw-r--r--. 1 root root     0 Sep 10 11:13 test5

[root@linuxmaster ~]# mdadm -r /dev/sdb1
mdadm: /dev/sdb1 does not appear to be an md device

[root@linuxmaster ~]# mdadm -r /dev/sdc2
mdadm: /dev/sdc2 does not appear to be an md device

[root@linuxmaster ~]# mdadm -D /dev/md0 
/dev/md0:
           Version : 1.2
     Creation Time : Sat Sep 10 10:34:50 2022
        Raid Level : raid5
        Array Size : 2093056 (2044.00 MiB 2143.29 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 3
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Sat Sep 10 11:31:18 2022
             State : clean 
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 2
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : centos-2gb-hel1-1:0
              UUID : 04d4ef0e:16a91fb8:f82fec5f:59832a86
            Events : 56

    Number   Major   Minor   RaidDevice State
       3       8       33        0      active sync   /dev/sdc1
       1       8       18        1      active sync   /dev/sdb2
       5       8       19        2      active sync   /dev/sdb3

       0       8       17        -      faulty   /dev/sdb1
       4       8       34        -      faulty   /dev/sdc2
### Adding Devices to an Existing RAID 
[root@linuxmaster ~]# mdadm -a /dev/md0 /dev/sdd1 /dev/sdd2
mdadm: added /dev/sdd1
mdadm: added /dev/sdd2

[root@linuxmaster ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdd2[7](S) sdd1[6](S) sdb3[5] sdc2[4](F) sdc1[3] sdb2[1] sdb1[0](F)
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>

[root@linuxmaster ~]# mdadm -D /dev/md0 
/dev/md0:
           Version : 1.2
     Creation Time : Sat Sep 10 10:34:50 2022
        Raid Level : raid5
        Array Size : 2093056 (2044.00 MiB 2143.29 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 3
     Total Devices : 7
       Persistence : Superblock is persistent

       Update Time : Sat Sep 10 11:35:36 2022
             State : clean 
    Active Devices : 3
   Working Devices : 5
    Failed Devices : 2
     Spare Devices : 2

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : centos-2gb-hel1-1:0
              UUID : 04d4ef0e:16a91fb8:f82fec5f:59832a86
            Events : 58

    Number   Major   Minor   RaidDevice State
       3       8       33        0      active sync   /dev/sdc1
       1       8       18        1      active sync   /dev/sdb2
       5       8       19        2      active sync   /dev/sdb3

       0       8       17        -      faulty   /dev/sdb1
       4       8       34        -      faulty   /dev/sdc2
       6       8       49        -      spare   /dev/sdd1
       7       8       50        -      spare   /dev/sdd2
### Updating the mdadm.conf File 
[root@linuxmaster ~]# mdadm -D -s -v > /etc/mdadm.conf 

[root@linuxmaster ~]# cat /etc/mdadm.conf 
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=1.2 spares=2 name=linuxmaster:0 UUID=04d4ef0e:16a91fb8:f82fec5f:59832a86
   devices=/dev/sdb1,/dev/sdb2,/dev/sdb3,/dev/sdc1,/dev/sdc2,/dev/sdd1,/dev/sdd2
## **204.2 Adjusting Storage Device Access (weight: 2)** {% hint style="success" %} Candidates should be able to configure kernel options to support various drives. This objective includes software tools to view & modify hard disk settings including iSCSI devices. **Key Knowledge Areas:** * Tools and utilities to configure DMA for IDE devices including ATAPI and SATA * Tools and utilities to configure Solid State Drives including AHCI and NVMe * Tools and utilities to manipulate or analyse system resources (e.g. interrupts) * Awareness of sdparm command and its uses * Tools and utilities for iSCSI * Awareness of SAN, including relevant protocols (AoE, FCoE) **The following is a partial list of the used files, terms and utilities:** * hdparm, sdparm * nvme * tune2fs * fstrim * sysctl * /dev/hd\*, /dev/sd\*, /dev/nvme\* * iscsiadm, scsi\_id, iscsid and iscsid.conf * WWID, WWN, LUN numbers {% endhint %} ### Direct Memory Access (DMA) and Interrupts Handling **Direct Memory Access** allows hardware devices (e.g. disk drivers, network cards, sound cards, etc.) to effectively bypass the CPU and send or recive data directly from the main system memory rather than taking up large quantities of proccessing time. This process is managed by a DMA controller (DMAC). 
[root@linuxmaster ~]# cat /proc/interrupts 
           CPU0       CPU1       
  0:        178          0   IO-APIC-edge      timer
  1:         10          0   IO-APIC-edge      i8042
  4:       2382          0   IO-APIC-edge      serial
  8:          1          0   IO-APIC-edge      rtc0
  9:          0          0   IO-APIC-fasteoi   acpi
 12:         15          0   IO-APIC-edge      i8042
 16:          0          0   IO-APIC-fasteoi   i801_smbus
 22:          9          0   IO-APIC-fasteoi   virtio3, virtio4
 24:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 25:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 26:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 27:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 28:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 29:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 30:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 31:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 32:          1          0   PCI-MSI-edge      aerdrv, PCIe PME, pciehp
 33:         25          0   PCI-MSI-edge      xhci_hcd
 34:          0          0   PCI-MSI-edge      xhci_hcd
A **hardware interrupt or interrupt request (IRQ)** is a signal that informs the processor that a hardware device (e.g. mouse, keyboard, etc.) needs attention. The interrupt suspends the current activity of the processor in order to handle the external event. 
[root@linuxmaster ~]# ls -l /proc/irq/
total 0
dr-xr-xr-x. 2 root root 0 Sep 10 12:02 0
dr-xr-xr-x. 3 root root 0 Sep 10 12:02 1
dr-xr-xr-x. 2 root root 0 Sep 10 12:02 10
dr-xr-xr-x. 2 root root 0 Sep 10 12:02 11
dr-xr-xr-x. 3 root root 0 Sep 10 12:02 12
dr-xr-xr-x. 2 root root 0 Sep 10 12:02 13
dr-xr-xr-x. 2 root root 0 Sep 10 12:02 14
dr-xr-xr-x. 2 root root 0 Sep 10 12:02 15
### The hdparm Utility This utility is used to view information about or set SATA/IDE device parameters. Devices of this type will be /dev/hd\*. | Command | Description | | ------------------------ | ---------------------------------------------------------------- | | `hdparm -I /dev/hda` | Display information about the drive | | `hdparm -d# /dev/hda` | Get or set the `using_dma` flag for the drive | | `hdparm -t(-T) /dev/hda` | Test timings of device reads (or cache with `-T`) for comparison | | `hdparm --help` | View commands and syntax | ### The sdparm Utility List or change SCSI/SATA device parameters and send simple SCSI commands. Devices of this type will be /dev/dd\*. | Command | Description | | ------------------------------- | ----------------------------------------------------------------- | | `sdparm -I /dev/sda` | List all known fields for a device | | `sdparm --command=CMD /dev/sda` | Send a SCSI command to the device (e.g. eject, start, stop, etc.) | | `sdparm --help` | View commands and syntax | ```bash [root@linuxmaster ~]# sdparm /dev/sda /dev/sda: QEMU QEMU HARDDISK 2.5+ Read write error recovery mode page: AWRE 1 [cha: n, def: 1] ARRE 0 [cha: n, def: 0] PER 0 [cha: n, def: 0] Caching (SBC) mode page: WCE 1 [cha: y, def: 1] RCD 0 [cha: n, def: 0] ``` ### The nvme Utility A command line utility for managing PCI Express-based devices. Devices of this type will be /dev/nvme\*. | Command | Description | | ---------------------------------------- | ------------------------------------ | | `nvme help` | Display a list of available commands | | `nvme list` | List all NVMe controllers | | `nvme smart-log /dev/nvme0n1` | Retrieve the smart log | | `nvme read/write /dev/nvme0n1 [options]` | Issue IO read or write command | {% code overflow="wrap" %} ```bash [root@linuxmaster ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 /dev/ng0n1 S5EFNE0N501948 SAMSUNG MZVLB256HBHQ-00A00 1 54.53 GB / 256.06 GB 512 B + 0 B EXH7201Q ``` {% endcode %} ### The fstrim Utility A command line utiliy for discarding unwanted block on a mounted filesystem. * Discards all unused blocks by default but provides options to modify this behavior based on range or size. * Systemd provides a timer for `fstrim` called `fstrim.timer` that runs on a weekly basis when enabled. * The `-a` option will run `fstrim` against all mounted file systems. * The `discard` option can be set in `/etc/fstab` for automatic online TRIM (not recommended). 
[root@linuxmaster ~]# fstrim -v /mnt/util
/: 615.5 MiB (173498368 bytes) trimmed

[root@linuxmaster ~]# cat /usr/lib/systemd/system/fstrim.timer

[Unit]
Description=Discard unused blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container
ConditionPathExists=!/etc/initrd-release

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=6000

[Install]
WantedBy=timers.target
### iSCSI * iSCSI (Internet Small Computer System Interface): A protocol that allows SCSI commands to be sent over a TCP/IP network. Communication takes place between an initiator (the cliant) and a target (a server). * ATA over Ethernet (AoE): Allows the Advanced Technology Attachment (ATA) protocol over an ethernet network * Fiber Channel over Ethernet (FCoE): Allows the Fiber Channel (FC) protocol over an ethernet network * `targetcli`: A shell for viewing, editing and saving the configuration of the kernel's target subsystem. **Creating an iSCSI Target:** * Create a backstore using `nvme1n1`. * Create a target with a unique iSCSI Qualified Name (IQN). * Create an ACL, LUN, and portal within the target portal group (TPG) * **ACLs**: Access control lists * **LUN**: Logical unit number * **Portal**: The IP and TCP port pair for an iSCSI target ```bash [root@linuxmaster ~]# targetcli ls ... ``` ### Files and Utilities for the iSCSI Initiator * `iscsid.conf`: The default configuration file for iSCSI. It contains settings used by `iscsid` and `iscsiadm`. * `initaltorname.iscsi`: Contains the iSCSI initiator name. This should mirror the name from the ACL of the iSCSI target. * `iscsid`: A system daemon that manages iSCSI connections. * `scsi_id`: A utility used to retrieve and generate a unique SCSI identifier. This is primarity used by other utilities such as udev. * `iscsiadm`: The open-iscsi administration utility. ```bash iscsiadm -m [options] ``` | Option | Description | | ------ | ------------------------------------------ | | `-m` | Specify the mode | | `-t` | Specify the discovery type | | `-T` | Specify the target name | | `-l` | Login to a specified record | | `-p` | Use target portal with IP address and port | | `-P` | Print in tree format | 
[root@linuxmaster ~]# cat /etc/iscsi/initiatorname.iscsi 
	<<Add custom initiator name

[root@linuxmaster ~]# iscsiadm -m discovery -t st -p ip_address
	<<Discover the iSCSI target

[root@linuxmaster ~]# iscsiadm -m node -T iqn.2022-09.com.name.host:t1 -l
	<<Login to the target with the discovered IQN

[root@linuxmaster ~]# grep "Attached SCSI" /var/log/messages
	<<View the iSCSI disk name

[root@linuxmaster ~]# iscsiadm -m node -T iqn.2022-09.com.name.host:t1 -u
	<<Log off the iSCSI target204.3 Logical Volume Manager (weight: 3)

[root@linuxmaster ~]# yum install -y targetctl
...

[root@linuxmaster ~]# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> backstores/block create block01 /dev/nvme1n1 
Created block storage object block01 using /dev/nvme1n1.

/> iscsi/ create iqn.2022-09.com.linuxmaster:linuxmaster1
Created target iqn.2022-09.com.linuxmaster:linuxmaster1.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.

/> ls
o- / ............................................................................................. [...]
  o- backstores .................................................................................. [...]
  | o- block ...................................................................... [Storage Objects: 1]
  | | o- block01 ........................................ [/dev/nvme1n1 (2.0GiB) write-thru deactivated]
  | |   o- alua ....................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................................... [ALUA state: Active/optimized]
  | o- fileio ..................................................................... [Storage Objects: 0]
  | o- pscsi ...................................................................... [Storage Objects: 0]
  | o- ramdisk .................................................................... [Storage Objects: 0]
  o- iscsi ................................................................................ [Targets: 1]
  | o- iqn.2022-09.com.linuxmaster:linuxmaster1 .............................................. [TPGs: 1]
  |   o- tpg1 ................................................................... [no-gen-acls, no-auth]
  |     o- acls .............................................................................. [ACLs: 0]
  |     o- luns .............................................................................. [LUNs: 0]
  |     o- portals ........................................................................ [Portals: 1]
  |       o- 0.0.0.0:3260 ......................................................................... [OK]
  o- loopback ............................................................................. [Targets: 0]

/> cd iscsi/iqn.2022-09.com.linuxmaster:linuxmaster1/tpg1/

/iscsi/iqn.20...xmaster1/tpg1> luns/ create /backstores/block/block01 
Created LUN 0.

/iscsi/iqn.20...xmaster1/tpg1> acls/ create iqn.2022-09.com.linuxmaster.acg02:master
Created Node ACL for iqn.2022-09.com.linuxmaster.acg02:master
Created mapped LUN 0.

/iscsi/iqn.20...xmaster1/tpg1> cd /

/> ls
o- / ............................................................................................. [...]
  o- backstores .................................................................................. [...]
  | o- block ...................................................................... [Storage Objects: 1]
  | | o- block01 .......................................... [/dev/nvme1n1 (2.0GiB) write-thru activated]
  | |   o- alua ....................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................................... [ALUA state: Active/optimized]
  | o- fileio ..................................................................... [Storage Objects: 0]
  | o- pscsi ...................................................................... [Storage Objects: 0]
  | o- ramdisk .................................................................... [Storage Objects: 0]
  o- iscsi ................................................................................ [Targets: 1]
  | o- iqn.2022-09.com.linuxmaster:linuxmaster1 .............................................. [TPGs: 1]
  |   o- tpg1 ................................................................... [no-gen-acls, no-auth]
  |     o- acls .............................................................................. [ACLs: 1]
  |     | o- iqn.2022-09.com.linuxmaster.acg02:master ................................. [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................................................... [lun0 block/block01 (rw)]
  |     o- luns .............................................................................. [LUNs: 1]
  |     | o- lun0 .................................... [block/block01 (/dev/nvme1n1) (default_tg_pt_gp)]
  |     o- portals ........................................................................ [Portals: 1]
  |       o- 0.0.0.0:3260 ......................................................................... [OK]
  o- loopback ............................................................................. [Targets: 0]
  
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json

[root@linuxmaster ~]# systemctl enable target.service --now
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.

[root@linuxmaster ~]# yum install -y iscsi-initiator-utils
...

[root@linuxmaster ~]# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2022-09.com.linuxmaster.acg02:master

[root@linuxmaster ~]# systemctl enable iscsid --now
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.

[root@linuxmaster ~]# systemctl status iscsid 
● iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-09-13 12:06:34 UTC; 20s ago
     Docs: man:iscsid(8)
           man:iscsiuio(8)
           man:iscsiadm(8)
 Main PID: 4125 (iscsid)
   Status: "Ready to process requests"
   CGroup: /system.slice/iscsid.service
           └─4125 /sbin/iscsid -f

Sep 13 12:06:34 linuxmaster systemd[1]: Starting Open-iSCSI...
Sep 13 12:06:34 linuxmaster systemd[1]: Started Open-iSCSI.

[root@linuxmaster ~]# iscsiadm -m discovery -t st -p 172.31.28.253
172.31.28.253:3260,1 iqn.2022-09.com.linuxmaster:linuxmaster1

[root@linuxmaster ~]# iscsiadm -m node -T iqn.2022-09.com.linuxmaster:linuxmaster1 -l 
Logging in to [iface: default, target: iqn.2022-09.com.linuxmaster:linuxmaster1, portal: 172.31.28.253,3260] (multiple)
Login to [iface: default, target: iqn.2022-09.com.linuxmaster:linuxmaster1, portal: 172.31.28.253,3260] successful.

[root@linuxmaster ~]# grep "Attached SCSI" /var/log/messages
Sep 13 12:08:17 ip-172-31-28-253 kernel: sd 0:0:0:0: [sda] Attached SCSI disk

[root@linuxmaster ~]# ll /dev/sda 
brw-rw----. 1 root disk 8, 0 Sep 13 12:08 /dev/sda

[root@linuxmaster ~]# ll /dev/disk/by-id/
total 0
lrwxrwxrwx. 1 root root 13 Sep 13 11:50 nvme-Amazon_Elastic_Block_Store_vol09bb3f1a4dc9b3cbe -> ../../nvme0n1
lrwxrwxrwx. 1 root root 15 Sep 13 11:50 nvme-Amazon_Elastic_Block_Store_vol09bb3f1a4dc9b3cbe-part1 -> ../../nvme0n1p1
lrwxrwxrwx. 1 root root 13 Sep 13 12:01 nvme-Amazon_Elastic_Block_Store_vol0d17c9387510b69db -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Sep 13 11:50 nvme-nvme.1d0f-766f6c3039626233663161346463396233636265-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001 -> ../../nvme0n1
lrwxrwxrwx. 1 root root 15 Sep 13 11:50 nvme-nvme.1d0f-766f6c3039626233663161346463396233636265-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001-part1 -> ../../nvme0n1p1
lrwxrwxrwx. 1 root root 13 Sep 13 12:01 nvme-nvme.1d0f-766f6c3064313763393338373531306236396462-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001 -> ../../nvme1n1
lrwxrwxrwx. 1 root root  9 Sep 13 12:08 scsi-36001405b58213e8e6e647d4ad9139dea -> ../../sda
lrwxrwxrwx. 1 root root  9 Sep 13 12:08 wwn-0x6001405b58213e8e6e647d4ad9139dea -> ../../sda

[root@linuxmaster ~]# lsblk 
NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda           8:0    0   2G  0 disk 
nvme0n1     259:0    0  20G  0 disk 
└─nvme0n1p1 259:1    0  20G  0 part /
nvme1n1     259:2    0   2G  0 disk

[root@linuxmaster ~]# iscsiadm -m session -P 1
Target: iqn.2022-09.com.linuxmaster:linuxmaster1 (non-flash)
	Current Portal: 172.31.28.253:3260,1
	Persistent Portal: 172.31.28.253:3260,1
		**********
		Interface:
		**********
		Iface Name: default
		Iface Transport: tcp
		Iface Initiatorname: iqn.2022-09.com.linuxmaster.acg02:master
		Iface IPaddress: 172.31.28.253
		Iface HWaddress: <empty>
		Iface Netdev: <empty>
		SID: 1
		iSCSI Connection State: LOGGED IN
		iSCSI Session State: LOGGED_IN
		Internal iscsid Session State: NO CHANGE

[root@linuxmaster ~]# iscsiadm -m node -P 1
Target: iqn.2022-09.com.linuxmaster:linuxmaster1
	Portal: 172.31.28.253:3260,1
		Iface Name: default
## 204.3 Logical Volume Manager (weight: 3) {% hint style="success" %} Candidates should be able to create and remove logical volumes, volume groups, and physical volumes. This objective includes snapshots and resizing logical volumes. **Key Knowledge Areas:** * Tools in the LVM suite * Resizing, renaming, creating, and removing logical volumes, volume groups, and physical volumes * Creating and maintaining snapshots * Activating volume groups **The following is a partial list of the used files, terms and utilities:** * /sbin/pv\* * /sbin/lv\* * /sbin/vg\* * mount * /dev/mapper/ * lvm.conf {% endhint %} ### LVM Architecture
### Working with Physical Volumes | Command | Purpose | | ----------- | --------------------------------------------------------- | | `pvcreate` | Create (initialize) a physical volume from a block device | | `pvs` | Display info about physical volumes | | `pvdisplay` | Display verbose info about physical volumes | | `pvscan` | Scan all supported LVM block devices | | `pvresize` | Update the size of a physical volume | | `pvremove` | Remove a physical volume | 
[root@linuxmaster ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0 19.1G  0 disk 
└─sda1   8:1    0 19.1G  0 part /
sdb      8:16   0   10G  0 disk 
├─sdb1   8:17   0    1G  0 part 
└─sdb2   8:18   0    1G  0 part 
sdc      8:32   0   10G  0 disk 
├─sdc1   8:33   0    1G  0 part 
└─sdc2   8:34   0    1G  0 part 
sr0     11:0    1 1024M  0 rom

[root@linuxmaster ~]# pvcreate /dev/sdb{1,2} /dev/sdc{1,2}
  Physical volume "/dev/sdb1" successfully created.
  Physical volume "/dev/sdb2" successfully created.
  Physical volume "/dev/sdc1" successfully created.
  Physical volume "/dev/sdc2" successfully created.

[root@linuxmaster ~]# pvs
  PV         VG Fmt  Attr PSize PFree
  /dev/sdb1     lvm2 ---  1.00g 1.00g
  /dev/sdb2     lvm2 ---  1.00g 1.00g
  /dev/sdc1     lvm2 ---  1.00g 1.00g
  /dev/sdc2     lvm2 ---  1.00g 1.00g

[root@linuxmaster ~]# pvscan 
  PV /dev/sdc1                      lvm2 [1.00 GiB]
  PV /dev/sdb2                      lvm2 [1.00 GiB]
  PV /dev/sdc2                      lvm2 [1.00 GiB]
  PV /dev/sdb1                      lvm2 [1.00 GiB]
  Total: 4 [4.00 GiB] / in use: 0 [0   ] / in no VG: 4 [4.00 GiB]

[root@linuxmaster ~]# pvdisplay 
  "/dev/sdc1" is a new physical volume of "1.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdc1
  VG Name               
  PV Size               1.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               yayebB-OerJ-U7IE-Ww8h-M1N6-l0dl-ezcXrS
   
  "/dev/sdb2" is a new physical volume of "1.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdb2
  VG Name               
  PV Size               1.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               O5PA7y-HqOx-q69t-Vret-V9UZ-RqH8-NJuHqB
   
  "/dev/sdc2" is a new physical volume of "1.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdc2
  VG Name               
  PV Size               1.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               bOKd2X-yHB3-w0xS-t8JU-uq95-VksQ-KZ0LM6
   
  "/dev/sdb1" is a new physical volume of "1.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdb1
  VG Name               
  PV Size               1.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               hzCENy-Pic3-8Z9D-Legi-JacD-G6oj-ucfuTM
### Working with Volume Group | Command | Purpose | | ----------- | ----------------------------------------------------------------- | | `vgcreate` | Create a volume group from physical volumes | | `vgextend` | Add a physical volume to a volume group | | `vgreduce` | Remove a physical volume from a volume group | | `vgs` | Display information about volume groups | | `vgdisplay` | Display verbose information about volume groups | | `vgscan` | Display information about volume groups and rebuild the LVM cache | | `vgchange` | Modify attributes of a volume group | | `vgrename` | Change the name of a volume group | | `vgremove` | Remove a volume group | 
[root@linuxmaster ~]# vgcreate data_vg /dev/sdb{1,2}
  Volume group "data_vg" successfully created

root@linuxmaster ~]# vgs
  VG      #PV #LV #SN Attr   VSize VFree
  data_vg   2   0   0 wz--n- 1.99g 1.99g

[root@linuxmaster ~]# vgscan
  Reading volume groups from cache.
  Found volume group "data_vg" using metadata type lvm2

[root@linuxmaster ~]# vgdisplay 
  --- Volume group ---
  VG Name               data_vg
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.99 GiB
  PE Size               4.00 MiB
  Total PE              510
  Alloc PE / Size       0 / 0   
  Free  PE / Size       510 / 1.99 GiB
  VG UUID               z5kweM-C1aR-egdJ-IefJ-f1Bz-VGDC-EXcQzx

[root@linuxmaster ~]# vgextend data_vg /dev/sdc{1,2}
  Volume group "data_vg" successfully extended

[root@linuxmaster ~]# vgs
  VG      #PV #LV #SN Attr   VSize VFree
  data_vg   4   0   0 wz--n- 3.98g 3.98g
### Working with Logical Volumes | Command | Purpose | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lvcreate` |

Create a logical volume from a volume group

  • -L - Specify the size
  • -l - Specify the number of extends or extent percentage
| | `lvs` | Display info about logical volumes | | `lvdisplay` | Display verbose info about logical volumes | | `lvscan` | List all logical volumes in all volume groups | | `lvrename` | Change the name of a logical volume | | `lvchange` | Adjust the parameters of a logical volume | | `lvremove` | Remove a logical volume from a volume group | 
[root@linuxmaster ~]# lvcreate -L 1G -n backup data_vg
  Logical volume "backup" created.

[root@linuxmaster ~]# lvcreate -l 500 -n temp data_vg
  Logical volume "temp" created.

[root@linuxmaster ~]# lvcreate -l 50%FREE -n current data_vg
  Logical volume "current" created.

[root@linuxmaster ~]# lvs
  LV      VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backup  data_vg -wi-a-----   1.00g                                                    
  current data_vg -wi-a----- 528.00m                                                    
  temp    data_vg -wi-a-----   1.95g

[root@linuxmaster ~]# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/data_vg/backup
  LV Name                backup
  VG Name                data_vg
  LV UUID                a2GDAc-WoFl-QVQ5-E4YD-ylhX-7RPT-w7B462
  LV Write Access        read/write
  LV Creation host, time linuxmaster, 2022-09-13 14:26:00 +0000
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Current LE             256
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/data_vg/temp
  LV Name                temp
  VG Name                data_vg
  LV UUID                AAtwqm-CtbZ-9AdO-SLDJ-F61m-gIXc-rxK22j
  LV Write Access        read/write
  LV Creation host, time linuxmaster, 2022-09-13 14:26:53 +0000
  LV Status              available
  # open                 0
  LV Size                1.95 GiB
  Current LE             500
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/data_vg/current
  LV Name                current
  VG Name                data_vg
  LV UUID                yRgoQq-u5d2-FiqW-ijZ5-538q-aH1I-UVWOQ0
  LV Write Access        read/write
  LV Creation host, time linuxmaster, 2022-09-13 14:28:10 +0000
  LV Status              available
  # open                 0
  LV Size                528.00 MiB
  Current LE             132
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

[root@linuxmaster ~]# lvremove /dev/data_vg/temp 
Do you really want to remove active logical volume data_vg/temp? [y/n]: y
  Logical volume "temp" successfully removed
  
[root@linuxmaster ~]# lvscan 
  ACTIVE            '/dev/data_vg/backup' [1.00 GiB] inherit
  ACTIVE            '/dev/data_vg/current' [528.00 MiB] inherit


[root@linuxmaster ~]# mkfs.ext4 /dev/data_vg/backup 
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

[root@linuxmaster ~]# mkfs.ext4 /dev/data_vg/current 
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
33840 inodes, 135168 blocks
6758 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=138412032
5 block groups
32768 blocks per group, 32768 fragments per group
6768 inodes per group
Superblock backups stored on blocks: 
	32768, 98304

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

[root@linuxmaster ~]# mkdir /mnt/backup

[root@linuxmaster ~]# mkdir /mnt/current

[root@linuxmaster ~]# mount -t ext4 /dev/data_vg/backup /mnt/backup/

[root@linuxmaster ~]# mount -t ext4 /dev/data_vg/current /mnt/current/

[root@linuxmaster ~]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     871M     0  871M   0% /dev
tmpfs                        895M     0  895M   0% /dev/shm
tmpfs                        895M  8.6M  887M   1% /run
tmpfs                        895M     0  895M   0% /sys/fs/cgroup
/dev/sda1                     19G  1.1G   17G   6% /
tmpfs                        179M     0  179M   0% /run/user/0
/dev/mapper/data_vg-backup   976M  2.6M  907M   1% /mnt/backup
/dev/mapper/data_vg-current  504M  804K  466M   1% /mnt/current
### Resizing Logical Volumes | Command | Purpose | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `lvextend` |

Grow the size of a logical volume

  • -L - Specify the size
  • -l - Specify the number of extens or extent percentage
  • -r - Resize the underlying file system
| | `lvs` | Display info about logical volumes | | `resize2fs` | Resize an ext2/ext3/ext4 filesystem | | `xfs_growfs` | Expand an XFS filesystem | | `lvreduce` | Reduce the size of a logical volume | 
[root@linuxmaster ~]# vgs
  VG      #PV #LV #SN Attr   VSize VFree 
  data_vg   4   2   0 wz--n- 3.98g <2.47g

[root@linuxmaster ~]# lvextend -L +500M /dev/data_vg/backup 
  Size of logical volume data_vg/backup changed from 1.00 GiB (256 extents) to <1.49 GiB (381 extents).
  Logical volume data_vg/backup successfully resized.

[root@linuxmaster ~]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     871M     0  871M   0% /dev
tmpfs                        895M     0  895M   0% /dev/shm
tmpfs                        895M  8.6M  887M   1% /run
tmpfs                        895M     0  895M   0% /sys/fs/cgroup
/dev/sda1                     19G  1.1G   17G   6% /
tmpfs                        179M     0  179M   0% /run/user/0
/dev/mapper/data_vg-backup   976M  2.6M  907M   1% /mnt/backup
/dev/mapper/data_vg-current  504M  804K  466M   1% /mnt/current

[root@linuxmaster ~]# resize2fs /dev/data_vg/backup 
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/data_vg/backup is mounted on /mnt/backup; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/data_vg/backup is now 390144 blocks long.

[root@linuxmaster ~]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     871M     0  871M   0% /dev
tmpfs                        895M     0  895M   0% /dev/shm
tmpfs                        895M  8.6M  887M   1% /run
tmpfs                        895M     0  895M   0% /sys/fs/cgroup
/dev/sda1                     19G  1.1G   17G   6% /
tmpfs                        179M     0  179M   0% /run/user/0
/dev/mapper/data_vg-backup   1.5G  3.0M  1.4G   1% /mnt/backup
/dev/mapper/data_vg-current  504M  804K  466M   1% /mnt/current

[root@linuxmaster ~]# lvs
  LV      VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backup  data_vg -wi-ao----  <1.49g                                                    
  current data_vg -wi-ao---- 528.00m

[root@linuxmaster ~]# lvreduce -l 100 -r data_vg/current
Do you want to unmount "/mnt/current" ? [Y|n] y
fsck from util-linux 2.23.2
/dev/mapper/data_vg-current: 11/33840 files (0.0% non-contiguous), 6428/135168 blocks
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/mapper/data_vg-current to 102400 (4k) blocks.
The filesystem on /dev/mapper/data_vg-current is now 102400 blocks long.

  Size of logical volume data_vg/current changed from 528.00 MiB (132 extents) to 400.00 MiB (100 extents).

  Logical volume data_vg/current successfully resized.

[root@linuxmaster ~]# lvs
  LV      VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backup  data_vg -wi-ao----  <1.49g                                                    
  current data_vg -wi-ao---- 400.00m
### Creating and Managing Snapshots | Command | Purpose | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lvcreate` |

Create a logical volume

  • -L - Specify the size
  • -l - Specify the number of extents or extent percentage
  • -s - Create a snapshot
| | `lvs` | Display info about logical volumes | | `lvextend` | Increate the size of a logical volume or snapshot (COW table) | | `lvreduce` | Reduce the size of a logical volume or snapshot (COW table) | | `lvremove` | Remove a logical volume or a snapshot | | `lvchange` | Adjust the parameters of a logical volume | | `lvconvert` |

Change logical volume layout

  • --merge - Merge a snapshot logical volume into its origin
| 
[root@linuxmaster ~]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     871M     0  871M   0% /dev
tmpfs                        895M     0  895M   0% /dev/shm
tmpfs                        895M  8.6M  887M   1% /run
tmpfs                        895M     0  895M   0% /sys/fs/cgroup
/dev/sda1                     19G  1.1G   17G   6% /
tmpfs                        179M     0  179M   0% /run/user/0
/dev/mapper/data_vg-current  976M  2.6M  907M   1% /mnt/current
/dev/mapper/data_vg-backup   976M  2.6M  907M   1% /mnt/backup

[root@linuxmaster ~]# touch /mnt/current/test{1..5}.txt

[root@linuxmaster ~]# ll /mnt/current/
total 16
drwx------. 2 root root 16384 Sep 13 15:02 lost+found
-rw-r--r--. 1 root root     0 Sep 13 15:03 test1.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test2.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test3.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test4.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test5.txt

[root@linuxmaster ~]# lvcreate -l 25%ORIGIN -s -n cur_snap1 /dev/mapper/data_vg-current 
  Logical volume "cur_snap1" created.

[root@linuxmaster ~]# lvcreate -L 100M -s -n cur_snap2 /dev/mapper/data_vg-current
  Logical volume "cur_snap2" created.

[root@linuxmaster ~]# lvs
  LV        VG      Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  backup    data_vg -wi-ao----   1.00g                                                     
  cur_snap1 data_vg swi-a-s--- 264.00m      current 0.01                                   
  cur_snap2 data_vg swi-a-s--- 100.00m      current 0.01                                   
  current   data_vg owi-aos---   1.00g                                                     
  temp      data_vg -wi-a-----   1.00g

[root@linuxmaster ~]# cp -r /etc/ /mnt/current/

[root@linuxmaster ~]# lvs
  LV        VG      Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  backup    data_vg -wi-ao----   1.00g                                                     
  cur_snap1 data_vg swi-a-s--- 264.00m      current 1.14                                   
  cur_snap2 data_vg swi-a-s--- 100.00m      current 3.01                                   
  current   data_vg owi-aos---   1.00g                                                     

[root@linuxmaster ~]# lvdisplay /dev/data_vg/cur_snap1 
  --- Logical volume ---
  LV Path                /dev/data_vg/cur_snap1
  LV Name                cur_snap1
  VG Name                data_vg
  LV UUID                giAvbe-z4kn-CWDp-cvZx-Vn8U-N86W-bUwEze
  LV Write Access        read/write
  LV Creation host, time linuxmaster, 2022-09-13 15:05:37 +0000
  LV snapshot status     active destination for current
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Current LE             256
  COW-table size         264.00 MiB
  COW-table LE           66
  Allocated to snapshot  14.26%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:5

[root@linuxmaster ~]# lvextend -l 50%ORIGIN /dev/mapper/data_vg-cur_snap1
  Size of logical volume data_vg/cur_snap1 changed from 264.00 MiB (66 extents) to 512.00 MiB (128 extents).
  Logical volume data_vg/cur_snap1 successfully resized.

[root@linuxmaster ~]# lvs
  LV        VG      Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  backup    data_vg -wi-ao----   1.00g                                                     
  cur_snap1 data_vg swi-a-s--- 512.00m      current 7.47                                   
  cur_snap2 data_vg swi-a-s--- 100.00m      current 38.23                                  
  current   data_vg owi-aos---   1.00g                                                     

[root@linuxmaster ~]# lvremove data_vg/cur_snap2
Do you really want to remove active logical volume data_vg/cur_snap2? [y/n]: y
  Logical volume "cur_snap2" successfully removed

[root@linuxmaster ~]# lvs
  LV        VG      Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  backup    data_vg -wi-ao----   1.00g                                                     
  cur_snap1 data_vg swi-a-s--- 512.00m      current 7.47                                   
  current   data_vg owi-aos---   1.00g                                                     

[root@linuxmaster ~]# mount /dev/mapper/data_vg-cur_snap1 /mnt/snap/

[root@linuxmaster ~]# ll /mnt/snap/
total 16
drwx------. 2 root root 16384 Sep 13 15:02 lost+found
-rw-r--r--. 1 root root     0 Sep 13 15:03 test1.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test2.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test3.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test4.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test5.txt

[root@linuxmaster ~]# ll /mnt/current/
total 20
drwxr-xr-x. 75 root root  4096 Sep 13 15:07 etc
drwx------.  2 root root 16384 Sep 13 15:02 lost+found
-rw-r--r--.  1 root root     0 Sep 13 15:03 test1.txt
-rw-r--r--.  1 root root     0 Sep 13 15:03 test2.txt
-rw-r--r--.  1 root root     0 Sep 13 15:03 test3.txt
-rw-r--r--.  1 root root     0 Sep 13 15:03 test4.txt
-rw-r--r--.  1 root root     0 Sep 13 15:03 test5.txt

[root@linuxmaster ~]# umount /mnt/current/

[root@linuxmaster ~]# umount /mnt/snap/

[root@linuxmaster ~]# lvchange -an /dev/data_vg/current 

[root@linuxmaster ~]# lvconvert --merge data_vg/cur_snap1 
  Merging of snapshot data_vg/cur_snap1 will occur on next activation of data_vg/current.

[root@linuxmaster ~]# lvchange -ay /dev/data_vg/current

[root@linuxmaster ~]# mount /dev/data_vg/current /mnt/current/

[root@linuxmaster ~]# ll /mnt/current/
total 16
drwx------. 2 root root 16384 Sep 13 15:02 lost+found
-rw-r--r--. 1 root root     0 Sep 13 15:03 test1.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test2.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test3.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test4.txt
-rw-r--r--. 1 root root     0 Sep 13 15:03 test5.txt

[root@linuxmaster ~]# lvs
  LV      VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backup  data_vg -wi-ao---- 1.00g                                                    
  current data_vg -wi-ao---- 1.00g
## Sample Questions **1.** Which of the following are protocols used in a Storage Area Network (SAN)? (Choose three.) * [ ] PaaS * [x] AoE * [x] FCoE * [x] iSCSI **2.** What command is used to initialize a physical volume? * [ ] `pvresize` * [x] `pvcreate` * [ ] `pvs` * [ ] `vgcreate` **3.** What command is used to add a physical volume to a volume group? * [ ] `pvresize` * [ ] `vgchange` * [ ] `vgrename` * [x] `vgextend` **4.** What option for the sdparm utility lists all known fields for a device? * [ ] `--help` * [ ] `-l` * [ ] `-v` * [x] `-a` **5.** What is the default configuration file for iSCSI? * [ ] `initiatorname.iscsi` * [ ] `iscsid.conf` * [ ] `iscsid` * [ ] `iscsiadm` **6.** What are two ways to gain information about a RAID? (Choose two.) * [x] `cat /proc/mdstat` * [x] `mdadm -D` * [ ] `mdamd -f` * [ ] `mdadm -C` **7.** What subcommand for the nvme utility will display all NVMe controllers? * [x] `list` * [ ] `read` * [ ] `smart-log` * [ ] `view-controllers` **8.** What option for the hdparm utility will retrieve or set parameters the using\_dma option? * [ ] `-t` * [ ] `-T` * [x] `-d` * [ ] `-l` **9.** What mode is used to log in to an iSCSI target? * [x] `node` * [ ] `discovery` * [ ] `session` * [ ] `connect` **10.** What component in a target portal group is added to initiatorname.iscsi for security? * [ ] `portal` * [x] `ACL` * [ ] `LUN` * [ ] `backstore` **11.** What option is used to designate a logical volume as a snapshot? * [x] `-s` * [ ] `-n` * [ ] `-r` * [ ] `-L` **12.** What option is used to resize the underlying filesystem of a logical volume? * [x] `-r` * [ ] `-l` * [ ] `-s` * [ ] `-L` **13.** What option is used to specify the number of extents for a logical volume? * [ ] `-L` * [x] `-l` * [ ] `-s` * [ ] `-n` **14.** What type of technology allows hardware devices to bypass the CPU and communicate with system memory? * [x] Direct Memory Access * [ ] Interrupt Requests * [ ] Memory Access Request * [ ] Internal Request **15.** What partition type specifies a RAID partition? * [ ] The 8E type * [ ] The 83 type * [ ] The 82 type * [x] The FD type **16.** What option is used to specify a RAID device as faulty? * [ ] `-C` * [ ] `-x` * [ ] `-n` * [x] `-f` **17.** What RAID level uses mirroring to prevent data loss? * [ ] RAID 5 * [x] RAID 1 * [ ] RAID 2 * [ ] RAID 0 **18.** What is the main configuration file for the management of software RAID with the mdadm utility? * [ ] `mdstat` * [ ] `md.conf` * [ ] `mdadm` * [x] `mdadm.conf` **19.** What command is used to discard unwanted blocks on a mounted filesystem? * [x] `fstrim` * [ ] `dump` * [ ] `mkfs` * [ ] `fsck` # 205: Networking Configuration ## **205.1 Basic networking configuration (weight: 3)** {% hint style="success" %} Candidates should be able to configure a network device to be able to connect to a local, wired or wireless, and a wide-area network. This objective includes being able to communicate between various subnets within a single network including both IPv4 and IPv6 networks. **Key Knowledge Areas:** * Utilities to configure and manipulate ethernet network interfaces * Configuring basic access to wireless networks **The following is a partial list of the used files, terms and utilities:** * ip * ifconfig * route * arp * iw * iwconfig * iwlist {% endhint %} ### Managing Network Interfaces * `ifconfig`: configure a network interface
CommandPurpose
ifconfigDisplay information for active network interfaces
ifconfig -aDisplay information for all network interfaces
ifconfig eth0Display information for a specific network interface
ifconfig eth0 upBring a device online
ifconfig eth0 downBring a device offline
ifconfig eth0 192.168.1.200Assign an IP address to a network interface
ifconfig eth0 netmask 255.255.255.0Assign a netmask to a network interface
ifconfig eth0 broadcast 192.168.1.255Assign a broadcast address to a network interface
ifconfig eth0 192.168.1.200Remove an IP address from a network interface
ifconfig eth0 mtu numberSet maximum transmission unit for a network interface
ifconfig eth0 promiscSet a network interface to promiscuous mode
* `ip`: Show/manipulate routing, devices, policy routing, and tunnels. ```bash ip [ OPTIONS ] OBJECT { COMMAND | help } ``` OBJECT: * `link` * `addr` * `addrlabel` * `route` * `rule` * `neigh` * `tunnel` * `maddr` * `mroute` * `monitor`
CommandPurpose
ip helpDisplay a list of commands and options for the ip command
ip addr helpDisplay a list of commands and options for the address subcommand
ip link helpDisplay a list of commands and options for the link subcommand
ip addrShow information for all address
ip addr show dev eth0Show information for a specific device
ip addr add 192.168.1.200/24 dev eth0Add a address to device
ip addr del 192.168.1.200/24 dev eth0Remove an address from a device
ip addr add 192.168.1.200/24 broadcast 192.168.1.255 dev eth0Add an IP address specific broadcast address to a device
ip linkShow information for all interfaces
ip link show dev eth0Show information for a single device
ip -s linkShow interface statistics
ip link setAlter the status of an interface
ip link set mtu numberSet maximum transmission unit for a network interface
ip link set eth0 promisc onSet a network interface to promiscuous mode
ip link set eth0 upBring a device online
ip link set eth0 downBring a device offline
* `iwconfig`: configure a wireless network interface * `iwlist`: Get more detailed wireless information from a wireless interface
CommandPurpose
iwconfigDisplay information about all available wireless interfaces
iwconfig wlan0Display information about a wireless interface
iwconfig --helpDisplay a list of commands and options
iwconfig wlan0 essid "MyNetwork" key my_keyConnect to a wireless network by providing a key
iwconfig wlan0 rate 24MSet the bitrate for an interface
iwlist wlan0 scanScan for available wireless networks
iwlist wlan0 freqList available frequencies
iwlist wlan0 rateList available bit rates
* `iw`: show / manipulate wireless devices and their configuration ```bash iw [ OPTIONS ] { help | OBJECT COMMAND } ``` | Command | Purpose | | ------------------------- | ------------------------------------------------ | | `iw help` | Print all supported commands | | `iw help command` | Print help information for specified command | | `iw dev` | View available wireless interfaces | | `iw list` | List all wireless devices and their capabilities | | `iw dev wlan0 link` | Display link information | | `iw dev wlan0 info` | Show information for an interface | | `iw phy phy0 info` | Show capabilities for a device | | `iw event` | Monitor event from the kernel | | `iw wlan0 scan` | Scan for available SSIDs | | `iw dev wlan0 connad` | Connect to a wireless network | | `iw dev wlan0 disconnect` | Disconnect from a wireless network | ### Discovering Network Devices * `arp`: manipulate the system ARP cache
CommandPurpose
arp [-avn]Display the contents of the ARP cache
arp -i eth1Display entries for an interface
arp -a 192.168.1.9Display entries for an IP address
arp -s 192.168.1.9 -i eth2 1:2:3:4:5:6Add an enrty to the ARP cache
arp -i eth1 -d 192.168.1.9Remove an entry from the ARP cache
* `ip neigh:` Display the neighbor objects or the ARP cache | | | | ---------------------------------------------------------- | -------------------------------------------------- | | `ip neigh` | Display neighbor objects | | `ip -s neigh` | Display neighbor objects in verbos with statistics | | `ip neigh show dev eth1` | Show o `arp` cache for a device | | `ip neigh add 192.168.1.9 lladdr 1:2:3:4:5:6 dev eth1` | Add an entry into the ARP table | | `ip neigh del 192.168.1.9 dev eth1` | Invalidate an entry in the ARP table | | `ip neigh replace 192.168.1.9 lladdr 1:2:3:4:5:6 dev eth1` | Replace an entry or add one if not defined | ## **205.2 Advanced Network Configuration (weight: 4)** {% hint style="success" %} Candidates should be able to configure a network device to implement various network authentication schemes. This objective includes configuring a multi-homed network device and resolving communication problems. **Key Knowledge Areas:** * Utilities to manipulate routing tables * Utilities to configure and manipulate ethernet network interfaces * Utilities to analyse the status of the network devices * Utilities to monitor and analyse the TCP/IP traffic **The following is a partial list of the used files, terms and utilities:** * ip * ifconfig * route * arp * ss * netstat * lsof * ping, ping6 * nc * tcpdump * nmap {% endhint %} ### Adjusting Network Routing * `ip route` and `route`: show / manipulate the IP routing table | Command | Purpose | | ------------------------------------------------- | ---------------------------------------------------- | | `ip route show` | Display the routing table | | `ip route add 10.0.2.0/24 via 10.0.2.10 dev eth1` | Add a route | | `ip route del 10.0.2.0/24 via 10.0.2.10 dev eth1` | Remove a route | | `ip route add default via 10.0.2.10` | Add a default gateway | | `ip route add prohibit 10.0.2.10/24` | Blockl the destinication route and send ICMP message | | `ip route add blackhole 10.0.2.0/24` | Block the destination route and silently discard | | Command | Purpose | | ------------------------------------------------------ | ----------------------------------------- | | `route (n)` | Display the routing table | | `route add -net 10.0.2.0/24 gw 10.0.2.10 eth1` | Add a route | | `route del -net 10.0.2.0/24 gw 10.0.2.10 eth1` | Remove a route | | `route add default gw 10.0.2.10` | Add a default gateway | | `route add -host 10.0.2.10 reject` | Block the destination route for a host | | `route add -net 10.0.2.0 netmask 255.255.255.0 reject` | Block the destination route for a network | ### Monitoring Network Sockets * `ss`: A utility used to investigate network sockets and dump socket statistics. | Option | Description | | -------------------- | ---------------------------------------- | | `-l`, `--listening` | Display listening server sockets | | `-a`, `--all` | Display all sockets (default: connected) | | `-i`, `--interfaces` | Display interfaces table | | `-s`, `--summary` | Show socket usage summary (like SNMP) | | `-e`, `--extended` | Show detailed socket information | | `-n`, `--numeric` | Don't resolve names | | `-p`, `--programs` | Display PID/Program name for sockets | | `-t`, `--tcp` | Display only TCP sockets | | `-u`, `--udp` | Display only UDP sockets | * `netstat`: Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. | Option | Description | | -------------------- | ---------------------------------------- | | `-l`, `--listening` | Display listening server sockets | | `-a`, `--all` | Display all sockets (default: connected) | | `-i`, `--interfaces` | Display interfaces table | | `-s`, `--statistics` | Show network statistics | | `-e`, `--extended` | Show detailed socket information | | `-v`, `--verbose` | Be verbose | | `-n`, `--numeric` | Don't resolve names | | `-p`, `--programs` | Display PID/Program name for sockets | | `-t`, `--tcp` | Display only TCP sockets | | `-u`, `--udp` | Display only UDP sockets | | `-r`, `-route` | Display routing table | * `lsof`: A utility that lists open files.
OptionDescription
-u usernameList open files by user
-u ^usernameList open files and exclude a user
-i [46][protocol][@hostname|hostaddr][:service|port]List open files by network connections
-p PIDList open files by PID
-p ^PIDList open files and exculde a PID
/directoryList open files by directory
/dev/sda1List open files by device
-cList open files by process name
### Monitoring Network Traffics * `tcpdump`: A network traffic monitoring tool. Can monitor protocols other than TCP. Lofical operators **and** and **or** can be used to combine filters. | Option | Description | | -------------------------- | ------------------------------------------------------------------ | | `-D` | List interfaces available for capture | | `-i eth0` | Capture packets on an interface or all interfaces (any) | | `-c` | Capture a specified count of packets | | `-n` | Disable hostname resolution | | `-nn` | Disable protocol, port and hostname resolution | | `-i any protocol` | Capture packets by protocol on all interfaces | | `-i any host 10.0.2.10` | Capture packets by a host on all interfaces | | `-i any src/dst 10.0.2.10` | Capture packets by source or destination address on all interfaces | | `-A` | View packet content in ASCII | | `-X` | View packet content in hex and ASCII | | `-w file_name.pcap` | Save the output of `tcpdump` to a file | | `-r file_name.pcap` | Read packets from a file | * `nmap`: Network Mapper is a network exploration and security scanner. The network mapper services file is located at `/usr/share/nmpa/nmpa-services`.
OptionDescription
hostnameScan using a hostname or multiple hostnames
10.0.2.10Scan using IP address or multi IP addresses
-v 10.0.2.10Increase verbosity
-iL hosts.txtScan a list of hosts from a file
-A 10.0.2.10Enable OS detection, version detection, script scanning, and traceroute
-O 10.0.2.10Enable OS detection
-sA 10.0.2.10Detect firewall or packet filters
-Pn 10.0.2.10Skip host discovery (formerly -PN)
-sn 10.0.2.10Perform a "ping scan" - Dot not detect open ports (formerly -sP)
-F 10.0.2.10Perform fast scan using less ports
-r 10.0.2.10Scan ports consecutively - don't randomize
--iflistView host interface and route information
-p 22, 443 10.0.2.10Specify ports to scan
-sU 58 10.0.2.10Scan for a UDP port
-sV 10.0.2.10Determine service/version information
-sS 10.0.2.10Perform TCP SYN scan (stealthy scan)
-sT 10.0.2.10Perform TCP connect scan
### Interacting with Remote Hosts * `ping` and `ping6`: Utilities used to send ICMP ECHO\_REQUEST to network hosts. Provided by the `iputils` packages. All options can be used by `ping` and `ping6` execpt for -F (not listed), which in used to allocated a 20-bit flow label on echo request packets. | Option | Description | | ----------------- | ------------------------------------------------------ | | `hostname` | Send a stream of ICMP packets to a hostname | | `10.0.2.10` | Send a stream of ICMP packets to an IP address | | `-c 5 10.0.2.10` | Send a specified amount of packets | | `-s 10.0.2.10` | Alter the size of the packets | | `-i 3 10.0.2.10` | Change the interval for sending packets | | `-q 10.0.2.10` | Only show the summary information | | `-w 5 10.0.2.10` | Set a timeout of when to stop sending packets | | `-f 10.0.2.10` | Flood ping. Send packets as soon as possible. | | `-p ff 10.0.2.10` | Fill a packet with data. ff fills the packet with ones | | `-b 10.0.2.10` | Send packets to a broadcast address | | `-t 10 10.0.2.10` | Limit the number of network hops | | `-v 10.0.2.10` | Increase verbosity | * `ncat (nc):` A network utility that provides several options for interacting with hosts using TCP or UDP over IPv4 and IPv6. Provided by the `nmap-ncat` package. | Option | Purpose | | ----------------------- | ------------------------------------------- | | `-l port` | Listen for inbound connections on a port | | `10.0.2.10 port` | Connect to remote system on a specific port | | `-u udp_port` | Specify a UDP port (TCP is the default) | | `-w time_count` | Terminate connection after specified time | | `-l -k port` | Accept multiple connections in listen mode | | `-v` | Increase verbosity | | `-z` | Report connection status only | | `-i` | Set an idle timeout | | `-v -z 10.0.2.10 22 80` | Scan multiple ports | | `-v -z 10.0.2.10 20-80` | Scan a range of ports | | `-c command` | Executes given command via /bin/sh | | `-e command` | Execute the given command | ## **205.3 Troubleshooting network issues (weight: 4)** {% hint style="success" %} Candidates should be able to identify and correct common network setup issues, to include knowledge of locations for basic configuration files and commands. **Key Knowledge Areas:** * Location and content of access restriction files * Utilities to configure and manipulate ethernet network interfaces * Utilities to manage routing tables * Utilities to list network states. * Utilities to gain information about the network configuration * Methods of information about the recognised and used hardware devices * System initialisation files and their contents (Systemd and SysV init) * Awareness of NetworkManager and its impact on network configuration **The following is a partial list of the used files, terms and utilities:** * ip * ifconfig * route * ss * netstat * /etc/network/, /etc/sysconfig/network-scripts/ * ping, ping6 * traceroute, traceroute6 * mtr * hostname * System log files such as /var/log/syslog, /var/log/messages and the systemd journal * dmesg * /etc/resolv.conf * /etc/hosts * /etc/hostname, /etc/HOSTNAME * /etc/hosts.allow, /etc/hosts.deny {% endhint %} ### Undrestanding Network Configuration Files and Locations 
cat ifcfg-eth0

BOOTPROTO=dhcp
DEVICE=eth0
DHCPV6C=yes
HWADDR=02:ne:5a:69:69:0f
IPV6INIT=yes
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
OptionDescription
IPADDR=10.0.1.10Specify the IPv4 address
PREFIX=24Specify the network prefix
NETMASK=10.0.10.1Specify the netmask
GATEWAY=10.0.10.1Specify the gateway
DNS1=192.168.154.3Specify a DNS server
DNS2=10.216.6.3Specify another DNS server
PEERDNS=yesModify the /etc/resolv.conf file (yes|no)
| Option | Description | | -------------------------- | ------------------------------------------------------- | | `TYPE=Ethernet` | The type of network interface device | | `BOOTPROTO=none` | Specify boot protocol (none\|dhcp\|bootp) | | `DEFROUTE=yes` | Specify default route for IPv4 traffic (yes\|no) | | `IPV6_DEFROUTE=yes` | Specify default route for IPv6 traffic (yes\|no) | | `IPV4_FAILURE_FATAL=no` | Disable the device if the configuration fails (yes\|no) | | `IPV6_FAILURE_FATAL=no` | Disable the device if the configuration fails (yes\|no) | | `IPV6INIT=yes` | Enable or disable IPv6 on the interface (yes\|no) | | `IPV6_AUTOCONF=yes` | Enable or disable autoconf configuration (yes\|no) | | `NAME=eth0` | Specify a name for the connection | | `UUID=...` | Specify the unique identifier for the device | | `ONBOOT=yes` | Activate interface on boot (yes\|no) | | `HWADDR=0e:a5:1a:b9:fc:89` | Specify the MAC address for the interface | * `/etc/hosts`: The host configuration file associates hostnames with an IP address. 
cat /etc/hosts

127.0.0.1 localhost.localdomain localhost
10.0.1.10 linuxmaster.example.com linuxmaster
* `/etc/resolv.conf`: The resolver configuration file specifies DNS servers and searches domains for the host. 
cat /etc/resolv.conf

search example.com
nameserver 192.168.20.4
nameserver 172.8.100.3
* `/etc/sysconfig/network`: This configuration file is used to specify global network settings. 
cat /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=linuxmaster.example.com
* `/etc/nsswitch.conf`: The Name Service Switch (NSS) configuration file is used to determine which sources to obtain name-service information and in what order. ```bash cat /etc/nsswitch.conf ... hosts: files dns ... ``` * `/etc/network/interfaces`: (Debian Based-System): ```bash cat /etc/network/interfaces # An example ethernet card setup: (broadcast and gateway are optional) # # auto eth0 # iface eth0 inet static # address 192.168.0.42 # network 192.168.0.0 # netmask 255.255.255.0 # broadcast 192.168.0.255 # gateway 192.168.0.1 ``` #### NetworkManager * Attempts to automate and simplify network configuration * Implements a dynamic network control and configuration daemon to ensure connections stay active * Proactivity creats (temporary) connections for detected network devices * Provides user-friendly administrative tools: GUI, `nmtui`, and `nmcli` > Lines beginning with the word "auto" are used to identify to be brought up when `ifup` is run with the `-a` optio. (This option is used by the system boot scripts.) ### Analyzing Network Diagnostics and Troubleshooting Network Issues * `traceroute`: Tracks the route packets take from an IP network on their way to a given host. * `traceroute6`: is identical to `traceroute` with the `-6` option. 
traceroute [option] hostname [packet_len]
| Option | Description | | --------------------- | --------------------------------------------------------------- | | `-I` | Use ICMP ECHO for probes | | `-T` | Use TCP SYN for probes | | `-f first_ttl` | Specifies what TTL to start (default is 1) | | `-g gateway` | Specify a gateway to route the packets | | `-i interface` | Specify an interface to send packets through | | `-m max_ttl` | Specify the maximum number of hopes (default is 30) | | `-n` | Do not attempt to resolve host names | | `-q` | Set the number of probe packet per hop (default is 3) | | `-w` | Set the time to wait, in seconds, for a response (default is 5) | | `-4` \| `-6` | Use IPv4 or IPv6 only | | `hostname packet_len` | Set the size of the probing packet (default is 60 bytes) | * `mtr`: A network diagnostic utility that combines the funcionality of the `traceroute` and `ping` command ```bash mtr [options] hostname [packet_size] ``` | Option | Description | | --------------- | ---------------------------------------------------------------------------- | | `-r -c 5` | Run `mtr` report mode and print out statistics based on the number of cycles | | `-w` | Run mtr in wide report mode and print out statistics | | `-c 5` | Specify the number of pings | | `-n` | Do not resolve hostnames | | `-b` | Show hostnames and IP addresses | | `-o "LSD NBAW"` | Specify the fields and order of fields | | `-a 10.0.2.20` | Send outgoing packes through a specific interface | | `-i seconds` | Specify the interval for sending packets (default is 1) | | `-m NUM` | Specify the maximum number of hops (default is 30) | | `-f NUM` | Specify the maximim number of hops (default is 1) | | `-u` | Use UDP datagrams instead of ICMP ECHO | | `-T` | Use TCP SYN packets instead of ICMP ECHO | | `-4` \| `-6` | Use IPv4 or IPv6 only | * `journalctl`: A logging system introduce by Systemd. Implemented by the `journald` daemon, which stores logs in a binary format that can viewed by using the `journalctl` utility. Settings for the Systemd journal can be updated bt modifying `/etc/systemd/journald.conf` or by adding configuration files to `/etc/systemd/journald.conf.d/`. | Option | Description | | ----------- | ---------------------------------------------------- | | `-u unit` | View messages for a particular Systemd unit | | `-f` | Follow the journal for the latest messages | | `-e` | Jump to the end of the journal | | `-o format` | Change the format of the messages displayed | | `-x` | Add explanation texts from the message catalogue | | `-p` | Filter messages based on priority specified | | `-S`, `-U` | Show entries from a specified date (since and until) | * `dmesg`: A utiliy used to examine or control the kernel ring buffer. By default, it reads all messages from the kernel ring buffer. | Option | Description | | --------- | ----------------------------------------------------- | | `-C` | Clear the ring buffer | | `-c` | Clear the ring buffer contents after printing | | `-D` | Disable printing message to the console | | `-E` | Enable printing message to the console | | `-e` | Display local time and delta in human-readable format | | `-H` | Enable human readable format | | `-F file` | Read log from a file | * `/var/log/syslog`: The main system log for Debian-based hosts. Stores all global system activity and startup messages. Options are controlled by `/etc/syslog.conf` or `/etc/rsyslog.conf` in newer versions. Additional configuration files can be added to \`/etc/rsyslog.d/. 
cat /var/log/syslog

...
Aug 27 14:26:13 linuxmaster systemd[3616238]: Listening on GnuPG network certificate management daemon.
Aug 27 14:26:13 linuxmaster systemd[3616238]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Aug 27 14:26:13 linuxmaster systemd[3616238]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Aug 27 14:26:13 linuxmaster systemd[3616238]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Aug 27 14:26:13 linuxmaster systemd[3616238]: Listening on GnuPG cryptographic agent and passphrase cache.
Aug 27 14:26:13 linuxmaster systemd[3616238]: Listening on debconf communication socket.
...
* `/var/log/messages`: The main system log on RHEL-based hosts. Stores all global system activity and startup messages. Options are controlled by `/etc/rsyslog.conf`. Additional configurations can be added to \`/etc/rsyslog.d/. 
/var/log/messages

...
Aug 21 03:41:13 linuxmaster rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-57.el7_9.3" x-pid="737" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Aug 21 03:41:13 linuxmaster pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Aug 21 03:41:13 linuxmaster pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__klgf3cDA7cymYCN1 is now logged in
Aug 21 03:41:13 linuxmaster pure-ftpd: (__cpanel__service__auth__ftpd__klgf3cDA7cymYCN1@127.0.0.1) [INFO] Logout.
Aug 21 03:41:41 linuxmaster PAM-hulk[9971]: Brute force detection active: 550 LOGIN DENIED -- TOO MANY FAILURES
...
### Managing Hostnames and Restricting Host-Level Access * `/etc/hostname` and `/etc/HOSTNAME`: The /etc/hostname file is used to store hostname of the system. On some distributions, the `/etc/HOSYNAME` file is used for this purpose but is ofen aliased to `/etc/hostname`. * `hostname` and `hostnamectl`: The `hostname` command is used to show ro set the system's hostname (i.e., `hostname HOSTNAME`). On Systemd systems, the `hostnamectl` command has replaced the `hostname` command (i.e., \`hostnamectl set-hostname HOSTNAME) 
cat /etc/hostname
linuxmaster.example.com

hostname
linuxmaster.example.com

hostnamectl status

   Static hostname: linuxmaster.example.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 7d9f417ed8ed4e2393f3dce9f5a89ef4
           Boot ID: 3179595ad0cd4454a4b0c7a5f33f27cc
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-1160.42.2.el7.x86_64
      Architecture: x86-64
* `/etc/hosts`: This file is used to map hostnames and aliases to IP addresses. * `/etc/hosts.allow` and `/etc/hosts.deny`: These files are used to determine whether a client has permission to connect to a network service on a remote host. The format of both files is as follow: `daemon_list:client_list [:command]`. The daemon list is a comma-seprated list of service daemons, the client list is a comma-separated list of clients, and command is an optional command that is executed when a client tries to access a server daemon. The keyword `ALL` may be used for the daemon and client lists in order to allow or deny access to all clients. ```bash cat /etc/hosts.deny sshd : ALL ``` ```bash cat /etc/hosts.allow sshd : 10.0.3.* ``` ```bash cat /etc/hosts.deny vsfpd : .example.com ``` ```bash cat /etc/hosts.allow vsftpd : linuxmaster.example.com ``` ## Sample Questions # 206: System Maintenance ## **206.1 Make and install programs from source (weight: 2)** {% hint style="success" %} Candidates should be able to build and install an executable program from source. This objective includes being able to unpack a file of sources. **Key Knowledge Areas:** * Unpack source code using common compression and archive utilities. * Understand basics of invoking make to compile programs. * Apply parameters to a configure script. * Know where sources are stored by default. **The following is a partial list of the used files, terms and utilities:** * /usr/src/ * gunzip * gzip * bzip2 * xz * tar * configure * make * uname * install * patch {% endhint %} ### Undrestanding Directories for **St**oring and Building Sources Files * `/usr/src`: The main default for storing sources files. According to the FHS, this directory should only be used for reference. Generally, sources should not be built in this directory heirarchy. ```bash [root@linuxmaster ~]# ls -l /usr/src/ drwxr-xr-x. 2 root root 4096 Apr 11 2018 linux-aws-5.3-headers-5.3.0-1035hbash ``` * `/usr/local`: For use by the system administrator when installing software locally (based on FHS). `/usr/local/src` can be used for storing source code. ```bash [root@linuxmaster ~]# ls -l /usr/local/ drwxr-xr-x. 2 root root 4096 Apr 11 2018 bin drwxr-xr-x. 2 root root 4096 Apr 11 2018 etc ... drwxr-xr-x. 5 root root 4096 Jul 18 12:47 share drwxr-xr-x. 2 root root 4096 Apr 11 2018 src ``` * `/opt`: Reserved for the installation of add-on application software packages (based on FHS). Files should be placed in a separate `/opt/` or `/opt/` directory tree. `` is a name that describes the software package, and `` is the provider's LANANA (Linux Assigned Names and Numbers Autority). ```bash [root@linuxmaster ~]# ls -l /opt drwxr-xr-x. 2 root root 4096 Apr 10 2018 websh ``` ### Compression Utilities * `gzip`: Compresses files using Lempel-Ziv coding (LZYY).`gzip` on only compress single files but may be used to compress tar archives as well. Compresses files should end in .gz, .z, or .tgz in the case of tar files. ```bash gzip [OPTIONS] [FILENAME(s)] ``` | Option | Description | | ------------ | ------------------------------------------------------ | | `-k` | Keep the original input file (not available in CentOS) | | `-c` | Write output to Standard out | | `-r` | Compress all files in a directory | | `-1` -> `-9` | Specify the compression level (default is -6) | | `-d` | Decompress a file | | `-l` | List information about a compressed file | | `-t` | Test the integrity of the compressed file | | `-v` | Increase verbosity | * `bzip2 (bunzip2)`: Compresses files using the Burrows-Wheeler block sorting text compression algorithm and Huffman coding. bzip2 files should end in .bz2, .bz, .tbz2 (tar), or .tbz (tar). ```bash bzip2 [OPTIONS] [FILENAME(s)] ``` | Option | Description | | ------------ | ----------------------------------------------- | | `-k` | Keep the original input file | | `-c` | Write output to standard out | | `-r` | Compress all files in a directory | | `-1` -> `-9` | Specify the compression level (default is -9) | | `-d` | Decompress a file | | `-z` | Force compression regardless of invocation name | | `-s` | Reduce memory usage | | `-l` | List information about a compressed file | | `-t` | Test the integrity of the compressed file | | `-v` | Increase verbosity | * `xz (unzx)`: General-purpose compression tool with a similar command-line syntax to `gzip` and `bzip2`. The file extensions for `xz` is .xz or .txz (tar). The alias `unxz` can be used for decompression. ```bash xz [OPTIONS] [FILENAME(s)] ``` | Option | Description | | ------------ | ------------------------------------------------ | | `-k` | Keep the original file (not available in CentOS) | | `-c` | Write output to standard out | | `-e` | Use a slower variant of the compression level | | `-0` -> `-9` | Specify the compression level (default is -6) | | `-d` | Decompress a file | | `-l` | List information about a compressed file | | `-t` | Test the integrity of the compressed file | | `-v` | Increase verbosity | * `tar`: Creates a tar file by converting groups of files and directories into an archive. It was originally used to create archives on magnetic tapes, which is where the name comes from. Tar files end with .tar extension followed by the compression that was used, if any (e.g. .tar.gz). ```bash tar [OPTIONS] [ARCHIVE_NAME] [FILENAME(s)] ``` | Option | Description | | ------------ | --------------------------------------------------------------------------------- | | `-c` | Create an archive | | `-x` | Extract an archive | | `-t` | List the content of an archive | | `-f` | Specifgy the archive file name | | `-v` | Show files being processed by the command | | `-z` | Filter the archive through gzip | | `-j` | Filter the archive through bzip2 | | `-J` | Filter the archive through xz | | `-C` | Choose a directory to extract the archive to | | `-A` | Append an archive to the end of another archives (does not work with compression) | | `-d` | Find differences between the archive and the file system | | `--delete` | Delete a member from the archive (does not work with compression) | | `-r` | Append a file to the end of an archive | | `--wildcard` | Wildcards match | ### Install a Program from Source * `configure:` This script checks the system to ensure that the project can be built and create the Makefile. Allows for options that can alter the build (see `--help` to see a list of all the options) * `Makefile`: Contain the instructions that are required in order to build the project. * `make`: Reads the project specific Makefile and performs all the actions required in order to build (compile) the program. * `make install`: Installs the compiled program to a default location or the one specified by the configuration script when using the `--prefix` option. #### Patching a Program from Source * `diff`: Compress files line by line. Output displayed can vary based on options passed to the `diff` command. ```bash diff [OPTION]... FILES ``` | Option | Description | | -------- | -------------------------------------------- | | `--help` | List options and descriptions | | `-u` | Output lines of unified text | | `-r` | Recursively compare any subdirectories found | | `-N` | Treat absent files as empty | | `-y` | Output in two columns | * `patch`: `patch` takes a patch file containing a difference listing produced by the `diff` command and applies those differences to one or more original files, producing patched versions. ```bash patch -pnum # or patch [options] [oroginalfile [patchfile]] ``` | Option | Description | | ----------- | ------------------------------------------------- | | `--help` | List options and descriptions | | `-d` | Change to the directory specified | | `-b` | Create a backup of the original file | | `-i` | Read the patch from the .patch file | | `-p[#]` | Strip slashes from the file path to the file name | | `-R` | Revese the previous patch | | `-u` | Interpret patch file as unified context `diff` | | `-n` | Interpret patch file as a normal `diff` | | `-s` | Run the command silently | | `--dry-run` | Print the result of a patch without applying them | ## **206.2 Backup operations (weight: 3)** {% hint style="success" %} Candidates should be able to use system tools to back up important system data. **Key Knowledge Areas:** * Knowledge about directories that have to be included in backups * Awareness of network backup solutions such as Amanda, Bacula, Bareos and BackupPC * Knowledge of the benefits and drawbacks of tapes, CDR, disk or other backup media * Perform partial and manual backups. * Verify the integrity of backup files. * Partially or fully restore backups. **The following is a partial list of the used files, terms and utilities:** * /bin/sh * dd * tar * /dev/st\* and /dev/nst\* * mt * rsync {% endhint %} ### Undrestanding Directories to Consider for Backup #### Reviewing Directories for Backup * `/home`: User home directories. Contains user-specific configuration files and data. * `/etc`: System-wide conf files are used to of a program. * `/usr/local`: Contains locally installed software. The directory structure is purposefully similar to that of the root directory. * `/opt`: Reserved for the installation of add-on application software packages. * `/root`: Home directory of the root user. operation Contains configuration files and data for the root user. * `/srv`: Data for services provided by the system. ```bash [root@linuxmaster ~]# tree -L 1 . ├── bin -> usr/bin ├── boot ├── dev ├── etc ├── home ├── lib -> usr/lib ├── lib32 -> usr/lib32 ├── lib64 -> usr/lib64 ├── libx32 -> usr/libx32 ├── lost+found ├── media ├── mnt ├── opt ├── proc ├── root ├── run ├── sbin -> usr/sbin ├── srv ├── sys ├── tmp ├── usr └── var ``` #### Reviewing Subdirectories in /var for Backup * `/var/backups` (Debian): Contains apt and dpkg package status backups, along with other important system file backups (e.g., `/etc/passwd`). * `/var/lib`: Contains variable state information for applications and the system. * `/var/log`: Contains system and application logs. * `/var/mail`: Contains local user mail. * `/var/www`: A default location for the web root of a web server, which can contain web content. ```bash [root@linuxmaster ~]# tree -L 1 /var /var/ ├── backups ├── cache ├── crash ├── lib ├── local ├── lock -> /run/lock ├── log ├── mail ├── opt ├── run -> /run ├── spool └── tmp ``` ### Comparing Backup Media #### Magnetic Tape Storage * **Pros:** Cost Efficient, Storage Capacity, Reliability, Security * **Cons:** Higher Upfront Cost, Slower Data Access, Susceptibility on Natural Disasters * The `mt` Utility: The `mt` utility is provided bt the `mt-st` package and is used to control magnetic tape drive operations on a Linux system. Tape devices appear as `/dev/st` for rewinding tape devices and `/dev/nst` for non-rewinding tape devices. #### Optical Media Storage * **Pros:** Compact Physical Size, Low Cost, Read/Write Speed * **Cons:** Limited Storage Capacity, Questionable Durability, Variable Speed #### Hard Disk Storage * **Pros:** Cost Efficient, High Storage Capacity, Reliability, Wide Availability * **Cons:** Slower Speed Compared to SSDs, Larger Physical Size, Higher Power Usage, Durability ### Comparing Network Backup Solutions #### AMANDA "AMANDA, the Advanced Maryland Automatic Network Disk Archiver, is a backup solution that allows the IT administrator to set up a single primary backup server to back up multiple hosts over network to tape drives/changers or disks or optical media. Amanda uses native utilities and formats (e.g., dump and/or GNU tar) and can back up a large number of servers and workstations running multiple versions of Linux or Unix. Amanda uses a native Windows client to back up Microsoft Windows desktops and servers." ([www.amanda.org](http://www.amanda.org)) * Developed at the University of Maryland * Used to back up multiple computers on a network * Implements a client-server model * Has a free community edition and a paid enterprise edition * Has the ability to back up multiple versions of Linux and Unix * Includes a native Windows client to backup Microsoft Windows desktops and servers * Supports tape-, disk-, and cloud-based backups #### Bacula "Bacula is a set of computer programs that permits the system administrator to manage backup, recovery, and verification of computer data across a network of computers of different kinds. Bacula can also run entirely upon a single computer and can backup to various types of media, including tape and disk." ([www.bacula.org](http://www.bacula.org)) * Implements a client-server model * Supports Linux, Unix, Windows, and macOS * Bacula Systems provides support contracts for Bacula Enterprise Edition * Uses various storage media for backups (e.g., tape and disk) * Has a modular design * A robust backup solution intended for more advanced users * Bacula components: Bacula Director, Bacula Console, Bacula File, Bacula Storage, Bacula Catalog, Bacula Monitor #### **Bareos** "Bareos (Backup Archiving Recovery Open Sourced) is a reliable, cross-network open source software for backup, archiving and recovery of data for all well-established operating systems. Emerged from the Bacula Project in 2010, Bareos was and is actively developed as a fork and enriched with lots of new features." ([www.bareos.com](http://www.bareos.com)) * Stands for Backup Archiving Recovery Open Sourced (Bareos) * Is a fork of Bacula that emerged in 2010 * Has an updated web interface as well as other features * The Bareos project and documentation can be found at [www.bareos.org](http://www.bareos.org) - Professional services are provided at [www.bareos.com](http://www.bareos.com) * Supports multiple storage media (e.g., disk, tape, and cloud) * Implements the same core components as Bacula #### **BackupPC** "BackupPC is a high-performance, enterprise-grade system for backing up Linux, Windows and macOS PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain." () * Hosted on GitHub and distributed under the GPL license * Provides a web interface for initiating/restoring backups and for viewing logs and configurations * Uses data deduplication (or pooling), so identical files across multiple backups are only stored once * Uses compression on new files (not already pooled) * Does not require any client-side software * The `smb` protocol is used to extract backup data on Windows clients * `tar over ssh/ rsh/nfs` is used to back up data on Linux systems (`rsync` is also supported) ### Creating and Restoring Backups * `rsync`: `rsync` stands for remote sync and is used for copying and synchronizing files and directories, both remotely and locally. ```bash rsync [OPTIONS] SOURCE DESTINATION ``` | Option | Description | | ------------------------- | ---------------------------------------------------------------- | | `-a` | Use archive mode which preserves permissions, ownership, etc. | | `-b` | Performs a backup during data synchronization | | `-e` | Specify a protocol | | `-h` | Output in a human-readable format | | `-n` | Performs a dry run | | `-r` | Copy data recursively (do not keep ownership, permissions, etc.) | | `-u` | Skip files that are newer on the destination | | `-v` | Use verbose output | | `-z` | Use compression | | `--progress` | Show transfer progress | | `--include` / `--exclude` | Choose files/directories to include | | `--delete` | Delete any file or directory not located in the source | | `--remove-sources-files` | Removes source files after a successful transfer | * `dd`: The `dd` command is used to vonvert and copy files. This extends to entire disks/partitions, which cane be transferred to and from files or other disks/partitions. ```bash dd [OPERAND] # or dd [OPTION] ``` | Option | Description | | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `if` | The input file (file/disk/partition) | | `of` | The output file (file/disk/partition) | | `bs` | Read and write BYTES at a time (default: 512) | | `count` | Number of blocks to transfer | | `ibs` | Read up to BYTES at a time (default: 512) | | `obs` | write BYTES at a time (default: 512) | | `conv` |

Applies data conversion (use command-seprated list):

noerror - dd will continue running even if it encounters error

notrunc - dd will not truncate the output file

sync - dd will pad input blocks with NULs

| ```bash # Examples ... # Copy an HDD to another HDD usin noerror and sync [root@linuxmaster ~]# dd if=/dev/sda of=/dev/sdb conv=noerror,sync # Create an image of the HDD [root@linuxmaster ~]# dd if=/dev/sda of=/home/sda.img # Restore an image of the HDD [root@linuxmaster ~]# dd if=/home/sda.img of=/dev/sda # Create a backup of a CDROM using a block size of 2048 [root@linuxmaster ~]# dd if=/dev/cdrom of=/home/backup.iso bs=2048 # Overwrite a disk write zero using a block size of 4096 [root@linuxmaster ~]# dd if=/dev/zero of=/dev/sda bs=4096 ``` ## 206.3 Notify users on system-related issues (weight: 1) {% hint style="success" %} Candidates should be able to notify the users about current issues related to the system. **Key Knowledge Areas:** * Automate communication with users through logon messages. * Inform active users of system maintenance **The following is a partial list of the used files, terms and utilities:** * /etc/issue * /etc/issue.net * /etc/motd * wall * shutdown * systemctl {% endhint %} #### Using Automated Messages to Communicate System-Related Issues * `issue`: `/etc/issue` is a text file that contains a message or system identification to be printed before the login prompt. It may contain various `@char` and `\char` sequences, if supported by the getty-type program employed on the system. * `issue.net`: `/etc/issue.net` is the historical location for pre-login banners for network logins using protocols lik: `telnet` or `rlogin` but can be used for SSH (escaped characters are not processed). **IMPORTANT:** To add a message for remote logins, the banner path in `/etc/ssh/sshd_config` must be updated and the `sshd` service must be restarted. | Option | Description | | ------ | ---------------------------------------------------------------------------------------------- | | `\b` | Insert the baud rate of the current line. | | `\d` | Insert the current date. | | `\s` | Insert the system name, the name of the operating system. | | `\l` | Insert the name of the current tty line. | | `\m` | Insert the architecture identifier of the machine. | | | Insert the node name of the machine, also known as the hostname. | | `\o` | Insert the domain name of the machine. | | | Insert the release number of the OS. Insert the current time. | | `\u` | Insert the number of current users logged in. | | `\U` | Insert the string "1 user" or" users", where "users" is the number of current users logged in. | | \\`v` | Insert the version of the OS, e.g., the build date, etc. | * `/etc/motd`: The contents of `/etc/motd` are displayed after a successful login before the login shell is executed. The abbreviation "motd" stands for "message of the day", and this file has been traditionally used for exactly that (it requires much less disk space than mail to all users). ```bash [root@linuxmaster ~]# cat /etc/motd Welcome to Ubuntu 18.04.05 LTS (GNU/Linux 5.4.0-1037 x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage * What are your preferred Linux desktop apps? Help us set the default desktop apps in Ubuntu 18.04 LTS: - https://ubu.one/apps1804 10 packages can be updated. 0 updates are security updates. Last login: Fri Aug 28 18:41:49 2021 from 192.168.0.11 ``` * `wall`: `wall` displays a message, the contents of a file, or otherwise its standard input on the terminals of all currently logged-in users. ```bash wall [-n] [-t timeout] [-g group] [message | file] ``` The `/usr/bin/wall` command send simple messages to certain system users. Only users who meet the following conditions will receive these messages: Users who are currently into a terminal (tty#) or a terminal-emulator (pts/#), and Users who have their message status set to "yes" ```bash [root@linuxmaster ~]# mesg is n [root@linuxmaster ~]# mesg y [root@linuxmaster ~]# mesg is y ``` | Option | Description | | ------ | ------------------------------------------------------------------- | | `-n` | Suppress the banner (do not print banner, works only for root) | | `-t` | Abandon the write attempt after the time specified (default is 300) | | `-g` | Limit message to members of a group (or GID) | | `-V` | Display version information | | `-h` | Display help text | * `shutdown`: `shutdown` is used to halt, power off, or reboot the machine. * A shutdown time may be specified in "hh:mm" format or with `+m` where `m` is the number of minutes from now (`now` is an alias for `+0`) * In order to send a wall message, you must specify a time argument. ```bash shutdown [OPTIONS...] [TIME] [WALL...] ``` | Option | Description | | -------------------------- | ---------------------------------------------------------- | |

-H

| Halt the machine | | `-P` | Power off the machine (this is the default) | | `-r` | Reboot the machine | | `-k` | Do not halt, power off, or reboot; just write wall message | | `-c` | Cancel a pending shutdown | | `--no-wall` | Do not send wall message before halt, power-off, reboot. | ## Commands and Notes * `/usr/src`: The main default for storing sources files. * `/usr/local`: For use by the system administrator when installing software locally (based on FHS). `/usr/local/src` can be used for storing source code. * `/opt`: Reserved for the installation of add-on application software packages (based on FHS). * `gzip`: Compresses files using Lempel-Ziv coding (LZYY). * `bzip2` (`bunzip2`): Compresses files using the Burrows-Wheeler block sorting text compression algorithm and Huffman coding. * `xz` (`unzx`): General-purpose compression tool with a similar command-line syntax to gzip and `bzip2`. * `tar`: Creates a tar file by converting groups of files and directories into an archive. * `configure`: This script checks the system to ensure that the project can be built and create the Makefile. * `Makefile`: Contain the instructions that are required in order to build the project. * `make`: Reads the project specific `Makefile` and performs all the actions required in order to build (compile) the program. * `make install`: Installs the compiled program to a default location or the one specified by the configuration script when using the `--prefix` option. * `diff`: Compress files line by line. Output displayed can vary based on options passed to the `diff` command. * `patch`: `patch` takes a patch file containing a difference listing produced by the `diff` command and applies those differences to one or more original files, producing patched versions. * `rsync`: `rsync` stands for remote sync and is used for copying and synchronizing files and directories, both remotely and locally. * `dd`: The `dd` command is used to vonvert and copy files. * `issue`: `/etc/issue` is a text file that contains a message or system identification to be printed before the login prompt. * `issue.net`: `/etc/issue.net` is the historical location for pre-login banners for network logins using protocols lik: `telnet` or `rlogin` but can be used for SSH (escaped characters are not processed). * `/etc/motd`: The contents of `/etc/motd` are displayed after a successful login before the login shell is executed. * `wall`: `wall` displays a message, the contents of a file, or otherwise its standard input on the terminals of all currently logged-in users. * `shutdown`: `shutdown` is used to halt, power off, or reboot the machine. ## Sample Questions **1.** What are the three steps in installing a program from source? (Choose three.) * [ ] Makefile * [x] make * [x] configure * [x] make install **2.** What option of the `tar` command uses `xz` compression? * [ ] `-j` * [x] `-J` * [ ] `-z` * [ ] `-c` **3.** What is the default directory for storing source code (as a reference location)? * [ ] `/etc` * [ ] `/opt` * [ ] `/usr/local` * [x] `/usr/src` **4.** What network backup solution is a fork of Bacula? * [ ] BackupPC * [ ] Nagios * [x] Bareos * [ ] AMANDA **5.** What command is used to create a patch file? * [ ] `make` * [ ] `patch` * [ ] `make patch` * [x] `diff` **6.** What option for the rsync command applies compression? * [x] `-z` * [ ] `-r` * [ ] `-a` * [ ] `-u` **7.** Which directory contains system-wide configuration files? * [x] `/etc` * [ ] `/srv` * [ ] `/usr/local` * [ ] `/home` **8.** What option for the wall command will limit a message to members of a group? * [ ] `-h` * [x] `-g` * [ ] `-n` * [ ] `-t` **9.** What file is presented to users who attempt to log in locally? * [x] `/etc/issue` * [ ] `/etc/ssh/sshd_config` * [ ] `/etc/motd` * [ ] `/etc/issue.net` **10.** Which of the following commands will allow a message to be sent to any tty terminals with write access allowed? (Choose all that apply.) * [ ] `mesg` * [ ] `echo` * [ ] `mail` * [x] `wall` **11.** To shut down your system without sending a message to users, use which of the following /sbin/shutdown options? * [ ] `-c` * [ ] `-k` * [x] `--no-wall` * [ ] `--wall` * [ ] None of the above **12.** Your company’s legal department has decided that a login notification is needed on a Linux system that you manage, warning that unauthorized system use is prohibited. Which method is the best one for deploying this legal notification? * [x] Edit the `/etc/issue` file, and place the notification there. * [ ] Edit the `/etc/motd` file, and place the notification there. * [ ] Use the `notify-send` command to send the notification to users. * [ ] Edit the `∼./bashrc` file, and place the notification there. * [ ] None of the above. **13.** Assume that money is plentiful, and you want the fastest media to use for your company’s backups. Based on that information, which is the best backup media choice? * [ ] Magnetic tapes * [ ] Optical discs * [ ] HDD * [x] SSD * [ ] NAS **14.** Your co-worker, also a system administrator, hands you an SSD and says, “This has all the files on it that were created and/or modified since the system’s last full backup.” Which of the following backup types can this backup be? (Choose all that apply.) * [ ] Full * [x] Incremental * [x] Differential * [ ] None of the above * [ ] All of the above **15.** Which of the following Linux directories should be included in a backup when you plan on using the backup data in full system restores? (Choose all that apply.) * [x] `/bin` * [x] `/home` * [ ] `/run` * [ ] `/tmp` * [ ] All of the above **16.** Which of the following software packages are GUI and/or web-based backup solutions? (Choose all that apply.) * [x] Amanda * [x] Duplicity * [ ] Nagios * [x] Bacula * [ ] `connectd` **17.** Which of the following `tar` command options would employ `gzip` compression for creating a backup tarball? (Choose all that apply.) * [ ] `-g` * [x] `--gzip` * [x] `-z` * [ ] `-g` * [x] `z` **18.** When using the tar command for an incremental backup, a snapshot file is created. This snapshot file should have which of the following file extensions as part of its name? * [ ] `.tar` * [ ] `.tgz` * [ ] `.tar.snap` * [x] `.snap` * [ ] `.snar` **19.** To verify a tar backup immediately after it is created, you use which of the following `tar` command options? (Choose all that apply.) * [x] `--compare` * [x] `--diff` * [x] `-d` * [x] `-W` * [ ] `-J` **Answer:** A, B, C, D. The --compare and --diff (options A and B) are long-style options of -d (option C). These three options compare a tar archive file’s members with external files and list the differences. Option D, the -W option (long style is --verify), automatically verifies a tar archive as soon as it is created. However, it cannot be used with tarballs, only tar archives. Option E, -J, is a tar option that causes the tar command to employ xz compression. **20.** Which of the following indicates a tape device that automatically rewinds? (Choose all that apply.) * [x] `/dev/st0` * [ ] `/dev/nst0` * [x] `/dev/ht1` * [ ] `/dev/nht1` * [ ] `/dev/nht0` **Answer:** A, C. The /dev/st0 (SCSI) and /dev/ht1 (PATA) tape device files are for automatically rewinding tape devices. Options B, D, and E all point to non-rewinding tape devices as indicated by the leading n in their device filename, /dev/n\*. **21.** To control a magnetic tape for backups, use the ...... program. * [ ] `tar` * [ ] `star` * [x] `mt` * [ ] `rsync` * [ ] `dd` **22.** Which of the following rsync commands is the proper way to send a backup over the network to the remote host, ServerA, while having the transfer encrypted using OpenSSH? * [ ] `rsync -av Project /run/media/ServerA` * [ ] `rsync -av Project ServerA` * [ ] `rsync -av ServerA Project` * [x] `rsync -av Project ServerA:∼/` * [ ] `rsync -av Project rsync://ServerA:/` **23.** Which of the following are true about the `dd` command? (Choose all that apply.) * [x] The `dd` command can be used to create a low-level copy of a hard drive. * [ ] The `dd` command is popular for daily incremental backups. * [ ] The `dd` command can be safely used to back up a disk mounted at /. * [ ] The `dd` command can be used for managing tapes. * [x] The `dd` command can be used to zero-out a hard disk. **24.** When installing a program from source code, which of the following installation files typically can provide you with additional installation information? (Choose all that apply.) * [x] `README` * [x] `INSTALL` * [ ] `COPYING` * [ ] `RELEASE-NOTES` * [ ] `NEWS` **25.** During the process of installing a program from source, which of the following creates or updates the Makefile with the Makefile.in file’s contents and what it finds on your system? * [ ] `INSTALL` * [x] `configure` * [ ] `make` * [ ] `make install` * [ ] `cp` **26.** to build and install a program. However, the installation fails. What could be done to install the program? (Choose two.) * [ ] Install the binaries manually with `suinstall` * [x] Run `make install` with root privileges * [ ] Do not run `./configure` in order to maintain the default configuration for correct installation * [x] Rerun `./configure` with a `--prefix` option where the user has permissions to write * [ ] Run make install\_local to install into `/usr/local/` **27.** The following command has just been run successfully:\ `cd /opt; tar xvf /dev/nst0;` What will happen if the command sequence is run again? * [ ] An error saying that there is no tape present is generated because the tape has been ejected after being used * [ ] The contents of `/opt` will be restored again * [ ] The entire contents of `/opt` will be replaced with the contents of the next file on the tape * [x] The contents of `/opt` will have additional content added from the next file on the tape **28.** A system has one hard disk and one CD writer which are both connected to SATA controllers. Which device represents the CD writer? * [ ] `/dev/hdb` * [ ] `/dev/sdd` * [ ] `/dev/scd1` * [x] `/dev/sr0` * [ ] `/dev/sr1` **29.** What command ensures that the file systems are written to disk after a lot of write operations? (Specify ONLY the command without any path or parameters.) **Answer:** `sync` **30.** Which of the following is a CD-ROM filesystem standard? * [ ] OSI 9660 * [x] ISO 9660 * [ ] SR0FS * [ ] ISO 8859 * [ ] ROM-EO **31.** Which of the following commands restores only those files containing `lpi` in their name from the archive `lpifiles.tar.gz`? * [x] `tar xvzf lpifiles.tar.gz --wildcards '*lpi*'` * [ ] `tar xvzwf lpifiles.tar.gz '*lpi*'` * [ ] `tar -xvfz lpifiles.tar.gz --deep '*lpi*'` * [ ] `tar -xvzf lpifiles.tar.gz --subdirs '*lpi*'` * [ ] `tar xvzf lpifiles.tar.gz --globbing '*lpi*'` **32.** Which of the following commands erases the contents of the `/dev/sdb3` partition? * [ ] `rm /dev/sdb3` * [x] `dd if=/dev/zero of=/dev/sdb3` * [ ] `dd of=/dev/zero if=/dev/sdb3` * [ ] `umount /dev/sdb3` **33.** Which of the following files will be looked for and used by GNU make, if one of them exists, unless a different file is specified on the command line when trying to compile software from source code? (Choose two.) * [ ] `configure` * [ ] `config.h.in` * [x] `makefile` * [x] `Makefile` * [ ] `Makefile.in` # Docker # Docker Basic ## Install Docker (Ubuntu) ### Uninstall Old Versions ```bash sudo apt-get remove docker docker-engine docker.io containerd runc ``` ### Set up the Repository ```bash sudo apt-get update sudo apt-get install ca-certificates curl gnupg lsb-release sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ``` ### Install Docker Engine ```bash sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin ``` ### Verify that Docker Engine is installed correctly by running the hello-world image. ```bash sudo docker run hello-world ``` ## Docker Images ```bash docker images COMMAND ``` ```bash docker image --help Usage: docker image COMMAND Manage images Commands: build Build an image from a Dockerfile history Show the history of an image import Import the contents from a tarball to create a filesystem image inspect Display detailed information on one or more images load Load an image from a tar archive or STDIN ls List images prune Remove unused images pull Pull an image or a repository from a registry push Push an image or a repository to a registry rm Remove one or more images save Save one or more images to a tar archive (streamed to STDOUT by default) tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE Run 'docker image COMMAND --help' for more information on a command. ``` Ref: ```bash root@linuxmaster:~# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest feb5d9fea6a5 11 months ago 13.3kB root@linuxmaster:~# docker image pull nginx Using default tag: latest latest: Pulling from library/nginx ... Digest: sha256:b95a99feebf7797479e0c5eb5ec0bdfa5d9f504bc94da550c2f58e839ea6914f Status: Downloaded newer image for nginx:latest docker.io/library/nginx:latest root@linuxmaster:~# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 2b7d6430f78d 8 days ago 142MB hello-world latest feb5d9fea6a5 11 months ago 13.3kB root@linuxmaster:~# docker pull alpine Using default tag: latest latest: Pulling from library/alpine ... Digest: sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad Status: Downloaded newer image for alpine:latest docker.io/library/alpine:latest root@linuxmaster:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 2b7d6430f78d 8 days ago 142MB alpine latest 9c6f07244728 3 weeks ago 5.54MB hello-world latest feb5d9fea6a5 11 months ago 13.3kB root@linuxmaster:~# docker image rm nginx Untagged: nginx:latest Untagged: nginx@sha256:b95a99feebf7797479e0c5eb5ec0bdfa5d9f504bc94da550c2f58e839ea6914f ... Deleted: sha256:6485bed636274e42b47028c43ad5f9c036dd7cf2b40194bd556ddad2a98eea63 root@linuxmaster:~# docker rmi alpine Untagged: alpine:latest Untagged: alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad... Deleted: sha256:994393dc58e7931862558d06e46aa2bb17487044f670f310dffe1d24e4d1eec7 root@linuxmaster:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest feb5d9fea6a5 11 months ago 13.3kB ``` ### Build an image from a Dockerfile ```bash docker image build --help Usage: docker image build [OPTIONS] PATH | URL | - Build an image from a Dockerfile Options: --add-host list Add a custom host-to-IP mapping (host:ip) --build-arg list Set build-time variables --cache-from strings Images to consider as cache sources --cgroup-parent string Optional parent cgroup for the container --compress Compress the build context using gzip --cpu-period int Limit the CPU CFS (Completely Fair Scheduler) period --cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota -c, --cpu-shares int CPU shares (relative weight) --cpuset-cpus string CPUs in which to allow execution (0-3, 0,1) --cpuset-mems string MEMs in which to allow execution (0-3, 0,1) --disable-content-trust Skip image verification (default true) -f, --file string Name of the Dockerfile (Default is 'PATH/Dockerfile') --force-rm Always remove intermediate containers --iidfile string Write the image ID to the file --isolation string Container isolation technology --label list Set metadata for an image -m, --memory bytes Memory limit --memory-swap bytes Swap limit equal to memory plus swap: '-1' to enable unlimited swap --network string Set the networking mode for the RUN instructions during build (default "default") --no-cache Do not use cache when building the image --pull Always attempt to pull a newer version of the image -q, --quiet Suppress the build output and print image ID on success --rm Remove intermediate containers after a successful build (default true) --security-opt strings Security options --shm-size bytes Size of /dev/shm -t, --tag list Name and optionally a tag in the 'name:tag' format --target string Set the target build stage to build. --ulimit ulimit Ulimit options (default []) ``` ```bash root@linuxmaster:~# docker image build -t firstapp:1.0 . Sending build context to Docker daemon 3.072kB Step 1/4 : FROM python:alpine alpine: Pulling from library/python 213ec9aee27d: Pull complete 6b2a141cd227: Pull complete a292fad6b52e: Pull complete 4593e4e33a59: Pull complete 9fc487f38654: Pull complete Digest: sha256:0c46c7f15ee201a2e2dc3579dbc302f989a20b1283e67f884941e071372eb2cc Status: Downloaded newer image for python:alpine ---> ce4168535f30 Step 2/4 : WORKDIR /opt ---> Running in a0731f6162d6 Removing intermediate container a0731f6162d6 ---> 0e888d12c01e Step 3/4 : COPY ./app.py ./ ---> 8152bff3fcb7 Step 4/4 : CMD python app.py ---> Running in 2e0cc2f2d3a9 Removing intermediate container 2e0cc2f2d3a9 ---> 44f6ee3b8fe9 Successfully built 44f6ee3b8fe9 Successfully tagged firstapp:1.0 root@linuxmaster:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE firstapp 1.0 44f6ee3b8fe9 17 seconds ago 48.7MB python alpine ce4168535f30 2 weeks ago 48.7MB hello-world latest feb5d9fea6a5 11 months ago 13.3kB root@linuxmaster:~# cat Dockerfile FROM python:alpine WORKDIR /opt COPY ./app.py ./ CMD python app.py root@linuxmaster:~# cat app.py print('Hello World!') ``` ## Docker Containers ```bash docker container COMMAND ``` ```bash docker container --help Usage: docker container COMMAND Manage containers Commands: attach Attach local standard input, output, and error streams to a running container commit Create a new image from a container's changes cp Copy files/folders between a container and the local filesystem create Create a new container diff Inspect changes to files or directories on a container's filesystem exec Run a command in a running container export Export a container's filesystem as a tar archive inspect Display detailed information on one or more containers kill Kill one or more running containers logs Fetch the logs of a container ls List containers pause Pause all processes within one or more containers port List port mappings or a specific mapping for the container prune Remove all stopped containers rename Rename a container restart Restart one or more containers rm Remove one or more containers run Run a command in a new container start Start one or more stopped containers stats Display a live stream of container(s) resource usage statistics stop Stop one or more running containers top Display the running processes of a container unpause Unpause all processes within one or more containers update Update configuration of one or more containers wait Block until one or more containers stop, then print their exit codes Run 'docker container COMMAND --help' for more information on a command. ``` Ref: ```bash root@linuxmaster:~# docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES root@linuxmaster:~# docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ddc30affeab2 hello-world "/hello" 19 minutes ago Exited (0) 19 minutes ago youthful_matsumoto root@linuxmaster:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ddc30affeab2 hello-world "/hello" 20 minutes ago Exited (0) 20 minutes ago youthful_matsumoto root@linuxmaster:~# docker build -t newapp -t newapp:1.0 . Sending build context to Docker daemon 3.072kB Step 1/6 : FROM python:alpine ---> ce4168535f30 Step 2/6 : RUN pip install flask ---> Running in 2355fe929f8a Collecting flask Downloading Flask-2.2.2-py3-none-any.whl (101 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.5/101.5 kB 8.1 MB/s eta 0:00:00 Collecting Werkzeug>=2.2.2 Downloading Werkzeug-2.2.2-py3-none-any.whl (232 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.7/232.7 kB 9.0 MB/s eta 0:00:00 Collecting itsdangerous>=2.0 Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB) Collecting Jinja2>=3.0 Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 14.1 MB/s eta 0:00:00 Collecting click>=8.0 Downloading click-8.1.3-py3-none-any.whl (96 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 14.8 MB/s eta 0:00:00 Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.1-cp310-cp310-musllinux_1_1_x86_64.whl (29 kB) Installing collected packages: MarkupSafe, itsdangerous, click, Werkzeug, Jinja2, flask Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.2 click-8.1.3 flask-2.2.2 itsdangerous-2.1.2 Removing intermediate container 2355fe929f8a ---> 5523d98df312 Step 3/6 : WORKDIR /opt ---> Running in 30fa670591f1 Removing intermediate container 30fa670591f1 ---> b14b85b397bd Step 4/6 : COPY ./app.py ./ ---> 23e52c7226bf Step 5/6 : EXPOSE 5000 ---> Running in 36bfefc3e95b Removing intermediate container 36bfefc3e95b ---> f35d9b43ba4d Step 6/6 : CMD python ./app.py ---> Running in 39bee1319ef3 Removing intermediate container 39bee1319ef3 ---> ae768a1f9832 Successfully built ae768a1f9832 Successfully tagged newapp:latest Successfully tagged newapp:1.0 root@linuxmaster:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE newapp 1.0 ae768a1f9832 10 seconds ago 60.4MB newapp latest ae768a1f9832 10 seconds ago 60.4MB firstapp 1.0 44f6ee3b8fe9 8 minutes ago 48.7MB python alpine ce4168535f30 2 weeks ago 48.7MB hello-world latest feb5d9fea6a5 11 months ago 13.3kB root@linuxmaster:~# docker container create --name newapp newapp 1aebfdb5997f7e0ecc4dfc6c2c5d9bd431533f2f109a287405bfde1b8bf43410 root@linuxmaster:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1aebfdb5997f newapp "/bin/sh -c 'python …" 36 seconds ago Created newapp ddc30affeab2 hello-world "/hello" 24 minutes ago Exited (0) 24 minutes ago youthful_matsumoto root@linuxmaster:~# docker container start newapp newapp root@linuxmaster:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1aebfdb5997f newapp "/bin/sh -c 'python …" 2 minutes ago Up 49 seconds 5000/tcp newapp root@linuxmaster:~# docker stop newapp newapp root@linuxmaster:~# docker rm newapp newapp ``` ### Run Container in Attach mode ```bash root@linuxmaster:~# docker run --name newapp -p 5000:5000 newapp * Serving Flask app 'app' * Debug mode: on WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:5000 * Running on http://172.17.0.2:5000 Press CTRL+C to quit * Restarting with stat * Debugger is active! * Debugger PIN: 426-547-533 88.248.106.175 - - [01/Sep/2022 15:48:00] "GET / HTTP/1.1" 200 - 88.248.106.175 - - [01/Sep/2022 15:48:01] "GET /favicon.ico HTTP/1.1" 404 - ``` ### Run Container in Detach mode ```bash root@linuxmaster:~# docker rm newapp newapp root@linuxmaster:~# docker run --name newapp -p 5000:5000 -d newapp 3f0328b58a6dfe22013256b3d22d028052aee0d8ab44c30446b01212d3cfd620 root@linuxmaster:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3f0328b58a6d newapp "/bin/sh -c 'python …" 8 seconds ago Up 7 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp newapp ``` ### Run a command in a running container ```bash root@linuxmaster:~# docker exec newapp top Mem: 1370968K used, 610780K free, 1116K shrd, 35424K buff, 1034988K cached CPU: 9% usr 0% sys 0% nic 90% idle 0% io 0% irq 0% sirq Load average: 0.01 0.02 0.00 2/160 14 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 6 1 root S 30220 2% 0 0% /usr/local/bin/python /opt/app.py 1 0 root S 28048 1% 0 0% python ./app.py 10 0 root R 1604 0% 0 0% top root@linuxmaster:~# docker exec -it newapp sh /opt # root@linuxmaster:~# ls app.py root@linuxmaster:~# exit ``` * `--interactive or -i: Keep STDIN open even if not attached` * `--tty or -t: Allocate a pseudo-TTY` ### Display detailed information on one or more containers ```bash docker container inspect [OPTIONS] CONTAINER [CONTAINER...] ``` ### List port mappings or a specific mapping for the container ```bash docker container port CONTAINER [PRIVATE_PORT[/PROTO]] # docker port newapp 5000/tcp -> 0.0.0.0:5000 5000/tcp -> :::5000 ``` ```bash root@linuxmaster:~# docker logs newapp * Serving Flask app 'app' * Debug mode: on WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:5000 * Running on http://172.17.0.2:5000 Press CTRL+C to quit * Restarting with stat * Debugger is active! * Debugger PIN: 120-117-977 88.248.106.175 - - [01/Sep/2022 15:52:36] "GET / HTTP/1.1" 200 - root@linuxmaster:~# docker top newapp UID PID PPID C STIME TTY TIME CMD root 4792 4763 0 15:52 ? 00:00:00 python ./app.py root 4823 4792 0 15:52 ? 00:00:01 /usr/local/bin/python /opt/app.py root 4868 4763 0 15:53 ? 00:00:00 top root@linuxmaster:~# docker stats newapp CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 3f0328b58a6d newapp 0.21% 35.48MiB / 1.89GiB 1.83% 2.71kB / 1.02kB 0B / 0B 4 ``` ### Remove one or more containers ```bash docker container rm [OPTIONS] CONTAINER [CONTAINER...] ``` ### Remove all stopped containers ```bash docker container prune [OPTIONS] ``` ```bash root@linuxmaster:~# docker stop newapp newapp root@linuxmaster:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3f0328b58a6d newapp "/bin/sh -c 'python …" 12 minutes ago Exited (0) 6 seconds ago newapp ddc30affeab2 hello-world "/hello" 45 minutes ago Exited (0) 45 minutes ago youthful_matsumoto root@linuxmaster:~# docker container prune WARNING! This will remove all stopped containers. Are you sure you want to continue? [y/N] y Deleted Containers: 3f0328b58a6dfe22013256b3d22d028052aee0d8ab44c30446b01212d3cfd620 ddc30affeab236d3023077f495603422c7b4a1b4ff781bbed696129c9c3fc3b1 Total reclaimed space: 111.4kB ``` ```bash root@linuxmaster:~# docker run firstapp:1.0 Hello World! root@linuxmaster:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES root@linuxmaster:~# docker run -d firstapp:1.0 be92a6137535579486213512d99caa6bdd557a1f4a1841558d666de0264862c8 root@linuxmaster:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` ## Docker Networking # Docker Compose # Ansible # Git ## Create Create an existing repository ```bash git clone ssh://user@domain.com/repo.git ``` Create a new local repository ```bash git init ``` ## Local changes Changed files in your working directory ```bash git status ``` Changes to tracked files ```bash git diff ``` Add all current changes to the next commit ```bash git add . ``` Add some changes in \ to the next commit ```bash git add -p ``` Commit all local changes in tracked files ```bash git commit -a ``` Commit previously staged changes ```bash git commit ``` Change the last commit *Don‘t amend published commits!* ```bash git commit --amend ``` ## Commit history Show all commits, starting with newest ```bash git log ``` Show changes over time for a specific file ```bash git log -p ``` Who changed what and when in ```bash git blame ``` ## Branchs and Tags List all existing branches ```bash dig branch -av ``` Switch HEAD branch ```bash git checkout ``` Create a new branch based on your current HEAD ```bash git branch ``` Create a new tracking branch based on a remote branch ```bash git checkout --track ``` Delete a local branch ```bash git branch -d ``` Mark the current commit with a tag ```bash git tag ``` ## Update and Publish List all currently configured remotes ```bash git remote -v ``` Show information about a remote ```bash git remote show ``` Add new remote repository, named \ ```bash git remote add ``` Download all changes from \, but don‘t integrate into HEAD ```bash git fetch ``` Download changes and directly merge / integrate into HEAD ```bash git pull ``` Publish local changes on a remote ```bash git push ``` Delete a branch on the remote ```bash git branch -dr ``` Publish your tags ```bash git push --tags ``` ## Merge and Rebase Merge \ into your current HEAD ```bash git merge ``` Rebase your current HEAD onto \ *Don‘t rebase published commits!* ```bash git rebase ``` Abort a rebase ```bash git rebase --abort ``` Continue a rebase after resolving conflicts ```bash git rebase --continue ``` Use your configured merge tool to solve conflicts ```bash git mergetool ``` Use your editor to manually solve conflicts and (after resolving) mark file as resolved ```bash git add ``` ```bash git rm ``` ## Undo Discard all local changes in your working directory ```bash git reset --hard HEAD ``` Discard local changes in a specific file ```bash git checkout HEAD ``` Revert a commit *by producing a new commit with contrary changes* ```bash git revert ``` Reset your HEAD pointer to a previous commit …and discard all changes since then ```bash git reset --hard ``` ... and preserve all changes as unstaged changes ```bash git reset ``` ... and preserve uncommitted local changes ```bash git reset --keep ``` ## Help and Documentation ```bash git help ``` ## Useful links * [Installing git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) * [Git refrence](https://git-scm.com/docs) * [Git commands](https://git-scm.com/docs/git#_git_commands) * [Git Tutorial (Youtube, Mosh)](https://www.youtube.com/watch?v=8JJ101D3knE) # Terraform Terraform is an infrastructure-as-code software tool created by HashiCorp. Users define and provide data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language, or optionally JSON. # Terraform Basic # Kubernetes Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. [Kubernetes.io - Overview](https://kubernetes.io/docs/concepts/overview/) ## Installation with k3s.io ```bash curl -sfL https://get.k3s.io | sh - ``` ## Basic Commands ```bash # kubectl version Client Version: v1.29.4+k3s1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.4+k3s1 # kubectl help | head kubectl controls the Kubernetes cluster manager. Find more information at: https://kubernetes.io/docs/reference/kubectl/ Basic Commands (Beginner): create Create a resource from a file or from stdin expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service run Run a particular image on the cluster set Set specific features on objects # kubectl get nodes NAME STATUS ROLES AGE VERSION k8s Ready control-plane,master 3d2h v1.29.4+k3s1 # kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s Ready control-plane,master 3d2h v1.29.4+k3s1 157.90.144.2 Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.7.15-k3s1 # kubectl api-resources | head NAME SHORTNAMES APIVERSION NAMESPACED KIND bindings v1 true Binding componentstatuses cs v1 false ComponentStatus configmaps cm v1 true ConfigMap endpoints ep v1 true Endpoints events ev v1 true Event limitranges limits v1 true LimitRange namespaces ns v1 false Namespace nodes no v1 false Node persistentvolumeclaims pvc v1 true PersistentVolumeClaim ``` ### Kind Versions | Kind | Version | | ---------- | ------- | | Pod | v1 | | Service | v1 | | ReplicaSet | apps/v1 | | Deployment | apps/v1 | ## Pods Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers. A Pod's contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific "logical host": it contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host. #### What is a Pod? The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation - the same things that isolate a container. Within a Pod's context, the individual applications may have further sub-isolations applied. A Pod is similar to a set of containers with shared namespaces and shared filesystem volumes. Pods in a Kubernetes cluster are used in two main ways: * Pods that run a single container. The "one-container-per-Pod" model is the most common Kubernetes use case; in this case, you can think of a Pod as a wrapper around a single container; Kubernetes manages Pods rather than managing the containers directly. * Pods that run multiple containers that need to work together. A Pod can encapsulate an application composed of multiple co-located containers that are tightly coupled and need to share resources. These co-located containers form a single cohesive unit. Grouping multiple co-located and co-managed containers in a single Pod is a relatively advanced use case. You should use this pattern only in specific instances in which your containers are tightly coupled. You don't need to run multiple containers to provide replication (for resilience or capacity); if you need multiple replicas, see Workload management. [Kubernetes.io - Pods](https://kubernetes.io/docs/concepts/workloads/pods/) ### Pod Example ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx tier: frontend spec: containers: - name: nginx image: nginx ``` ### Pod Commands ```bash # kubectl run nginx --image=nginx pod/nginx created # kubectl get pods NAME READY STATUS RESTARTS AGE nginx 1/1 Running 0 7s # kubectl apply -f pod.yaml pod/myapp-pod created # # kubectl get pods NAME READY STATUS RESTARTS AGE nginx 1/1 Running 0 4s # kubectl describe pod nginx Name: nginx Namespace: default Priority: 0 Service Account: default Node: k8s/157.90.144.2 Start Time: Mon, 20 May 2024 11:37:00 +0000 Labels: app=nginx tier=frontend Annotations: Status: Running IP: 10.42.0.90 IPs: IP: 10.42.0.90 Containers: nginx-controller: Container ID: containerd://bdd2731ff6673fe732d0bf4a9f002db8ed624ee9c2e3c21834abd30b20792d61 Image: nginx Image ID: docker.io/library/nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c Port: Host Port: State: Running Started: Mon, 20 May 2024 11:37:02 +0000 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9rtkw (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-9rtkw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 20s default-scheduler Successfully assigned default/nginx to k8s Normal Pulling 20s kubelet Pulling image "nginx" Normal Pulled 19s kubelet Successfully pulled image "nginx" in 767ms (767ms including waiting) Normal Created 19s kubelet Created container nginx-controller Normal Started 19s kubelet Started container nginx-controller ``` ## ReplicaSets A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods. #### How a ReplicaSet works A ReplicaSet is defined with fields, including a selector that specifies how to identify Pods it can acquire, a number of replicas indicating how many Pods it should be maintaining, and a pod template specifying the data of new Pods it should create to meet the number of replicas criteria. A ReplicaSet then fulfills its purpose by creating and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod template. A ReplicaSet is linked to its Pods via the Pods' metadata.ownerReferences field, which specifies what resource the current object is owned by. All Pods acquired by a ReplicaSet have their owning ReplicaSet's identifying information within their ownerReferences field. It's through this link that the ReplicaSet knows of the state of the Pods it is maintaining and plans accordingly. A ReplicaSet identifies new Pods to acquire by using its selector. If there is a Pod that has no OwnerReference or the OwnerReference is not a Controller and it matches a ReplicaSet's selector, it will be immediately acquired by said ReplicaSet. #### When to use a ReplicaSet A ReplicaSet ensures that a specified number of pod replicas are running at any given time. However, a Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features. Therefore, we recommend using Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don't require updates at all. This actually means that you may never need to manipulate ReplicaSet objects: use a Deployment instead, and define your application in the spec section. [Kubernetes.io - ReplicaSet](https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/) ### ReplicaSet Example ```yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: myapp-replicaset labels: app: myapp type: frontend spec: replicas: 3 selector: matchLabels: type: frontend template: metadata: name: myapp-replicas labels: app: myapp type: frontend spec: containers: - name: nginx-container image: nginx:latest ``` ### ReplicaSet Commands ```bash # kubectl create -f replicaset.yaml replicaset.apps/myapp-replicaset created # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-replicaset-fqtxn 1/1 Running 0 31s 10.42.0.91 k8s myapp-replicaset-z566r 1/1 Running 0 31s 10.42.0.93 k8s myapp-replicaset-kbv9l 1/1 Running 0 31s 10.42.0.92 k8s # kubectl get replicasets NAME DESIRED CURRENT READY AGE myapp-replicaset 3 3 3 56s # kubectl get rs NAME DESIRED CURRENT READY AGE myapp-replicaset 3 3 3 59s # kubectl describe rs/myapp-replicaset Name: myapp-replicaset Namespace: default Selector: type=frontend Labels: app=myapp type=frontend Annotations: Replicas: 3 current / 3 desired Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: app=myapp type=frontend Containers: nginx-container: Image: nginx:latest Port: Host Port: Environment: Mounts: Volumes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 5m56s replicaset-controller Created pod: myapp-replicaset-fqtxn Normal SuccessfulCreate 5m56s replicaset-controller Created pod: myapp-replicaset-kbv9l Normal SuccessfulCreate 5m56s replicaset-controller Created pod: myapp-replicaset-z566r # kubectl get pods myapp-replicaset-kbv9l -o yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: "2024-05-21T11:24:51Z" generateName: myapp-replicaset- labels: app: myapp type: frontend name: myapp-replicaset-kbv9l namespace: default ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: myapp-replicaset uid: d401b705-74e2-4f4b-a84c-f16956aff1c3 resourceVersion: "109572" uid: 7108b120-c428-4af7-a5ef-67afd49ddb06 spec: containers: - image: nginx:latest imagePullPolicy: Always name: nginx-container resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-64w99 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: k8s preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: kube-api-access-64w99 projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2024-05-21T11:24:53Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2024-05-21T11:24:51Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2024-05-21T11:24:53Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2024-05-21T11:24:53Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2024-05-21T11:24:51Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://f54298a7f82ef810cde637c0c80bc3a14029b18ae0aae50c756b09301c407b6e image: docker.io/library/nginx:latest imageID: docker.io/library/nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c lastState: {} name: nginx-container ready: true restartCount: 0 started: true state: running: startedAt: "2024-05-21T11:24:53Z" hostIP: 157.90.144.2 hostIPs: - ip: 157.90.144.2 phase: Running podIP: 10.42.0.92 podIPs: - ip: 10.42.0.92 qosClass: BestEffort startTime: "2024-05-21T11:24:51Z" ``` #### Scaling a ReplicaSet A ReplicaSet can be easily scaled up or down by simply updating the .spec.replicas field. The ReplicaSet controller ensures that a desired number of Pods with a matching label selector are available and operational. When scaling down, the ReplicaSet controller chooses which pods to delete by sorting the available pods to prioritize scaling down pods based on the following general algorithm: 1. Pending (and unschedulable) pods are scaled down first 2. If `controller.kubernetes.io/pod-deletion-cost` annotation is set, then the pod with the lower value will come first. 3. Pods on nodes with more replicas come before pods on nodes with fewer replicas. 4. If the pods' creation times differ, the pod that was created more recently comes before the older pod (the creation times are bucketed on an integer log scale when the `LogarithmicScaleDown` feature gate is enabled) If all of the above match, then selection is random. ### Scaling Example ```yaml ... spec: replicas: 8 ... ``` ### Scaling Commands ```bash # kubectl replace -f replicaset.yaml replicaset.apps/myapp-replicaset replaced # kubectl get pods NAME READY STATUS RESTARTS AGE myapp-replicaset-fqtxn 1/1 Running 0 15m myapp-replicaset-z566r 1/1 Running 0 15m myapp-replicaset-kbv9l 1/1 Running 0 15m myapp-replicaset-cnh5v 1/1 Running 0 3s myapp-replicaset-jzmk2 1/1 Running 0 3s myapp-replicaset-5l5kj 1/1 Running 0 3s myapp-replicaset-qfjzp 1/1 Running 0 3s myapp-replicaset-2ww4g 1/1 Running 0 3s # kubectl scale --replicas=5 replicaset myapp-replicaset replicaset.apps/myapp-replicaset scaled # kubectl get pods NAME READY STATUS RESTARTS AGE myapp-replicaset-fqtxn 1/1 Running 0 17m myapp-replicaset-z566r 1/1 Running 0 17m myapp-replicaset-kbv9l 1/1 Running 0 17m myapp-replicaset-cnh5v 1/1 Running 0 92s myapp-replicaset-5l5kj 1/1 Running 0 92s # kubectl delete replicasets myapp-replicaset replicaset.apps "myapp-replicaset" deleted # kubectl get pods No resources found in default namespace. ``` ## Deployments A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments. #### Use Case The following are typical use cases for Deployments: * Create a Deployment to rollout a ReplicaSet. The ReplicaSet creates Pods in the background. Check the status of the rollout to see if it succeeds or not. * Declare the new state of the Pods by updating the PodTemplateSpec of the Deployment. A new ReplicaSet is created and the Deployment manages moving the Pods from the old ReplicaSet to the new one at a controlled rate. Each new ReplicaSet updates the revision of the Deployment. * Rollback to an earlier Deployment revision if the current state of the Deployment is not stable. Each rollback updates the revision of the Deployment. * Scale up the Deployment to facilitate more load. * Pause the rollout of a Deployment to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout. * Use the status of the Deployment as an indicator that a rollout has stuck. * Clean up older ReplicaSets that you don't need anymore. [Kubernetes.io - Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) ### Deployment Example ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp-deployment labels: app: myapp type: frontend spec: replicas: 2 selector: matchLabels: type: frontend template: metadata: name: myapp-deployment labels: app: myapp type: frontend spec: containers: - name: nginx-container image: nginx:latest ``` ### Deployment Commands ```bash # kubectl create -f deployment.yaml deployment.apps/myapp-deployment created # kubectl get pods NAME READY STATUS RESTARTS AGE myapp-deployment-69c58b88c6-bn7mm 1/1 Running 0 13s myapp-deployment-69c58b88c6-5q6st 1/1 Running 0 13s # kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE myapp-deployment 2/2 2 2 38s # kubectl get replicasets NAME DESIRED CURRENT READY AGE myapp-deployment-69c58b88c6 2 2 2 66s # kubectl get all NAME READY STATUS RESTARTS AGE pod/myapp-deployment-69c58b88c6-bn7mm 1/1 Running 0 2m1s pod/myapp-deployment-69c58b88c6-5q6st 1/1 Running 0 2m1s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.43.0.1 443/TCP 7d4h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/myapp-deployment 2/2 2 2 2m1s NAME DESIRED CURRENT READY AGE replicaset.apps/myapp-deployment-69c58b88c6 2 2 2 2m1s # kubectl exec -it pod/myapp-deployment-69c58b88c6-5q6st -- nginx -v nginx version: nginx/1.25.5 ``` #### Upgrade an application 1. Update yaml file with new configuration ```yaml ... image: nginx:1.26.0 ... ``` 2. Apply new changes ```bash # kubectl apply -f deployment.yaml deployment.apps/myapp-deployment configured ``` ```bash # kubectl set image deployment/myapp-deployment nginx-container=nginx:1.24.0 deployment.apps/myapp-deployment image updated ``` ## Service In Kubernetes, a Service is a method for exposing a network application that is running as one or more Pods in your cluster. A key aim of Services in Kubernetes is that you don't need to modify your existing application to use an unfamiliar service discovery mechanism. You can run code in Pods, whether this is a code designed for a cloud-native world, or an older app you've containerized. You use a Service to make that set of Pods available on the network so that clients can interact with it. If you use a Deployment to run your app, that Deployment can create and destroy Pods dynamically. From one moment to the next, you don't know how many of those Pods are working and healthy; you might not even know what those healthy Pods are named. Kubernetes Pods are created and destroyed to match the desired state of your cluster. Pods are ephemeral resources (you should not expect that an individual Pod is reliable and durable). Each Pod gets its own IP address (Kubernetes expects network plugins to ensure this). For a given Deployment in your cluster, the set of Pods running in one moment in time could be different from the set of Pods running that application a moment later. [Kubernetes.io - Service](https://kubernetes.io/docs/concepts/services-networking/service/) ### Service Example ```yaml apiVersion: v1 kind: Service metadata: name: redis-db spec: type: ClusterIP ports: - targetPort: 6379 port: 6379 selector: app: myapp name: redis-db ``` ```yaml apiVersion: v1 kind: Service metadata: name: web-service spec: type: NodePort ports: - targetPort: 80 port: 80 nodePort: 30008 selector: app: my-app name: front-end ``` ### Service Commands ```bash # kubectl apply -f service.yaml service/redis-db created # kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 443/TCP 7d13h redis-db ClusterIP 10.43.16.42 6379/TCP 9s # kubectl apply -f service.yaml service/web-service created # kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 443/TCP 7d14h redis-db ClusterIP 10.43.16.42 6379/TCP 7m50s web-service NodePort 10.43.38.70 80:30008/TCP 5s ``` ## DeamonSet A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created. Some typical uses of a DaemonSet are: * running a cluster storage daemon on every node * running a logs collection daemon on every node * running a node monitoring daemon on every node In a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon. A more complex setup might use multiple DaemonSets for a single type of daemon, but with different flags and/or different memory and cpu requests for different hardware types. [Kubernetes.io - DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) ### DaemonSet Example ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: promethues spec: selector: matchLabels: name: node-exporter part: monitoring template: metadata: labels: name: node-exporter part: monitoring spec: containers: - name: node-exporter image: prom/node-exporter:latest ``` ### DaemonSet Commands ```bash # kubectl create ns promethues namespace/promethues created # kubectl apply -f ds.yaml daemonset.apps/node-exporter created # kubectl get pods -n promethues NAME READY STATUS RESTARTS AGE node-exporter-pvkm7 1/1 Running 0 59s # kubectl get daemonsets -n prometheus NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE node-exporter 1 1 1 1 1 6d19h ``` ## StatefulSets StatefulSet is the workload API object used to manage stateful applications. Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of its Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed. [Kubernetes.io - StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) ### StatefulSet Example ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: redis labels: name: redis spec: replicas: 3 selector: matchLabels: name: redis template: metadata: labels: name: redis spec: containers: - name: redis image: redis:latest ``` #### StatefulSet Commands ```bash # kubectl apply -f sts.yaml statefulset.apps/redis created # kubectl get statefulsets NAME READY AGE redis 3/3 14s # kubectl get sts NAME READY AGE redis 3/3 55s # kubectl get pods NAME READY STATUS RESTARTS AGE redis-0 1/1 Running 0 67s redis-1 1/1 Running 0 63s redis-2 1/1 Running 0 61s # kubectl delete sts redis statefulset.apps "redis" deleted ``` ### StatefulSet Example with Volumes ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: redis labels: name: redis spec: replicas: 3 selector: matchLabels: name: redis template: metadata: labels: name: redis spec: containers: - name: redis image: redis:latest volumeMounts: - name: redis-data mountPath: /var/lib/redis volumeClaimTemplates: - metadata: name: redis-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` ```bash # kubectl apply -f sts.yaml statefulset.apps/redis created # kubectl get pods NAME READY STATUS RESTARTS AGE redis-0 1/1 Running 0 41s redis-1 1/1 Running 0 33s redis-2 1/1 Running 0 27s # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE redis-data-redis-0 Bound pvc-053e82b4-8965-43bd-9953-bd22231294a7 1Gi RWO local-path 89s redis-data-redis-1 Bound pvc-4b0ffa32-5fc2-415f-a812-281272722875 1Gi RWO local-path 81s redis-data-redis-2 Bound pvc-463e9ac4-64fa-474f-b633-f87e7c36c90c 1Gi RWO local-path 75s # kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE pvc-053e82b4-8965-43bd-9953-bd22231294a7 1Gi RWO Delete Bound default/redis-data-redis-0 local-path 2m14s pvc-4b0ffa32-5fc2-415f-a812-281272722875 1Gi RWO Delete Bound default/redis-data-redis-1 local-path 2m8s pvc-463e9ac4-64fa-474f-b633-f87e7c36c90c 1Gi RWO Delete Bound default/redis-data-redis-2 local-path 2m2s ``` # AWS Certified Cloud Practitioner * [AWS Certified Cloud Practitioner](https://aws.amazon.com/certification/certified-cloud-practitioner/) * [AWS Certified Cloud Practitioner (CLF-C01) Exam Guide](https://d1.awsstatic.com/training-and-certification/docs-cloud-practitioner/AWS-Certified-Cloud-Practitioner_Exam-Guide.pdf) * [Overview of Amazon Web Services - AWS Whitepaper](https://d1.awsstatic.com/whitepapers/aws-overview.pdf) * [AWS Cloud Practitioner Essentials E-Learning](https://www.aws.training/Details/eLearning?id=60697) * [AWS Certified Cloud Practitioner Practice Exam Questions](https://www.youtube.com/results?search_query=AWS+Certified+Cloud+Practitioner+Practice+Exam+Questions) # Foundations of Cloud Computing ## Understanding Cloud Computing ### What is cloud computing? Cloud computing is the delivery of computing services over the internet. * **Compute:** EC2, Lamba * **Networking:** VPC, Direct Connect * **Storage:** S3, EBS * **Analytics:** Athena, Redshift * **Development:** Cloud9, CodeCommi * **Security:** IAM, Macie * **Database:** RDS, DynamoDB ### Virtual Machines Virtualization lets you divide hardware resources on a single physical server into smaller units. ### Usage Your pay only when you access it and only for what you use. * **On-Demand:** No long-term commitments or upfront payments. * **Pay as You Go:** Pay by the hour or the second for only what you use. ## Exploring the Advantages of Cloud Computing ### Six advantages of cloud computing * **Trade fixed expense for variable expense:** Instead of having to invest heavily in data centers and servers before you know how you’re going to use them, you can pay only when you consume computing resources, and pay only for how much you consume. * **Benefit from massive economies of scale:** By using cloud computing, you can achieve a lower variable cost than you can get on your own. Because usage from hundreds of thousands of customers is aggregated in the cloud, providers such as AWS can achieve higher economies of scale, which translates into lower pay-as-you-go prices. * **Stop guessing capacity:** Eliminate guessing on your infrastructure capacity needs. When you make a capacity decision prior to deploying an application, you often end up either sitting on expensive idle resources or dealing with limited capacity. With cloud computing, these problems go away. You can access as much or as little capacity as you need, and scale up and down as required with only a few minutes’ notice. * **Increase speed and agility:** In a cloud computing environment, new IT resources are only a click away, which means that you reduce the time to make those resources available to your developers from weeks to just minutes. This results in a dramatic increase in agility for the organization since the cost and time it takes to experiment and develop is significantly lower. * **Stop spending money running and maintaining data centers:** Focus on projects that differentiate your business, not the infrastructure. Cloud computing lets you focus on your own customers, rather than on the heavy lifting of racking, stacking, and powering servers. * **Go global in minutes:** Easily deploy your application in multiple regions around the world with just a few clicks. This means you can provide lower latency and a better experience for your customers at minimal cost. ### Benefits of cloud computing * **High Availability:** High-available systems are designed to operate continuously without failure for a long time. loss of service by reducing or managing failures. * **Elasticity:** With elasticity, you don't have to plan ahead of time how much capacity you need. You can provision only what you need, and then grow and shrink based on demand. * **Agility:** The cloud gives you increased agility. All the services you have access to help you innovate faster, giving your speed to market. * **Durability:** Durability is all about long-term data protection. This means your data will remain intact without corruption. ### CapEx vs. OpEx * **Capital expenditures (CapEx):** Capital expenditures are upfront purchases toward fixed assets. * **Operating Expenses (OpEx):** Operating expenses are funds used to run day-to-day operations. ## Reviewing Cloud Computing and Deployment Models ### Cloud Computing Models The are 3 common cloud computing models. **1. Infrastructure as a Service (IaaS)**: * **Building Blocks:** Fundamental building blocks that can be rented. * **Web Hosting:** Monthly subscription to have a hosting company serve your website. **2. Software as s Service (SaaS):** * **Complete Application:** Using a complete application, on-demand, that someone offers to users. * **Email Provider:** Your personal email that you access through a web browser is SaaS. **3. Platform as a Service (PaaS):** * **Used by Developers:** Develop software using web-based tools without worrying about the underlying infrastructure. * **Storefront Website:** Tools provided to build a storefront application that runs on another company's server. ### Cloud Deployment Models There are 3 common cloud deployment models **1. Private Cloud:** * Also called "on-premises" * Exists in your internal data center * Doesn't offer the advantages of cloud computing **2. Public Cloud:** * Offered by AWS * You aren't responsible for the physical hardware * Provides all the advantages of cloud computing **3. Hybrid Cloud:** * Sample architecture for a hybrid solution * Highly sensitive data stored locally * Web application run on AWS infrastructure * AWS provides tools so they talk to each other ## Leveraging the AWS Global Infrastructure ### Regions A Region is a **physical location**. * How does AWS group Regions? AWS logically groups its Regions into geographic locations. ### Availability Zones Availability Zones (AZs) consist of one or more physically separated data centers, each with redundant power, networking, and connectivity, housed in separate facilities. ### Edge Locations Edge locations cache content for fast delivery to your users. * **Latency:** Latency is the time that passes between a user request and the resulting response. ## Exploring Your Amazon Web Services (AWS) Account ### The AWS Management Console The AWS Management Console allows you to access your AWS account and manage applications running in your account from a web browser. ### AWS Command Line Interface (CLI) The AWS Command Line Interface (CLI) allows you to access your AWS account through a terminal or command window. # Compute Services {% hint style="success" %} **AWS Overview Links** * [EC2](https://aws.amazon.com/ec2/) * [Lambda](https://aws.amazon.com/lambda/) * [Fargate](https://aws.amazon.com/fargate/) * [Lightsail](https://aws.amazon.com/lightsail/) * [Outposts](https://aws.amazon.com/outposts/) * [Batch](https://aws.amazon.com/batch/) {% endhint %} ## EC2 EC2 (Elastic Compute Cloud) allows you to rent and merge virtual servers in the cloud. EC2 is a foundational service used for managing your virtual instances. 1. You can provision an EC2 instance at the click of a button. 2. You can use a preconfigured template called an Amazon Machine Image (AMI) to launch your instance. 3. You can deploy your applications directly to EC2 instances. 4. You receive 750 compute hours per month on the [Free Tier](https://aws.amazon.com/free/) plan. ### Methods to access an EC2 instance There are several ways to access your EC2 instances. * AWS Management Console: You can configure and manage your instances via a web browser. * Secure Shell (SSH): SSH lets you establish a secure connection to your instance from your local laptop. * EC2 Instance Connect (EIC): EIC allows you to use IAM policies to control SSH access to your instances, removing the need to manage SSH keys. * AWS Systems Manager: Systems Manager allows you to manage your EC2 instances via a web browser or the AWS CLI. ### EC2 Pricing Options There are several pricing options to choose from for your EC2 instances. #### 1. On-Demand A fixed price in which you are billed down to the second based on the instance type. There is no contract, and you pay only for what you use. **Use On-Demand instances when:** * You care about low cost without any upfront payment or long-term commitment. * Your applications have unpredictable workloads that can’t be interrupted. * Your applications are under development. * Your workloads will not run longer than a year. You can reserve capacity using On-Demand Capacity Reservations. The EC2 capacity is held for you whether or not you run the instance. #### 2. Spot Spot instances let you take advantage of unused EC2 capacity. Your request is fulfilled only if capacity is available. **Use Spot instances when:** * You are not concerned about the start or stop time of your application. * Your workload can be interrupted. * Your application is only feasible at very low compute prices. You can save up to **90%** off On-Demand prices. You pay the spot price that’s in effect at the beginning of each hour. #### 3. Reserved Instances (RIs) RIs allow you to commit to a specific instance type in a particular Region for 1 or 3 hours. **Use Reserved Instances when:** * Your application has steady-state usage, and you can commit to 1 or 3 years. * You can pay monthly upfront in order to receive a discount on On-Demand prices. * Your application requires a capacity reservation. You can save up to **75%** off On-Demand prices. You are required to sign a contract. You can reserve capacity in an Availability Zone for any duration. You can pay All Upfront, Partial Upfront, or No Upfront. All Upfront for the max term earns the highest discount. Provides convertible types at a **54%** discount. #### 4. Dedicated Hosts Dedicated Hosts allow you to pay for a physical server that is fully dedicated to running your instances. **Use Dedicated Hosts when:** * You want to bring your own server-bound software license from vendors like Microsoft or Oracle. * You have regulatory or corporate compliance requirements around the tenancy model. You can save up **70%** off On-Demand prices. You bring your existing per-socket, per-core, or per-VM software licenses. There is no multi-tenancy, meaning the server is not shared with other customers. A Dedicated Host is a physical server, whereas a Dedicated Instance runs on the host. #### 5. Saving Plan Saving plan allows you to commit to compute usage (measured per hour) **Use Saving Plans when:** * You want to lower your bill across multiple compute services. * You want the flexibility to change compute services, instance types, operating systems, or Regions. You can save up to 72% off On-Demand prices. You are not making a commitment to a Dedicated Host, just compute usage. Saving can be shared across various compute services like EC2, Fargate, and Lambda. This does not provide a capacity reservation. ### Features EC2 instances offer load balancing and Auto Scaling. #### Elastic Load Balancing Automatically distributes your **incoming application traffic** across multiple EC2 instances. #### EC2 Auto Scaling Adds or replaces EC2 instances automatically across AZs, **based on need and changing demand**. ## Lambda Lambda is a serverless computing service that lets you run code without managing servers. Lambda allows developers to focus on core business logic for the apps they are developing instead of worrying about managing servers. ### Features 1. Supports popular programming languages like Java, Go, Powershell, Node.js, C#, Python, and Ruby. 2. You another code using your favorite development environment or via the console. 3. Lambda can execute your code in response to events. 4. Lambda functions have a 15-minute timeout. ### Pricing Model You are charged based on the duration and number of requests. 1. **Compute time:** Pay only for compute time used. There is no charge if your code is not running. 2. **Request count:** A request is counted each time it starts execution. Test invokes in the console count as well. 3. **Always free:** The free usage tier includes 1 million free requests each month. ## Additional Compute Services ### AWS Fargate Fargate is a serverless computing engine for containers. * Fargate allows you to manage containers, like Docker. * Scanles automatically * Serverless means you don’t worry about provisioning, configuring, or scaling servers. ### Amazon Lightsail Lightsail allows you to quickly launch all the resources you need for small projects. * Deploy pre-configured applications, like Wordpress websites, at the click of a button * Simple screens for people with no cloud experience * Includes a virtual machine, SSD-bashed storage, data transfer, DNS Management, and a static IP * Provides a low, predictable monthly fee, as low as $3.50 ### AWS Outposts Outposts allow you to run cloud services in your internal data center. * Supports workloads that need to remain on-premises due to latency or data sovereignty needs * AWS delivers and installs servers in your internal datacenter * Use for a hybrid experience * Have access to the cloud services and APIs to develop apps on-premises ### AWS Batch Batch allows you to process large workloads in smaller chunks (or batches). * Runs hundreds and thousands of smaller batch processing jobs * Dynamically provisions instances based on volume # Storage Services {% hint style="success" %} **AWS Overview Links** * [S3](https://aws.amazon.com/s3/) * [EBS](https://aws.amazon.com/ebs/) * [EFS](https://aws.amazon.com/efs/) * [Storage Gateway](https://aws.amazon.com/storagegateway/) * [Backup](https://aws.amazon.com/backup/) {% endhint %} Companies today need to collect, store, and analyze the data they’ve accumulated over the years on a massive scale. Storage services in the cloud provide a place for companies to store data. ## Amazon Simple Storage Services (Amazon S3) S3 is an object storage service for the cloud that is highly available. * Objects (or files) are stored in buckets (or directories). * Essentially unlimited storage that can hold millions of objects per bucket. * Objects can upload objects via the console, the CLI, or programmatically from within code using SKDs. You can set security at the bucket level or individual object level using access control lists (ACLs), bucket policies, or access point policies. You can enable versioning to create multiple versions of your file in order to protect against accidental deletion and to use a previous version. You can use S3 access logs to track access to your buckets and objects. S3 is a regional service, but bucket names must be globally unique. ### Data Accessibility Durability and availability are 2 very different aspects of data accessibility. * **Durability:** Durability is important so your objects are never lost or compromised. (Amazon S3 Standard is designed for 99.999999999% of durability. * **Availability:** Availability is important so you cam access your data quickly when you need it. (Amazon S3 Standard is designed for 99.99% availability.) ### S3 Storage Classes Amazon S3 offers several storage classes designed for different use cases. #### 1. S3 Standard * General-purpose storage * Data stored across multiple Available Zones * Low Latency and high throughput * Recommended for “Frequently accessed data” * Durability: 99.999999999% * Availability: 99.99% #### 2. S3 Intelligent-Tiering * Automatically moves your data to the most cost-effective storage class * Automatic cost saving * No retrieval fees * Data stored across multiple Availability Zones * Recommended for “Data with unknown or changing access pattern” * Durability: 99.999999999% * Availability: 99.9% #### 3. S3 Standard-Infrequent Access (IA) * Data is accessed less frequently but requires rapid access * Data stored across multiple Availablity Zones * Cheaper than S3 Standard * Recommended for “Long-lived data, Infrequently accessed, Milisecond access when needed” * Durability: 99.999999999% * Availability: 99.9% #### 4. S3 One Zone-Infrequent Access (IA) * Like S3 Standard-IA but data stored in a single AZ * Costs 20% less than S3 Standard-IA * Data stored in this storage class can be lost * Recommended for “Re-creatable data, Infrequently accessed with millisecond access, Availability, and durability no essential” * Durability: 99.999999999% * Availability: 99.95% #### 5. S3 Glacier * Long-term data storage and archival for lower costs * Data retrieval takes longer * 3 retrieval options: 1-5 minutes, 3-5 hours, or 5-12 hours * Data stored across multiple AZ * Recommended for “Long-term backups, Cheaper storage options” * Durability: 99.999999999% #### 6. S3 Glacier Deep Archive * Like S3 Glacier but longer access time * 2 retrieval options: 12 hours or 48 hours * Cheapest of all S3 options * Data stored across multiple AZ * Recommended for “Long-term data archival accessed once or twice a year, Retaining data for regulatory compliance requirements” * Durability: 99.999999999% #### 7. S3 Outposts * Provides object storage on-premises * A single storage class * Store data across multiple devices and servers * Recommended for “Data that need to be kept local, Demanding application performance needs” ### S3 in the Real World 1. **Static Websites:** Deploy static websites to S3 and use CloudFront for global distribution. 2. **Data archive:** Archive data using Amazon Glacier as a storage option for Amazon S3. 3. **Analytics systems:** Store data in Amazon S3 for use with analytics services like Redshift and Athena. 4. **Mobile applications:** Mobile applications users can upload files to Amazon S3 buckets. ## Additional Storage Services ### Amazon Elastic Block Store (EBS) EBS is storage device (called a volume) that can be attache to (or removed from) your instance. * Data persists when the instance is not running. * Can only be attached to one instance in the same AZ * Tied to one AZ * Recommended for “Quickly accessible data, Running a database on a instance, Long-term data storage” ### EC2 Instance Store An instance store is local storage that is physically attached to the host computer and cannot be removed. * Storage on disks physically attached to an instance * Storage is temporary since data loss occurs when the EC2 instance is stopped * Faster with higher I/O speeds * Recommended for “Temporary storage needs, Data replicated across multiple instances” ### Amazon Elastic File System (EFS) EFS is a serverless network file system for sharing files. * Only supports the Linux file system * Accessible across different AZ in the same Region * More expensive than EBS * Recommended for “Main directories for business-critical apps, Lift-and-shift existing enterprise apps” ### Storage Gateway Storage Gateway is a hybrid storage service. * Connect on-premises and cloud data * Support a hybrid model * Recommended for “Moving backups to the cloud, Reducing costs for hybrid cloud storage, Low latency access to data” ### AWS Backup AWS Backup helps you manage data backups across multiple AWS services. * Integrates with resources like EC2, EBS, EFS, and more * Create a backup plan that includes frequency and retention # Content Delivery Services {% hint style="success" %} **AWS Overview Links** * [CloudFront](https://aws.amazon.com/cloudfront/) * [Global Accelerator](https://aws.amazon.com/global-accelerator/) * [S3 Transfer Accelerator](https://aws.amazon.com/s3/transfer-acceleration/) {% endhint %} #### Content Delivery Network (CDN) A CDN is a mechanism to deliver content quickly and efficiently based on geographic location. Latency simply means the time it takes to respond to a request, and a CND provides low latency. ## Amazon CloudFront CloudFront is a CDN that delivers data and applications globally with low latency * Makes content available globally or restricts it based on location * Speeds up delivery of static and dynamic web content * Uses edge locations to cache content #### CloudFront in the real world 1. **S3 static websites:** CloudFront is often used with S3 to deploy content globally. 2. **Prevent attacks:** CloudFront can stop certain web attacks, like DDoS. 3. **IP address blocking:** Geo-restriction prevents users in certain countries from accessing content. ## Amazon Global Accelerator Global Accelerator sends your users through the AWS global network when accessing your content, speeding up delivery. * Improves latency and availability of single-Region applications * Sends traffic through the AWS global network infrastructure * 60% performance boost * Automatically re-routes traffic to healthy available regional endpoints ## Amazon S3 Transfer Acceleration Amazon S3 Transfer Acceleration improves content uploads and downloads to and from S3 buckets. * Fast transfer of files over long distances * Uses CloudFront’s globally distributed edge locations * Customers around the world can upload to a central bucket # Networking Services {% hint style="success" %} **AWS Overview Links** * [VPC](https://aws.amazon.com/vpc/) * [Route 53](https://aws.amazon.com/route53/) * [Direct Connect](https://aws.amazon.com/directconnect/) * [VPN](https://aws.amazon.com/vpn/) * [API Gateway](https://aws.amazon.com/api-gateway/) {% endhint %} Networking connects computers together and allows for the sharing of data and applications, around the globe, in a secure manner using virtual routers, firewalls, and network management services. ## Amazon Virtual Private Cloud (VPC) VPC is a foundational service that allows you to create a secure private network in the AWS cloud where you launch your resources. * Private virtual network * Launch resources like EC2 instances inside the VPC * Isolate and protect resources * A VPC spans AZs in a Region VPS peering allows you to connect 2 VPCs together. Peering facilitates the transfer of data in a secure manner. ## Additional Network Services * **What is DNS?** DNS stands for Domain Name System and directs internet traffic by connecting domain names with web servers. ### Amazon Route 53 Route 53 is a DNS service that routes users to applications. * Domain name registration * Performs health checks on AWS resources * Supports hybrid cloud architectures ### AWS Direct Connect Direct Connect is a dedicated physical network connection from your on-premises data center to AWS. * Dedicated physical network connection * Connects your on-premises data center to AWS * Data travels over a private network * Supports a hybrid environment #### Direct Connect in the real world 1. **Large datasets:** Transfer large datasets to AWS 2. **Business-critical data:** Transfer internal data directly to AWS bypassing your internet service provider 3. **Hybrid model:** Build hybrid environments ### AWS VPN Site-to-Site VPN creates a secure connection between your internal networks and your AWS VPCs. * Similar to Direct Connect, but data travels over the public internet * Data is automatically encrypted * Connects your on-premises data center to AWS * Supports a hybrid environment ### API Gateway API Gateway allows you to build and manage APIs. * Share data between systems * Integrate with services like Lambda # Utilizing Databases {% hint style="success" %} **AWS Overview Links** * [RDS](https://aws.amazon.com/rds/) * [Aurora](https://aws.amazon.com/rds/aurora/) * [DynamoDB](https://aws.amazon.com/dynamodb/) * [DocumentDB](https://aws.amazon.com/documentdb/) * [ElastiCache](https://aws.amazon.com/elasticache/) * [Neptune](https://aws.amazon.com/neptune/) {% endhint %} ### Database Key Takeaways * Databases allow us to collect, store, retrieve, sort, graph, and manipulate data. * A database is an organized collection of various forms of data. * Databases are used by many applications: web, mobile, services, and more. * Databases are necessary to persist data through runs of an application. * You access the data stored in a database by querying it. * Databases are typically controlled by a database management system. In the AWS ecosystem, there are many different types of databases that support different use cases. ## Amazon Relational Database Service (RDS) RDS is a service that makes it easy to launch and manage relational databases. * Support popular database engines. like Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle, SQL Server * Offers high availability and fault tolerance using Multi-AZ deployment option * AWS manages the database with automatic software patching, automated backups, operating system maintenance, and more. * Launch read replicas across Regions in order to provide enhanced performance and durabilty. ## Amazon Aurora Aurora is a relational database compatible with MySQL and PostgreSQL that was created by AWS. * Supports MySQL and PostgreSQL database engines * 5x faster than normal MySQL and 3x faster that normal PostgreSQL * Scales automatically while providing durability and high availability * Managed by RDS ## Amazon DynamoDB DynamoDB is a fully managed NoSQL key-value and document database. * NoSQL key-value database * Fully managed and serverless * Non-relational * Scales automatically to massive workloads with fast performance ## Amazon DocumentDB DocumentDB is a fully managed document database that supports MongoDB * Document database * MongoDB compatible * Fully managed and serverless * Non-relational ## Amazon ElastiCache ElastiCache is a fully managed in-memory datastore compatible with Redis or Memcahced. * In-memory datastore * Compatible with Redis or Memcached engines * Data can be lost * Offers high performance and low latency ## Amazon Neptune Neptune is a fully managed graph database that supports highly connected datasets. * Graph database service * Supports highly connected datasets like social media networks * Fully managed and serverless * Fast and reliable # Migration and Transfer Services {% hint style="success" %} **AWS Overview Links** * [DMS](https://aws.amazon.com/dms/) * [SMS](https://aws.amazon.com/server-migration-service/) * [Snow Family](https://aws.amazon.com/snow/) * [Snowcone](https://aws.amazon.com/snowcone/) * [Snowball](https://aws.amazon.com/snowball/) * [Snowmobile](https://aws.amazon.com/snowmobile/) * [DataSync](https://aws.amazon.com/datasync/) {% endhint %} ## Database Migration Service (DMS) DMS helps you migrate databases to or within AWS. * Migrate on-premises databases to AWS * Continuous data replication * Supports homogeneous and heterogeneous migrations * Virtually no downtime DMS in the real world 1. **Oracle to Aurora MySQL:** Migrate an on-premises Oracle database to Aurora MySQL 2. **Oracle to Oracle:** Migrate an on-premises Oracle database to Oracle on EC2 3. **RDS Oracle to Aurora MySQL:** Migrate an RDS Oracle database to Aurora MySQL ## Server Migration Service (SMS) SMS allows you to migrate on-premises servers to AWS. * Migrates on-premises servers to AWS * Servers saved as a new Amazon Machine Image (AMI) * Use AMI to launch servers as EC2 instances ## Snow Family The Snow Family allows you to transfer large amounts of on-premises data to AWS using a physical device. ### Snowcone * The smallest member of data transport devices * 8 terabytes of usable storage * Offline shipping * Online with DataSync ### Snowball and Snowball Edge * Petabyte-scale data transport solution * Transfer data in and out * Cheaper than Internet transfer * Snowball Edge supports EC2 and Lambda ### Snowmobile * Multi-petabyte or exabyte scale * Data loaded to S3 * Securely transport ## DataSync DataSync allows for online data transfer from on-premises to AWS storage services like S3 or EFS. * Migrates data from on-premises to AWS * Copy data over Direct Connect or the internet * Copy data between AWS storage services * Replicate data cross-Region or cross-account # Analytics Services {% hint style="success" %} **AWS Overview Links** * [Redshift](https://aws.amazon.com/redshift/) * [Athena](https://aws.amazon.com/athena/) * [Glue](https://aws.amazon.com/glue/) * [Kinesis](https://aws.amazon.com/kinesis/) * [EMR](https://aws.amazon.com/emr/) * [Data Pipeline](https://aws.amazon.com/datapipeline/) * [QuickSight](https://aws.amazon.com/quicksight/) {% endhint %} ## Amazon Redshift Redshift is a scalable data warehouse solution. * Data warehousing solution * Improves speed and efficiency * Handles exabyte-scale data ### Redshift in the real world 1. **Data consolidation:** When you need to consolidate multiple data resources for reporting 2. **Relational databases:** When you want to run a database that doesn’t require real-time transaction processing (insert, update, and delete) Analytics is the act of querying or processing your data. There are several services that allow you to gain deeper insights, enhance decision-making, and act in real-time to what your data is telling you. ## Athena Athena is a query service for Amazon S3. * Query service * Analyze S3 data using SQL * Pay per query * Considered serverless ## Glue Gle prepares your data for analytics. * Extract, transform, load (ETL) service * Prepare and load data * Helps to better understand your data ## Kinesis Kinesis allows you to analyze data and video streams in real-time. * Analyze real-time, streaming data * Supports video, audio, application logs, website clickstreams, and IoT ## Elastic MapReduce (EMR) EMR helps you process large amounts of data. * Process big data * Analyze data using Hadoop * Works with big data frameworks ## Data Pipeline Data Pipeline helps you move data between compute and storage services running either on AWS or on-premises * Moves data at specific intervals * Moves data based on conditions * Sends notifications on success or failure ## QuickSight QuickSight helps you visualize your data. * Build interactive dashboards * Embed dashboards in your applications ### Analytics in the real world 1. **Search data in S3:** Athena helps you query historical data stored in S3 as if they were relational data using standard SQL. 2. **Log analytics:** Kinesis helps you analyze logs in near real-time for application monitoring or fraud detection. # Machine Learning Services {% hint style="success" %} **AWS Overview Links** * [Rekognition](https://aws.amazon.com/rekognition/) * [Comprehend](https://aws.amazon.com/comprehend/) * [Polly](https://aws.amazon.com/polly/) * [SageMaker](https://aws.amazon.com/sagemaker/) * [Translate](https://aws.amazon.com/translate/) * [Lex](https://aws.amazon.com/lex/) {% endhint %} Artificial intelligence (AI) teaches computers to do thing that normaly require human intelligence. Businesses leverage AI and machine learning to add intelligence to their applications and leverage trends and patterns in data. ## Rekognition Rekognition allows you to automate your image and video analysis. * Image and video analysis * Identify custom labels in images and videos * Face and text detection in images and videos ## Comprehend Comprehend is a natural-language processing (NLP) service that finds relationships in text. * Natural-language processing (NLP) service * Uncovers insights and relationships * Analyzes text ## Polly Polly turns text into speech. * Mimics natural-sounding human speech * Several voices across many languages * Can create a custom voice ## SageMaker SageMaker helps you build, train, and deploy machine learning models quickly * Prepare data for models * Train and deploy models * Provides Deep Learning AMIs ## Translate Translate provides language translation. * Provide real-time and batch language translation * Support many languages * Translates many content formats ## Lex Lex helps you build conversational interfaces like chatbots. * Recognizes speech and understands language * Build highly engaging chatbots * Powers Amazon Alexa # Developer Tools {% hint style="success" %} **AWS Overview Links** * [Cloud9](https://aws.amazon.com/cloud9/) * [CodeCommit](https://aws.amazon.com/codecommit/) * [CodeBuild](https://aws.amazon.com/codebuild/) * [CodeDeploy](https://aws.amazon.com/codedeploy/) * [CodePipeline](https://aws.amazon.com/codepipeline/) * [X-Ray](https://aws.amazon.com/xray/) * [CodeStar](https://aws.amazon.com/codestar/) {% endhint %} Software developers use tools to accelerate the software development and release cycle. ## environmentCloud9 Cloud9 allows you to write code within an integrated development environment (IDE) from within your web browser. * Integrated development denironment (IDE) * Write and debug code * Supports popular programming languages ## CodeCommit CodeCommit is a source control system for private Git repositories. * Create repositories to store code * Commit, branch, and merge code * Collaborate with other software developers ## CodeBuild CodeBuild allows you to build and test your application source code * Compiles source code and runs tests * Enables continuous integration and delivery * Products build artifacts ready to be deployed ## CodeDeploy CodeDeploy manages the deployment of code to compute services in the cloud or on-premises. * Deploys code to EC2, Fargate, Lambda, and on-premises. * Maintains application uptime ## CodePipeline CodePipeline automates the software release process. * Quickly deliver new features and updates * Integrates with CodeBuild to run builds and unit tests * Integrates with CodeCommit to retrieve source code * Integrates with CodeDeploy to deploy your changes ## X-Ray X-Ray helps you debug production applications. * Analyze and debug production applications * Map application components * View requests end to end ## CodeStar CodeStar helps developers collaboratively work on development projects. * Developers connect their development environment * Integrates with CodeCommit, CodeBuild, and CodeDeploy * Contains issue tracking dashboard # Deployment and Infrastructure Management Service {% hint style="success" %} **AWS Overview Links** * [CloudFormation](https://aws.amazon.com/cloudformation/) * [Elastic Beanstalk](https://aws.amazon.com/elasticbeanstalk/) * [OpsWorks](https://aws.amazon.com/opsworks/) {% endhint %} These services help you quickly stand up new applications, automate the management of infrastructure, and provide real-time visibility into system health. ### IAC (Infrastructure as Code) IaC allows you to write a script to provision AWS resources. The benefit is that you provision resources in a reproducible manner that saves time. ## CloudFormation CloudFormation allows you to provision AWS resources using Infrastructure as Code (IaC). * Provides a repeatable process for provisioning resources. * Works with most AWS services * Create templates for the resources you want to provision ### CloudFormation in the real world Automate the infrastructure-provisioning process for EC2 servers. You can use CloudFormation to automate the creation of EC2 instances in your AWS account. ## Elastic Beanstalk Elastic Beanstalk allows you to deploy your web applications and web services to AWS. * Orchestration service that provisions resources * Automatically handles the deployment * Monitors application health via a health dashboard ### Elastic Beanstalk in the real world Quickly deploy a scalable Java-based web application to AWS. After you upload your Java code, Elastic Beanstalk deploys it and handles capacity provisioning, load balancing, and Auto Scaling. Elastic Beanstalk even monitors the health of your application. ## OpsWorks OpsWorks allows you to use Chef or Puppet to automate the configuration of your servers and deploy code. * Deploy code and manage applications * Manage on-premises servers or EC2 instances in AWS Cloud * Works with Chef and Puppet automation platforms ### OpsWork in the real world Automate software configuration and infrastructure management for your application. OpsWorks allows you to define software installation scripts and automate configuration for your application servers. # Managing and Integration Services {% hint style="success" %} **AWS Overview Links** * [SQS](https://aws.amazon.com/sqs/) * [SNS](https://aws.amazon.com/sns/) * [SES](https://aws.amazon.com/ses/) {% endhint %} Coupling defines the interdependencies or connections between components of a system. Loose coupling helps reduce the risk of cascading failures between components. * **Tight coupling:** Tightly coupled components are highly dependent on each other. (**Monolithic Application**) * **Loose coupling:** Loosly coupled components are connected but not dependent on each other. (**Microservices**) Queue are used to implement loosely coupled systems. ## Simple Queue Service (SQS) SQS is a message quering service that allows you to build loosely coupled systems. * Allows component-to-component communication using messages * Multiple components (or producers) can add messages to the queue * Messages are processed in an asynchronous manner There are often times that users of your applications need to notified when certain events happen. ## Simple Notification Service (SNS) SNS allows you to send emails and text messages from your applications. * Send emails and messages * Publish messages to a topic * Subscribers receive messages ### SNS in the real world Send an email when the CPU utilization of an EC2 instance goes above 80%. SNS works with CloudWatch when an alarm’s metric threshold is breached to send an email. ## Simple Email Service (SES) SES is an email service that allows you to send richly formatted HTML emails from your applications. * Ideal choice for marketing campaigns or professional emails * Unlike SNS, SES sends HTML emails ### SES in the real world Send a marketing email and track open or click-through rates. SES allows you to send richly formatted HTML emails in bulk and gain valuable insights about the effectiveness of your campaign. # Auditing, Monitoring, and Logging Services {% hint style="success" %} **AWS Overview Links** * [CloudWatch](https://aws.amazon.com/cloudwatch/) * [CloudTrail](https://aws.amazon.com/cloudtrail/) {% endhint %} These services give you insight into your systems' performance and help you proactively find and resolve errors. ## CloudWatch CloudWatch is a collection of services that help you monitor and observe your cloud resources. * Collects metrics, logs, and events * Detect anomalies in your environment * Set alarms * Visualize logs ### CloudWatch Services * **CloudWatch Alarms:** Set high-resolution alarms * **CloudWatch Logs:** Monitor application logs * **CloudWatch Metrics:** Visualize time-series data * **CloudWatch Events:** Trigger an event based on a condition ### CloudWatch in the real world Provide real-time monitoring on EC2 instances. CloudWatch Alarms can notify you if an EC2 instance goes into the stopped state or usage goes above a specific utilization. Receive a notification when root user activity is detected in your account. Create a CloudWatch event rule to notify you when root user API calls are detected in your account indicating root user activity. ## CloudTrail CloudTrail tracks user activity and API calls within your account. * Log and retain account activity * Track activity through the console, SDKs, and CLI * Identify which user made changes * Detect unusual activity in your account ### CloudTrail in the real world Track the time a particular event occurred in your account. You can troubleshoot events over the past 90 days using the CloudTrail event history log to find the specific time an event occurred per region. You can create a custom trail to extend past 90 days. Things you can track with CloudTrail: * Username * Event time and name * IP address * Access key * Region * Error code # Additional Services {% hint style="success" %} **AWS Overview Links** * [WorkSpaces](https://aws.amazon.com/workspaces/) * [Connect](https://aws.amazon.com/connect/) {% endhint %} ## Amazon WorkSpaces Amazon WorkSpaces allows you to host virtual desktops in the cloud. * Virtualise Windows or Linux desktops * Enables employees to work from home ## Amazon Connect Amazon Connect is a cloud contact center service. * Provides customer service functionality * Improves productivity of help desk agents # Security and Compliance ## Shared Responsibility Model In the public cloud, there is shared security responsibility between you and AWS. * AWS responsibility: Security of the Cloud * Your responsibility: Security in the Cloud ### Security of the Cloud AWS is responsible for protecting and securing its infrastructure. * AWS Global Infrastructure: AWS is responsible for its global infrastructure elements: Regions, edge locations, and Availability Zones. * Building Security: AWS controls access to its data centers where your data resides. * Networking Components: AWS maintains networking components: generators, uninterruptible power supply (UPS) systems, computer rooms, air conditioning (CRAC) units, fire suppression systems, and more. * Software: AWS is responsible for any managed service like RDS, S3, ECS, or Lambda, patching of host operating systems, and data access endpoints. ### Security in the Cloud You are responsible for implementing the services and managing your application data. * **Application Data:** You are responsible for managing your application data, which includes encryption options. * **Security Configuration:** You are responsible for securing your account and API calls, rotating credentials, restricting internet access from your VPCs, and more. * **Patching:** You are responsible for the guest operating system (OS), which includes updates and security patches. * **Identity and Access Management:** You are responsible for application security and identity and access management. * **Network Traffic:** You are responsible for network traffic protection, which includes security group firewall configuration. * **Installed Software:** You are responsible for your application code, installed software, and more. It would help if you frequently scan for and patch code vulnerabilities. #### EC2 Shared Responsibility Model | Your responsibility | AWS responsibility | | --------------------------------------- | -------------------------------------- | | Installed applications | EC2 service | | Patching the **guest operating** system | Patching the **host operating** system | | Security controls | Security of the physical server | #### Lambda Shared Responsibility Model | Your responsibility | AWS responsibility | | ------------------------- | -------------------------- | | Security of the code | Lambda service | | Storage of sensitive data | Upgrading Lambda languages | | IAM for permissions | Lambda endpoints | | | Operaing system | | | Underlying infrastructure | | | Software dependencies | #### Which security responsibilities are shared?
Your responsibilityAWS responsibility
Patch ManagementPatching guest OS and applicationsPatching infrastructure
Configuration ManagementConfiguring databases and applicationsConfigure infrastructure devices
Awareness and TrainingYour employeesAWS employees
## Well-Architected Framework The AWS Well-Architected Framework is a collection of best practices and guidelines for designing and operating reliable, secure, efficient, and cost-effective systems in the cloud. ### **Six pillars of AWS Well-Architected Framework** #### **1.** Operational Excellence This pillar focuses on creating applications that effectively support production workloads. * Plans for and anticipate failure * Deploy smaller, reversible changes * Script operations as code * Learn from failure and refine #### 2. Security This pillar focuses on putting mechanisms in place that help protect your systems and data. * Automate security tasks * Encrypt data in transit and at rest * Assign only the least privileges required * Track who did what and when * Ensure security at all application layers #### 3. Reliability This pillar focuses on designing systems that work consistently and recover quickly. * Recover from failure automatically * Scale horizontally for resilience * Stop guessing capacity * Manage change through automation * Test recovery procedures #### 4. Performance Efficiency This pillar focuses on the effective use of computing resources to meet system and business requirements while removing bottlenecks. * Use serverless architectures first * Use multi-region deployments * Delegate tasks to a cloud vendor * Experiment with virtual resources #### 5. Cost Optimization This pillar focuses on delivering optimum and resilient solutions at the least cost to the user. * Utilize consumption-based pricing * Implement Cloud Financial Management * Measure overall efficiency * Pay only for resources your application requires #### 6. Sustainability This pillar focuses on environmental impacts, especially energy consumption and efficiency. * Understand your impact * Establish sustainability goals * Maximize utilization * Use managed services * Reduce downstream impact ### 6 Pillars in the Real World 1. Operational Excellence (**CodeCommit**): You can use AWS CodeCommit for version control to enable tracking of code changes and to version-control CloadFormation templates of your infrastructure. 2. Security (**CloudTrail**): You can configure central logging of all actions performed in your account using CloudTrail. 3. Reliability (**RDS**): You can use Multi-AZ deployments for enhanced availability and reliability of RDS databases. 4. Performance Efficiency (**Lambda**): You can use AWS Lambda to run code with zero administration. 5. Cost Optimization (**S3**): You can use S3 Intelligent-Tiering to automatically move your data between access tiers based on usage patterns. 6. Sustainability (**Auto Scaling**): You can use EC2 Auto Scaling to ensure you are maximizing utilization. ## IAM Users Identify and Access Management (IAM) IAM allows you to control access to your AWS services and resources. * Helps you **secure** your cloud resources * You define **who** has access * You define **what** they can do * A free **global** service ### Identities vs. Access #### **Identities** **Who** can access your resources * Root user * Individual users * Groups * Roles #### Access **What** resources they can access * Policies * AWS managed policies * Customer managed policies * Permissions boundaries ### Authentication (Who) vs. Authorization (What) * **Authentication:** Authentication is where you present your identify (**username**) and provide verification (**password**). * **Authorization:** Authorization determines which services and resources the authenticated identity has **access** to. ## Users Users are entities you create in IAM to represent the person or application needing to access your AWS resources. ### ROOT User The root user is created when you first open your AWS account. What can only the root user do? * Close your account * Change email address * Modify your support plan ### Individual Users Individual users are created in IAM and are used for everyday tasks. What can individual users do? * Perform administrative tasks * Access application code * Launch EC2 instances * Configure databases ### Applications You’ll create a user in IAM so you can generate access keys for an application running on-premises that needs access to your cloud resources. #### Users in the Real World * Create access keys for an IAM user that needs access to the AWS CLI: The AWS Command Line Interface (CLI) allows you to access resources in your AWS account through a terminal or command window. Access keys are needed when using the CLI and can be generated using IAM. ## Groups A group is a collection of IAM users that helps you apply common access control to all group members. * Used to group users that perform similar tasks * Access permissions apply to all members of the group. * Access is assigned using policies and roles. Do not confuse security groups for EC2 with IAM groups. EC2 security groups act as firewalls, while IAM groups are collections of users. * **Administrators:** Administrators perform administrative tasks such as creating new users. * **Developers:** Developers use computing and database services to build applications. * **Analysts:** Analysts run to budget and usage reports. #### Groups in the Real World * Apply the same access control to a large set of users: Groups save you time by allowing you to apply the same access permissions to more than one user at once. When a user no longer needs access, they can be removed from the group. ## IAM Permissions ### Roles Roles define access permissions and are temporarily assumed by an IAM user or service. * You assume a role to perform a task in a single session. * Assumed by any user or service that needs it. * Access is assigned using policies. * You grant users in one AWS account access to resources in another AWS account. **Examples:** * User with “DevOps-Engineer Role”: Create code branch in CodeCommit, List pipelines in CodePipeline * Lambda with “Lambda-Execution Role”: List contents of S3 bucket, Query DynamoDB #### Roles in the Real World * Attach a role to an EC2 instance for access to S3: You can attach a role to an instance that provides privileges (e.g. uploading files to S3) to applications running on the instance. Roles help you avoid sharing long-term credentials like access keys and protect your instances from unauthorized access. ### Policies You manage permissions for IAM users, groups, and roles by creating a policy document in JSON format and attaching it. **Sample JSON Policy** ```bash { "Version": …, "Statement": [ { "Action": rds.*, "Effect": "Allow", "Resource": […] } ] } ``` #### Policies in the Real World * You can limit access to an Amazon S3 bucket to specific users: You can add a bucket access policy to an Amazon S3 bucket to grant IAM users access permissions for the bucket and the objects in it. ### IAM Best Practices There are several recommended best practices for IAM. 1. **Enable MFA for privileged users:** You should enable multi-factor authentication (MFA) for the root user and other administrative users. 2. **Implement strong password policies:** You should require IAM users to change their password after a specified period of time, prevent users from reusing previous passwords, and rotate security credentials regularly. 3. **Create individual users instead of using root:** You shouldn’t use the root user for daily tasks. 4. **Use roles for Amazon EC2 instances:** You should use roles for applications that run on EC2 instances instead of long-term credentials like access keys. ### IAM Credential Report The IAM credential report lists all users in your account and the status of their various credentials. * List all users and the status of passwords, access keys, and MFA devices * Used for auditing and compliance ## Security ### Web Application Firewall (WAF) WAF helps protect your web applications against common web attacks. * Protects apps against common attack patterns * Protects against SQL injection * Protects against cross-site scripting #### WAF in the Real World * Protect your web application from cross-site scripting attacks: You can deploy a web application directly to an EC2 instance and protect it from cross-site scripting attacks using WAF. You can even deploy WAF on CloudFront as part of your CDN solution to block malicious traffic. ### Distributed Denial of Service (DDoS) A DDoS attack causes a traffic jam on a website or web application in an attempt to cause it to crash. ### Shield Shield is a managed Distributed Denial of Service (DDoS) protection service. * Always-on detection * Shield Standard is free * Shield Advanced is a paid service 1. **Shield Standard:** Provides free protection against common and frequently occurring attacks 2. **Shield Advanced:** Provides enhanced protections and 24/7 access to AWS experts for a fee DDoS protection via **Shield Advanced** is supported on several services. 1. CloudFront 2. Route 53 3. Elastic Load Balancing 4. AWS Global Accelerator #### Shield in the Real World * Receive real-time notifications of suspected DDoS incidents and assistance from AWS during the attack: Shield Advanced will give you notifications of DDoS attacks via CloudWatch metrics. Additionally, with Shield Advanced, you have 24/7 access to AWS experts to assist during an attack. ### Macie Macie helps you discover and protect sensitive data. * Use machine learning * Evaluates S3 environment * Uncovers personally identifiable information (PII) #### Macie in the Real World * Discover passport numbers stored on S3: Macie can be used to find sensitive data like passport numbers, social security numbers, and credit card numbers on S3. ### Config Config allows you to assess, audit, and evaluate the configurations of your resources. * Track configuration changes over time * Delivers configuration history file to S3 * Notifications via Simple Notification Service (SNS) of every configuration change #### Config in the Real World * Identify system-level configuration changes made to your EC2 instances: Config allows you to record configuration changes within your EC2 instances. You can view network, software, and operating system (OS) configuration changes, system-level updates, and more. ### GuardDuty GuardDuty is an intelligent threat detection system that uncovers unauthorized behavior. * Use machine learning * Built-in detection for EC2, S3, and IAM * Reviews CloudTrail, VPC, Flow Logs, and DNS logs #### GuardDuty in the Real World * Detect unusual API calls in your account: GuardDuty’s anomaly detection feature evaluates all API requests in your account and identifies events that are associated with common techniques used by attackers. ### Inspector Inspector works with EC2 instances to uncover and report vulnerabilities, * Agent installed on EC2 instance * Reports vulnerabilities found * Checks access from the internet, remote root login, vulnerable software version, etc. #### Inspector in the Real World * Identify unintended network access to an EC2 instance via a detailed report of security findings: Inspector has several built-in rules to access your EC2 instances to find vulnerabilities and report them prioritized by level of security. ### Artifact Artifact offers on-demand access to AWS security and compliance reports. * Central repository for compliance reports from third-party auditors * Service Organization Controls (SOC) reports * Payment Card Industry (PCI) reports #### Artifact in the Real World * You need to access AWS certification for ISO compliance: Artifact provides a central repository for AWS security and compliance repots via a self-service portal. ### Cognito Cognito helps you control access to mobile and web applications. * Provides authentication and authorization * Helps you manage users * Assists with user sign-up and sign-in #### Cognito in the Real World * You need to add a social media sign-in to your web application: Cognito provides functionality that allows your users to sign in to your application through social media accounts like Facebook and Google. ## Encryption and Secret Management Understand the difference between data in flight vs. data at rest * **Data in Flight:** Data that is moving from one location to another * **Data at Rest:** Data that is inactive or stored for later use ### Key Management Service (KMS) KMS allows you to generate and store encryption keys. * Key generator * Store and control keys * AWS manages encryption keys * Automatically enabled for certain services #### KMS in the Real World * Create encrypted Amazon EBS volumes: When you create an encrypted Amazon EBS volume, you’re able to specify a KMS customer master key. ### CloudHSM CloudHSM is a hardware security module (HSM) used to generate encryption keys. * Dedicated hardware for security * Generate and manage your own encryption keys * AWS does not have access to your keys #### CloudHSM in the Real World * Meet compliance requirements for data security by using dedicated hardware: CloudHSM allows you to meet corporate, contractual, and regulatory compliance requirements for data security by using dedicated hardware in the cloud. ### Secret Manager Secret Manager allows you to manage and retrieve secrets (passwords or keys). * Rotate, manage, and retrieve secrets * Encrypt secrets at rest * Integrates with services like RDS, Redshift, and DocumentDB #### Secret Manager in the Real World * Retrieve database credentials needed for your application code: Secret Manager allows you to retrieve database credentials with a call to Secrets Manager APIs, removing the need to hardcode sensitive information in plain text within your application code. # Pricing, Billing, and Governance ## AWS Pricing ### Pricing 1. **Compute:** Hourly from launch to termination 2. **Storage:** The data you store in the cloud 3. **Outbound data transfer:** Data in flight moving between systems ### Free Offer Types 1. **12 months free:** 12 months free usage following your initial sign-up date to AWS 2. **Always free**: Offers do not expire and are available to all AWS customers 3. **Trials:** Short-term free trials staring from the date you activate a particular service ### EC2 Pricing 1. **On-Demand:** You pay by the hour or by the second without pre-paying 2. **Saving Plan:** Commit to compute usage measured per hour for 1- or 3-year term 3. **Reserved Instances:** Commit to use for 1 or 3 years; pay regardless of usage 4. **Spot Instances:** Instances only launch if spare capacity is available 5. **Dedicated Hosts:** An entire physical server just for you ### Lambda Pricing 1. **Number of requests:** Include test invokes from the console 2. **Code execution time:** From execution start, in response to events, to stop 3. **Always free:** 1 million requests per month ### S3 Pricing 1. **Storage class:** Various storage classes 2. **Storage:** Number and size of objects 3. **Data transfer:** Data transferred out of S3 Region 4. **Request and data retrieval:** Request made for data and amount of requests ### RDS Pricing 1. Running clock hours 2. Type of database 3. Storage 4. Purchase type 5. Database count 6. API requests 7. Deployment type 8. Data transfer ### Total Cost of Ownership (TCO) TCO is a financial estimate that helps you underestand both the direct and indirect costs of AWS. ### Application Discovery Service Application Discovery Service helps you plan migration projects to the AWS Cloud. * Plan migration projects * Used to estimate TCO * Works with other services to migrate servers ### A Few Ways to Reduce Your TCO Using AWS 1. **Minimize capital expenditures:** AWS helps you minimize large capital expenditures, which reduces your TCO. 2. **Utilize Reserved Instances:** AWS provides Reserved Instances to help you lock in saving and reduce your TCO. 3. **Right size your resources:** AWS helps you match the provisioning of resources to your usage needs to reduce your TCO. ### AWS Price List API The Price List API allows you to query the price of AWS services. * Query using JSON or HTML * Receive price alerts when prices change ## Billing Services ### Budgets Budgets allow you to set a custom budget that alerts you when your costs or usage exceed your budgeted amount. * Improve planning and cost control * Cost, usage, and reservation budgets * Budget alerts ### Budget Types 1. Cost Budgets: Plan how much you want to spend on a service. 2. Usage Budgets: Plan how much you want to use on one or more services. 3. Reservation Budgets: Set RIs or Saving Plans utilization or coverage targets. #### Budgets in the Real World * Monitor Free Tier usage so you don’t incur unwanted costs: You can monitor Free Tier usage to ensure you don’t accidentally exceed Free Tier limits and incur unwanted costs. You can set up an alert notification for when your account is approaching a particular dollar amount. ### Cost and Usage Report The Cost and Usage Report contains the most comprehensive set of cost and usage data. * A downloadable detailed and comprehensive report * List usage for each service category * Aggregate usage data on a daily, hourly, or monthly level #### Cost and Usage Report in the Real World * View the most granular data about your AWS bill: The Cost and Usage Report gives you the ability to do a deep dive into your AWS cost and usage data. Once set up, you can download the report using the Amazon S3 console. ### Cost Explorer Cost Explorer allows you to visualize and forecast your costs and usage over time. * Visualize costs over time * View the past 12 months * Forecast for up to 12 months #### Cost Explorer in the Real World * Analyze your EC2 usage over the past 7, 30, or 60 days: If you considering your options for Saving Plans, AWS Cost Explorer can analyze your EC2 usage over the past 7, 30, or 60 days. ### Cost Allocation Tags Tags are useful for tracking spending. 1. Tags allow you to label resources using a key and value pair. 2. Tags allow you to track costs via the cost allocation report. ## Governance Services ### Organizations Organizations allow you to centrally manage multiple AWS accounts under one umbrella. * Group multiple accounts * Single payment for all accounts * Automate account creation * Allocate resources and apply policies across accounts #### What are the benefits of Organizations? 1. Consolidated Billing: The advantage of consolidated billing is that you receive one bill for multiple accounts. 2. Cost Saving: You’ll receive volume discounts since usage is combined across accounts. 3. Account Governance: You have a quick and automated way to create accounts or invite existing accounts. #### Organizations in the Real World * Reduce costs by sharing resources across accounts: Organizations allow you to save money using Reserved Instance (RI) sharing. RI sharing allows all accounts in the organization to receive the hourly cost-benefit of RIs purchased by any other account. You can always turn off RI sharing using the master payer (or root) organization. ### Control Tower Control Tower helps you ensure your accounts conform to company-wise policies. * Helps set up new accounts using a multi-account strategy * Works directly with AWS Organizations * Enforce the best use of services across accounts * Provides a dashboard to manage accounts #### Control Tower in the Real World * Disallow public write access to all S3 buckets across your accounts: Control Tower allows you to govern your multi-account environment by enabling cross-account security audits or preventing or detecting security issues through mandatory or optional guardrails. ### Systems Manager Systems Manager gives you visibility and control over your AWS resources. * Automate operational tasks on your resources * Group resources and take action * Patch and run commands on multiple EC2 instances or manage RDS instances #### Systems Manager in the Real World * Deploy operating system and software patches automatically across a large group of instances: Systems Manager allows you to auto-patch software running on EC2 instances according to a schedule. ### Trusted Advisor Trusted Advisor provides real-time guidance to help you provision your resources following AWS best practices. * Checks your account and makes recommendations * Helps you see service limits * Helps you understand best practices #### A few popular recommendations made by Trusted Advisor 1. Checks for unrestricted access for specific ports on EC2 instances (free) 2. Checks S3 buckets permissions to determine if public access (free) 3. Checks for MFA on root account (free) 4. Checks IAM password policy (non-free, Enterprise or Business Support required) 5. Checks for RDS public snapshots (free) 6. Checks for service usage greater than 80% over service limit (non-free, Enterprise or Business Support required) 7. Checks for exposed access keys (non-free, Enterprise or Business Support required) 8. Checks for CloudFront content delivery optimization (non-free, Enterprise or Business Support required) #### Trusted Advisor in the Real World * Check read and write capacity service limits for DynamoDB: Trusted Advisor helps reduce your overall costs by monitoring service limits. ### License Manager License Manager helps you manage software licenses. * Manage on-premises and AWS licenses * Track licenses for Oracle, Microsoft, SAP, and more ### Certificate Manager Certificate Manager helps you provision and manage SSL/TLS certificates. * Provides public and private certificates for free * Integrates with Elastic Load Balancing, API Gateways, and more ## Management Services ### Managed Services Managed Services helps you efficiently operate your AWS infrastructure. * Augments your internal staff * Provides ongoing management of your infrastructure * Reduces operational risks and overhead #### Managed Services in the Real World * Develop application-specific health monitoring using CloudWatch: Managed Services can increase your operational efficiency by helping you develop application-specific health monitoring using CloudWatch. Professional Services Professional Services helps enterprise customers move to a cloud-based operating model. * Proposes solutions * Architects solutions * Implements solutions ### Professional Services in the Real World * Get helps with evaluating an application for migration to the cloud: You can quickly move on-premises applications to the cloud using AWS Professional Services. #### AWS Partner Network APN is a global community of approved partners that offer software solutions and consulting services for AWS. * Offers technology partners that provide software solutions * Provides consulting partners that offer professional services * Find approved vendors with deep AWS expertise #### APN in the Real World * You need help designing and building a new application: If your team lacks the technical expertise to build and deploy cloud applications, the APN cloud helps you get up and running quickly. ### Marketplace Marketplace is a digital catalog or prebuilt solutions you can purchase or license. You may also sell your own solutions to others via Marketplace. * Buy third-party software * Sell solutions to AWS customers * Search the catalog of software listings and install them with the click of a button #### Marketplace in the Real World * Try out an application before making a long-term commitment: Some products listed on Marketplace offer free trials. The free trial allows you to try the software before you buy it. ### Personal Health Dashboard Personal Health Dashboard alerts you to events that might impact your AWS environment. * Provides troubleshooting guidance * Feedback tailored to your specific environment ### Support Plans There are 4 support plans. 1. **Basic:** Basic support is included for free for all AWS accounts. 2. **Developer:** Developer support starts at $29 a month and is recommended for testing and development. 3. **Business:** Business support starts at $100 a month and is recommended for production workloads. 4. **Enterprise:** Enterprise support starts at $15,000 a month and is recommended for business or mission-critical production workloads. ### Support Case Types There are 3 types of support cases you can open with AWS Support. 1. **Account and billing:** Account-related and billing cases can be opened by customers. 2. **Service limit increases:** Default service quota (or limit) increases can be opened by all customers. 3. **Technical support:** Technical support cases can only be opened by customers on the Developer, Business, or Enterprise plans. AWS Support does not allow cases for code development, debugging custom software, or performing system administration tasks. #### Basic Support Plan Basic Support is included for free for all AWS accounts. * Access to account and billing support cases * Access to service limit increases * Customer Services: 24/7 access via email only * Access to discussion forums #### Developer Support Plan Developer Support starts at $29 a month and is recommended for testing and development. * Access to account and billing support cases * Access to service limit increases * Access to technical support * Allowed to 1 primary contact * Access to open unlimited support cases * Cloud Support Associate: Business-hours access via email only * Response Times: <24 hours general guidance and <12 hours system impaired #### Business Support Plan Business Support starts at $100 a month and is recommended for production workloads. * Access to account and billing support cases * Access to service limit increases * Access to technical support * Allowed unlimited contact * Access to open unlimited support cases * Access to a Full Set of Trusted Advisor Checks * Cloud Support Engineers: 24/7 access via email, phone, or chat * Response Times: <24 hours general guidance, <4 hours production system impaired, <12 system impaired, < 1-hour production system down #### Enterprise Support Plan Enterprise Support starts at $15,000 a month and is recommended for business or mission-critical production workloads. * Access to account and billing support cases * Access to service limit increases * Access to technical support * Allowed unlimited contact * Access to open unlimited support cases * Access to Technical Account Manager (TAM) * Access to Concierge Support Team * Access to Infrastructure Event Manager * Access to a Full Set of Trusted Advisor Checks * Cloud Support Engineers: 24/7 access via email, phone, or chat * Response Times: <24 hours general guidance, <4 hours production system impaired, <12 system impaired, < 1-hour production system down, <15 minutes business-critical system down