Who Has Rebooted The Linux System?

There are many times that we get puzzled with this question "Who has rebooted the system?". Was this triggered by a user or done by some kernel threads? What was the reason why the server was rebooted? Of course, a running system should not be rebooted without a reason, as this incurs downtime and service disruptions. There are many ways to avoid service disruption by setting up High Availability (HA) so that one server down/reboot would not cause any downtime as the application/service would continue to work from another server in this setup. However, in Load Balancer (LB) mode where there are multiple nodes serving the applications and node down would certainly be balanced by other nodes in the LB setup. Let's come back to our main agenda which is to track the system reboot activity. Yes, in this blog post we would talk about this topic and different ways to detect/identify this from a Linux system. 

Step 1: Tracing through the last command

--------------------------------------------------------

As we all know, the "last" command would get the required results. This command would basically read the data from the file "/var/log/wtmp" to list out all users logged in and logged out. 

From the man page of "last":

The pseudo user reboot logs in each time the system is rebooted. Thus the last reboot will show a log of all the reboots since the log file was created.

 lastb is the same as last, except that by default it shows a log of the /var/log/btmp file, which contains all the bad login attempts.

Since the system shutdown is also a user "shutdown", one could track the system reboot/shutdown events using the "last" command as shown below:

# last -x shutdown

# last -x reboot OR # last reboot

Example: Tracing the most latest system reboot and shutdown event:

These events tell about the date & time when the system shutdown/reboot happened, what kernel version was running that time and how long the system was down in minutes. But there is no way to identify who has rebooted the system, was that initiated by the system or triggered by a user. 

Note: This "shutdown" user is defined by the system as shown below:

[root@rhel6upgraded ~]# grep shutdown /etc/passwd

shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown

Step 2: Tracing through default /var/log/messages

-----------------------------------------------------------------

Lets try to search through the popular /var/log/messages file and see if we could get any trace of system reboot/shutdown events:

# egrep -i "shutdown|reboot" /var/log/messages

This would not provide the required details regarding who rebooted the system. However, we could get to know the date and time when the reboot/shutdown event was triggered.

Step 3: Tracing through bash history logs

-----------------------------------------------------

Let's search using bash history.

Yes, this is another useful way to search through the default bash history file to find out which commands were executed by a specific user. However, if a user clears the bash history before logout then this may not be effective.

Yes, from the above output we come to know that the reboot/shutdown/poweroff commands were executed by two different users. But we are not sure if those commands were executed successfully or failed, also there is no information about when those commands were executed. In such a case, one option is to set the "HISTTIMEFORMAT" in users ".bashrc" file that could display the timestamp of each command. But whether the command was a success or failure is not known yet. Another caveat with the history command is that if a user runs history clear ($ history -c) command then previously recorded events get removed. So, this may not be a reliable option.

Step 4: Tracing through /var/log/secure logs

---------------------------------------------------------

All the sudo commands that any user executes would by default get logged into /var/log/secure. So, this log file is very helpful in analyzing the sudo commands which were executed by any user, their time stamp and if those were allowed commands or not. 

Let's say that the user by name "redhat" is allowed to run all privileged commands. This is done by either adding the user to the "wheel" group or adding separate entry for the user either in /etc/sudoers file or /etc/sudoers.d/<username> file. In this case this user "redhat" is added to the wheel group as shown below:

[root@rhel6upgraded ~]# grep wheel /etc/group

wheel:x:10:redhat

So, when this user runs any sudo command, it would get recorded in the /var/log/secure file as shown below (applicable to allowed sudo users and respective commands):

However, when a non-root non-privileged user runs such command then it would get recorded as “user NOT in sudoers” entry in /var/log/secure as shown below:

This is good in understanding the sudo commands run by users. However, if a user elevates (redhat in this example) himself as root by changing the identity as "sudo su -" and then runs the command then that would not get recorded here. Anyways, there is no need to run the command as sudo since it is at the root level. In such cases this may not be of much help. Btw, it is good practice not to allow all commands except "/bin/su" for sudo users. So, any commands that need root privileges would get executed as sudo by users, and that gets recorded. Overall, this seems to be a good option but no 100% accuracy. 

Step 4: Tracing through native auditd

-------------------------------------------------

Let's assume that the native auditd is active/running. This is capable enough to log all commands that the user executes which would require root privileges by default. So, when a non-root user "redhat" runs the "reboot" command with sudo privileges here are the initial audit logs that gets recorded:

Looking at the above log i.e. /var/log/audit/audit.log file, we could understand that the command "/usr/bin/sudo" executed with 3 arguments ( argc=3 a0="sudo" a1="/usr/bin/systemctl" a2="reboot" ). Lets further break this message to understand better as shown below:

[1] Who executed this command? 

To understand this we need to look at the below statement:

auid=500 uid=500 gid=500 euid=0 suid=0 

This says that this command was run by the user with UID 500 (redhat) and Effective-UID is 0 which is root. Here auid denotes Audit-UID which is the user ID which was used to login to the system originally.

[2] What command was executed? 

To understand this we need to look at the below line: 

 comm="sudo" exe="/usr/bin/sudo"    

This says that the main command executed was "/usr/bin/sudo" and further arguments are shown as:

argc=3 a0="sudo" a1="/usr/bin/systemctl" a2="reboot"

So, that makes the complete command as:

/usr/bin/sudo /usr/bin/systemctl reboot

[3] Was this command run successfully? 

We could get to know this by looking at the "success=yes" parameter in the above statement. 

[4] From which location the command was executed? 

This can be identified by looking at this section "cwd="/home/redhat" in the above statement. Here "cwd" denotes Current Working Directory. 

Same way, when a user elevates the privileges as root user and executes commands, this is how the auditd records the events (this is just small snip of it):

In this case the "UID" of the user is 0 (root) who executed the commands. However, look at the "AUID" (Audit UID) which is "500" means that the user "redhat" has elevated the privileges as root and then executed the command "reboot" as shown above. By default, commands executed by root users are not recorded, we need to add rules to get it done. 

Watch my next blog page dedicated to auditd covering more on this.