Linux Commands Every Sysadmin Must Know

Minutes

Nowadays there are countless ways to manage a Linux system – sysadmins can easily feel like sitting back and simply relying on managed environments to help them through their workload. These command line tools and packages help developers find problems on their Linux-powered machines, and can often make it much simpler and easier to organize their workflow. Linux commands is the path for optimizing apps and to help troubleshoot, providing essential information.

It doesn’t matter what your experience level is when it comes to managing a Linux system: we think the following commands will help you understand your system better. You can also use many of these tools for troubleshooting. For example, why does your application refuse to work on a remote host – but work just fine locally? The following tools cover everything from a development environment across to Linux environments that are virtualized – Linux containers are covered too, and so are bare metal machines. Let’s get started.

curl

Do you want to test the endpoint of an application, or check its connectivity to an upstream service? curl is your friend because curl can transfer a URL. Use it to determine whether your app is able to contact a different service (a database for example) or to check if a service is healthy.

Here is an example: if your app shows an HTTP 500 error that says a MongoDB cannot be reached, you can use the following command

$ curl -I -s superapp:5000
HTTP/1.0 500 INTERNAL SERVER ERROR

Here, using -l will show the header while -s produces a silenced response. In another example, you can check the endpoint of a database using your PC:

$ curl -I -s database:27017
HTTP/1.0 200 OK

The response you get should look fine, but next you can try to get to your database – by reaching it from the application host. Don’t forget, your application uses the same name as the hostname of the database, so this is something you can try:

$ curl -I -s https://superapphost.com
HTTP/1.1 200 OK

When you examine this response you will see that the application is unable to find the database because the address of the database cannot be reached, or because the host (whether it is a virtual machine or a container) does not have a nameserver that allows it to find your hostname.

ls

Want a list of files in a directory? Use the ls command. It’s a frequently used command by both developers and sysadmins. If you’re using a container, it’s a good way to find out what the files and directories look like on your container image. Another really useful aspect of ls is that it shows you the permissions on files.

In the example we show below you will see why you are unable to run myapp – it’s because of a problem with permissions. You can check permissions using the command ls-l; in turn you will see from the results below that the permissions it shows does not include an “x” in the sequence “-rw-r–r—”; this sequence of course means that the file is write and read only.

$ ./superapp
bash: ./superapp: Permission denied
$ ls -l superapp
-rw-r--r--. 1 root root 33 Jul 21 18:36 superapp

tail

Using the tail command shows you the very last part of a file. It is a useful command because, in most cases when you need to examine a log file, you only need to see the final part – not the entire file. Using this command you can check just the recent requests made to the application.

One example is using tail to see what has happened in your Apache web server logs. Use -f and you will see the requests to your Apache server as they happen.

Using the -f option tells tail to “follow”, doing this means that the log lines are sent to your screen as these lines are being written to the log. In our example you will see a script running in the background which is trying to access an endpoint – every few seconds, with the log making a note of all of these requests. Another tail option is -n which shows you the last 100 lines in a file.

$ tail -n 100 /var/log/httpd/access_log

cat

The function you need to print and to concatenate a file is – cat. You can make use of it if you want to check your dependencies file or if you want to double check the version of an app which you have been building on your local machine.

$ cat requirements.txt
flask
flask_pymongo

In our example we make use of the function to see whether the Python Flash app has a specific dependency – Flask, in this instance.

env

env is the command you need if you want to set environment variables, or print your environment variables. Checking for incorrect environment variables can help you find issues with applications. Below, we show how you can use env to check environment variables on the machine hosting your app:

$ env
PYTHON_PIP_VERSION=9.0.1
HOME=/root
DB_NAME=test
PATH=/usr/local/bin:/usr/local/sbin
LANG=C.UTF-8
PYTHON_VERSION=3.4.6
PWD=/
DB_URI=mongodb://database:27017/test

grep

If you’re looking for file patterns grep is your friend. You can also use it to find a pattern in the output of a different command that you used – grep will use highlighting to show you the lines you’re looking for. So, you can use grep if you want to find something in a log file or to find a specific process.

For example, if you’re trying to figure out whether Apache Tomcat has successfully started up you could find that there are too many lines to go through. However, a grep command can help you find the right line – simply pipe the output to grep and it will show you the line that says the server has started.

$ cat tomcat.log | grep org.apache.catalina.startup.Catalina.start
15-Jan-2020 14:08:48.543 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 516 ms

netstat

You can display the network status using netstat. It includes an outline of the ports currently being used as well as inbound connections. Note that netstat is not included in Linux by default, you need to add it separately by installing the net-tools package.

How it can be used? E.g. developer tests an application on a local machine and then publishes their app on the host machine – in turn, getting an error which says a port is being used by another application – or that a specific address is currently being used.

ip address

ip address is the command you use if you want to see all the interfaces and IP addresses associated with an application, on a specific host. One way to use ip address is to check the IP address for your container – or to check the IP address of the host. Also, where a host gets connectivity from two networks ip address will tell you which network is connected to which network interface.

Note that if you can find ip address on your host you need to download a specific package to install it – it’s called iproute2.

chmod

Sometimes when you run an app binary for the first time you can get an error which states “permission denied”. With ls, of course, you can check the permission in place for a specific application’s binary:

$ ls -l
total 4
-rw-rw-r--. 1 user user 34 Jan 15 12:17 testx.sh

Our example displays how in this case you do not have the execution rights to be able to run this binary – there is no “x” in place. Chmod is the command you need to change these permissions so that you are able to execute the binary as the current user.

$ chmod +x test.sh
[user@localhost ~]$ ls -l
total 4
-rwxrwxr-x. 1 user user 34 Jan 15 12:17 testx.sh

This example shows how the permissions are now updated with the right execution privileges. When you now try to run your binary you won’t see the same permission error. Note that you can also use chmod where you have loaded a binary into your container – chmod can make sure that the container has the right permissions.

id

You can use the id command to determine the identity of a user that is running a specific app. In our example below, Vagrant is in place to test apps and to isolate the app’s development environment.

Once you log in to Vagrant you will encounter a problem installing Apache: you’ll be told you don’t have the ability to do it while logged in as root. You can use id to see what your user and group is – in our example, you will see you are running as user “vagrant”, which on its own is in the “vagrant” group.

$ yum -y install httpd
Loaded plugins: fastestmirror
You need to be root to perform this command.
$ id
uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

By the way, to fix this particular problem, you will need to execute the command as a superuser – this will give you sufficient privileges.

dig and nslookup

DNS – domain name servers – is the server that links a URL to the server containing that application. Sometimes you can’t resolve a URL, for whatever reason, and often the blame is on some connectivity problem with that application.

Say that, from your host, you wanted to access a database that is at the “mydatabase” URL – but you get an error which says “cannot resolve”. Both dig and nslookup are great tools for trying to find what the real problem is. You use dig to look up a DNS, while nslookup performs a full query of an internet name server.

$ nslookup superdatabase
Server:   10.0.2.3
Address:  10.0.2.3#53
** server can't find superdatabase: NXDOMAIN

So, with nslookup you can now see that your domain “mydatabase” cannot be resolved. Dig will give you a similar result:

$ dig superdatabase
; <<>> DiG 9.9.4-RedHat-9.9.4-50.el7_3.1 <<>> superdatabase
;; global options: +cmd
;; connection timed out; no servers could be reached

It can be tricky to find the root cause behind errors like this – it could be as a result of many different problems. Your sysadmin can often help you because these can be very technical, networking problems. If your problem is on a local test server it could mean that you have not configured the nameservers on your test host the right way.

sestatus

When using an application host that is enterprise-grade or managed by a large company you might encounter a Linux security module called SELinux. The goal of SELinux is to enable processes to run on a host using the absolutely minimum security privileges possible. As a result, any malicious process cannot do much damage because by default it does not have the security privileges.

However, it can happen that an app needs to legitimately access a file – only giving an error when it tries to. You can check whether SELinux is causing the problem by making use of grep and tail to check whether there is a message in /var/log/audit logs which says “denied”. Another option is to try and see whether SELinux is enabled – you do this using sestatus.

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Our example shows a case where the Linux instance has SELinux configured and enabled. If it is your own host you manage locally you can choose to reconfigure SELinux to be less restrictive. On the other hand, if the Linux host is not under your control your sysadmin can help you change the configuration so your app has the required permissions.

du

For more insight into exactly which files use the most disk space you can use the du command to show space usage in a specific directory. In the example below, you can find out which log uses the most disk space in /var/log, we added the -h parameter for a human readable result, while -s tells du to show the total file size.

$ du -sh /var/log/*
358K  /var/log/audit
5.0K  /var/log/boot.log
0 /var/log/chrony
4.0K  /var/log/cron
4.0K  /var/log/maillog
48K /var/log/messages

You can see from the above output that the biggest directory is in fact /var/log/audit. Using df and du together will help you figure out what is using the most disk space on your host.

Understanding Basic Linux Commands Can Help You

To wrap it up, the basic tools we have outlined above are simple commands which can all help you find out why one of your apps are not working as expected – and why an application is working fine in one development environment, but refuses to co-operate in another development environment.

Sysadmins make use of these tools to debug system and application issues. If you understand these commands you’ll have a head start – you can fix some problems yourself, you can help your sysadmin find a problem and at the very least you can grow to understand what sysadmins are trying to tell you while they try to fix your application.