Unattended upgrades on Ubuntu & Debian

It's super simple! Just install with apt and check the configuration files to make sure it's enabled (often it will be after install).

$ sudo apt install unattended-upgrades
$ cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

Unattended upgrades on RHEL & CentOS

On Red Hat variants, we generally have to modify the /etc/yum/yum-cron.conf config file with the entries shown below to enable updates, but it is very simple. The preferred package is yum-cron.

$ yum install -y yum-cron
$ cat /etc/yum/yum-cron.conf | grep -E 'update_cmd|download_updates|apply_updates'
update_cmd = security
download_updates = yes
apply_updates = yes
$ systemctl start yum-cron.service

Installing UFW

It comes by default in Ubuntu, but if you are missing it you can run apt get install ufw. In RHEL variants, it comes with the EPEL repository:

$ sudo yum install -y epel-release
$ sudo yum install -y ufw

Configuring Basic Firewall Rules

Let's configure some rules for our web server. I am assuming that this is for a basic site that only runs on ports 80 or 443, and allows remote SSH connections, so we'll need port 22 as well.

When configuring this, the order of commands is important - blocking port 22 and enabling the firewall will disconnect any open connection you have with SSH!

The following commands will allow all outgoing traffic, and all incoming traffic on these three ports, then start the firewall. At any time you can see the status of the firewall and all active rules by running ufw status verbose

$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow ssh
# Before running this next command, make sure you have allowed SSH!
$ sudo ufw enable
$ sudo ufw allow http
$ sudo ufw allow https

You might wonder why this is necessary if you aren't running any other services on the server currently. This helps ensure that if one day someone mis-configures a local database to listen on a port, or installs and runs redis, or any other service, then a conscious choice to update the firewall must be made before any remote connections can happen.

Configuring Advanced Firewall Rules (I promise it's still simple!)

I call these advanced because they need to be thought through. Decide if they make sense for your specific implementation, and think about what other needs your environment might have.

Limit SSH only to known IP address blocks

If possible, it is best to limit SSH only to your own company's IP address ranges, or specific jump host servers on your network with static IP's. This can be dangerous if you don't have control of your IP's, though you could limit it to specific countries.

Make sure that you also remove the rule allowing SSH from anywhere after setting this rule, and that you are currently connected through a valid IP or you risk being locked out of SSH.

# Limit to one IP (Jump host perhaps). Note that 10. addresses are unroutable.
$ sudo ufw allow from 10.10.10.10 to any port 22
# or an IP CIDR range
$ sudo ufw allow from 10.10.10.0/24 to any port 22

Limit outgoing connections

Most servers don't need to communicate outside on all other ports. Limit it only to the web and DNS ports. These are generally always needed since updates will be pulled from those ports. This can help prevent malware that uses hard coded ports from reaching back out of your server if you end up infected.

Add other rules as needed - and make them as specific as possible (for instance, a database IP and port combination), while not making them so specific that they would require frequent changes.

$ sudo ufw default deny outgoing
$ sudo ufw allow out 53
$ sudo ufw allow out http
$ sudo ufw allow out https

Deleting rules

Need to remove a rule? List them by number and delete the numbered rule.

$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 80                         ALLOW IN    Anywhere
$ sudo ufw delete 1

Installing mod_security with Apache

It's really easy with Apache!

# Debian / Ubuntu
$ sudo apt install libapache2-modsecurity
$ sudo service apache restart

# RHEL / CentOS
$ sudo yum install mod_security
$ sudo /etc/init.d/httpd restart

Install default & OWASP rulesets

The default and OWASP rules are a good starting point. From there, you can determine additional rules that are specific to your environment.

Navigate to the default modsecurity folder (it should have been created and populated on install) - /etc/modsecurity/ for Apache. Then download and re-name the rulesets. The first (default) ruleset I show may have been installed automatically. If so, it is up to you whether you wish to download the latest directly or leave what was installed.

$ cd /etc/modsecurity
# Default rules
$ wget https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/modsecurity.conf-recommended
$ wget https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/unicode.mapping
$ mv modsecurity.conf-recommended modsecurity_default.conf
# OWASP rules
$ git clone https://github.com/SpiderLabs/owasp-modsecurity-crs
$ cp owasp-modsecurity-crs/crs-setup.conf.example owasp-modsecurity-crs/crs-setup.conf

Let's now go ahead and actually turn the rules on within our webserver.

Turn the rules on in Apache

Now we need to update the Apache configuration. Open the /etc/apache2/mods-available/security2.conf file and update it to contain the following:

<IfModule security2_module>
        # Default dir for modsecurity's persistent data
        SecDataDir /var/cache/modsecurity

        IncludeOptional /etc/modsecurity/*.conf
        Include /etc/modsecurity/rules/*.conf
        IncludeOptional /etc/modsecurity/owasp-modsecurity-crs/*.conf
        Include /etc/modsecurity/owasp-modsecurity-crs/rules/*.conf
</IfModule>

Test the configuration changes using apachectl configtest and if all looks good restart Apache to enable the WAF.

Actively block caught traffic

I recommend actively blocking malicious traffic. If you feel comfortable doing more than logging attack requests, then it's a simple config update. Only do this on production systems if you are confident it won't break existing functionality and have viewed the logs in /var/log/modsec_audit.log

In the modsecurity_default.conf file, update the line near the top with SecRuleEngine DetectionOnly to SecRuleEngine On

Restart the web server to have these take effect.

Installing mod_security with NGINX

It's a bit more involved with NGINX, and many guides do not seem to clarify what needs to be in place to get this installed and working.

First, check if the NGINX you are running is compiled with the --with-compat flag by running nginx -V. Any installation without this flag will need to re-install NGINX with this flag enabled. As of this writing, most distribution default installs do not distribute with the --with-compat flag, so I assume you will have to reinstall NGINX.

Installing the official NGINX repository & version

There are a couple of options, but I find the easiest one to be just installing the version in the official NGINX repository. If you already have NGINX installed, remove it first with sudo apt remove nginx before installing new (this will create a service interruption for a few minutes, but no configuration files will be lost)

On Ubuntu, to install the official NGINX :

$ wget https://nginx.org/keys/nginx_signing.key
$ sudo apt-key add nginx_signing.key
$ echo deb https://nginx.org/packages/mainline/ubuntu/ `lsb_release -c | cut -f2` nginx >> /etc/apt/sources.list.d/nginx.list
$ echo deb-src https://nginx.org/packages/mainline/ubuntu/ `lsb_release -c | cut -f2` nginx >> /etc/apt/sources.list.d/nginx.list
$ sudo apt update
# Check that the official nginx source is planned. 
$ apt policy nginx
nginx:
  Installed: (none)
  Candidate: 1.15.8-1~bionic
  Version table:
     1.15.8-1~bionic 500
        500 https://nginx.org/packages/mainline/ubuntu bionic/nginx amd64 Packages
$ sudo apt install nginx

Building the mod_security module

Unfortunately, we have to build this module manually, but it isn't too hard. NGINX has an excellent walk-through on building and installing the module, so follow their instructions to install mod_security.

The commands I ran to get this running on my latest server are here if you just want to get moving quickly. You may have to install certain build tools such as make first.

# Download and build mod_security - may take 15-20 minutes
$ git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
$ cd ModSecurity/
$ git submodule init
$ git submodule update
$ ./build.sh
$ ./configure
$ make
$ make install
$ cd ..
# Build custom NGINX package to make the module - update the 
# version to match the latest one in mod_security repo. 
# Start only after above commands finish successfully.
$ wget http://nginx.org/download/nginx-1.14.0.tar.gz
$ tar zxvf nginx-1.14.0.tar.gz nginx-1.14.0/
$ cd nginx-1.14.0/
$ ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
$ make modules
$ cp objs/ngx_http_modsecurity_module.so /usr/share/nginx/modules/ngx_http_modsecurity_module.so

Install default & OWASP rulesets

The default and OWASP rules are a good starting point. From there, you can determine additional rules that are specific to your environment.

Create mod_security folder (/etc/nginx/modsec), then download and re-name the rulesets.

# Default rules
$ wget https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/modsecurity.conf-recommended
$ wget https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/unicode.mapping
$ mv modsecurity.conf-recommended modsecurity_default.conf
# OWASP rules
$ git clone https://github.com/SpiderLabs/owasp-modsecurity-crs
$ cp owasp-modsecurity-crs/crs-setup.conf.example owasp-modsecurity-crs/crs-setup.conf

Let's now go ahead and actually turn the rules on within our webserver.

Turn the rules on in NGINX

First, let's create a single configuration file that loads all of our other conf files and rules (Still from within our ruleset directory /etc/nginx/modsec).

$ cat main.conf
Include /etc/nginx/modsec/modsecurity_default.conf
Include /etc/nginx/modsec/owasp-modsecurity-crs/crs-setup.conf
Include /etc/nginx/modsec/owasp-modsecurity-crs/rules/*.conf

Now update the NGINX config file at /etc/nginx/conf.d/default.conf to include the following two lines right below the server_name directive.

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Test the changes with nginx -t and if everything looks ok, restart the web server!

Actively block caught traffic

I recommend actively blocking malicious traffic. If you feel comfortable doing more than logging attack requests, then it's a simple config update. Only do this on production systems if you are confident it won't break existing functionality and have viewed the logs in /var/log/modsec_audit.log

In the modsecurity_default.conf file, update the line near the top with SecRuleEngine DetectionOnly to SecRuleEngine On

Restart the web server to have these take effect.

Installing & Configuring OSSEC

I choose OSSEC for this purpose because it is a well known standard, open source, and pretty easy to get started.

Installing from a repository is more likely to help you get setup as a client or a server as part of a larger system like I discussed above, but today we just want to consider monitoring a single isolated server. To do this, we will pull down and run a setup manually.

Let's start by downloading and installing OSSEC 3.1. You can check for more recent updates if you prefer. You can see a full install log below - I chose to enable everything, but think about what makes sense for your installation.

The important part is to choose a local install, and active response. You can also see that I added my own (fake) IP to the whitelist so that I cannot be banned.

$ wget https://github.com/ossec/ossec-hids/archive/3.1.0.tar.gz
$ tar -xvzf 3.1.0.tar.gz
$ cd ossec-hids-3.1.0/
$ ./install.sh

- What kind of installation do you want (server, agent, local, hybrid or help)? local

  - Local installation chosen.

2- Setting up the installation environment.

 - Choose where to install the OSSEC HIDS [/var/ossec]: 

    - Installation will be made at  /var/ossec .

    - The installation directory already exists. Should I delete it? (y/n) [y]: 

3- Configuring the OSSEC HIDS.

  3.1- Do you want e-mail notification? (y/n) [y]: 
   - What's your e-mail address? [email protected]

   - We found your SMTP server as: mail.nullsweep.com.
   - Do you want to use it? (y/n) [y]: 

   --- Using SMTP server:  mail.nullsweep.com.

  3.2- Do you want to run the integrity check daemon? (y/n) [y]: 

   - Running syscheck (integrity check daemon).

  3.3- Do you want to run the rootkit detection engine? (y/n) [y]: 

   - Running rootcheck (rootkit detection).

  3.4- Active response allows you to execute a specific 
       command based on the events received. For example,
       you can block an IP address or disable access for
       a specific user.  
       More information at:
       http://www.ossec.net/en/manual.html#active-response
       
   - Do you want to enable active response? (y/n) [y]: 

     - Active response enabled.
   
   - By default, we can enable the host-deny and the 
     firewall-drop responses. The first one will add
     a host to the /etc/hosts.deny and the second one
     will block the host on iptables (if linux) or on
     ipfilter (if Solaris, FreeBSD or NetBSD).
   - They can be used to stop SSHD brute force scans, 
     portscans and some other forms of attacks. You can 
     also add them to block on snort events, for example.

   - Do you want to enable the firewall-drop response? (y/n) [y]: 

     - firewall-drop enabled (local) for levels >= 6

   - Default white list for the active response:
      - 127.0.0.53

   - Do you want to add more IPs to the white list? (y/n)? [n]: y
   - IPs (space separated): 10.10.10.10

  3.6- Setting the configuration to analyze the following logs:
    -- /var/log/auth.log
    -- /var/log/syslog
    -- /var/log/dpkg.log
    -- /var/log/nginx/access.log (apache log)
    -- /var/log/nginx/error.log (apache log)

 - If you want to monitor any other file, just change 
   the ossec.conf and add a new localfile entry.
   Any questions about the configuration can be answered
   by visiting us online at http://www.ossec.net .
   
   
   --- Press ENTER to continue ---

Now we will want to customize the rules. I notice that OSSEC default log locations are often not quite right. For instance, it was looking for my nginx log in /var/www (despite what it said in the install output), which does not exist on my server. Instead, I opened /var/ossec/etc/ossec.conf and added the following entry:

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/error.log</location>
  </localfile>

Look around in this file and update any other log files that need monitoring. A few other things to consider looking at and updating:

• The directory list under the syscheck section. Consider adding your web directory or some parts of it if it does not change frequently
• Other log files that are important: web, database, system, root commands, mailserver. Find the full list of supported formats in the documentation (and you can write your own rulesets if so inclined)
• Consider email notifications - off by default. This might be noisy depending on your configuration, so try it and tune.
• Active Response settings. By default, OSSEC will block IP's for 5 minutes that trigger a ruleset, such as multiple invalid SSH login attempts.