Wednesday, October 1, 2014

How to install Snort with BASE & barnyard2 - Part II

Continuing with the last post Snort installation part I now I’ll explain how to install BASE and barnyard2. BASE (Basic Analysis and Security Engine) provides a web front-end to query and analyze the alerts coming from Snort. The alerts will send to a MySQL database, this feature is provided by barnyard2. Barnyard2 is an output system for Snort, it reads the binary logs from snort using the unified2 format and then it will resend the information of this logs to a database backend, for this We’ll configure Snort to output alerts to this format.

Install BASE dependencies





# yum install -y mysql-server mysql-devel php-mysql php-adodb php-pear php-gd httpd
# pear channel-update pear.php.net
# pear install Numbers_Roman

Preparing MySQL environment
– Initializing mysql and configuring to start the daemon at boot time:


# service mysql start
# chkconfig --levels 235 mysql on
– Preparing the new database for snort:

# mysql -u root -p



<pre>mysql> create database snort;
mysql> grant select,insert,update,delete,create on snort.* to snort@localhost;
mysql> set password for snort@localhost=PASSWORD('snortpassword');

Setup snort to log out in unified2 format

# vi /etc/snort/snort.conf

output unified2: filename snort.u2, limit 128
Installing barnyard2










# tar -xzvf barnyard2-1.9.tar.gz
# cd barnyard2-1.9
# ./configure --with-mysql
# make && make install
# cp etc/barnyard2.conf /etc/snort/
# mysql -u snort -psnortpassword snort < schemas/create_mysql
# touch /etc/snort/barnyard2.waldo
# chmod 777 /etc/snort/barnyard2.waldo
# chown snort:snort /etc/snort/barnyard2.waldo
– Edit barnyard2 configuration:

# vi /etc/snort/barnyard2.conf







config reference_file: /etc/snort/reference.config
config classification_file: /etc/snort/classification.config
config gen_file: /etc/snort/rules/gen-msg.map
config sid_file: /etc/snort/rules/sid-msg.map
input unified2
config hostname: localhost
config interface: eth0
config alert_with_interface_name
output database: log, mysql, user=snort password=snortpassword dbname=snort host=localhost
Adapting our init script to work with barnyard2

# vi /etc/init.d/snortd

BARNYARD2=/usr/local/bin/barnyard2
start()
{
[ -x $SNORTD ] || exit 5
echo -n $"Starting $prog: "
daemon --pidfile=$PID_FILE $SNORTD $LINK_LAYER $NO_PACKET_LOG $DUMP_APP -D $PRINT_INTERFACE $INTERFACE -u $USER -g $GROUP $CONF -l $LOGDIR $PASS_FIRST $BPFFILE $BPF && success || failure
RETVAL=$?
$BARNYARD2 -c /etc/snort/barnyard2.conf -d /var/log/snort -f snort.u2 -w /etc/snort/barnyard2.waldo -u snort -g snort -D
[ $RETVAL -eq 0 ] && touch $lockfile
echo
return $RETVAL
}
stop()
{
echo -n $"Stopping $prog: "
killproc $SNORTD
killproc $BARNYARD2
if [ -e $PID_FILE ]; then
chown -R $USER:$GROUP /var/run/snort_eth0.* && rm -f /var/run/snort_eth0.pi*
fi
RETVAL=$?
if [ "x$runlevel" = x0 -o "x$runlevel" = x6 ] ; then
trap TERM
killall $prog 2>/dev/null
trap TERM
fi
[ $RETVAL -eq 0 ] && rm -f $lockfile
echo
return $RETVAL
}
– Restart Snortd:

# /etc/init.d/snortd restart

Installing BASE

# tar -xzvf base-1.4.5.tar.gz
# cp -r base-1.4.5/ /var/www/base
# cd /var/www/base/
# cp base_conf.php.dist base_conf.php
– Edit BASE scripts configuration:

# vi base_conf.php

$BASE_urlpath = '/base';
$DBlib_path = '/usr/share/php/adodb';
$alert_dbname = 'snort';
$alert_host = 'localhost';
$alert_port = '3306';
$alert_user = 'snort';
$alert_password = 'snortpassword';

Configuring Apache

# vi /etc/httpd/conf.d/base.conf

Alias /base /var/www/base/
<directory "/var/www/base/">
AllowOverride None
Order allow,deny
Allow from all
AuthName "Snort IDS"
AuthType Basic
AuthUserFile /etc/snort/base.passwd
Require valid-user
</directory>
– Generating password file for web access for BASE:

# htpasswd -c /etc/snort/base.passwd snortadmin
– Restart apache:

# service httpd restart

Accessing to the BASE web environment
http://IP-WEB-SERVER/base/base_db_setup.php
and click create BASE AV


How to install Snort - Part I


An IDS is a security tool, that allow us to monitor our network events searching attempts to compromise the security of our systems. It’s possible matching predefinied rules emulating the behaviour of an attack and it’s possible to deny the package or simply alert us to an email or sending messages to log. Basically we can find two types of IDS:
  • HIDS: Host based IDS, monitors the activity of a single machine, searching anomaling behaviors.
  • NID: Network IDS, capture and analyze network packages to search attack patterns.
Generally an IDS can be located in each network segment, for example front of the firewall or back of the firewall or also can be implemented in the same firewall if we have a small network traffic, with this way we can analyze all input and output traffic.

SNORT
Snort is a NIDS, implements real time scanning of attack detection and port scanning detecting. The basic architecture of snort:
  • Packet capture module: Used to capture network traffic using libpcap library.
  • Decoder: It ensures to form the data structures of the packages captured and identify the network protocol.
  • Preprocessor: prepocessors are plugins developed generally in C and process the packets provided by the decoder and ensambles the packets received. This preprocessors are configured in snort.conf file configuration. Some preprocessor examples may be:
    – sfPortscan
    – Frag3
    – HTTP
    – SSH
    – To see a complete list visit: http://manual.snort.org/node17.html
  •  Detection engine: Analyze the packets based in our rules configued.
  •  Detection plugins: Used to modify the behaviour of the detection engine.
  • Output plugins: Defines how and where saves the alters and the packages generated.


For this post I’ll explain how to install and configure snort from the source code in CentOS 6 and download free ruleset for snort and configure for be used.

Installing dependencies and preparing the environment
– Installing rpmforge repository:
– Install prerequisites packages:

# yum -y install libdnet libdnet-devel libpcap libpcap-devel daq gcc make flex bison pcre pcre-devel zlib zlib-devel
– Downloading and installing daq:






# cd /tmp ; wget http://www.snort.org/downloads/1850 -O daq-1.1.1.tar.gz
# tar -xzvf daq-1.1.1.tar.gz
# cd daq-1.1.1/
# ./configure
# make && make install
# ldconfig -v
– Creating snort user and tree directories:








# groupadd snort
# useradd -g snort snort
# mkdir /usr/local/snort
# mkdir /etc/snort
# mkdir /var/log/snort
# mkdir /var/run/snort
# chown snort:snort /var/log/snort
# chown snort:snort /var/run/snort
Installing Snort and configuring the ruleset
– Downloading and installing snort:











# cd /tmp ; wget http://www.snort.org/downloads/1862 -O snort-2.9.3.1.tar.gz
# tar -xzvf snort-2.9.3.1.tar.gz
# cd snort-2.9.3.1/
# ./configure --prefix /usr/local/snort --enable-sourcefire --enable-ipv6
# make && make install
# ln -s /usr/local/snort/bin/snort /usr/bin/snort
# cp /tmp/snort-2.9.3.1/etc/snort.conf /etc/snort/
# cp /tmp/snort-2.9.3.1/etc/unicode.map /etc/snort/
# cp /tmp/snort-2.9.3.1/etc/classification.config /etc/snort/
# cp -r /usr/local/snort/lib/snort_dynamicpreprocessor/ /usr/local/lib/
# cp -r /usr/local/snort/lib/snort_dynamicengine /usr/local/lib/
– Downloading open source ruleset from emerging:




# tar -xzvf emerging.rules.tar.gz
# touch /etc/snort/rules/white_list.rules /etc/snort/rules/black_list.rules
# chown -R snort:snort /etc/snort/
– Edit snort configuration:

# vi /etc/snort/snort.conf







ipvar HOME_NET 192.168.1.0/24
var RULE_PATH /etc/snort/rules
var SO_RULE_PATH /etc/snort/so_rules
var PREPROC_RULE_PATH /etc/snort/preproc_rules
var WHITE_LIST_PATH /etc/snort/rules
var BLACK_LIST_PATH /etc/snort/rules
include $RULE_PATH/emerging.conf
Configuring the init script for Snort
– Create sysconfig snort configuration:

# vi /etc/sysconfig/snort














#### General Configuration
INTERFACE=eth0
CONF=/etc/snort/snort.conf
USER=snort
GROUP=snort
PASS_FIRST=0
#### Logging & Alerting
LOGDIR=/var/log/snort
ALERTMODE=fast
DUMP_APP=1
BINARY_LOG=1
NO_PACKET_LOG=0
PRINT_INTERFACE=0
– Adding the init script:

# vi /etc/init.d/snortd

#!/bin/bash
#
# snort Start up the Snort Intrusion Detection System daemon #
# chkconfig: 2345 55 25
# description: Snort is a Open Source Intrusion Detection System
# This service starts up the snort daemon. #
# processname: snort
# pidfile: /var/run/snort_eth0.pid
### BEGIN INIT INFO
# Provides: snort
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $syslog
# Should-Start: $syslog
# Should-Stop: $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start up the Snort Intrusion Detection System daemon
# Description: Snort is an application for Open Source Intrusion Detection.
# This service starts up the Snort IDS daemon.
### END INIT INFO
# source function library
. /etc/rc.d/init.d/functions
# pull in sysconfig settings
[ -f /etc/sysconfig/snort ] && . /etc/sysconfig/snort
RETVAL=0
prog="snort"
lockfile=/var/lock/subsys/$prog
# Some functions to make the below more readable
SNORTD=/usr/bin/snort
#OPTIONS="-A fast -b -d -D -i eth0 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort"
#PID_FILE=/var/run/snort_eth0.pid
# Convert the /etc/sysconfig/snort settings to something snort can
# use on the startup line.
if [ "$ALERTMODE"X = "X" ]; then
ALERTMODE=""
else
ALERTMODE="-A $ALERTMODE"
fi
if [ "$USER"X = "X" ]; then
USER="snort"
fi
if [ "$GROUP"X = "X" ]; then
GROUP="snort"
fi
if [ "$BINARY_LOG"X = "1X" ]; then
BINARY_LOG="-b"
else
BINARY_LOG=""
fi
if [ "$LINK_LAYER"X = "1X" ]; then
LINK_LAYER="-e"
else
LINK_LAYER=""
fi
if [ "$CONF"X = "X" ]; then
CONF="-c /etc/snort/snort.conf"
else
CONF="-c $CONF"
fi
if [ "$INTERFACE"X = "X" ]; then
INTERFACE="-i eth0"
PID_FILE="/var/run/snort_eth0.pid"
else
PID_FILE="/var/run/snort_$INTERFACE.pid"
INTERFACE="-i $INTERFACE"
fi
if [ "$DUMP_APP"X = "1X" ]; then
DUMP_APP="-d"
else
DUMP_APP=""
fi
if [ "$NO_PACKET_LOG"X = "1X" ]; then
NO_PACKET_LOG="-N"
else
NO_PACKET_LOG=""
fi
if [ "$PRINT_INTERFACE"X = "1X" ]; then
PRINT_INTERFACE="-I"
else
PRINT_INTERFACE=""
fi
if [ "$PASS_FIRST"X = "1X" ]; then
PASS_FIRST="-o"
else
PASS_FIRST=""
fi
if [ "$LOGDIR"X = "X" ]; then
LOGDIR=/var/log/snort
fi
# These are used by the 'stats' option
if [ "$SYSLOG"X = "X" ]; then
SYSLOG=/var/log/messages
fi
if [ "$SECS"X = "X" ]; then
SECS=5
fi
if [ ! "$BPFFILE"X = "X" ]; then
BPFFILE="-F $BPFFILE"
fi
runlevel=$(set -- $(runlevel); eval "echo $$#" )
start()
{
[ -x $SNORTD ] || exit 5
echo -n $"Starting $prog: "
daemon --pidfile=$PID_FILE $SNORTD $ALERTMODE $BINARY_LOG $LINK_LAYER $NO_PACKET_LOG $DUMP_APP -D $PRINT_INTERFACE $INTERFACE -u $USER -g $GROUP $CONF -l $LOGDIR $PASS_FIRST $BPFFILE $BPF && success || failure
RETVAL=$?
[ $RETVAL -eq 0 ] && touch $lockfile
echo
return $RETVAL
}
stop()
{
echo -n $"Stopping $prog: "
killproc $SNORTD
if [ -e $PID_FILE ]; then
chown -R $USER:$GROUP /var/run/snort_eth0.* && rm -f /var/run/snort_eth0.pi*
fi
RETVAL=$?
# if we are in halt or reboot runlevel kill all running sessions
# so the TCP connections are closed cleanly
if [ "x$runlevel" = x0 -o "x$runlevel" = x6 ] ; then
trap TERM
killall $prog 2> /dev/null
trap TERM
fi
[ $RETVAL -eq 0 ] && rm -f $lockfile
echo
return $RETVAL
}
restart() {
stop
start
}
rh_status() {
status -p $PID_FILE $SNORTD
}
rh_status_q() {
rh_status > /dev/null 2>&;1
}
case "$1" in
start)
rh_status_q && exit 0
start
;;
stop)
if ! rh_status_q; then
rm -f $lockfile
exit 0
fi
stop
;;
restart)
restart
;;
status)
rh_status
RETVAL=$?
if [ $RETVAL -eq 3 -a -f $lockfile ] ; then
RETVAL=2
fi
;;
*)
echo $"Usage: $0 {start|stop|restart|status}"
RETVAL=2
esac
exit $RETVAL
– Start snort at system boot time:


# chmod +x /etc/init.d/snortd
# chkconfig --levels 235 snortd on
– Starting snort:

# /etc/init.d/snortd start
Testing the basic functionality of port scanning detection with nmap

# tail -f /var/log/snort/alert

Wednesday, September 24, 2014

Tips for creating better bash scripts

For today's article I wanted to cover a few tips that will make your scripts better, not better for you but better for the next person who has to figure out why it isn't working anymore.

Always start with a shebang

The first rule of shell scripting is that you always start your scripts with a shebang. While the name might sound funny the shebang line is very important, this line tells the system which binary to use as the interpreter for the script. Without the shebang line, the system doesn't know what language to use to process the script script.
A typical bash shebang line would look like the following:
#!/bin/bash
I have seen many scripts were this line is missing, one would think if it wasn't there than the script wouldn't work but that's not necessarily true. If a script does not have an interpreter specified then some systems will default to /bin/sh. While defaulting to /bin/sh would be ok if the script is written for bourne shell, if the script is written for KSH or uses something specific to bash and not bourne than the script may produce unexpected results.
Unlike some of the other tips in this article this one is not just a tip but rather a rule. You must always start your shell scripts with the interpreter line; without it your scripts will eventually fail.

Put a description of the script in the header

Whenever I create a script or any program for that matter I always try to put a description of what the script will do in the beginning of the script. I also include my name and if I am writing these scripts for work, I will include my work email address as well as a date that the script was written.
Here is an example header.
#!/bin/bash
#### Description: Adds users based on provided CSV file 
#### CSV file must use : as separator
#### uid:username:comment:group:addgroups:/home/dir:/usr/shell:passwdage:password
#### Written by: Benjamin Cane - ben@example.com on 03-2012
Why do I put all of this? Well it's simple. The description is there to explain to anyone who is reading through the script what this script does and any information they need to know about it. I include my name and email on the chance that if someone was reading this script and they had a question for me they could reach out and ask. I include the date so that when they are reading the script they at least have some context as to how long ago the script was written. The date also adds a bit of nostalgia when you find a script you've written long ago and ask yourself "What was I thinking when I wrote this?".
A descriptive heading in your scripts can be as custom as you want it to be, there is no hard fast rule of what needs to be in there and what doesn't. In general, just keep it informative and make sure you put it at the top of the script.

Indent your code

Making your code readable is very important, it's something that a lot of people seem to forget as well. Before we get too far into why indentation is important let's look at an example.
NEW_UID=$(echo $x | cut -d: -f1)
NEW_USER=$(echo $x | cut -d: -f2)
NEW_COMMENT=$(echo $x | cut -d: -f3)
NEW_GROUP=$(echo $x | cut -d: -f4)
NEW_ADDGROUP=$(echo $x | cut -d: -f5)
NEW_HOMEDIR=$(echo $x | cut -d: -f6)
NEW_SHELL=$(echo $x | cut -d: -f7)
NEW_CHAGE=$(echo $x | cut -d: -f8)
NEW_PASS=$(echo $x | cut -d: -f9)    
PASSCHK=$(grep -c ":$NEW_UID:" /etc/passwd)
if [ $PASSCHK -ge 1 ]
then
echo "UID: $NEW_UID seems to exist check /etc/passwd"
else
useradd -u $NEW_UID -c "$NEW_COMMENT" -md $NEW_HOMEDIR -s $NEW_SHELL -g $NEW_GROUP -G $NEW_ADDGROUP $NEW_USER
if [ ! -z $NEW_PASS ]
then
echo $NEW_PASS | passwd --stdin $NEW_USER
chage -M $NEW_CHAGE $NEW_USER
chage -d 0 $NEW_USER 
fi
fi
 
Does the above code work, yes but it's not very pretty and if this was a 500 line bash script without any indentation it would be pretty hard to understand what's going on. Now let's look at the same code with indentation.

NEW_UID=$(echo $x | cut -d: -f1)
NEW_USER=$(echo $x | cut -d: -f2)
NEW_COMMENT=$(echo $x | cut -d: -f3)
NEW_GROUP=$(echo $x | cut -d: -f4)
NEW_ADDGROUP=$(echo $x | cut -d: -f5)
NEW_HOMEDIR=$(echo $x | cut -d: -f6)
NEW_SHELL=$(echo $x | cut -d: -f7)
NEW_CHAGE=$(echo $x | cut -d: -f8)
NEW_PASS=$(echo $x | cut -d: -f9)    
PASSCHK=$(grep -c ":$NEW_UID:" /etc/passwd)
if [ $PASSCHK -ge 1 ]
then
  echo "UID: $NEW_UID seems to exist check /etc/passwd"
else
  useradd -u $NEW_UID -c "$NEW_COMMENT" -md $NEW_HOMEDIR -s $NEW_SHELL -g $NEW_GROUP -G $NEW_ADDGROUP $NEW_USER
  if [ ! -z $NEW_PASS ]
  then
      echo $NEW_PASS | passwd --stdin $NEW_USER
      chage -M $NEW_CHAGE $NEW_USER
      chage -d 0 $NEW_USER 
  fi
fi
 
With the indented version it is a lot more apparent that the second if statement is nested within the first, you may not catch that at first glance if you were looking at the un-indented code.
The style of indentation is up to you, whether you want to use 2 spaces, 4 spaces or just a generic tab it doesn't really matter. What matters is that the code is consistently indented the same way every time.

Add Spacing

Where indentation can help make code understandable, spacing helps make code readable. In general I like to space code out based on what the code is doing, again this is a preference and really the point is just make the code more readable and easy to understand.
Below is an example of spacing with the same code as above.

NEW_UID=$(echo $x | cut -d: -f1)
NEW_USER=$(echo $x | cut -d: -f2)
NEW_COMMENT=$(echo $x | cut -d: -f3)
NEW_GROUP=$(echo $x | cut -d: -f4)
NEW_ADDGROUP=$(echo $x | cut -d: -f5)
NEW_HOMEDIR=$(echo $x | cut -d: -f6)
NEW_SHELL=$(echo $x | cut -d: -f7)
NEW_CHAGE=$(echo $x | cut -d: -f8)
NEW_PASS=$(echo $x | cut -d: -f9)

PASSCHK=$(grep -c ":$NEW_UID:" /etc/passwd)
if [ $PASSCHK -ge 1 ]
then
  echo "UID: $NEW_UID seems to exist check /etc/passwd"
else
  useradd -u $NEW_UID -c "$NEW_COMMENT" -md $NEW_HOMEDIR -s $NEW_SHELL -g $NEW_GROUP -G $NEW_ADDGROUP $NEW_USER

  if [ ! -z $NEW_PASS ]
  then
      echo $NEW_PASS | passwd --stdin $NEW_USER
      chage -M $NEW_CHAGE $NEW_USER
      chage -d 0 $NEW_USER 
  fi
fi
 
As you can see the spacing is subtle but every little bit of cleanliness can help make the code easier to troubleshoot later.

Comment your code

Where the header is great for adding a description of the scripts function adding comments within the code is great for explaining whats going on within the code itself. Below I will show the same code snippet from above but this time I will add comments to the code that explains what it does.

## Parse $x (the csv data) and put the individual fields into variables
NEW_UID=$(echo $x | cut -d: -f1)
NEW_USER=$(echo $x | cut -d: -f2)
NEW_COMMENT=$(echo $x | cut -d: -f3)
NEW_GROUP=$(echo $x | cut -d: -f4)
NEW_ADDGROUP=$(echo $x | cut -d: -f5)
NEW_HOMEDIR=$(echo $x | cut -d: -f6)
NEW_SHELL=$(echo $x | cut -d: -f7)
NEW_CHAGE=$(echo $x | cut -d: -f8)
NEW_PASS=$(echo $x | cut -d: -f9)

## Check if the new userid already exists in /etc/passwd
PASSCHK=$(grep -c ":$NEW_UID:" /etc/passwd)
if [ $PASSCHK -ge 1 ]
then
  ## If it does, skip
  echo "UID: $NEW_UID seems to exist check /etc/passwd"
else
  ## If not add the user
  useradd -u $NEW_UID -c "$NEW_COMMENT" -md $NEW_HOMEDIR -s $NEW_SHELL -g $NEW_GROUP -G $NEW_ADDGROUP $NEW_USER

  ## Check if new_pass is empty or not
  if [ ! -z $NEW_PASS ]
  then
      ## If not empty set the password and pass expiry
      echo $NEW_PASS | passwd --stdin $NEW_USER
      chage -M $NEW_CHAGE $NEW_USER
      chage -d 0 $NEW_USER 
  fi
fi
 
If you were to happen upon this snippet of bash code and didn't know what it did you could at least look at the comments and get a pretty good understand of what the goal is. Adding comments to your code will be extremely helpful to the next guy, and it might even help you out. I've found myself looking through a script I wrote maybe a month earlier wondering what I was doing. If you add comments religiously it can save you and others a lot of time later.

Create descriptive variable names

Descriptive variable names might seem like an obvious thing but I find myself using generic variable names all the time. Most of the time these variables are temporary and never used outside of that single code block, but even with temporary variables it is always good to put an explanation of what values they contain.
Below is an example of variable names that are mostly descriptive.

for x in `cat $1`
do
    NEW_UID=$(echo $x | cut -d: -f1)
    NEW_USER=$(echo $x | cut -d: -f2)
 
While it might be pretty obvious what goes into $NEW_UID and $NEW_USER it is not necessarily obvious what the value of $1 is, or what is being set as $x. A more descriptive way of writing this same code can be seen below.

INPUT_FILE=$1
for CSV_LINE in `cat $INPUT_FILE`
do
  NEW_UID=$(echo $CSV_LINE | cut -d: -f1)
  NEW_USER=$(echo $CSV_LINE | cut -d: -f2)
 
With the rewritten block of code it is very apparent that we are reading an input file and that file is a CSV file. It is also more apparent where we are getting the new UID and new USER information to store in the $NEW_UID and $NEW_USER variables.
The exmaple above might seem like a bit of overkill but someone may thank you later for taking a little extra time to be more descriptive with your variables.

Use $(command) for command substitution

If you want to create a variable that's value is derived from another command there are two ways to do it in bash. The first is to wrap the command in back-ticks such as the example below.

DATE=`date +%F`
 
The second method uses a different syntax.

DATE=$(date +%F)
 
While both are technically correct, I personally prefer the second method. This is purely personal prefrence, but in general I think that the $(command) syntax is more obvious than using back-ticks. Let's say for example you are digging through hundreds of lines of bash code; you may find as you read and read that sometimes those back-ticks start looking like single quotes. On top of that, sometimes a single quote tends to look like a back-tick. At the end of the day, it all comes down to preference. So use what works best for you; just make sure you are being consistent with the method you choose to use.

Before you exit on error describe the problem

We have gone though several examples of items that make it easier to read and understand code, but this last one is useful before the troubleshooting process even gets to that point. By adding descriptive errors in your scripts you can save someone a lot troubleshooting time early on. Let's take a look at the following code and see how we can make it more descriptive.

if [ -d $FILE_PATH ]
then
  for FILE in $(ls $FILE_PATH/*)
  do
    echo "This is a file: $FILE"
  done
else
  exit 1
fi
 
The first thing this script does is check if the value of the $FILE_PATH variable is a directory, if it isn't it will exit with a code of 1 which denotes an error. While it's great that we used an exit code that will tell other scripts that this script was not successful, it doesn't explain that to the humans running this script.
Let's make the code a little more human friendly.

if [ -d $FILE_PATH ]
then
  for FILE in $(ls $FILE_PATH/*)
  do
    echo "This is a file: $FILE"
  done
else
  echo "exiting... provided file path does not exist or is not a directory"
  exit 1
fi
 
If you were to run the first snippet, you would expect a huge amount of output. If you didn't get that output you would have to open the script up to see what could have possibly gone wrong. If you were to run the second code snippet however, you would know instantly that the path you gave the script wasn't valid. Adding just one line of code can save a lot of troubleshooting later.
The above examples are just a few things I try to use whenever I write scripts. I'm sure there are other great tips for writing clean and readable bash scripts, if you have any feel free to drop them in the comments box. It's always good to see what tricks others come up with.

MySQL: tee saving your output to a file

Tee is a unix command that takes the standard out output of a Unix command and writes it to both your terminal and a file. Until recently I never knew there was a MySQL client command that performed the same function. Today I will show you an example of how to use it.

First login to the MySQL CLI (command line interface)
 $ mysql -uroot -p
 Enter password:
 Welcome to the MySQL monitor.  Commands end with ; or g.
 Your MySQL connection id is 24839

Once you are in you will have a command prompt. From there simply type tee and the file path and name you want to save to.
 mysql> tee /var/tmp/mysql_tee.out
 Logging to file '/var/tmp/mysql_tee.out'
 mysql> use mysql;
 Reading table information for completion of table and column names
 You can turn off this feature to get a quicker startup with -A

 Database changed
 mysql> show tables;
 +---------------------------+
 | Tables_in_mysql           |
 +---------------------------+
 | columns_priv              |
 | db                        |
 | event                     |
 | func                      |
 | general_log               |
 | help_category             |
 | help_keyword              |
 | help_relation             |
 | help_topic                |
 | host                      |
 | ndb_binlog_index          |
 | plugin                    |
 | proc                      |
 | procs_priv                |
 | servers                   |
 | slow_log                  |
 | tables_priv               |
 | time_zone                 |
 | time_zone_leap_second     |
 | time_zone_name            |
 | time_zone_transition      |
 | time_zone_transition_type |
 | user                      |
 +---------------------------+
 23 rows in set (0.00 sec)
 
Now just check your file and make sure your commands were logged.
 $ ls -la /var/tmp/mysql_tee.out
 -rw-r--r-- 1 ubuntu ubuntu 1030 2011-12-22 01:43 /var/tmp/mysql_tee.out

IPtables: Linux firewall rules for a basic Web Server

What is iptables?

iptables is a package and kernel module for Linux that uses the netfilter hooks within the Linux kernel to provide filtering, network address translation, and packet mangling. iptables is a powerful tool for turning a regular Linux system into a simple or advanced firewall.

Firewall & iptables basics

Rules are first come first serve

In iptables much like other (but not all) firewall filtering packages the rules are presented in a list. When a packet is being processed, iptables will read through its rule-set list and the first rule that matches this packet completely gets applied.
For example if our rule-set looks like below, all HTTP connections will be denied:
  1. Allow all SSH Connections
  2. Deny all connections
  3. Allow all HTTP Connections
If the packet was for SSH it would be allowed because it matches rule #1, HTTP traffic on the other hand would be denied because it matches both rule #2 and rule #3. Because rule #2 says Deny all connections the HTTP traffic would be denied.
This is an example of why order matters with iptables, keep this in mind as we will see this later in this article.

Two policies for filtering

When creating filtering rule-sets there are two methods explicit allow or explicit deny.

Explicit Allow (Default Deny)

Explicit Allow uses a default deny policy, when creating these types of rules the base policy is a deny all rule. Once you have a base policy that denies all traffic you must add rules (in the proper order) that explicitly allow access.
Explicit Allow is by default more secure than Explicit Deny, because only traffic that you explicitly allow is allowed into the system. The trade off, however is that you must manage a sometimes complicated rule set. If the rule set is not correct you may even end up locking yourself out of the system. It is not that uncommon to lock yourself out of your system when creating a explicit allow policy.

Explicit Deny (Default Allow)

Explicit Deny uses a default allow policy, when creating these types of rules the base policy is to allow all. Once you have a base policy that allows all traffic you can create additional rules that explicitly deny access.
When using Explicit Deny only traffic that you specifically block is denied, this means that your system is more vulnerable if you forget to block a port that should be blocked. However this also means that you are less likely to lock yourself out of the system.
Because we are creating a filtering released for a public facing web server we will be covering a Explicit Allow policy, as it is more secure and will show you some best practices when creating iptables rules.

Chains and Tables

Tables

Tables are used by iptable to group rules based on their function. The 3 table types available are Filter, NAT, and Mangle.
  • Filter - The filter table is used to filter network traffic based on allow or deny rules the user has put in place. This is the default table to use if no tables were given in a command.
  • NAT - The NAT Table is used for Network Address Translation. This is useful for configuring your Linux System as a router or port forwarding.
  • Mangle - Mangle is used for packet mangling, this is mainly useful for changing the TOS or TTL on packets
For our examples today we are focusing on only the filter table, which is also the default. When running iptables commands we will not need to specify the table we are using in our examples.

Chains

Chains on the other hand are used to organize rules based on the source of the packets. Filter has 3 default chains INPUT, OUTPUT, and FORWARD these chains are used to separate the packets based on the source of the packet.
  • INPUT - This chain is used for packets that are incoming on the system from an outside source.
  • OUTPUT - This chain is used for packets that are outgoing from the system to an outside source.
  • FORWARD - This chain is used for packets that are being forwarded through NAT rules, this allows you to filter traffic that is also NAT'ed
In addition to the chains above custom chains can be created as well, placing custom chains in can allow for packets from specific sources to match faster without being run through unnecessary rules.
We will save creating custom chains for another day as you do not really need them for a basic web server.

Creating a rule set for a basic web server

Verify iptables is installed

Before starting to write iptables rules; lets validate that the package is installed and the iptables module is loaded.

Verifying the package is installed (Ubuntu/Debian)

 # dpkg --list | grep iptables
 ii iptables 1.4.12-1ubuntu4 administration tools for packet filtering and NAT

Verifying the Kernel Module is loaded

 # lsmod | grep ip_tables
 ip_tables 18106 1 iptable_filter
Once you have validated that iptables is loaded and ready, we can start creating rules.

Creating iptables rules

When implementing iptables there are two methods. Rules can be added to a file that gets loaded on reboot/restart of the iptables services, or rules can be added live and then saved to the file that is loaded on reboot/restart of iptables. For today's examples I will show adding the rules live and saving them to a file once complete.
I prefer this method as it provides me with the ability to simply reboot the system and regain access if I mess up any rules. While this may work for virtual machines and physical machines that are close by, this is always a concern when applying firewall rules to remote machines that you do not have quick access to.
Always triple check your rules before applying them. As a best practice if you are applying the rules to a remote machine it is best to test your rules on a machine that is close by first.

Show current rules

Before we start creating rules I want to cover how to show rules that are currently being enforced. We will use this command often to verify whether our rules are in place or not. The below command shows no rules are in place.
 # iptables -L
 Chain INPUT (policy ACCEPT)
  target prot opt source destination

 Chain FORWARD (policy ACCEPT)
  target prot opt source destination

 Chain OUTPUT (policy ACCEPT)
  target prot opt source destination

Add port 22, Before you add the base policy

No matter if I am adding a explicit deny or explicit allow base policy I always add a rule that allows me access to port 22 (SSH) as the first rule. This allows me to ensure that SSH is always available to me even if I mess up the other rules.
Since my SSH connections are coming from the IP 192.168.122.1 I will explicitly allow all port 22 connections from the IP 192.168.122.1.
 # iptables -I INPUT -p tcp --dport 22 -s 192.168.122.1 -j ACCEPT
 # iptables -L
 Chain INPUT (policy ACCEPT)
 target prot opt source destination
 ACCEPT tcp -- 192.168.122.1 anywhere tcp dpt:ssh
Lets break this command down a bit, and get a better understanding of iptables syntax.
-I INPUT
The -I flag tells iptables to take this rule and insert it into the INPUT chain. This will put the given rule as the first rule in the set; there are other methods of adding rules such as appending the rule or adding the rule into a specific line that we will show in the next few examples.
-p tcp
The -p flag is part of the rule that tells iptables whatprotocol that we want to match on, in this case it is TCP.
--dport 22
The --dport flag is short for destination port and is used to specify that we are only looking to match on traffic destined for port 22.
-s 192.168.122.1
The -s flag as you can probably guess is used to specify the source of the traffic, in our case the source of our port 22 traffic will be 192.168.122.1.As a note you must be careful to ensure you have a proper source when applying this rule.
-j ACCEPT
The -j flag tells iptables to jump to the action for this packet, these are known in iptables as targets. A target can be a chain or one of the predefined targets within iptables, the target in our example will tell iptables to ACCEPT the packet and allow it to pass through the firewall.
If you have a dynamic IP the IP address that you are trying to access the server from may change. If that is the case it may be preferable to leave out the source which would allow all port 22 traffic.
Example to allow SSH traffic from all sources:
 # iptables -I INPUT -p tcp --dport 22 -j ACCEPT

Changing the base policy to default deny (Explicit Allow)

Now that we have the SSH traffic allowed we can change the base policy of the INPUT chain to deny all, this means that anything that does not match any of our rules will be denied. In some filtering systems this needs to be done by adding a rule at the end of the set that denies everything. In iptables however this can be changed by changing the INPUT chains default policy.
 # iptables -P INPUT DROP
 # iptables -L
 Chain INPUT (policy DROP)
 target prot opt source destination
 ACCEPT tcp -- 192.168.122.1 anywhere tcp dpt:ssh
The -P flag stands for policy, in this command we are setting the default policy of the INPUT chain to the target DROP.
DROP vs REJECT
Within the default policy I showed using the DROP target. There are two targets in iptables that can deny a packet, DROP and REJECT. There is a very small but possibly very impactful difference in these two targets.
The REJECT target will send a reply icmp packet to the source system telling that system that the packet has been rejected. By default the message will be "port is unreachable".
The DROP target simply drops the packet without sending any reply packets back.
The REJECT target is vulnerable to DoS style attacks as every packet that is rejected will cause iptables to send back an icmp reply, when an attack is at volume this causes your system to also send icmp replies in volume. In this scenario it is better to simply drop the packet and reduce traffic congestion as best you can.
For this reason I suggest using DROP as a default reply.

Appending our web server rules

Now that we have the default policy and SSH rules in place we need to add rules for the other services we want to make accessible. The first rule we will add will accept and allow all traffic destined to port 80 and port 443 for web traffic.
 # iptables -A INPUT -p tcp --dport 80 -j ACCEPT
 # iptables -A INPUT -p tcp --dport 443 -j ACCEPT
 # iptables -L -n
 Chain INPUT (policy DROP)
 target prot opt source destination
 ACCEPT tcp -- 192.168.122.1 0.0.0.0/0 tcp dpt:22
 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
As you can see in the above example we added the rule by using -A INPUT. While -I is used to insert a rule into the beginning or a chains rule-set, the -A flag is used to append a rule at the end of the rule-set.

Allowing Loopback Traffic

In addition to allowing web server traffic we should also allow all traffic on the loopback interface, this is necessary if you are running a local MySQL server and connecting via localhost; but also a good idea in general for the loopback interface.
 # iptables -I INPUT -i lo -j ACCEPT
 # iptables -L -n
 Chain INPUT (policy DROP)
 target prot opt source destination
 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
 ACCEPT tcp -- 192.168.122.1 0.0.0.0/0 tcp dpt:22
We can tell iptables to allow all traffic on a specific interface with the -i flag, and because we didn't add -p the default action is to allow all protocols.
I placed this rule first in the chain because in general there is quite a bit of traffic on the loopback interface and having the rule earlier in the ruleset provides faster matching and allows the kernel to spend less time trying to match the packets.

Allowing FTP Traffic

FTP traffic can be tricky when it comes to firewalls; that is because FTP uses multiple ports when transferring data and sometimes not the same port.
Active mode FTP uses port 21 for a control channel and port 20 for data. This is easily enough accounted for with the first two iptables rules in the example below.
Passive mode FTP however is not as simple, once a client has connected to the control port the FTP server, the FTP client can establish a higher port for the client to connect to. Because this higher port is often within a large range it is difficult to open up the entire range without possibly allowing malicious traffic to unwanted ports.
For this scenario iptables uses another module called ip_conntrack; ip_conntrack tracks established connections and allows iptables to create rules that allows related connections to be accepted.
This allows for the FTP connection to establish on port 21 with the first rule in the list and then establish a connection with a higher port via the third rule.
 # iptables -A INPUT -p tcp --dport 21 -j ACCEPT
 # iptables -A INPUT -p tcp --dport 20 -j ACCEPT
 # iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
 # iptables -L -n
 Chain INPUT (policy DROP)
 target prot opt source destination
 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:21
 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:20
 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
In the third rule being applied the -m flag stands for match, this allows iptables to match connections based on the connection state (-ctstate) ESTABLISHED or RELATED.
RELATED is a connection type within iptables that matches new connections that are related to already established connections. This is really mainly used with FTP based traffic.
You may also want to restrict FTP traffic to a certain network, to do this simply add the -s flag and the appropriate ip/ip range to the first and second iptables rules.
Loading the ip_conntrack_ftp module
In order for conntrack to work properly you must ensure that the ip_conntrack_ftp module is loaded.
 # modprobe ip_conntrack_ftp
 # lsmod | grep conntrack
 nf_conntrack_ftp 13452 0
 nf_conntrack_ipv4 19716 1
 nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
 xt_conntrack 12760 1
 nf_conntrack 81926 4 nf_conntrack_ftp,xt_state,nf_conntrack_ipv4,xt_conntrack
 x_tables 29846 7 xt_state,ip6table_filter,ip6_tables,xt_tcpudp,xt_conntrack,iptable_filter,ip_tables

Allowing DNS Traffic

DNS based traffic is not as tricky as FTP however DNS traffic primarily uses UDP rather than TCP. However some DNS traffic can be over TCP traffic. In order to allow DNS traffic you must specify 2 iptables commands one opening the port for TCP and the other opening the port for UDP.
 # iptables -A INPUT -p tcp --dport 53 -j ACCEPT
 # iptables -A INPUT -p udp --dport 53 -j ACCEPT
 # iptables -L -n
 Chain INPUT (policy DROP)
 target prot opt source destination
 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53

Blocking IP's

Sometimes the internet is not as friendly as one would like, eventually you may need to block a specific IP address or range of IP's. For this example we are going to block the IP range of 192.168.123.0/24.
 # iptables -I INPUT 3 -s 192.168.123.0/24 -j DROP
 # iptables -L -n
 Chain INPUT (policy DROP)
 target prot opt source destination
 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
 ACCEPT tcp -- 192.168.122.1 0.0.0.0/0 tcp dpt:22
 DROP all -- 192.168.123.0/24 0.0.0.0/0
 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
The above rule is a little trickier than others because we already have a rule that accepts all traffic on port 80.
Our rule is to block all traffic from the IP range 192.168.123.0/24; in order for this to be blocked it must be listed in a specific spot within our ruleset. With iptables the first rule that matches a packet will be applied. For our rule to work we must add it before the port 80 rule otherwise the iprange will still be able to connect to port 80.
The number 3 after INPUT is a specification that tells iptables to place the rule 3rd in the list. Because we didn't specify any protocol this rule will block all protocols from the specified ip range.

Saving the rules for reboot

Each distribution of Linux has a different method for saving and restoring the iptables rules for reboot. In my opinion Red Hat variants have the best default.

Red Hat

In the Red Hat distributions you can save the current live iptables rules by using the init script. This saves the rules into the file /etc/sysconfig/iptables. This file is later read by the init script during reboot of the server or a simple restart given to the init script.
Saving your active rules
 # /etc/init.d/iptables save
Loading ip_conntrack on boot
In order to load the ip_conntrack_ftp module on boot you will need to edit the /etc/sysconfig/iptables-config file. This file is used by Red Hat's init script to load any related modules on boot.
 # vi /etc/sysconfig/iptables-config
Modify to add the following
 IPTABLES_MODULES=ip_conntrack_netbios_ns ip_conntrack ip_conntrack_ftp

Ubuntu/Debian

Debain based distributions which includes Ubuntu, do not offer such init scripts by default. There is however a package called iptables-persistent that provides an init script for Ubuntu/Debian variants with similar functionality as the Red Hat version.
Installing iptables-persistent
You can install this package with apt-get
 # apt-get install iptables-persistent
Saving your active rules
Once iptables-persistent is installed it will automatically save your current configuration into /etc/iptables/rules.v4. Any future modifications however will not be automatically saved, from then on you must save the rules similar to the Red Hat version.
 # /etc/init.d/iptables-persistent save
  * Saving rules...
  * IPv4...
  * IPv6...
  ...done.

 # cat /etc/iptables/rules.v4
 # Generated by iptables-save v1.4.12 on Mon Sep 17 05:56:17 2012
 *filter
 :INPUT DROP [0:0]
 :FORWARD ACCEPT [0:0]
 :OUTPUT ACCEPT [42:4296]
 -A INPUT -i lo -j ACCEPT
 -A INPUT -s 192.168.122.1/32 -p tcp -m tcp --dport 22 -j ACCEPT
 -A INPUT -s 192.168.123.0/24 -j DROP
 -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 21 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 20 -j ACCEPT
 -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
 -A INPUT -p udp -m udp --dport 53 -j ACCEPT
 COMMIT
 # Completed on Mon Sep 17 05:56:17 2012
Making sure iptables-persistent is started on boot
Once your rules are saved, iptables-persistent will start them on boot. To verify that the script is added properly simply check that it exists in /etc/rc2.d/.
 # runlevel
 N 2

 # ls -la /etc/rc2.d/ | grep iptables
 lrwxrwxrwx 1 root root 29 Sep 16 21:44 S37iptables-persistent -> ../init.d/iptables-persistent
Loading ip_conntrack on boot
For Debian the iptables-persistent package does not include a iptables-config file. You could add this module into the init script itself to ensure it is loaded before iptables starts.
 # vi /etc/init.d/iptables-persistent
Go to line #25 and add:
 /sbin/modprobe -q ip_conntrack_ftp