HowTo: Run Network Service Per System / VM Instance To Improve Overall Security Of A Web Stacks (LAMP)

A web stack is nothing but collection of many open source software such as an operating system, Web server, database server, server side programming language. The most commonly known web stacks is LAMP. It is an acronym for a solution stack of free, open source software, referring to the first letters of Linux (operating system), Apache Web server, MySQL database software and PHP (or sometimes Perl or Python). All of our security related tutorials recommends running different network services on separate systems or vm instance. Naturally, this limits the number of other services that can be cracked in the event that an attacker is able to successfully exploit a software flaw in one network service. This is also one of the most requested article via email. In this guide, I will explain how to setup a solution that can serve static content, dynamic content, database, and caching by running on separate servers or vm instance.

LAMP: Traditional vs split setup

You end us setting up everything using a single server or vm instance to serve up all content as follows:

                   One Big Server / VM instance
               +-----------------------------------------+
               | Apache+PHP/Perl @ 75.126.153.206:80     |
               | Mysql @ 127.0.0.1:3306 (or unix socket) |
               | Pgsql @ 127.0.0.1:5432 (or unix socket) |
               | netfilter for filtering traffic         |
               +-----------------------------------------+
                 *** Dedicated LAMP server ***
                 OS: RHEL/CentOS/Debian/Ubuntu/*BSD/Unix
                 RAM: 4-8GiB ECC
                 CPU: Single or dual Intel/Amd
                 Storage: RAID-1/5 server class sata/sas

Say if an Apache web server compromised, an attacker will gain access to your database, cache, and other part of network too. So you need to split services per server as follows:

/////////////////////////
/ Internet/ISP router  /
///////////////////////
          \
            |
             \
              ---------| vm00
             75.126.153.206:80 - eth0
             192.168.1.1       - eth1
         +----------------------------+
         | Reverse Proxy              |
         | Firewall                   |       eth0:192.168.1.10/vm01
         +----------------------------+      +----------------------+
         |                                   | Lighttpd static files|
         +-----------------------------------+ /var/www/static      |
         |                                   +----------------------+
         |
         |                                    eth0:192.168.1.11/vm02
         +-----------------------------------+-----------------------+
         |                                   | Apache+php+perl+python|
         |                                   | /var/www/html         |
         |                                   +-----------------------+
         |
         |                                    eth0:192.168.1.12/vm03
         +-----------------------------------+-----------------------+
         |                                   |Database sql caching   |
         |                                   |Redis/Memcached etc    |
         |                                   +-----------------------+
         |
         |                                   eth0:192.168.1.13/vm04 (or a dedicated server for database w/ RAID-10)
         +-----------------------------------+------------------------+
         |                                   | Mysql/pgsql db server  |
         |                                   | @192.168.1.13:3306/5432|
         |                                   +------------------------+
         |
         |                                   eth0:192.168.1.14/vm05 (or a dedicated server for storing all files via NFSv4 w/ RAID-10)
         +-----------------------------------+------------------------+
         |                                   | NFSv4 running on Linux |
         |                                   | /export/{static,html   |
         |                                   +------------------------+

The spit setup offers many benefits such as:

Security
Scalability and growth
Optimization
Ease of use
Ease of monitoring
Add advanced feature such as failover (VIP at vm00 level), load-balancing, cdn and much more easily.

Role for each vm / server:

The following provides more details. A wordpress based blog or drupal based site or custom made app hosted on the following setup can easily serve millions of page views per month.

VM/Server	Description	Software (part of stack)
vm00 w/ 2 core and 2GiB RAM	This is the only part you are going to expose to the world. This node / vm must have two network interfaces. One interface, called public is used by clients to connect to the your website. The other interface, called private (VLAN) is used for network traffic only between nodes.This vm or server must have two network interfaces.	nginx/haproxy as reverse proxy and netfilter as firewall.
vm01 w/ 1 core and 1GiB RAM	Use this node to serve static files such as images/js/css. You need to store files on nfs server called vm05 and mount it at /var/www/static/	Lighttpd/tux/nginx and Linux nfs client
vm2 w/ 2-4 core and 4GiB RAM	This node is used to serve dynamic pages. For example, you can install wordpress or your custom application written in PHP5/Python/Perl on this node. You need to store application files on nfs server called vm05 and mount it at /var/www/html/. You need store log locally at /var/log/apache directory.	Apache 2 with fastcgi+php5+perl+python etc and Linux nfs client.
vm3 w/ 1 core 1-2GiB RAM	This node is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source.	Memcached or Redis etc.
vm4 w/ 2-4 core and 2-4GiB RAM+RAID hybrid storage	This node stores your data in RDBMS such as mysql or postgresql.	Postgresql or Mysql server.
vm5 w/ 4 core and 4-6GiB RAM+RAID hybrid storage	This node act as Network File System (NFS) server which is a distributed file system. You store images,js,css in /var/www/static directory and php,perl,python files in /var/www/html directory on this server only. You need to export /var/www/html and /var/www/static directories using the /etc/exports configuration file. The client machine vm01 and vm02 requests access to exported data, typically by issuing a mount command. Apache and lighttpd server daemons on the client machine can then view and interact with mounted filesystems on the server. You also need to create ssh / sftp users on this server so that they can upload files.	Linux NFSv4 server
ISP router or uplink port at the hosting data center.	The vm00 node connects to the Internet using uplink provided by your ISP. You can start with 100Mbps or 1000Mbps uplink port for eth0. The private port (eth1) is used for network traffic only between all vm nodes and it can be on 100/1000Mbps based VLAN setup.	Site specific so no recommendations for uplink.

How does it works?

Let’s see how our setup and a reverse proxy works. In this example, I will be placing reverse proxy and http servers inside my firewall. See Figure 1. The web site name is www.example.com which resolves to a static public IPv4 address 202.54.1.1 which is assigned to eth0. 192.168.1.1 is our internal IP and assigned to eth1. This site is your reverse proxy. Rest of the all servers are inside the LAN and can not be accessed directly over the Internet.

The hardware (or OpenBSD / Linux software) firewall rules only allow 202.54.1.1 on ports 80 and 443. All other ports are blocked. Each vm node also runs iptables and only allows access to required ports. Your reverse proxy defines pool of http server as follows:

## our upstream for www.example.com ##
upstream mybackend {
      server 192.168.1.10:80; #server1
      server 192.168.1.11:80; #server2
      ....
      ..
      ..
      server 192.168.1.100:80; # server100
}

The Apache and lighttpd server will access its files using nfs server configured at vm05. The Apache web server is configured with PHP. Our PHP application is configured to connect to the database server hosted on vm04. Our PHP application uses vm03 to cache sql quires using memcached server.

NOTE: You can also place reverse proxy on a DMZ and http and other servers inside firewall to increase security. This will increase the cost of the project.

Enough talk, show me the actual server configuration

Most of the actions listed in this post are written with the assumption that they will be executed by the root user running the bash on CentOS 6.x / Red Hat Enterprise Linux 6.x. However, you can easily replicate the setup on any other *nix like operating systems.