Sep 122015

This is a large project, so I’ll be breaking this into multiple parts. Some of this information was previously published, here and on other blogs I’ve operated in the past. Where possible, when information comes from external sources, I will credit it here.

Boring Backstory

I had budgetary reasons to migrate from my old cPanel server (I’ve been using cPanel to host my websites since, I dunno, 2005-2006, and always stuck with it for legacy/can’t be arsed reasons) to a new infrastructure. As technology improved, I had moved from a cPanel shared hosting account, to cPanel reseller accounts, to cPanel on a VPS, to cPanel on a dedicated server, back to cPanel as a virtual machine (using a custom KVM hypervisor/public cloud product provided by an old employer, and then to a VMware VM on ESXi)… but it was time to stop paying for a cPanel license when I know how to admin servers and have no need for it any more, and it was time to get a cheaper, more efficient way to manage my various projects, websites, e-mail infrastructure, etc. This is that story. Are there better ways to do most of these things? Probably. I will highlight specifics that I would do differently, were I to do this again, when we get to those. But otherwise, I feel like this is a good overview, not only for the specific topic in the post title, but in general for how to set up a home lab, or “private cloud” of sorts.

The pieces and parts

A very brief overview of what comprises my infrastructure.

  1. A cheap, dedicated server; or, a workstation in your closet, depending on your public-facing needs.
    • Don’t pay more than $75/mo for a hosted dedicated server.
    • If you want to self-host, pick up an old tower-style IBM ThinkServer on eBay. I have a friend who picked one up at their current going-rate of $200-$300.
    • For the purposes of this guide, you’ll need something no older than a Nehalem-class Xeon CPU. That’s something like an L/X/E5520.
    • Aim for >=16GB RAM. Preference for a >=1TB spinning disk and =>250GB SSD at the very least; RAID1 at a minimum is preferred. I recommend it, but we’re going budget here, so I’m not actually using it to be honest.
    • OS on the bare metal depends on item #2 below.
  2. A virtualization system of your choice. Because of my experience with it, and the fact that it provides a WUI, I prefer oVirt. You may prefer VMware ESXi, the free one. Apparently it also comes with a WUI appliance now. This guide assumes you will use oVirt (don’t worry, I’ll walk you through it).
  3. opnSense as a NATing firewall.
  4. Ajenti and Ajenti V to manage vhosts. You can create all the nginx config files by hand, if you prefer, but I had a large number to manage, and I prefer Ajenti.

Part 1 – Installing oVirt

While I call this product a “private cloud,” we are going to start with a single management/compute node, because it’s necessary to keep within the budget. All of these items can be seamlessly expanded by simply repeating Part 1 as many times as you like to expand your cloud to multiple nodes. The technologies in oVirt and opnSense will bridge across any nodes you add, so this actually can be deployed as a private cloud; you can even span multiple data centers and geographic locations, if you like, although have fun with the latency.

This part of the guide comes from an old oVirt blog I used to run when I was running an oVirt infrastructure full-time. Modifications are made to the previous post to allow for oVirt to properly handle the NATing of the opnSense firewall.

This guide assumes that you will be using Intel-based servers that have the following BIOS settings enabled: Virtualization, and Disable Execute. Dig around in your BIOS settings before installing oVirt to make sure both these options are enabled.

This guide will also be using CentOS rather than Fedora or oVirt Node for all host installations. I found oVirt Node very difficult to deal with, and you will not be able to install “shared local” storage using oVirt Node because you do not have root read/write access to the filesystem with that OS. There is a much greater memory usage overhead using CentOS, so be warned.

This guide assumes a basic familiarity with CentOS, NFS, SSH, and to a lesser extent, oVirt itself. Further explanation and documentation on these technologies exists on their developers’ websites.

Install CentOS

First, install CentOS on your first machine. This should be fairly simple. Install the basic minimum server distribution. When the OS starts for the first time, make sure sshd is up, iptables is empty, and that you save the sshd and iptables configurations. If you are using a dedicated server provider, you may want to uninstall some packages from their default OS deployment. I would strongly recommend going with a dedicated server provider that provides some sort of IP over KVM so that you can ensure you are running as lean as possible.

Once the OS is deployed, prepare the environment by wiping out the iptables rules.

iptables -F
/etc/init.d/iptables save

Install the oVirt repository

yum localinstall

Install oVirt Engine

yum install -y ovirt-engine

Engine setup

Okay, so you’ve got all the packages installed, it’s time to set up the oVirt engine. This is done via a simple command-line function:


This will guide you through the installation of the oVirt Engine. Historically, I’ve had issues with the firewall configuration, so I do not let the engine setup perform this for me. For security, you should have the engine setup configure a firewall, but make sure to manually edit /etc/sysconfig/iptables to punch holes in it for ports 80, 443, and the NFS/portmapper ports, as you’ll need them later. Accept all the other defaults, but don’t configure an ISO domain at this time.

Set up NFS

yum -y install nfs-utils
/etc/init.d/rpcbind start
/etc/init.d/nfs start
chkconfig nfs on
chkconfig rpcbind on

In addition to starting the NFS services, you need to configure /etc/exports. You should have lines in it for the loopback and public IPs of your current server, as well as any other servers you will be adding to the cluster. oVirt also requires very specific NFS settings. If you’re running with separate spinning disk and SSD volumes, I would recommend setting these up as separate exports in NFS. Make sure these mount points exist and you have your filesystems set up the way you want them.


Set up your NFS storage domain

Set the appropriate permissions on your NFS mount points.

chown -R 36:36 /isos /vmhdd /vmssd
exportfs -a

Add the host in oVirt

Now, it’s time to use the web browser. Log into your server via HTTPS; for example,, and click on the Administration Portal link. Enter the username “admin” and the password you specified during engine-setup.

Now go to the Hosts tab and click new. Put in the IP address and root password of your first node. Warning: click Advanced Settings and disable the automatic firewall configuration, or ports 80 and 443 will be blocked and you won’t be able to access the web admin any more. Wait for the node installation to finish (several yum packages will be installed). If you’re setting up a true private cloud, and you’ve been running these steps on multiple servers, you can add all your hosts at the same time.

Add the storage domain

Now it’s time to add your storage. Click the Storage tab and click New Storage. Select your new data center and give your new storage a name; for example, VM-HDD. Spaces are not allowed in storage domains. For Use Host, select the node that contains the directory you created; i.e. for your first node, select your first node from the drop-down list. This is important; if you have a mis-match of the node and directory names, this will fail. Finally, in the location, specify the bound IP address of your node, followed by the path of your storage domain. For example:

It will take several minutes for the cluster to come online and your storage to become usable. Add all your exports except your ISO domain in this step.

ISO domain

Now you can create an ISO domain. This has the same setup as a storage domain, except under Domain Function / Storage Type, select ISO/NFS. Point to your ISO folder:

P.S.: Uploading to ISO domains

I found it very difficult to parse how to upload ISOs to storage domains. The easiest way is to download your files locally on the same host that your ISO domain exists on, and then use the ovirt-iso-uploader.

ovirt-iso-uploader -r --iso-domain=ISOs upload Fedora-Live-KDE-x86_64-22-3.iso

This will copy the file from your “local” filesystem to your “NFS” filesystem. Since it’s a local copy, it won’t take long.

Tweaks for opnSense

oVirt networking is not set up to pass NAT traffic properly. Make the following tweaks to fix this.

First, look at all of your networks:

ifconfig -a

This is the first caveat; I’ve said that there are probably better ways to do this; I don’t know which of these networks actually require these settings, so I’ve disabled these settings on all of the networks I could. This is just a trial-and-error dump of getting the NAT to pass traffic properly.

ethtool -K vnet3 tx off
ethtool -K vnet gso off
ethtool -K vnet3 gso off
ethtool -K vnet2 gso off
ethtool -K vnet1 gso off
ethtool -K vnet0 gso off
ethtool -K ovirtmgmt gso off
ethtool -K bond0 gso off
ethtool -K ovirtmgmt tx off
ethtool --offload ovirtmgmt rx off tx off
ethtool --offload eth0 rx off tx off
ethtool -k eth0
ethtool -K eth0 tso off
ethtool -K eth1 tso off
ethtool --offload eth1 rx off tx off
ethtool --offload bond0 rx off tx off
ethtool -K bond0 gso off
ethtool -K bond0 tso off

That’s as far as we’ll cover in part 1. Take a look around your first node (or nodes). Explore oVirt a bit. We’ll cover adding a VM and installing opnSense next.

Jul 012013

Well, it’s time to re-write the rules again. The latest releases of cPanel/WHM have changed the game in a lot of ways, so it’s about time for a re-written guide. This will guide you through installing the latest version of ffmpeg on a cPanel server, along with a number of required libraries. The last update to this post was November 8, 2013.

Some Prerequisites

There are a few things you’ll need to prepare if you’re on a fresh installation.

  1. Subversion and some libraries
  2. Subversion is required to download and install ffmpeg and mplayer. We also need to make sure you’ve got a bunch of libraries for all the crap we’re going to custom-compile, so we might as well get that out of the way first.

    yum install subversion libXext libXext-devel libcurl libcurl-devel curl curl-devel zlib-devel openssl-devel gettext-devel expat-devel

  3. Git
  4. Git is required because for some reason, when you install mplayer, it forces you to download a Git repository of ffmpeg. Don’t use this repository for installing ffmpeg – it breaks with a strange error message. How you install Git will vary across OSes.
    cd /usr/local/src
    tar -zxf git-
    cd git-
    make prefix=/usr/local all
    make prefix=/usr/local install

  5. Ruby
  6. Ruby is required for flvtool. You can install this using the cPanel script:


    If you’re not using cPanel, you can probably install this using your package manager.

  7. faac
  8. faac doesn’t compile any more, so we’re going to install the rpmforge RPM for it instead.

    rpm -ivh

    If you’re not on a CentOS 5.x 64-bit install, check for the package you want.

  9. libxvid
  10. The source code for libxvid doesn’t compile properly on CentOS in a way that lets it be used as a shared package, so we’re going to install the rpmforge RPM for it instead.

    rpm -ivh

    If you’re not on a CentOS 5.x 64-bit install, check for the package you want.

    Guess what, now the RPM doesn’t work! You will need to download and install the source, see below.

  11. ldconfig
  12. To make sure mplayer and ffmpeg install using the correct libs, we need to make sure ld knows where to find our libraries. First, check:

    nano /etc/

    Make sure that this file contains the line:


    If you’re on a 64-bit system you should also add:


    Save (CTRL+O), exit (CTRL+W), and then update:


    If you run in to any “file not found” errors, it’s probably ldconfig.

  13. Remove any other installations
  14. You might already have ffmpeg installed. If it was compiled in, that’s fine, this will overwrite it. But if you have RPM binaries installed, this will cause issues. Run the following:

    rpm -qa | grep ffmpeg
    rpm -qa | grep mplayer
    rpm -qa | grep mencoder

    Remove with rpm -e any libraries, common files, or binary installations of these packages to prevent conflicts.

Download All The Things

One note; the official distribution release of ffmpeg-php no longer compiles. Someone forked it and is maintaining a git repository here: so this is the version we will use for ffmpeg.

  1. Let’s get started! This step can and probably should be copy/pasted into your terminal all in one block.

  2. cd /usr/local/src
    tar zxf flvtool2-1.0.6.tgz
    tar zxf lame-3.99.5.tar.gz
    git clone ffmpeg-php
    tar xzf libvorbis-1.3.3.tar.gz
    tar xzf libogg-1.3.1.tar.gz
    tar xzf opencore-amr-0.1.3.tar.gz
    tar xjf libtheora-1.1.1.tar.bz2
    tar zxf faad2-2.7.tar.gz
    tar zxf yasm-1.2.0.tar.gz
    mkdir /usr/local/lib/codecs
    tar xjf essential-20071007.tar.bz2
    svn co svn:// gpac
    git clone git://
    tar zxf xvidcore-1.3.2.tar.gz
    cp /usr/local/src/essential-20071007/* /usr/local/lib/codecs/
    chmod -R 755 /usr/local/lib/codecs/
    mkdir /usr/local/src/tmp
    chmod 777 /usr/local/src/tmp
    export TMPDIR=/usr/local/src/tmp

  3. Now it’s time to grab mplayer and ffmpeg, the latest versions possible.

  4. git clone git:// ffmpeg
    svn checkout svn:// mplayer

Let’s Compile!

Alright, now we’re down to the good stuff – compilation! Hopefully if the above steps went correctly this will just fly by. However I do not recommend that you copy and paste the entire command tree into the terminal at this point. Run everything one step at a time to catch any errors if they do occur. Google is your friend, but feel free to leave a comment here if you run in to any problems.

  1. Lame

  2. cd /usr/local/src/lame-3.99.5/
    make install

  3. Libogg

  4. cd /usr/local/src/libogg-1.3.1
    make install

  5. Libvorbis

  6. cd /usr/local/src/libvorbis-1.3.3
    make install

  7. yasm

  8. cd /usr/local/src/yasm-1.2.0
    make install

  9. Libx264

  10. cd /usr/local/src/x264
    ./configure --enable-shared
    make install

  11. Flvtool

  12. cd /usr/local/src/flvtool2-1.0.6
    ruby setup.rb config
    ruby setup.rb setup
    ruby setup.rb install

  13. Opencore-amr

  14. cd /usr/local/src/opencore-amr-0.1.3
    make install

  15. Libtheora

  16. cd /usr/local/src/libtheora-1.1.1
    make install

  17. faad2

  18. cd /usr/local/src/faad2-2.7
    make install

  19. MP4Box

  20. cd /usr/local/src/gpac
    make install

  21. xvidcode

  22. cd /usr/local/src/xvidcore/build/generic
    make install

  23. Mplayer – remember, this will download a new version of ffmpeg for some crazy reason. When it asks, just hit enter and allow it to download. It’s only used for compiling mplayer.

  24. cd /usr/local/src/mplayer/
    ./configure --enable-jpeg
    make install

    If you run into an error here that looks something like this:

    libavfilter/avfiltergraph.c:140: error: #pragma GCC diagnostic not allowed inside functions

    This appears to be a bug in GCC. Use the following lines to comment out the code causing the issue:

    sed -i 's%FF_ENABLE_DEPRECATION_WARNINGS%/* FF_ENABLE_DEPRECATION_WARNINGS */%g' /usr/local/src/mplayer/ffmpeg/*/*.c
    sed -i 's%FF_DISABLE_DEPRECATION_WARNINGS%/* FF_DISABLE_DEPRECATION_WARNINGS */%g' /usr/local/src/mplayer/ffmpeg/*/*.c
    sed -i 's%FF_ENABLE_DEPRECATION_WARNINGS%/* FF_ENABLE_DEPRECATION_WARNINGS */%g' /usr/local/src/mplayer/ffmpeg/*/*.h
    sed -i 's%FF_DISABLE_DEPRECATION_WARNINGS%/* FF_DISABLE_DEPRECATION_WARNINGS */%g' /usr/local/src/mplayer/ffmpeg/*/*.h

    And then recompile mplayer.

  25. FFMpeg

  26. cd /usr/local/src/ffmpeg
    ./configure --enable-libmp3lame --enable-libvorbis --enable-shared --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-nonfree --enable-libtheora --enable-version3 --enable-gpl --enable-libxvid
    make install

  27. ffmpeg-php

  28. cd /usr/local/src/ffmpeg-php
    make install

  29. php.ini – All that’s left is to modify your php.ini file! Add the following near the top of the file /usr/local/lib/php.ini (to avoid conflicts)


  31. Restart Apache
  32. /etc/init.d/httpd restart

That’s It!

At this point you should have a fully functional ffmpeg installation compiled in to PHP. You can test this with the following:

php -m | grep ffmpeg

You should get the output:


Thank you very much to for providing a very stable base for me to build this tutorial on!

Mar 012013

Today, we’ll be setting up a base CentOS 5.9 server with nginx, php-fpm, mySQL, and extracting a cPanel account into it. This is useful if you have a single website you need to migrate from one provider into a dedicated environment, and you need it to run as predictably as possible with as little RAM as possible. Let’s get started by updating the operating system. Then we’ll proceed to setting up each individual service before extracting the backup into them all at once.

1. Update the Operating System

The first thing to do is make sure you’re running your latest-and-greatest before you open yourself up to the public.

yum update -y

Review the packages you just installed. If you see a kernel, reboot the system before continuing.

Next, let’s add the nginx repo. That just makes things easier, we probably don’t need to manually compile nginx.

rpm -Uvh

We also need the EPEL and IUS repositories in order to get some more updated PHP binaries as well as php-fpm.

rpm -Uvh
rpm -Uvh

2. Install and Configure nginx

And now, let’s install (and start) nginx.

yum -y install nginx
/etc/init.d/nginx start

At this point, you should be able to go to your server’s main IP address and see the nginx default page.

3. Install mySQL

Let’s install mySQL next, so we have the client and server set up before we try to install PHP to connect to it.

yum install mysql-server

Once you’ve installed the package, you still need to configure the mysql server:

/etc/init.d/mysqld start

Answer some questions, and then create the file /root/.my.cnf in your favorite text editor. Add the following lines:

pass=Your new root password

This will allow you to connect to mySQL from the root user command line without specifying the mySQL root password.

4. Install PHP/php-fpm

Time to install PHP! At a minimum, you need the following packages:

yum install php53u php53u-mysql php53u-fpm

A more complete installation that would more closely mirror a typical cPanel server will look something like this:

yum install php53u php53u-fpm php53u-mysql php53u-gd php53u-mbstring php53u-devel php53u-mcrypt php53u-imap php53u-pear php53u-pdo php53u-xml php53u-soap php53u-ioncube-loader

And finally, start the php-fpm listener:

/etc/init.d/php-fpm start

5. Configure nginx to use php-fpm

Now it’s time to configure your nginx configuration file to (securely!) pass PHP files to nginx. This information comes via this blog entry, which contains some nifty information on why other blog entries on how to set up nginx and php-fpm leave you open for a really nasty attack. I’ve made a lot of modifications to the code though, so don’t copy directly from that blog, as it may not work. I found that I needed to set “root” under the PHP location as well to get the path info working.
First, open your /etc/nginx/conf.d/default.conf file in your favorite text editor. You’ll likely see the following:

# pass the PHP scripts to FastCGI server listening on
#location ~ \.php$ {
# root html;
# fastcgi_pass;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;

I like to leave the default configuration example/comments intact in these files so I have something to refer back to if I break my server, so just add some empty lines below those comments and add the following:

# Pass all .php files onto a php-fpm/php-fcgi server.
location ~ \.php$ {
root /home/user/public_html;

fastcgi_split_path_info ^(.+\.php)(/.+)$;
include fastcgi_params;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
# fastcgi_intercept_errors on;
add_header request $request_filename;
if (-f $request_filename) {

You’re also going to want to uncomment the lines below that in the default file:

location ~ /\.ht {
deny all;

At this point you can make any other changes you want, including changing the “root” location (your document root; I recommend setting the same location you had in cPanel), your listen IP/port, your server_name, your error_pages, your “index” (you probably want to add “index.php” here), etc. At this point, you should also review /etc/php-fpm.conf and /etc/php-fpm.d/www.conf. The latter file especially impacts your permissions, as you will set the user PHP runs as on this. It might be a good idea to create a system user named the same thing as your cPanel user, and make sure your files are chowned and chmoded appropriately.

6. Extract your cPanel backup

You’re halfway there! It’s time to start picking apart the elements of a cPanel backup file. Let’s look at an extracted cpmove backup file.

addons homedir.tar mysql/ resellerpackages/ userconfig/
bandwidth/ httpfiles/ mysql.sql sds userdata/
counters/ interchange/ mysql-timestamps/ sds2 va/
cp/ locale/ nobodyfiles shadow vad/
cron/ logaholic/ pds shell version
dnszones/ logs/ proftpdpasswd sslcerts/ vf/
domainkeys/ meta/ psql/ ssldomain/
fp/ mm/ quota sslkeys/
homedir/ mma/ resellerconfig/ suspended/
homedir_paths mms/ resellerfeatures/ suspendinfo/

Ouch! Okay, let’s just mess with what’s absolutely essential to get this up and running.

mv homedir.tar homedir/
cd homedir/
tar -xf homedir.tar

Now this looks a little more familiar, right?

bin/ cpmove.psql/ homedir.tar php/ tmp/
cpanel3-skel/ cpmove.psql.1339066964/ mail/ public_ftp/ www/
cpbackup-exclude.conf etc/ perl5/ public_html/

Your directory structure may vary somewhat. The most important thing to us here is that public_html directory. Move it to wherever you want your document_root to be and set the ownership properly.

mv public_html /home/user/
chown -R user:user /home/user/public_html

Now it’s time to import the mySQL databases and users. Users are stored in the “mysql.sql” file at the root of the backup; individual databases are stored in the mysql directory.

cd ../
mysql < mysql.sql for i in `ls | grep .sql | grep -v horde | grep -v roundcube | cut -d\. -f1`; do mysql < $i.create; mysql $i < $i.sql;done

6. Manual configs

Okay, now we're into some more dicey stuff this tutorial won't cover. You should check your backup file for things like PostgreSQL, cron jobs, SSL certificates, restoring log files/webalizer information, DNS, and e-mail. We might cover that in a later tutorial. At this point, however, you should have a fully-functional website! Just host DNS elsewhere, point it at your new server, and go!

Aug 032012

Apache’s mod_security is a system that scans URL data and data POSTed to the server in forms for malicious attacks such as XSS, SQL injections, etc. It comes with a default ruleset that always trips up a number of standard AJAX functions in applications like WordPress and Joomla, necessitating that rules be removed from the mod_security configuration to allow the site to function properly. The method of removing these rules under Apache 2 is by whitelisting the individual rules in the VHost entry for the domain. Here’s how to do it in cPanel.

Is mod_security enabled?

[email protected] [~]# httpd -M | grep sec
security2_module (shared)


A typical mod_security error message

[Sun Jul 29 18:29:47 2012] [error] [client] ModSecurity: Access denied with code 403 (phase 2). Match of "rx (?:/event\\\\.ng/|horde/services/go\\\\.php|tiki-view_cache\\\\.php|^/\\\\?out=http://|homecounter\\\\.php\\\\?offerid=.*ureferrer=http|__utm\\\\.gif\\\\?|/plugins/wpeditimage/editimage\\\\.html|/spc\\\\.php)" against "REQUEST_URI" required. [file "/usr/local/apache/conf/modsec/50_asl_rootkits.conf"] [line "53"] [id "390145"] [rev "10"] [msg " UNSUPPORTED DELAYED Rules: Rootkit attack: Generic Attempt to install rootkit"] [data "="] [severity "CRITICAL"] [hostname ""] [uri "/index.php"] [unique_id "UBVB02B-nBoACs9n0p4AAAAH"]

The error log for mod_security is /usr/local/apache/logs/error_log. There are 2 important pieces of information contained in this error:

  • [id “390145”]
    This is the mod_security rule ID number. We will use this to tell mod_security which rule we want to whitelist.

  • [hostname “”]
    This is the name of the VirtualHost under which the error was generated. This is important because whitelisting a mod_security rule for a domain doesn’t whitelist it for the whole account; subdomains, addon domains, etc all have separate whitelists.

Creating the userdata includes

mkdir -p /usr/local/apache/conf/userdata/std/2/user/

[email protected] [~]# mkdir -p /usr/local/apache/conf/userdata/std/2/user/

In the above example, we are setting up an Inside VHost include. The italic std indicates this is being done in a port 80 (standard/non-secure) VirtualHost, the bold user is the cPanel username of the account, and the bold and italic is the hostname from the error message above (we discard www because that’s just a ServerAlias under the same VHost). Once we create the userdata include directory, it’s time to set up the configuration file to include. We’ll use nano because it’s the best text editor.

nano -w /usr/local/apache/conf/userdata/std/2/user/

[email protected] [~]# /usr/local/apache/conf/userdata/std/2/user/

You can name the file anything as long as it ends in .conf; I prefer to use for clarity and simplicity. Now let’s whitelist some mod_security rules.

SecRuleRemoveById 390145

Notice that we used 390145 from the error message above. If this site was generating errors from multiple IDs, we’d put in multiple SecRuleRemoveById lines.

Enabling the userdata includes in Apache conf

/scripts/ensure_vhost_includes --user=user

This uncomments a line in the VirtualHost entry in httpd.conf for this domain to include any .conf files in /usr/local/apache/conf/userdata/std/2/user/

/usr/local/cpanel/bin/apache_conf_distiller --update

These compile and distill the changes into Apache’s configuration.


The first thing to check is to ensure that the following line in the VirtualHost entry for is uncommented:

Include "/usr/local/apache/conf/userdata/std/2/user/*.conf"

The only other way to test this is to reproduce the issue and tail the Apache error log. Make sure you have reproduction instructions available; many scripts will trigger more than one SecRule, but you’ll only see the first one due to the request being blocked.

Jul 232012

Uninstalling CloudLinux is a fairly involved process, as the CloudLinux repos ship their own version of many software packages that need to be re-downgraded to CentOS defaults. I will be using this guide as a basis, with a few tweaks because uninstalling kernels with yum makes me nervous (dependency issues mean that you could be uninstalling the entire glibc system from your server…)

  • Run the CloudLinux uninstaller
    /usr/local/cpanel/bin/cloudlinux_system_install -c

  • Update any packages available from your newly-default CentOS repos
    yum upgrade -y

  • Recompile Apache to remove liblve and mod_hostinglimits
    /scripts/easyapache --build

  • Install the CentOS kernel
    yum --disableexcludes=all install kernel

  • Manually uninstall the CloudLinux kernel
    rpm -qa |awk '/^kernel.*lve/ {print $1|"xargs rpm -e --nodeps"}'

  • Make sure the uninstaller updated grub
    grep -r lve /boot/*
    If this finds anything in menu.lst or any other GRUB config files, manually remove the entries for kernels with “lve” in the name.

  • Reinstall the packages CloudLinux supplied with CentOS defaults
    rpm -qa --qf "[%{VENDOR} %{NAME}\n]"|awk '/CloudLinux/ {print $2|"xargs yum reinstall -y"}'

  • Downgrade any packages supplied by CloudLinux with CentOS versions
    rpm -qa --qf "[%{VENDOR} %{NAME}\n]"|awk '/CloudLinux/ {print $2|"xargs yum downgrade -y"}'

  • Remove any packages supplied by CloudLinux not available in CentOS
    rpm -qa --qf "[%{VENDOR} %{NAME}\n]"|awk '/CloudLinux/ {print $2|"xargs yum erase -y"}'

  • Final upgrade. Make sure kernel-headers and kernel-devel are installed.
    yum upgrade -y
    yum install kernel-headers kernel-devel

  • Reboot
    shutdown -rf now

    That should be it. If Apache fails to start, try another EasyApache build.

  • Apr 282012

    It’s been a long time coming, but here’s my guide to install the latest ffmpeg and associated packages for a video upload site on base CentOS, no cPanel required (if Google got you here and you have cPanel, try the cPanel ffmpeg guide). This article assumes you’ve already set up your HTTP daemon (Apache, nginx, lighttpd, litespeed, etc), scripting agent (PHP), and database provider (mySQL, PostgreSQL, etc). The commands in this guide assume you are running the latest CentOS 5.8 64-bit. Some modifications may be necessary for other architectures and versions of CentOS.

    1. Make sure you’re up to date
    2. Make sure you’re on the latest CentOS branch. We’re installing the latest version of ffmpeg, and if you’re not running the latest version of your CentOS branch, you may run into unexpected issues.

      yum update

      Install whatever is required, and compatible with your other software. If you update your kernel, reboot before continuing.

    3. Extra Repositories
    4. RPMForge is a useful, fully-compatible extra repository. We will use it for a number of packages that don’t need to be bleeding-edge. For more information, see the CentOS wiki.

      rpm --import
      rpm -K rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
      rpm -i rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm

    5. Development Tools
    6. You must have the required packages to download and compile the source code.

      yum groupinstall "Development Tools"
      yum install subversion git

    7. Install RPM libraries
    8. Some libraries don’t need to be bleeding-edge. We’ll install those now.

      yum install gettext-devel expat-devel curl-devel zlib-devel openssl-devel libXext libXext-devel flvtool2

    9. ldconfig
    10. We need to make sure ldconfig is checking the correct directories for libraries.

      echo /usr/local/lib >> /etc/; ldconfig

    11. Download libraries
    12. Okay, here’s the big one. It’s probably a good idea to go ahead and copy+paste this whole code block into your terminal at once to save time. One note; the official distribution release of ffmpeg-php no longer compiles. The SVN version available here works, but the SVN tarball link here can’t easily be downloaded in a terminal session, so I’ve rehosted the file here on this server. This is not my file, I haven’t modified it, and I take no responsibility for it – it’s merely a copy of the file available from the above link.

      cd /usr/local/src
      tar zxf lame-3.99.5.tar.gz
      tar xzf ffmpeg-php.tar.gz
      tar xzf libvorbis-1.3.3.tar.gz
      tar xzf libogg-1.3.0.tar.gz
      tar xzf opencore-amr-0.1.3.tar.gz
      tar xjf libtheora-1.1.1.tar.bz2
      tar zxf xvidcore-1.3.2.tar.gz
      tar zxf faad2-2.7.tar.gz
      tar zxf faac-1.28.tar.gz
      mkdir /usr/local/lib/codecs
      tar -jxf all-20110131.tar.bz2
      tar zxf yasm-1.2.0.tar.gz
      cp all-20110131/* /usr/local/lib/codecs/
      chmod -R 755 /usr/local/lib/codecs/
      mkdir /usr/local/src/tmp
      chmod 777 /usr/local/src/tmp
      export TMPDIR=/usr/local/src/tmp
      svn co gpac
      git clone git://
      git clone git:// ffmpeg
      svn checkout svn:// mplayer

    13. Compile everything
    14. Here we go! This is the hard part, where we compile bleeding-edge libraries. At the time this was posted, this all worked. If you run into any errors, try Googling them. If you still can’t figure it out (or more importantly, if you do!) please comment here and I’ll edit this post with the latest fixes.

      1. Lame

      2. cd /usr/local/src/lame-3.99.5/
        make install

      3. Libogg

      4. cd /usr/local/src/libogg-1.3.0
        make install

      5. Libvorbis

      6. cd /usr/local/src/libvorbis-1.3.3
        make install

      7. yasm

      8. cd /usr/local/src/yasm-1.2.0
        make install

      9. Libxvid

      10. cd /usr/local/src/xvidcore/build/generic
        make install

      11. Libx264

      12. cd /usr/local/src/x264
        ./configure --enable-shared
        make install

      13. Opencore-amr

      14. cd /usr/local/src/opencore-amr-0.1.3
        make install

      15. Libtheora

      16. cd /usr/local/src/libtheora-1.1.1
        make install

      17. Fadd2

      18. cd /usr/local/src/faad2-2.7
        make install

      19. Faac

      20. cd /usr/local/src/faac-1.28
        make install

      21. MP4Box

      22. cd /usr/local/src/gpac
        make install

      23. mplayer

      24. cd /usr/local/src/mplayer/
        ./configure --enable-jpeg
        make install

      25. ffmpeg

      26. cd /usr/local/src/ffmpeg
        ./configure --enable-libmp3lame --enable-libvorbis --enable-shared --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-nonfree --enable-libtheora --enable-version3 --enable-gpl --enable-libxvid
        make install

      27. ffpmeg-php

      28. cd /usr/local/src/ffmpeg-php
        make install

      29. php.ini
      30. If you’re adding in the ffmpeg module to PHP (which this guide assumes you are), you need to add the extension to PHP. Again, this guide assumes you’re using PHP on Apache with CentOS 5.

        echo "" > /etc/php.d/ffmpeg.ini
        /etc/init.d/httpd restart

    15. ldconfig
    16. To ensure that all of the libraries are loaded and linkable by ffmpeg, etc, run:


    That’s it! There are a few tests you can run to ensure that this is actually installed properly. Here are the two most important.

    php -m | grep ffmpeg

    Apr 272012

    So, you just did something that broke your entire everything. You’ve tried restoring backups, or you don’t have any. You’ve tried correcting the cPanel userdata files, but for some reason, httpd.conf just isn’t changing. And then you notice this file httpd.conf,v – this beautiful file, that contains all of the changes you’ve ever made to httpd.conf, and has exactly the information you need in it. But how do you convert httpd.conf,v to httpd.conf? Using RCS!

    1. Make backups
    2. You got into this situation because you didn’t make enough backups, you silly little lamb, so let’s not make that mistake again.

      cp /usr/local/apache/conf/httpd.conf /usr/local/apache/conf/httpd.conf.broken
      cp /usr/local/apache/conf/httpd.conf /root
      cp /usr/local/apache/conf/httpd.conf\,v /root

    3. Find the revision you need
    4. The file is formatted with revision numbers, dates, author information, etc. It’ll look something like this.

      date 2012.; author root; state Exp;
      next 1.516;

      Here’s an example of a revision with actual data. Let’s say you accidentally deleted this vhost and you need it back. We’ll be using the revision from this code block in the following examples.

      @"Modified by /usr/local/cpanel/scripts/killvhost After removing vhosts"
      @a8725 32
      DocumentRoot /home/test/public_html
      ## User test # Needed for Cpanel::ApacheConf
      <IfModule mod_suphp.c>
      suPHP_UserGroup test test
      <IfModule concurrent_php.c>
      php4_admin_value open_basedir "/home/test:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
      php5_admin_value open_basedir "/home/test:/usr/lib/php:/usr/local/lib/php:/tmp"
      <IfModule !concurrent_php.c>
      <IfModule mod_php4.c>
      php_admin_value open_basedir "/home/test:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
      <IfModule mod_php5.c>
      php_admin_value open_basedir "/home/test:/usr/lib/php:/usr/local/lib/php:/tmp"
      <IfModule sapi_apache2.c>
      php_admin_value open_basedir "/home/test:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
      <IfModule !mod_disable_suexec.c>
      SuexecUserGroup onst onst
      CustomLog /usr/local/apache/domlogs/ "%{%s}t %I .\n%{%s}t %O ."
      CustomLog /usr/local/apache/domlogs/ combined
      Options -ExecCGI -Includes
      RemoveHandler cgi-script .cgi .pl .plx .ppl .perl

      Let’s take this apart a bit.
      1.365 – this is our revision number. This tells the RCS command to roll back to the version with this data. Location the correct revision to restore is the most important part of the process.

    5. Revert
    6. Now it’s time to go ahead and revert the files. Make sure you’re in /root and not overwriting your existing httpd.conf first.

      cd /root
      co -l -r1.365 httpd.conf

      This will ask you if you wish to replace the existing httpd.conf. Say yes. The /root/httpd.conf file will be overwritten with the RCS version from revision 1.356.

    7. Restore httpd.conf

    8. cp /root/httpd.conf /usr/local/apache/conf
      /etc/init.d/httpd restart

      Overwrite your existing httpd.conf file and restart Apache. That’s it!

      Final notes: since this is cPanel, the httpd.conf you just made will eventually get overwritten with the broken one from cPanel configs. This is just a stop-gap measure to get the sites up while you investigate the real problem. Good luck!

    Apr 232012

    Just a few quick commandlets. These will enable SpamAssassin on all existing e-mail accounts:

    for i in `mysql -uadmin -p\`cat /etc/psa/.psa.shadow\` psa -Ns -e "select concat(mail.mail_name,\"@\", as address from mail,domains,accounts where and order by address"`; do /usr/local/psa/bin/spamassassin -u $i -status true

    And enable the anti-virus scanner for inbound and outbound mail on all existing e-mail accounts:

    for i in `mysql -uadmin -p\`cat /etc/psa/.psa.shadow\` psa -Ns -e "select concat(mail.mail_name,\"@\", as address from mail,domains,accounts where and order by address"`; do /usr/local/psa/bin/mail -u $i -antivirus inout; done

    To delete all spam messages with a score of 5 or higher:

    for i in `mysql -uadmin -p\`cat /etc/psa/.psa.shadow\` psa -Ns -e "select concat(mail.mail_name,\"@\", as address from mail,domains,accounts where and order by address"`; do /usr/local/psa/bin/spamassassin -u $i -status true -hits 5 -action del; done


    Apr 132012

    Need to run 2 versions of memcached? I have no idea why you’d need to do this, but I had a request for it tonight, and all of the Google results for it are terrible. Here’s what you need to know.

    1. /etc/sysconfig/memcached2

    2. PORT="11212"

    3. /etc/init.d/memcached2

    4. #! /bin/sh
      # chkconfig: - 55 45
      # description: The memcached daemon is a network memory cache service.
      # processname: memcached
      # config: /etc/sysconfig/memcached
      # pidfile: /var/run/memcached/

      # Standard LSB functions
      #. /lib/lsb/init-functions

      # Source function library.
      . /etc/init.d/functions


      if [ -f /etc/sysconfig/memcached2 ];then
      . /etc/sysconfig/memcached2

      # Check that networking is up.
      . /etc/sysconfig/network

      if [ "$NETWORKING" = "no" ]
      exit 0


      start () {
      echo -n $"Starting $prog: "
      # insure that /var/run/memcached has proper permissions
      if [ "`stat -c %U /var/run/memcached`" != "$USER" ]; then
      chown $USER /var/run/memcached

      daemon --pidfile /var/run/memcached/ memcached -d -p $PORT -u $USER -m $CACHESIZE -c $MAXCONN -P /var/run/memcached/ $OPTIONS
      [ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcached2
      stop () {
      echo -n $"Stopping $prog: "
      killproc -p /var/run/memcached/ /usr/bin/memcached
      if [ $RETVAL -eq 0 ] ; then
      rm -f /var/lock/subsys/memcached2
      rm -f /var/run/

      restart () {

      # See how we were called.
      case "$1" in
      status memcached
      [ -f /var/lock/subsys/memcached2 ] && restart || :
      echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
      exit 1

      exit $?

    5. Configure the service to run on start-up

    6. chmod +x /etc/init.d/memcached2
      chkconfig memcached2 on

      That’s it!

    Apr 132012

    Google’s mod_pagespeed is an Apache module for performance optimization. It can be used to increase the speed of pages served through Apache. This module is compatible with cPanel and WHM’s Apache setup, and it’s pretty easy to install.

    1. Download the correct mod_pagespeed RPM for your architecture.
    2. mod_pagespeed has RPMs available for 32-bit and 64-bit systems. First, let’s download the RPM files.
      For 32-bit:

      cd /usr/local/src
      mkdir mod_pagespeed
      cd mod_pagespeed

      For 64-bit:

      cd /usr/local/src/
      mkdir mod_pagespeed

    3. Extract the necessary files.
    4. I’m going to assume a 64-bit system from here on out. If you’re on a 32-bit system, the extracted files exist in slightly different paths; adjust accordingly.

      rpm2cpio mod-pagespeed-beta_current_x86_64.rpm | cpio -idmv
      cp usr/lib64/httpd/modules/ /usr/local/apache/modules/
      chmod 755 /usr/local/apache/modules/

    5. Create pagespeed directories.

    6. mkdir -p /var/mod_pagespeed/{cache,files}
      chown nobody:nobody /var/mod_pagespeed/*

    7. Enable mod_deflate (required for mod_pagespeed).
    8. The location of your httpd source directory will vary depending on the version of Apache you have installed. The quickest way to find the correct location is tab completion.

      /usr/local/apache/bin/apxs -c -i /home/cpeasyapache/src/httpd-[tab]/modules/filters/mod_deflate.c

    9. Set up your mod_pagespeed configuration.
    10. Add the following to the Apache configuration files. The simplest way to do this is to create a new configuration file called /usr/local/apache/conf/pagespeed.conf and then include that using the cPanel include files. Place the following in pagespeed.conf:

      LoadModule pagespeed_module modules/

      # Only attempt to load mod_deflate if it hasn't been loaded already.
      <IfModule !mod_deflate.c>
      LoadModule deflate_module modules/
      <IfModule pagespeed_module>
      # Turn on mod_pagespeed. To completely disable mod_pagespeed, you
      # can set this to "off".
      ModPagespeed on

      # Direct Apache to send all HTML output to the mod_pagespeed
      # output handler.
      AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER text/html

      # The ModPagespeedFileCachePath and
      # ModPagespeedGeneratedFilePrefix directories must exist and be
      # writable by the apache user (as specified by the User
      # directive).
      ModPagespeedFileCachePath "/var/mod_pagespeed/cache/"
      ModPagespeedGeneratedFilePrefix "/var/mod_pagespeed/files/"

      # Override the mod_pagespeed 'rewrite level'. The default level
      # "CoreFilters" uses a set of rewrite filters that are generally
      # safe for most web pages. Most sites should not need to change
      # this value and can instead fine-tune the configuration using the
      # ModPagespeedDisableFilters and ModPagespeedEnableFilters
      # directives, below. Valid values for ModPagespeedRewriteLevel are
      # PassThrough, CoreFilters and TestingCoreFilters.
      # ModPagespeedRewriteLevel PassThrough

      # Explicitly disables specific filters. This is useful in
      # conjuction with ModPagespeedRewriteLevel. For instance, if one
      # of the filters in the CoreFilters needs to be disabled for a
      # site, that filter can be added to
      # ModPagespeedDisableFilters. This directive contains a
      # comma-separated list of filter names, and can be repeated.
      # ModPagespeedDisableFilters rewrite_images

      # Explicitly enables specific filters. This is useful in
      # conjuction with ModPagespeedRewriteLevel. For instance, filters
      # not included in the CoreFilters may be enabled using this
      # directive. This directive contains a comma-separated list of
      # filter names, and can be repeated.
      # ModPagespeedEnableFilters rewrite_javascript,rewrite_css
      # ModPagespeedEnableFilters collapse_whitespace,elide_attributes

      # ModPagespeedDomain
      # authorizes rewriting of JS, CSS, and Image files found in this
      # domain. By default only resources with the same origin as the
      # HTML file are rewritten. For example:
      # ModPagespeedDomain
      # This will allow resources found on to be
      # rewritten in addition to those in the same domain as the HTML.
      # Wildcards (* and ?) are allowed in the domain specification. Be
      # careful when using them as if you rewrite domains that do not
      # send you traffic, then the site receiving the traffic will not
      # know how to serve the rewritten content.

      # Other defaults (cache sizes and thresholds):
      # ModPagespeedFileCacheSizeKb 102400
      # ModPagespeedFileCacheCleanIntervalMs 3600000
      # ModPagespeedLRUCacheKbPerProcess 1024
      # ModPagespeedLRUCacheByteLimit 16384
      # ModPagespeedCssInlineMaxBytes 2048
      # ModPagespeedImageInlineMaxBytes 2048
      # ModPagespeedCssImageInlineMaxBytes 2048
      # ModPagespeedJsInlineMaxBytes 2048
      # ModPagespeedCssOutlineMinBytes 3000
      # ModPagespeedJsOutlineMinBytes 3000

      # Bound the number of images that can be rewritten at any one time; this
      # avoids overloading the CPU. Set this to 0 to remove the bound.
      # ModPagespeedImageMaxRewritesAtOnce 8

      # Settings for image optimization:
      # Jpeg recompression quality (0 to 100, -1 strips metadata):
      # ModPagespeedJpegRecompressionQuality -1
      # Percent of original image size below which optimized images are retained:
      # ModPagespeedImageLimitOptimizedPercent 100
      # Percent of original image area below which image resizing will be
      # attempted:
      # ModPagespeedImageLimitResizeAreaPercent 100

      # When Apache is set up as a browser proxy, mod_pagespeed can record
      # web-sites as they are requested, so that an image of the web is built up
      # in the directory of the proxy administrator's choosing. When ReadOnly is
      # on, only files already present in the SlurpDirectory are served by the
      # proxy.
      # ModPagespeedSlurpDirectory ...
      # ModPagespeedSlurpReadOnly on

      # The maximum URL size is generally limited to about 2k characters
      # due to IE: See
      # Apache servers by default impose a further limitation of about
      # 250 characters per URL segment (text between slashes).
      # mod_pagespeed circumvents this limitation, but if you employ
      # proxy servers in your path you may need to re-impose it by
      # overriding the setting here. The default setting is 1024
      # characters.
      # ModPagespeedMaxSegmentLength 250

      # Uncomment this if you want to prevent mod_pagespeed from combining files
      # (e.g. CSS files) across paths
      # ModPagespeedCombineAcrossPaths off

      # Explicitly tell mod_pagespeed to load some resources from disk.
      # This will speed up load time and update frequency.
      # This should only be used for static resources which do not need
      # specific headers set or other processing by Apache.
      # Both URL and filesystem path should specify directories and
      # filesystem path must be absolute (for now).
      # ModPagespeedLoadFromFile "" "/var/www/static/"

      # Enables server-side instrumentation and statistics. If this rewriter is
      # enabled, then each rewritten HTML page will have instrumentation javacript
      # added that sends latency beacons to /mod_pagespeed_beacon. These
      # statistics can be accessed at /mod_pagespeed_statistics. You must also
      # enable the mod_pagespeed_statistics and mod_pagespeed_beacon handlers
      # below.
      # ModPagespeedEnableFilters add_instrumentation

      # Uncomment the following line so that ModPagespeed will not cache or
      # rewrite resources with Vary: in the header, e.g. Vary: User-Agent.
      # ModPagespeedRespectVary on

      # This handles the client-side instrumentation callbacks which are injected
      # by the add_instrumentation filter.
      # You can use a different location by adding the ModPagespeedBeaconUrl
      # directive; see the documentation on add_instrumentation.
      <Location /mod_pagespeed_beacon>
      SetHandler mod_pagespeed_beacon

      # Uncomment the following line if you want to disable statistics entirely.
      # ModPagespeedStatistics off

      # This page lets you view statistics about the mod_pagespeed module.
      <Location /mod_pagespeed_statistics>
      Order allow,deny
      # You may insert other "Allow from" lines to add hosts you want to
      # allow to look at generated statistics. Another possibility is
      # to comment out the "Order" and "Allow" options from the config
      # file, to allow any client that can reach your server to examine
      # statistics. This might be appropriate in an experimental setup or
      # if the Apache server is protected by a reverse proxy that will
      # filter URLs in some fashion.
      Allow from localhost
      Allow from
      SetHandler mod_pagespeed_statistics

      # Page /mod_pagespeed_message lets you view the latest messages from
      # mod_pagespeed, regardless of log-level in your httpd.conf
      # ModPagespeedMessageBufferSize is the maximum number of bytes you would
      # like to dump to your /mod_pagespeed_message page at one time,
      # its default value is 100k bytes.
      # Set it to 0 if you want to disable this feature.

      ModPagespeedMessageBufferSize 100000

      # mod_pagespeed has the ability to collect statistics about page visits as
      # well as page, resource, and div location (see div_structure_filter)
      # referrals. These will eventually be used to speed up pages with at least
      # resource pre-fetch, if not Chrome's new pre-render, technology. See
      # We recommend enabling the div_structure filter if turning on statistics
      # collection below. Enabling the div_structure filter will increase the
      # effectiveness of pre-rendering prediction, because it will take into
      # account both URLs and page locations when aggregating user click through
      # behavior. To enable the div_structure filter, uncomment the appropriate
      # line below or add div_structure to the enabled filters at the top of this
      # configuration file.
      # Page /mod_pagespeed_referer_statistics lets you view the accumulated
      # referral statistics.
      # ModPagespeedCollectRefererStatistics on enables collection (default off)
      # ModPagespeedHashRefererStatistics obscures collected info (default off)
      # ModPagespeedRefererStatisticsOutputLevel can be changed if the page
      # /mod_pagespeed_referer_statistics is slow to load:
      # - Organized (default) is the most readable and ordered logically, but
      # involves computation
      # - Simple is readable but unordered
      # - Fast is the fastest and contains all necessary information, but is
      # fairly unreadable

      # ModPagespeedCollectRefererStatistics on
      # ModPagespeedHashRefererStatistics on
      # ModPagespeedRefererStatisticsOutputLevel Simple
      # ModPagespeedEnableFilters div_structure

      <Location /mod_pagespeed_message>
      Allow from localhost
      Allow from
      SetHandler mod_pagespeed_message
      <Location /mod_pagespeed_referer_statistics>
      Allow from localhost
      Allow from
      SetHandler mod_pagespeed_referer_statistics

      And then open /usr/local/apache/conf/includes/pre_main_global.conf and add:

      Include conf/pagespeed.conf

    11. Rebuild Apache config and restart apache.

    12. /scripts/buildhttpdconf
      /etc/init.d/httpd restart

    That’s it!