Whitelisting mod_security rules

 cPanel, Fixing Shit, Linux, Shell  No Responses »
Aug 032012
 

Apache’s mod_security is a system that scans URL data and data POSTed to the server in forms for malicious attacks such as XSS, SQL injections, etc. It comes with a default ruleset that always trips up a number of standard AJAX functions in applications like WordPress and Joomla, necessitating that rules be removed from the mod_security configuration to allow the site to function properly. The method of removing these rules under Apache 2 is by whitelisting the individual rules in the VHost entry for the domain. Here’s how to do it in cPanel.

Is mod_security enabled?

root@server [~]# httpd -M | grep sec
security2_module (shared)

Yep.

A typical mod_security error message

[Sun Jul 29 18:29:47 2012] [error] [client 1.2.3.4] ModSecurity: Access denied with code 403 (phase 2). Match of "rx (?:/event\\\\.ng/|horde/services/go\\\\.php|tiki-view_cache\\\\.php|^/\\\\?out=http://|homecounter\\\\.php\\\\?offerid=.*ureferrer=http|__utm\\\\.gif\\\\?|/plugins/wpeditimage/editimage\\\\.html|/spc\\\\.php)" against "REQUEST_URI" required. [file "/usr/local/apache/conf/modsec/50_asl_rootkits.conf"] [line "53"] [id "390145"] [rev "10"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Rootkit attack: Generic Attempt to install rootkit"] [data "=http://www.hackers-web-site.com/what/e.txt?"] [severity "CRITICAL"] [hostname "www.domain.com"] [uri "/index.php"] [unique_id "UBVB02B-nBoACs9n0p4AAAAH"]

The error log for mod_security is /usr/local/apache/logs/error_log. There are 2 important pieces of information contained in this error:

  • [id "390145"]
    This is the mod_security rule ID number. We will use this to tell mod_security which rule we want to whitelist.

  • [hostname "www.domain.com"]
    This is the name of the VirtualHost under which the error was generated. This is important because whitelisting a mod_security rule for a domain doesn’t whitelist it for the whole account; subdomains, addon domains, etc all have separate whitelists.

Creating the userdata includes

mkdir -p /usr/local/apache/conf/userdata/std/2/user/domain.com

root@server [~]# mkdir -p /usr/local/apache/conf/userdata/std/2/user/domain.com

In the above example, we are setting up an Inside VHost include. The italic std indicates this is being done in a port 80 (standard/non-secure) VirtualHost, the bold user is the cPanel username of the account, and the bold and italic domain.com is the hostname from the error message above (we discard www because that’s just a ServerAlias under the same VHost). Once we create the userdata include directory, it’s time to set up the configuration file to include. We’ll use nano because it’s the best text editor.

nano -w /usr/local/apache/conf/userdata/std/2/user/domain.com/domain.com.conf

root@server [~]# /usr/local/apache/conf/userdata/std/2/user/domain.com/domain.com.conf

You can name the file anything as long as it ends in .conf; I prefer to use domain.com.conf for clarity and simplicity. Now let’s whitelist some mod_security rules.

SecRuleRemoveById 390145

Notice that we used 390145 from the error message above. If this site was generating errors from multiple IDs, we’d put in multiple SecRuleRemoveById lines.

Enabling the userdata includes in Apache conf

/scripts/ensure_vhost_includes --user=user

This uncomments a line in the VirtualHost entry in httpd.conf for this domain to include any .conf files in /usr/local/apache/conf/userdata/std/2/user/domain.com/

/usr/local/cpanel/bin/apache_conf_distiller --update
/usr/local/cpanel/bin/build_apache_conf

These compile and distill the changes into Apache’s configuration.

Testing

The first thing to check is to ensure that the following line in the VirtualHost entry for domain.com is uncommented:

Include "/usr/local/apache/conf/userdata/std/2/user/domain.com/*.conf"

The only other way to test this is to reproduce the issue and tail the Apache error log. Make sure you have reproduction instructions available; many scripts will trigger more than one SecRule, but you’ll only see the first one due to the request being blocked.

RCS, or, oh my god how do I just get my old httpd.conf back

 cPanel, Fixing Shit, Shell  No Responses »
Apr 272012
 

So, you just did something that broke your entire everything. You’ve tried restoring backups, or you don’t have any. You’ve tried correcting the cPanel userdata files, but for some reason, httpd.conf just isn’t changing. And then you notice this file httpd.conf,v – this beautiful file, that contains all of the changes you’ve ever made to httpd.conf, and has exactly the information you need in it. But how do you convert httpd.conf,v to httpd.conf? Using RCS!

  1. Make backups

    2. You got into this situation because you didn’t make enough backups, you silly little lamb, so let’s not make that mistake again.

    cp /usr/local/apache/conf/httpd.conf /usr/local/apache/conf/httpd.conf.broken
    cp /usr/local/apache/conf/httpd.conf /root
    cp /usr/local/apache/conf/httpd.conf\,v /root

  2. Find the revision you need

    3. The file is formatted with revision numbers, dates, author information, etc. It’ll look something like this.

    1.517
    date 2012.04.11.10.27.03; author root; state Exp;
    branches;
    next 1.516;

    Here’s an example of a revision with actual data. Let’s say you accidentally deleted this vhost and you need it back. We’ll be using the revision from this code block in the following examples.

    1.365
    log
    @"Modified by /usr/local/cpanel/scripts/killvhost After removing vhosts"
    @
    text
    @a8725 32
    <VirtualHost 1.2.3.4:80>
    ServerName test.com
    ServerAlias www.test.com
    DocumentRoot /home/test/public_html
    ServerAdmin webmaster@@test.com
    ## User test # Needed for Cpanel::ApacheConf
    <IfModule mod_suphp.c>
    suPHP_UserGroup test test
    </IfModule>
    <IfModule concurrent_php.c>
    php4_admin_value open_basedir "/home/test:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
    php5_admin_value open_basedir "/home/test:/usr/lib/php:/usr/local/lib/php:/tmp"
    </IfModule>
    <IfModule !concurrent_php.c>
    <IfModule mod_php4.c>
    php_admin_value open_basedir "/home/test:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
    </IfModule>
    <IfModule mod_php5.c>
    php_admin_value open_basedir "/home/test:/usr/lib/php:/usr/local/lib/php:/tmp"
    </IfModule>
    <IfModule sapi_apache2.c>
    php_admin_value open_basedir "/home/test:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
    </IfModule>
    <IfModule !mod_disable_suexec.c>
    SuexecUserGroup onst onst
    </IfModule>
    CustomLog /usr/local/apache/domlogs/test.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
    CustomLog /usr/local/apache/domlogs/test.com combined
    Options -ExecCGI -Includes
    RemoveHandler cgi-script .cgi .pl .plx .ppl .perl
    </VirtualHost>
    @

    Let’s take this apart a bit.
    1.365 – this is our revision number. This tells the RCS command to roll back to the version with this data. Location the correct revision to restore is the most important part of the process.

  3. Revert

    4. Now it’s time to go ahead and revert the files. Make sure you’re in /root and not overwriting your existing httpd.conf first.

    cd /root
    co -l -r1.365 httpd.conf

    This will ask you if you wish to replace the existing httpd.conf. Say yes. The /root/httpd.conf file will be overwritten with the RCS version from revision 1.356.

  4. Restore httpd.conf


    5. cp /root/httpd.conf /usr/local/apache/conf
    /etc/init.d/httpd restart

    Overwrite your existing httpd.conf file and restart Apache. That’s it!

    Final notes: since this is cPanel, the httpd.conf you just made will eventually get overwritten with the broken one from cPanel configs. This is just a stop-gap measure to get the sites up while you investigate the real problem. Good luck!

Increasing cPanel /tmp size

 cPanel, Fixing Shit, Linux, Shell  No Responses »
May 072011
 

By default, cPanel sets up a paltry <512MB /tmp disk. This isn't nearly large enough for large file uploads or other disk-intensive purposes. While it's better to use RAM for temporary storage, sometimes you need a place to dump huge files (such as uploads). Luckily, raising the /tmp disk size has been fairly simple. cPanel's script to secure the /tmp partition against drop-in hacks by making it noexec can also resize the /tmp directory.

The file we'll be modifying is:


/scripts/securetmp

Until recently, modifying one variable was enough to change this, but it seems like there was a change recently that has caused that method to no longer remount /tmp properly. Luckily, the fix for this is two additional small changes to the file.

Let's open up /scripts/securetmp in your favorite editor:


nano /scripts/securetmp

First, we're going to modify line 49:


my $auto = 1

If this isn't already set to 1, set it. Just makes things easier. Next, let's set the /tmp size, line 148:


my $tmpdsksize = 2097152;

This size is in KB - 2GB aught to do it. Now, to fix the issue of mounting /tmp, line 289:


system 'mount', '-o', $mountkeyword . ',loop,noexec,nosuid', $tmpmnt, '/tmp';

We're adding "loop," to the options passed to the mount command to ensure that the system understands /tmp is a loopback device being created on /usr/tmpDSK. Save and exit your file.

Next, we need to shut off anything using /tmp:


/etc/init.d/mysql stop
/etc/init.d/httpd stop

And unmount it and /var/tmp:


umount /tmp
umount /var/tmp

If you get errors, retry a few times, it'll usually unmount after the 2nd or 3rd try. If you're still getting errors, make sure nothing is open in /tmp:


lsof | grep /tmp

Shut it down or delete it. Next, we need to remove the existing /tmp partition file:


rm -f /usr/tmpDSK

And finally, create the new device:


/scripts/securetmp

Depending on the size of your partition, this may take up to 15-20 minutes. After you're done, start everything back up and ensure /tmp is mounted and the right size with a simple:


df -h

Installing New Plesk Components on Linux

 Fixing Shit, Installing Shit, Plesk, Shell  No Responses »
Apr 092011
 

Installing New Components

Ever been in a Plesk box but can’t find certain components, like ASP support, or the backup manager? This is because Plesk installs whatever you tell it to at the initial install, and only whatever you tell it to. This leads to a lot of missing components that you might be used to having access to. Plesk won’t tell you anything in the GUI except that the component is not installed, so you must hunt down this binary:

/usr/local/psa/admin/bin/autoinstaller

This is the Plesk Autoinstaller, which will allow you, through its byzantine menu corridors, to find the gold that is the component you need. Navigating it is fairly simple, although you’ll probably want to read all of the text on the screen if this is your first time.

However, a more sinister condition awaits you – the fact that the Plesk autoinstaller apparently doesn’t know how to fucking resolve rpm dependencies. Why is it called an autoinstaller if it doesn’t automatically install anything extra? I don’t know.

Resolving Plesk Component Dependencies

Luckily, it’s not that difficult to resolve RPM dependency errors. Did you get an error message that looks something like this?


Retrieving information about the installed packages...
File downloading PSA_9.3.0/dist-rpm-CentOS-5-x86_64/build-9.3.0-cos5-x86_64.hdr.gz: 10%..20%..30%..40%..50%..60%..70%..80%..90%..100% was finished.
File downloading PSA_9.3.0/update-rpm-CentOS-5-x86_64/update-9.3.0-cos5-x86_64.hdr.gz: 10%..20%..30%..40%..50%..60%..70%..80%..90%..100% was finished.
File downloading PSA_9.3.0/thirdparty-rpm-CentOS-5-x86_64/thirdparty-9.3.0-cos5-x86_64.hdr.gz: 13%..25%..31%..40%..54%..60%..72%..83%..95%..100% was finished.
Determining the packages that need to be installed.
ERROR: Unable to install the "psa-backup-manager-9.3.0-cos5.build93091230.06.x86_64" package.
Not all packages were installed.
Please, contact product technical support.
[root@host2 plesk]#

Not to worry! First, try Google to find that RPM – Plesk allows their FTP server to be directory indexed, so it shouldn’t be hard to find the exact RPM it’s erroring out on. Try downloading the .rpm file and installing it manually using rpm -i:

wget http://autoinstall.swsoft.com.cn/PSA_9.3.0/dist-rpm-CentOS-5-x86_64/opt/backup/psa-backup-manager-9.3.0-cos5.build93091230.06.x86_64.rpm
rpm -i psa-backup-manager-9.3.0-cos5.build93091230.06.x86_64.rpm

If all goes well, you’ll get output messages listing the failed dependencies. Ignore any Plesk-looking dependencies, the auto-installer actually fixes these. Use yum to install any other system packages you may need.

Troubleshooting Slow mySQL Performance with mtop

 Fixing Shit, Linux, Shell  No Responses »
Apr 082011
 

mySQL is the database backbone behind a wide variety of software programs and scripts, but most people don’t know how to code for it properly. As such, mySQL-related overloads and optimization of sub-par mySQL databases can take a lot of time. The easiest way to track and troubleshoot troublesome threads is with mtop – live mySQL query querying.

Prerequisites

mtop requires curses. In CentOS, this is as simple as a few yum commands. There are a few extra steps for cPanel compatibility.

  1. Install curses and curses-devel:

    yum install curses curses-devel

  2. Install curses-Perl:

    yum install curses-Perl

    1. If you’re on cPanel, you’ll first need to remove the perl* exclude from yum.conf.

      nano /etc/yum.conf

      In the exclude= line, remove “perl*” – save (CTRL+O) and exit (CTRL+X). Be sure to add it back after installing curses-Perl as per the above command.

Install mtop

Now it’s time to fetch the latest version of mtop and compile it. The homepage is here if you wish to check for the latest version. As of this post, we will be using 0.6.6, the latest version.

  1. Grab the mtop source:

    wget http://softlayer.dl.sourceforge.net/project/mtop/mtop/v0.6.6/mtop-0.6.6.tar.gz

  2. Configure, make, and install:

    tar -zxf mtop-0.6.6.tar.gz
    cd mtop-0.6.6
    perl Makefile.PL
    make
    make install

Configure and run

By default, mtop doesn’t connect as the root user. For most server installations, we want to connect as root. Here’s the quickest and easiest way to run mtop.

  1. Find your current mySQL root password:

    cat /root/.my.cnf

  2. Copy the root password (and hostname, if different than localhost) and run the command as follows:

    mtop --host=YOURHOSTNAME --dbuser=root --password=YOURPASSWORD --seconds=1

Adding a New mySQL Access Host

 cPanel, Fixing Shit, Linux  No Responses »
Apr 082011
 

So you’ve migrated to a new database server, or a new web server, or you’ve added a new database server to your cluster. The problem is, all of your mySQL users only have grants on the old server. Luckily, the guys at cPanel know your pain and have added an easy way to make sure all of your users have access to their databases.

Add the new hostname

  1. Back up the “mysql” database before making ANY changes.

  2. Log in to WHM as root
  3. Navigate to SQL Services->Additional SQL Access Hosts
  4. Put the new IP address that you’ll be accessing FROM in the box

If you’ve got a deadline, you can add the host % here – this will allow access to mySQL from all hosts. SQL authentication is still required, so you’re not exposing any passwords, but if your clients use insecure mySQL usernames, passwords, and database prefixes, this could still leave you open to a huge SQL injection vulnerability risk, so it is highly not recommended.

Add the grants

  1. Navigate back to the Additional SQL Access Hosts page, and at the bottom of the page, click the link labeled “click here.”
  2. Wait for the process to finish and confirm access.

This quick-and-dirty method will ensure your users will still be able to authenticate using their same passwords, and saves a lot of nasty mySQL batch work.