##################
Compromised System
##################


Abstract
========

The following is a checklist of tasks to perform when a hosting system has been compromised, to ensure you have all the appropriate data to recover the system and ensure that it will not be compromised again. A key to rapid recovery is to use AO to minimize the forensic investigation time required to recover. Ideally the specific exploits should be identified in advance, however given time constraints this might not be possible until later. The goal of this checklist is Rapid Recovery.

----------

Prerequisites
=============

   * Backup server, to store 2 copies of data from the compromised system.
   
   * 1 valid AO subscription
   
   
   
--------------

Step 1) Find out how the system was compromised
===============================================

First determine the level of compromise. Is this a full compromise, or just an individual site.

   * Start with your desktop and the desktop of anyone who accesses your system(s) as a privileged user. About 1/3 of all our forensic cases originate here. Of that 1/3, most of the time users believe that the desktop has not been compromised, and unfortunately most of the time it is!

     Please dont do that to yourself, time is of the essence, assume the desktop has been compromised. Its the easiest way to compromise a system, just steal the credentials on the desktop and log in!


   \
   
   * general rootkit detection (note these tools are LIMITED. They are best used for initial inspections, they will miss a lot)
   
      * Using Rkhunter, run the following commands
	  
           .. code-block:: console
		   
              rkhunter --update
              rkhunter -c -sk		   
			  
			  
      \
	  
      * Using Chrootkit, run the following commands
	  
           .. code-block:: console
		   
              chkrootkit
			  
			  
   \
   
   * Look for suspicious processes.
   
   \
   
   * Look for suspicious files.
   
   \
   
   * Create snapshots of memory
   
   \
   
   * Boot system from trusted media (CD, PXE, etc)
   
   \
   
   * Run "trusted" versions of chkrootkit and rkhunter against compromised drives
   
   \
   
   * Investigate the following logfiles:
   
      .. code-block:: console
	  
         /var/log/messages
         /var/log/secure
		 
   \
   
   * Read the :doc:`Compromised System FTP </AEO/faq/compromisedSystemsFTP>` article.
   
   
   \
   
   * Confirm that your backups have not been compromised. Dont restore from a backup until you know you can trust it.
   
   \
   
   * Image the compromised system if you can, but don't trust anything it tells you.
   
   
------------

Step 2) Back up data from the compromised host. We make 2 copies
================================================================

   * Rsync back of compromised host from the backup server (it is because migration tools amost always miss something. This task will give you a complete copy of the old system)
   
         .. code-block:: console
		 
            rsync -av -e ssh root@<IP>:/ /var/backups/<IP>/
			
			
   \
   
   * On the compromised system, create a backup:
   
         .. code-block:: console
		 
            mkdir /root/backups
		 
     **PSA 7.5 and Lower:**

         .. code-block:: console

            /usr/loca/psa/bin/psadump -f | split -b1000m /root/backups/backup.
			
			
     **PSA 8.0 and Higher:**
	  
         .. code-block:: console
		 
            /usr/local/psa/bin/pleskbackup all --split=1G /root/backups/backup
		 

   \

   * Rsync back of compromised host from the backup server (this gets those backups too):  
   
      .. code-block:: console
	  
         rsync -av -e ssh root@<IP>:/ /var/backups/<IP>/
	  
----------

Step 3) Reinstall the system
============================

   * Re-image the system
   
      .. code-block:: console
	  
         wget -q -O - https://www.atomicorp.com/installer/aooi |sh


   \

   * Update the system by running the following command

      .. code-block:: console

         yum -y update


   \

   * Install/Configure Atomic Secured Linux run the following commands:

      .. code-block:: console

         wget -q -O - https://www.atomicorp.com/installers/awp |sh

         aum -u

         awp -s -f 
		 
		 
   \
   
   * Install Plesk by running the following command:
   
      .. code-block:: console
	  
         wget -q -O - https://www.atomicorp.com/installer/atomic |sh

   \
   
   * Install PSA and supported packages by running the following command:
   
      .. code-block:: console
	  
         yum -y install psa psa-bu mailman psa-spamassassin frontpage
	  
   \
   
   * Copy psa.key from rsync backup on the backup server to **/etc/psa/psa.key** on the new system by running the following command:
   
      .. code-block:: console
	  
         scp /backup/<IP>/etc/psa/psa.key  root@<IP>:/etc/psa/psa.key
	
   \

   * Restart PSA by running the following command:

      .. code-block:: console

         /etc/init.d/psa restart


   \

   * Log into PSA, and reconfigure settings. Specifically set the shared IP's.

      .. code-block:: console

         https://<IP>:8443
	  

--------------

Step 5) Restore system
======================

   * Copy plesk backup to the re-imaged system:
   
      .. code-block:: console
	  
         scp /var/backups/<IP>/root/backups/* root@<IP>:/root/

		 
   \
   
   * Use psastore/pleskstore to recover data:
   
      .. code-block:: console
	  
         /usr/local/psa/bin/pleskrestore
