= mkstage4 ==

//mkstage4// is a bash script to backup a GNU/Linux installation. At the moment it's aimed for a [[http://gentoo.org|Gentoo GNU/Linux]] installation - although it can be customised easily for any other distribution as well. The resulting backup file is called a stage4 (meaning a full system backup).

== How it works ==

//mkstage4// works right out of the box and merely uses the following tools:

* cut 
* date 
* echo 
* find 
* grep 
* hostname 
* mount 
* sh 
* split 
* tar 
* umount 
* uname 
* which

It detects error/misconfiguration (wrong //$PATH//, wrong commands, wrong exclude list, overwriting existing files ...). After checking that everything is set up the script will provide you with four options:

<code> 
Backup script v3.5
==================
What do you want to do? (Use CONTROL-C to abort)
 
Fast (tar.gz):
(1) Minimal backup
(2) Interactive backup

Best (tar.bz2):
 (3) Minimal backup
 (4) Interactive backup

Please enter your option:
</code>

=== Minimal backup ===
This option allows you to quickly build a stage4.tar.bz2/stage4.tar.gz without any further fuss, which means

# it will exclude (i.e. NOT make a backup of) any files/directories listed in the //$custom_list// variable
# result in a minimal stage4.


=== Interactive backup ===

This option will ask you for any file/directory listed in the //custom_list// variable if you want it to be included in your backup. This might result in the same stage4.tar.bz2/stage4.tar.gz if you answer all question with "no" or in a considerably bigger stage4 file.


=== Parameters ===

The script only shows errors on stdout per default. If you want to run the backup in verbose mode and watch the files processed by tar, call the script like this:
    
<code>
 ./mkstage4.sh --verbose
 ./mkstage4.sh -v
</code>

If you intend to backup your system on a cd/dvd with a predefined size call the script like:

<code>
 ./mkstage4.sh --split
 ./mkstage4.sh -s
</code>

Don't forget to adjust the //$split_options// variable, i.o. to set the desired chunk size. Default is 685MB.

Of course you may combine command line parameters in any way.

== Features ==

* works right out of the box.
* checks if all the needed tools exist.
* checks that any file/folder listed exists (no spelling errors).
* names backups according to their hostname and the date it was created -> unique, meaningful file name.
* prevents the overwriting of a backup file created on the same day.
* runs an integrity check after the stage4 has been created.
* allows fine-tuning and easy change.
* verbose mode.
* command line option --split for splitting the tarball on the fly.


== Portability/Prerequisites ==

Against some misconceptions you may use your stage4 on a box with a different CPU as well. If - and only if - you plan to use your stage4 for another CPU type (not just another box with the same CPU), you must check your ///etc/make.conf// for the CFLAGS entry:

( < GCC-3.4 )
  CFLAGS="-Os -mcpu=athlon -funroll-loops -pipe"

( >= GCC-3.4):
  CFLAGS="-Os -mtune=athlon -funroll-loops -pipe"

If your CFLAGS looks like this:
  CFLAGS="-Os -march=athlon -funroll-loops -pipe"

you won't be able to use your stage4 on a box with a different CPU type. //march// breaks compatibility, //mcpu/mtune// doesn't. There is a simple workaround: replace "march" with //mcpu/mtune// and recompile your whole system:

<code>
# emerge -e world
</code>

== How to customise ==

As mentioned above there are several variables to customise. The script itself has a few comments which should get you started. You might want to change the location where the stage4.tar.bz2 is put (currently /mnt/backups/stage4), or you might not like the filename (hostname-stage4-date.tar.bz2), or you might think that the kernel sources are part of a minimal system backup and hence you won't exclude them (and so on) ...

//$default_exclude_list//: put any file/directory in here which is never needed nor wanted for a minimal system backup.

//$default_exclude_pattern//: exclude patterns for the //$default_include_folders/files//.

//$default_include_files//: files which are needed for a minimal working system. Don't add folders here which should be included in the backup recursively. These files provide solely the needed folder structure.

//$default_include_folders//: folders which need to be included in the backup recursively for a minimal working system.

//$custom_include_list//: directories which are not imperative for a working system but which may be desirable to be also saved by your backup interactively (like /home or /usr/src/).

//$custom_exclude_list//: files/folders which are subfolders of a folder listed in //$custom_include_list// which should NOT be included.

//$custom_exclude_pattern//: exclude patterns for the //$custom_include_list//.

== The result ==

=== stage4.tar.gz ===

You'll get a stage4 which will take about half the time it does for a bzip2 stage4. On my laptop it takes about 20 minutes to create the stage4, my amd64 with a software raid0 needs 8 minutes. Unpacking takes about 2/3 of the time. A stage4.tar.gz will be about 10% bigger than a stage4.tar.bzip2.

=== stage4.tar.bz2 ===

You'll get a stage4.tar.bz2 (mine is about 800MB generated from 3.5GB of data) which will be a fully working backup of your system. On my laptop it takes some 50 minutes to create the
bz2 stage4.

== The Script ==

<code bash>
#!/bin/bash

# Backup script for Gentoo Linux
#
# mkstage4.sh is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# mkstage4.sh is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# Copyright: Reto Glauser aka blinkeye
# Mailto: stage4 at blinkeye dot ch
# Homepage: http://blinkeye.ch
# Forum post: http://forums.gentoo.org/viewtopic-t-312817.html
# Date: 2009-04-02

version=v3.7
basename=`basename $0`

find=/usr/bin/find
tar=/bin/tar

# these are the commands we actually need for the backup
command_list=(cut date echo $find grep hostname mount sh split $tar umount uname which)

# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.
for command in ${command_list[@]}; do
        if [ ! -x "`which $command 2>&1`" ]; then
                echo -e "\nERROR: $command not found! "
                base=`basename $command`
                if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then
                        echo -e "ERROR: $base is not in your \$PATH."
                fi
                exit -1
        fi
done

help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"

# Defaults to creating one tarball
tar_output="--file"

# split command
split_options="--suffix-length=1 --bytes=685m"

# options for the tar command
tarOptions=" --preserve-permissions --create --absolute-names --totals --ignore-failed-read"

# where to put the stage4
stage4Location=/mnt/backups/stage4

# name prefix
stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d`

# patterns which should not be backed up (like iso files).
# example: default_exclude_pattern="*.iso *.divx"
# These pattern count only for files NOT listed in the $custom_include_list.
default_exclude_pattern=""

# these files/directories are always excluded. don't add trailing slashes.
# don't touch it unless you know what you are doing!
# /var/db and /var/cache/edb are intentionally added here. they are listed
# in $default_include_folders
default_exclude_list="
/dev
/lost+found
/mnt
/proc
/sys
/tmp
/usr/portage
/usr/src
/var/log
/var/tmp
/var/db
/var/cache/edb
$stage4Location
`echo $CCACHE_DIR`"

# files/devices/folders, which need to be backed up (preserve folder structure).
# don't touch it unless you know what you are doing! no recursive backup of folders.
# use $default_include_folders instead.
default_include_files="
/dev/null
/dev/console
/home
/mnt
`find /mnt -name .keep`
/proc
/sys
/tmp
/usr/portage
/usr/src
/var/log/emerge.log
/usr/src/linux-`uname -r`/.config"

# folders, which need to be backed up recursively on every backup.
# don't touch it unless you know what you are doing! the reason for this
# variable is that some users add /var to the $default_exclude_list. here
# we ensure that portage's memory is backed up in any case.
default_include_folders="
/var/db"

# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will
# include/exclude these files/folders depening on your answer.
custom_include_list="
/home/*
/usr/src/linux-`uname -r`"

# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT
# be backed up. eg.
#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"
custom_exclude_list=""

# Only files/folders within the $custom_include_list are checked against these patterns
# custom_exclude_pattern="*.mp3 *.iso"
custom_exclude_pattern=""

# the find_command
find_command="$find /*"

# don't backup anything which matches pattern listed in $default_exclude_pattern
for pattern in $default_exclude_pattern; do
        find_command="$find_command -not -name $pattern"
done

# assemble the find_command
function find_files()
{
        for folder in $default_exclude_list; do
                find_command="$find_command -path $folder -prune -o"
        done

        find_command="$find_command -print"

        for i in $default_include_files; do
                find_command="echo $i; $find_command"
        done

        for i in $default_include_folders; do
                if [ -d $i ]; then
                        find_command="$find $i; $find_command"
                else
                        find_command="echo $i; $find_command"
                fi
        done
}

# check the exclude/include variables for non-existing entries
function verify()
{
        for i in $1; do
                if [ ! -e "`echo "$i" | cut -d'=' -f2 | cut -d'*' -f1`" -a "$i" != "/lost+found" -a "$i" != "$stage4Location" ]; then
                        echo "ERROR: `echo "$i" | cut -d'=' -f2` not found! Check your "$2
                        exit 0
                fi
        done
}

# check input parameters
while [ $1 ]; do
        case  $1 in
        "-h" | "--help")
                echo -e $help
                exit 0;;
        "-v" | "--verbose")
                verbose=$1;;
        "-s" | "--split")
                tar_output="--split";;
        "");;
        *)
                echo -e $help
                exit 0;;
        esac
        shift
done

echo ""

# check folder/files listed in $default_exclude_list exist
verify "$default_exclude_list" "\$default_exclude_list"

# check files listed in $default_include_files exist
verify "$default_include_files" "\$default_include_files"

# check folder listed in $default_include_folders exist
verify "$default_include_folders" "\$default_include_folders"

#check folder listed in $custom_include_list exist
verify "$custom_include_list" "\$custom_include_list"

#check folder listed in $custom_exclude_list exist
verify "$custom_exclude_list" "\$custom_exclude_list"

# print out the version
 echo -e "\nBackup script $version"
 echo -e "=================="

# how do you want to backup?
echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n
Fast (tar.gz):
 (1) Minimal backup
 (2) Interactive backup


Best (tar.bz2):
 (3) Minimal backup
 (4) Interactive backup\n"

while [ "$option" != '1' -a "$option" != '2' -a "$option" != '3' -a "$option" != '4' ]; do
        echo -en "Please enter your option: "
        read option
done

case $option in
[1,3])
        stage4Name=$stage4Location/$stage4prefix-minimal.tar;;

[2,4])
        stage4Name=$stage4Location/$stage4prefix-custom.tar

        for folder in $custom_include_list; do
                echo -en "\nDo you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "
                read answer
                while [ "$answer" != 'y' -a "$answer" != 'n' ]; do
                        echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "
                        read answer
                done
                if [ "$answer" == 'n' ]; then
                        find_command="$find_command -path $folder -prune -o"
                else
                        custom_find="$find $folder"
                        for i in $custom_exclude_pattern; do
                                custom_find="$custom_find -name $i -o"
                        done
                        for i in $custom_exclude_list; do
                                custom_find="$custom_find -path $i -prune -o"
                        done
                        find_command="$custom_find -print; $find_command"
                fi
        done ;;
esac

# add $custom_include_list to the $default_exclude_list as we assembled
# $custom_find with $custom_include_list already.
default_exclude_list="$default_exclude_list $custom_include_list"

case $option in
[1,2])
        stage4postfix="gz"
        zip="--gzip";;

[3,4])
        stage4postfix="bz2"
        zip="--bzip2";;
esac

# mount boot
echo -e "\n* mounting boot"
mount /boot >/dev/null 2>&1

# find the files/folder to backup
find_files
find_command="($find_command)"

# create the final command
if [ "$tar_output" == "--file" ]; then
        tar_command="$find_command | $tar $zip $tarOptions $verbose --file $stage4Name.$stage4postfix --no-recursion -T -"
else
        tar_command="$find_command | $tar $zip $tarOptions $verbose --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"
fi

if [ "$verbose" ]; then
        echo -e "\n* creating the stage4 in $stage4Location with the following command:\n\n"$tar_command
fi

# everything is set, are you sure to continue?
echo -ne "\nDo you want to continue? (y/n) "
read answer
while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
        echo -ne "Do you want to continue? (y/n) "
        read answer
done

if [ "$answer" == 'y' ]; then
        # check whether the file already exists.
        if [ "$tar_output" == "--split" ]; then
                overwrite="`ls "$stage4Name.$stage4postfix"_* 2>&1 | grep -v 'No such file'`"
        else
                overwrite="$stage4Name.$stage4postfix"
        fi

        if [ -a "`echo "$overwrite" | grep "$overwrite" -m1`" ]; then
                echo -en "\nDo you want to overwrite $overwrite? (y/n) "
                read answer
                while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
                        echo -en "Do you want to overwrite $overwrite? (y/n) "
                        read answer
                done
                if [ "$answer" == 'n' ]; then
                        echo -e "\n* There's nothing to do ... Exiting"
                        exit 0;
                fi
        fi

        # if necessary, create the stage4Location
        if [ ! -d "$stage4Location" ] ; then
                echo "* creating directory $stage4Location"
                mkdir -p $stage4Location
        fi

        echo -e "\n* Please wait while the stage4 is being created.\n"

        # do the backup.
        sh -c "$tar_command"

        # finished, clean up
        echo -e "\n* stage4 is done"
        echo "* umounting boot"
        umount /boot >/dev/null 2>&1

        # Integrity check
        echo -e "* Checking integrity"
        if [ "$zip" == "--gzip" ]; then
                zip="gzip"
        else
                zip="bzip2"
        fi

        if [ "$tar_output" == "--split" ]; then
                if [ "`cat "$stage4Name.$stage4postfix"_*"" | $zip --test 2>&1`" != "" ]; then
                        echo -e "* Integrity check failed. Re-run the script and check your hardware."
                        exit -1
                fi
        else
                if [ "`$zip --test  $stage4Name.$stage4postfix 2>&1`" != "" ]; then
                        echo -e "* Integrity check failed. Re-run the script and check your hardware."
                        exit -1
                fi
        fi

        # everything went smoothly
        echo -e "* Everything went smoothly. You successfully created a stage4."

else
        echo -e "\n* There's nothing to do ... Exiting"
fi
</code>

== Run it ==

# chmod +x  mkstage4.sh
# ./mkstage4.sh

This will execute the script (you must be root to succesfully backup all folders).

For available parameters see section [[mkstage4#Parameters]].

== Automating ==
If you like to run this script from a cron job create a file with the commands you have to enter to get the stage4 you want:

<code>
1
y
</code>

Just write down the exact commands you enter if you do the backup yourself. Save the file and call the script 

<code>
$ mkstage4.sh < file
</code>


== Restore ==

# boot off a live-cd and repartition your harddisks and create filesystems as necessary (make sure you remove all remaining files of your to-be-intallation).
# eventually reboot, using option: gentoo docache
# # umount /mnt/cdrom
# remove the live-cd and insert the cd with the stage4
# # mount /dev/cdrom /mnt/cdrom
# # mount /dev/hdaX /mnt/gentoo
# # mkdir /mnt/gentoo/boot
# # mount /dev/hdaX /mnt/gentoo/boot
# # tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/   <br>or
# # tar xjvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.bz2 -C /mnt/gentoo/
# # mount -t proc none /mnt/gentoo/proc
# # mount -o bind /dev /mnt/gentoo/dev
# # chroot /mnt/gentoo /bin/bash
# # env-update
# # source /etc/profile
# if in need adjust necessary files (/etc/fstab, /boot/grub/grub.conf) and/or install grub
# # emerge sync      (rebuild portage tree)
# # exit
# # cd /
# # umount /mnt/cdrom
# # remove backup cd
# # umount /mnt/gentoo/boot
# # umount /mnt/gentoo/dev
# # umount /mnt/gentoo/proc
# # umount /mnt/gentoo
# # reboot


== Re-assemble split-ed stage4 ==

If you split your stage4 either with the command line parameter //--split/-s// or uncommented the split section within the script you need to re-assemble the split chunks if you intend to restore such a stage4:

<code>
# cat stage4.tar.gz_* > stage4.tar.gz
</code>

or

<code>
# cat stage4.tar.bz2_* > stage4.tar.bz2
</code>

substitute stage4.tar.gz/stage4.tar.bz2 with your stage4 name.



== Changelog ==

====2005-03-23====

* fixed split command

====2005-03-27====

* added new option for faster backup (gzip)

====2005-03-28====

* Added ///dev/console// ///dev/null// to be backed up - gets rid of those "Unable to open initial console" messages.

====2005-03-27====

* added /dev/console, /dev/null to both final_command variables

====2005-03-31====

* fixed issue of included files/folder from excluded parent folder

====2005-04-15====

* fixed typ (zip -> gzip) in command_list

====2005-04-20====

* Added new //$exclude_pattern// variable thanks to a hint by //saskatchewan46//

====2005-04-24====

* The stage4 is compressed in-place -> no more temporary additional space is needed. It's faster now

====2005-04-28====

* Fixed command (bzip2 options now create a bzip2 stage4 again). Small change in the code

====2005-04-30====

* Fixed Restore commands. Thanks to Alpo Nestori

====2005-05-08====

* Removed warning about the /boot/boot symlink. No longer necessary

====2005-05-09====

* Added //Warning// section (//$default_include_list// doesn't recursively perform a backup of folders)

====2005-05-09====

* Fixed issue with $default_include_list, i.e. added new variable $custom_include_list

====2005-05-15====

* include /tmp without its content to preserve folder structure. Thanks to Lorijho. Verifying custom_include_list for wrong entries

====2005-06-07====

* Added integrity check of the stage4. Changed default filename to Year-Month-ay for easier listing

====2005-06-08====

* Fixed issue about not preserving all permissions

====2005-06-11====

* Fixed confusing variable names
* Removed old debug echo statement
* Split $default_include_list in $default_include_files and $default_include_folders
* Added portage's memory (/var/db) to $default_include_folders to prevent any mistakes
* Added portage's /var/cach/edb to $default_include_folders
* Included files/folders now take precedence of excluded files/folders
* Not existing files/folders in any list exits the script as it could be ignored too easily
* Added ftp link to the script for those having issues copy & pasting it
* Updated this Howo - explained some variables

====2005-06-13====

* Added //$custom_include_list//, //$custom_exclude_list//, //$custom_exclude_pattern// to provide the abilty of backing up directories interactively while excluding files/folders deeper in the tree

====2005-06-14====

* Added /var/db and /var/cache/edb to the //$default_exclude_list// to prevent backing it up twice
* Removed udev warning sections as there seem to be no futher issues
* Added verbose mode to the script. Default mode is now non-verbose, i.e. only errors are shown. For verbose mode use "mkstage4.sh --verbose" or "mkstage4.sh -v"
* Hardcoded tar and find. Added routine for suggestion path if tar and/or find could not be found
* Updated Howto. New //Features// and //Run it// sections 

====2005-06-15====

*Added Portability/Prerequesites section

====2005-06-21====

* Added new parameter --split and new Parameters section 

====2005-06-30====

* Fixed issue with command line parameters and mutual exclusion. --split and --verbose or -s and -v may now be combined in any order
* Fixed integrity check for splitted files
* Removed /var/cache/edb as this is only cache and portage does not need it to be backed up
* Removed /var/log/portage per default as new users get confused about adjusting the script
* Added check of needed commands at the beginning

====2005-11-14====

* Layout changes. 
* Removed unnecessary mkdir commands from the Restore section (thanks to [[http://forums.gentoo.org/profile.php?mode=viewprofile&u=121896&sid=de6734e84166a2ff43953eaf67b8713d|-valheru-)]].
* Added command to keep mountpoints. Thanks to [[http://forums.gentoo.org/profile.php?mode=viewprofile&u=14121|Hagar]].

====2009-04-13====

* Ported tutorial from Mediawiki to Dokuwiki