Wednesday, May 18, 2011

Creating a Local Ubuntu Mirror using apt-mirror

The pre-requisites:
  1. Have a running Ubuntu installation.
Installation:
  • Type the command below in the terminal:
    sudo apt-get install apt-mirror
  • or install the package "apt-mirror" from Synaptic if you prefer a GUI or you can alternatively use aptitude from the command line.
Configuration:
  • Edit the configuration file /etc/apt/mirror.list file. You might find the lines below by default:

    ############# config ##################
    #
    # set base_path    /var/spool/apt-mirror
    #
    # set mirror_path  $base_path/mirror
    # set skel_path    $base_path/skel
    # set var_path     $base_path/var
    # set cleanscript $var_path/clean.sh
    # set defaultarch  
    # set postmirror_script $var_path/postmirror.sh
    # set run_postmirror 0
    set nthreads     20
    set _tilde 0
    #
    ############# end config ##############
    
    deb http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
    #deb http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
    #deb http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse
    
    deb-src http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
    deb-src http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
    deb-src http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
    #deb-src http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
    #deb-src http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse
    
    clean http://archive.ubuntu.com/ubuntu
    
    
  • This may be rewritten as:

    ############# config ##################
    #
    set base_path    /home/myusername/apt-mirror
    set mirror_path  $base_path/mirror
    set skel_path    $base_path/skel
    set var_path     $base_path/var
    set cleanscript $var_path/clean.sh
    set defaultarch  i386
    set postmirror_script $var_path/postmirror.sh
    set run_postmirror 0
    set nthreads     20
    set _tilde 0
    #set limit_rate 2.5k
    #
    ############# end config ##############
    
    
    deb http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
    deb http://archive.canonical.com/ubuntu maverick partner
    deb http://extras.ubuntu.com/ubuntu maverick main
    
    
    clean http://archive.ubuntu.com/ubuntu
    clean http://archive.canonical.com
    clean http://extras.ubuntu.com
    

What I have done in this configuration:
  • base_path: The place where are the downloaded packages go.
  • mirror_path,skel_path,var_path,cleanscript,postmirror_script,run_postmirror,_tilde: Better leave them as they are.
  • defaultarch: When you specify "deb <deb-url>" it'll fetch the URL of this type, or you can explicitly specify which architecture packages to pick up by specifying it as "deb-i386 <deb-url>"
  • nthreads: The number of threads of wget that have to be launched for fetching the mirror contents
  • limit_rate: This variable can be set to limit the download rate of the wget but this is per thread. So it boils down to (20 Threads)*2.5KBps = 50 KBps (Notice the 'B' it's for bytes).
  • I just picked up the deb URL's from my file "/etc/apt/sources.list".
  • I skipped the "deb-src" url's because I wouldn't be needing the sources and it'll just eat up my space. It all depends on the need of  the mirror if you need 'em just include them in the package.
  • deb-urls: These are of the form "deb http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse". Replace maverick with the codename that you want to mirror, maverick is the codename for Ubuntu 10.10. You can find out which version you are using by the command "lsb_release -a". If you just need the codename use "lsb_release -c". You can mirror any number of archives.
Start Mirroring:
  • I would recommend mirroring the archive as a normal user than as a root user using "sudo"
  • Run the command "apt-mirror" after you're done with the configuration, wait for it to complete
  • If you'd like to detach the command from the terminal you can issue the command "apt-mirror &"
  • Warning: Set the base_path variable only to the folder to which the current user or the user using whom you are trying to run "apt-mirror" has write permissions
Settting it for daily update using a cron job:
  • Edit the file "/etc/cron.d/apt-mirror". This file would be available at your disposal after successful installation of the "apt-mirror" package
  • You'll find some default commented lines you can change it and uncomment or you can insert "@daily myusername /usr/bin/apt-mirror > /home/myusername/apt-mirror/cron.log && /home/myusername/apt-mirror/var/clean.sh"
  • Adding the above command will schedule a cron job in your operating system such that it'll start that command everyday at midnight which in-turn does the downloading for you and clean up older(if a newer deb package comes)/unwanted(if you change your configuration file after  download and re-run the script) packages.
  • Warning: Try the command manually before adding it to the cron job and check whether the download process is starting successfully and it is having write permissions. This is just a fail-safe
The mirroring is complete. What to do now?
  • Install Apache http server using the command "sudo apt-get install apache2".
  • Now all you have to do is create a symbolic link in the /var/www folder which is the root of apache2 web server in Ubuntu:
    • I'll create a new folder in /var/www with name ubuntu by using the command "sudo mkdir /var/www/ubuntu".
    • Now browse to that folder using "cd /var/www/ubuntu" and create symbolic links to mirrors by issuing the commands below:
      ln -s /home/myusername/apt-mirror/mirror/archive.canonical.com/ archive-canonical
      ln -s /home/myusername/apt-mirror/mirror/archive.ubuntu.com/ archive-ubuntu
      ln -s /home/myusername/apt-mirror/mirror/extras.ubuntu.com/ extras-ubuntu
      
    • I've used the names archive-canonical, archive-ubuntu, extras-ubuntu only for my reference you are free to use whichever names you desire.
    • The folders under apt-mirror/mirror such as archive.canonical.com and archive.ubuntu.com etc are there because you have specified them in your configuration file i.e., mirror.list file. Suppose you specify a deb url as "deb http://extras.ubuntu.com/ubuntu maverick main" the script will create a folder extras.ubuntu.com under the apt-mirror/mirror directory in our case or under the path where you've specified the "mirror_path" variable.
  • Well ah, that is all for the configuration.
The final step using the mirror to update the packages:
  • Pre-requisite: Make sure that your port 80 is open.
  • Edit the /etc/apt/sources.list file in the host machine from (use the below only as an example).

    deb http://archive.canoncial.com/ubuntu maverick partner
    deb http://extras.ubuntu.com/ubuntu maverick main
    deb http://security.ubuntu.com/ubuntu/ maverick-security main restricted
    deb http://security.ubuntu.com/ubuntu/ maverick-security universe
    deb http://security.ubuntu.com/ubuntu/ maverick-security multiverse
    
  • To the following:
    deb http://localhost/ubuntu/archive-canonical/ubuntu maverick partner
    deb http://localhost/ubuntu/extras-ubuntu/ubuntu maverick main
    deb http://localhost/ubuntu/security-ubuntu/ubuntu/ maverick-security main restricted
    deb http://localhost/ubuntu/security-ubuntu/ubuntu/ maverick-security universe
    deb http://localhost/ubuntu/security-ubuntu/ubuntu/ maverick-security multiverse
    
  • Replace localhost with an IP address if on a network.
  • Run the commands:
    sudo apt-get update
    sudo apt-get upgrade
    
  • Well that's should do it your system is having the latest updates(if you've mirrored it recently) installed.

1 comment:

  1. great post thanks, I just wish you elaborated more on the speed variable. i don't know my bits from my bytes from my megabits or bytes. I have a 3.0 x768 internet, I have no idea what I would need to prevent lag during download

    ReplyDelete