Htcondor Repos

Overview

HTCondor currently provide native packages in Debian and RPM format. Location of the repositories are listed below:

http://www.cs.wisc.edu/condor/yum
http://www.cs.wisc.edu/condor/debian

Background

Providing package via repository offers many advantages. Firstly, many OS distributions automatically detect package change and notify users to update. Secondly, package management programs such as yum or apt-get automatically resolve dependency

Repository Layout

There are 3 separate repositories: stable, development and testing (pre-release testing repo).

Yum

/p/condor/public/html/yum/stable/rhel5/*.rpm
/p/condor/public/html/yum/stable/rhel4/*.rpm
/p/condor/public/html/yum/development/rhel5/*.rpm
/p/condor/public/html/yum/development/rhel4/*.rpm

Debian

/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386/*.deb
/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-amd64/*.deb
...
/p/condor/public/html/debian/development/dists/etch/contrib/binary-i386/*.deb
/p/condor/public/html/debian/development/dists/etch/contrib/binary-amd64/*.deb
...

Manually Managing Repositories

Yum

Add/remove packages

  1. Go to a specific branch such as /p/condor/public/html/yum/stable/rhel5/ and add/remove package from the directory.
  2. Invoke createrepo to update the repo index.
    createrepo /p/condor/public/html/yum/stable/rhel5/

Add a new OS distribution

  1. Create new branch such as /p/condor/public/html/yum/stable/rhel6/ and follows the same step as add/remove package.

Debian

Add/remove packages

  1. Add/remove package from its branch such as /p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386
  2. Set appropriate environment variable for invoking repository utility
    export PATH=$PATH:/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386/
    export PERL5LIB=$PERL5LIB:/p/condor/workspaces/vdt/debian-build-tools/lib/perl5/vendor_perl/5.8.8 
  3. Go to the target folder and invoke dpkg-scanpackages to generate index file
    dpkg-scanpackages --arch i386 --multiversion . "/dev/null" > Packages 
  4. Create compressed version of index file (apt-get or synaptic will download only the compressed version)
    gzip -9c Packages > Packages.gz
    bzip2 -9k Packages 

Add a new OS distribution

  1. Copy entire distribution for existing one
    cp -r /p/condor/public/html/debian/stable/dists/lenny /p/condor/public/html/debian/stable/dists/squeeze 
  2. Remove all packages (.deb) and index files (Packages.gz, Packages.bz)
  3. Edit archive tag in the Release file. Possible values are: stable, oldstable, testing or unstable. Release file is used for APT pinning.
  4. Follows Step 2 of add/remove packages

Automate Script for Managing Repositories

Related Files

File Description
/p/condor/public/html/yum/files/manage_repo.pl Script
/p/condor/public/html/yum/files/mapping.txt Config file

Overview

The main goal of this script is to facilitate the release process. The script accepts NMI RUNID as a parameter. It uses the same logic as nmi_crowbar to query URLs from NMI database of a given RUNID. Then it will read the mapping.txt which provides mapping between NMI platform and platform name on each repo in order to download packages and put them into the correct location. Finally, it will update the repositories' index files.

In normal mode of operation 1-2 GB should be enough for scratch disk requirement because it only download and extract one package at time.

mapping.txt Format

NMI_PLATFORM TYPE ARCH TARGET_OS
ia64_rhas_3 RPM ia64 rhel4
x86_deb_5.0 DEB i386 lenny

  • The first column specifies a NMI platform name.
  • Type must be DEB or RPM
  • ARCH is the architecture string.For RPM, it should match the arch string in RPM's name. For Debian, it must match Debian-style arch string (i386, ia64, alpha, amd64, .. )

Note that we use packages built on RHEL3 on RHEL4. The script will only download packages from platforms listed in the mapping.

The script is not capable of add a new OS distribution via mapping.txt. Please use the manual method to add a new OS distribution into a repository before add new mapping in mapping.txt

Script Usage

  1. To download packages from a given RUNID, use /scratch/kooburat as scratch folder and push them into the stable repos. It will also update the repos when finishes.

manage_repo.pl -i 24009 -s /scratch/kooburat -b stable

  1. To download packages from a given RUNID into scratch folder.

manage_repo.pl -i 24009 -s /scratch/kooburat  

  1. To remove a specific version of package from a given set of repos.

manage_repo.pl -b development -r 7.3.1-1 

Using Repositories

Stable and Development Repositories

Please see below for detailed instructions
http://www.cs.wisc.edu/condor/yum
http://www.cs.wisc.edu/condor/debian

Internal Testing Repositories

To test pre-release packages. Testing repositories are created for both Debian and Yum repository. Please use the instruction below to access these repositories.
Note: Access to these repositories is limited only to wisc.edu network.

Yum

Use the following repo files (Depends on platform)
http://www.cs.wisc.edu/condor/yum/repo.d/condor-testing-rhel4.repo
http://www.cs.wisc.edu/condor/yum/repo.d/condor-testing-rhel5.repo 

Debian

Add the following lines to /etc/apt/sources.list (Depends on platform)
deb http://www.cs.wisc.edu/condor/debian/testing/ etch contrib
deb http://www.cs.wisc.edu/condor/debian/testing/ lenny contrib

Download Statistics

We use CSL's facility to capture the download statistic on these repositories. A weekly statistic email will be sent to the addresses listed in these files.

/p/condor/public/html/yum/.statinfo
/p/condor/public/html/debian/.statinfo

Please see below documentation for more detail.
http://www.cs.wisc.edu/twiki/bin/view/CSDocs/WebServerFAQs#How%20Do%20I%20Get%20Statistics%20For%20My%20W