OverviewHTCondor currently provide native packages in Debian and RPM format. Location of the repositories are listed below:
BackgroundProviding package via repository offers many advantages. Firstly, many OS distributions automatically detect package change and notify users to update. Secondly, package management programs such as yum or apt-get automatically resolve dependency
Repository LayoutThere are 3 separate repositories: stable, development and testing (pre-release testing repo).
/p/condor/public/html/yum/stable/rhel5/*.rpm /p/condor/public/html/yum/stable/rhel4/*.rpm /p/condor/public/html/yum/development/rhel5/*.rpm /p/condor/public/html/yum/development/rhel4/*.rpm
/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386/*.deb /p/condor/public/html/debian/stable/dists/lenny/contrib/binary-amd64/*.deb ... /p/condor/public/html/debian/development/dists/etch/contrib/binary-i386/*.deb /p/condor/public/html/debian/development/dists/etch/contrib/binary-amd64/*.deb ...
Manually Managing Repositories
- Go to a specific branch such as /p/condor/public/html/yum/stable/rhel5/ and add/remove package from the directory.
Invoke createrepo to update the repo index.
Add a new OS distribution
- Create new branch such as /p/condor/public/html/yum/stable/rhel6/ and follows the same step as add/remove package.
- Add/remove package from its branch such as /p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386
Set appropriate environment variable for invoking repository utility
export PATH=$PATH:/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386/ export PERL5LIB=$PERL5LIB:/p/condor/workspaces/vdt/debian-build-tools/lib/perl5/vendor_perl/5.8.8
Go to the target folder and invoke dpkg-scanpackages to generate index file
dpkg-scanpackages --arch i386 --multiversion . "/dev/null" > Packages
Create compressed version of index file (apt-get or synaptic will download only the compressed version)
gzip -9c Packages > Packages.gz bzip2 -9k Packages
Add a new OS distribution
Copy entire distribution for existing one
cp -r /p/condor/public/html/debian/stable/dists/lenny /p/condor/public/html/debian/stable/dists/squeeze
- Remove all packages (.deb) and index files (Packages.gz, Packages.bz)
- Edit archive tag in the Release file. Possible values are: stable, oldstable, testing or unstable. Release file is used for APT pinning.
- Follows Step 2 of add/remove packages
Automate Script for Managing Repositories
OverviewThe main goal of this script is to facilitate the release process. The script accepts NMI RUNID as a parameter. It uses the same logic as nmi_crowbar to query URLs from NMI database of a given RUNID. Then it will read the mapping.txt which provides mapping between NMI platform and platform name on each repo in order to download packages and put them into the correct location. Finally, it will update the repositories' index files.
In normal mode of operation 1-2 GB should be enough for scratch disk requirement because it only download and extract one package at time.
- The first column specifies a NMI platform name.
- Type must be DEB or RPM
- ARCH is the architecture string.For RPM, it should match the arch string in RPM's name. For Debian, it must match Debian-style arch string (i386, ia64, alpha, amd64, .. )
Note that we use packages built on RHEL3 on RHEL4. The script will only download packages from platforms listed in the mapping.
The script is not capable of add a new OS distribution via mapping.txt. Please use the manual method to add a new OS distribution into a repository before add new mapping in mapping.txt
- To download packages from a given RUNID, use /scratch/kooburat as scratch folder and push them into the stable repos. It will also update the repos when finishes.
manage_repo.pl -i 24009 -s /scratch/kooburat -b stable
- To download packages from a given RUNID into scratch folder.
manage_repo.pl -i 24009 -s /scratch/kooburat
- To remove a specific version of package from a given set of repos.
manage_repo.pl -b development -r 7.3.1-1
Stable and Development RepositoriesPlease see below for detailed instructions
Internal Testing RepositoriesTo test pre-release packages. Testing repositories are created for both Debian and Yum repository. Please use the instruction below to access these repositories.
Note: Access to these repositories is limited only to wisc.edu network.
YumUse the following repo files (Depends on platform)
DebianAdd the following lines to /etc/apt/sources.list (Depends on platform)
deb http://www.cs.wisc.edu/condor/debian/testing/ etch contrib deb http://www.cs.wisc.edu/condor/debian/testing/ lenny contrib
Download StatisticsWe use CSL's facility to capture the download statistic on these repositories. A weekly statistic email will be sent to the addresses listed in these files.
Please see below documentation for more detail.