Skip to content

Installing HTCondor-CE 3

HTCondor-CE 3 end-of-life: October 2021

HTCondor-CE 3 is distributed alongside HTCondor 8.8 in the HTCondor Yum repositories. so HTCondor-CE 3 will reach it's end-of-life in October 2021. Consider updating to HTCondor-CE 5 at your earliest convenience.

Joining the Open Science Grid (OSG)?

If you are installing an HTCondor-CE for the OSG, consult the OSG-specific documentation.

The HTCondor-CE software is a job gateway based on HTCondor for Compute Entrypoints (CE) belonging to a computing grid (e.g. European Grid Infrastructure, Open Science Grid). As such, HTCondor-CE serves as an entry point for incoming grid jobs — it handles authorization and delegation of jobs to a grid site's local batch system. See the overview page for more details on the features and architecture of HTCondor-CE.

Use this page to learn how to install, configure, run, test, and troubleshoot HTCondor-CE 3 from the HTCondor Yum repositories.

Before Starting

Before starting the installation process, consider the following points (consulting the reference page as necessary):

  • User IDs: If they do not exist already, the installation will create the condor Linux user (UID 4716)
  • SSL certificate: The HTCondor-CE service uses a host certificate at /etc/grid-security/hostcert.pem and an accompanying key at /etc/grid-security/hostkey.pem
  • DNS entries: Forward and reverse DNS must resolve for the HTCondor-CE host
  • Network ports: The pilot factories must be able to contact your HTCondor-CE service on port 9619 (TCP)
  • Submit host: HTCondor-CE should be installed on a host that already has the ability to submit jobs into your local cluster running supported batch system software (Grid Engine, HTCondor, LSF, PBS/Torque, Slurm)
  • File Systems: Non-HTCondor batch systems require a shared file system between the HTCondor-CE host and the batch system worker nodes.

There are some one-time (per host) steps to prepare in advance:

  • Ensure the host has a supported operating system (Red Hat Enterprise Linux variant 7)
  • Obtain root access to the host
  • Prepare the EPEL and HTCondor Yum repositories
  • Install CA certificates and VO data into /etc/grid-security/certificates and /etc/grid-security/vomsdir, respectively

Installing HTCondor-CE


HTCondor-CE must be installed on a host that is configured to submit jobs to your batch system. The details of this setup is site-specific by nature and therefore beyond the scope of this document.

  1. Clean yum cache:

    root@host # yum clean all --enablerepo=*
  2. Update software:

    root@host # yum update

    This command will update all packages

  3. Install the fetch-crl package, available from the EPEL repositories.

    root@host # yum install fetch-crl
  4. Select the appropriate convenience RPM:

    If your batch system is... Then use the following package...
    Grid Engine htcondor-ce-sge
    HTCondor htcondor-ce-condor
    LSF htcondor-ce-lsf
    PBS/Torque htcondor-ce-pbs
    SLURM htcondor-ce-slurm
  5. Install the CE software:

    root@host # yum install <PACKAGE>

    Where <PACKAGE> is the package you selected in the above step.

Configuring HTCondor-CE

There are a few required configuration steps to connect HTCondor-CE with your batch system and authentication method. For more advanced configuration, see the section on optional configurations.

Configuring authentication

To authenticate job submission from external users and VOs, HTCondor-CE can be configured to use a built-in mapfile or to make Globus callouts to an external service like Argus or LCMAPS. THe former option is simpler but the latter option may be preferred if your grid supports it or your site already runs such a service.

Built-in mapfile

The built-in mapfile is a unified HTCondor mapfile located at /etc/condor-ce/condor_mapfile. This file is parsed in line-by-line order and HTCondor-CE will use the first line that matches. Therefore, mappings should be added to the top of the file.


condor_mapfile.rpmnew files may be generated upon HTCondor-CE version updates and they should be merged into condor_mapfile.

To configure authorization for users submitting jobs with an X.509 proxy certificate to your HTCondor-CE, add lines of the following format:


Replacing <DISTINGUISHED NAME> (escaping any '/' with '\/') and <USERNAME> with the distinguished name of the incoming certificate and the unix account under which the job should run, respectively.

VOMS attributes of incoming X.509 proxy certificates can also be used for mapping:


Replacing <DISTINGUISHED NAME> (escaping any '/' with '\/'), <VOMSFQAN> fields, and <USERNAME> with the distinguished name of the incoming certificate, the VOMS roles and groups, and the unix account under which the job should run, respectively.

Additionally, you can use regular expressions for mapping certificate and VOMS attribute credentials. For example, to map any certificate from the GLOW VO with the htpc role to the glow user, add the following line:

GSI ".*,\/GLOW\/Role=htpc.*"


You should only add mappings to the mapfile. Do not remove any of the default mappings:

GSI "(/CN=[-.A-Za-z0-9/= ]+)" \
CLAIMTOBE .* anonymous@claimtobe
FS (.*) \1

Globus Callout

To use a Globus callout to a service like LCMAPS or Argus, you will need to have the relevant library installed as well as the following HTCondor-CE configuration:

  1. Add the following line to the top of /etc/condor-ce/condor_mapfile:

  2. Create /etc/grid-security/gsi-authz.conf with the following content:

    • For LCMAPS:

      globus_mapping lcmaps_callout
    • For Argus:

      globus_mapping /usr/lib64/ argus_pep_callout

Configuring the batch system

Before HTCondor-CE can submit jobs to your local batch system, it has to be configured to do so. The configuration will differ depending on if your local batch system is HTCondor or one of the other supported batch systems. Choose the section corresponding to your batch system below.

HTCondor batch systems

To configure HTCondor-CE for an HTCondor batch system, set JOB_ROUTER_SCHEDD2_POOL to your site's central manager host and port:


Additionally, set JOB_ROUTER_SCHEDD2_SPOOL to the location of the local batch SPOOL directory on the CE host if it is different than the default location (/var/lib/condor/spool).

Non-HTCondor batch systems

Configuring the BLAHP

HTCondor-CE uses the Batch Language ASCII Helper Protocol (BLAHP) to submit and track jobs to non-HTCondor batch systems. To work with the HTCondor-CE, modify /usr/libexec/condor/glite/etc/batch_gahp.config using the following steps:

  1. Disable BLAHP handling of certificate proxies:

  2. (Optional) If your batch system tools are installed in a non-standard location (i.e., outside of /usr/bin/), set the corresponding *_binpath variable to the directory containing your batch system tools:

    If your batch system is... Then change the following configuration variable...
    LSF lsf_binpath
    PBS/Torque pbs_binpath
    SGE sge_binpath
    Slurm slurm_binpath

    For example, if your Slurm binaries (e.g. sbatch) exist in /opt/slurm/bin, you would set the following:

Sharing the SPOOL directory

Non-HTCondor batch systems require a shared file system configuration to support file transfer from the HTCondor-CE to your site's worker nodes. The current recommendation is to run a dedicated NFS server on the CE host. In this setup, HTCondor-CE writes to the local spool directory, the NFS server shares the directory, and each worker node mounts the directory in the same location as on the CE. For example, if your spool directory is /var/lib/condor-ce (the default), you must mount the shared directory to /var/lib/condor-ce on the worker nodes.


If you choose not to host the NFS server on your CE, you will need to turn off root squash so that the HTCondor-CE daemons can write to the spool directory.

You can control the value of the spool directory by setting SPOOL in /etc/condor-ce/config.d/99-local.conf (create this file if it doesn't exist). For example, the following sets the SPOOL directory to /home/condor:

SPOOL = /home/condor


The shared spool directory must be readable and writeable by the condor user for HTCondor-CE to function correctly.

Optional Configuration

The following configuration steps are optional and will not be required for all sites. If you do not need any of the following special configurations, skip to the section on next steps.

Configuring for multiple network interfaces

If you have multiple network interfaces with different hostnames, the HTCondor-CE daemons need to know which hostname and interface to use when communicating with each other. Set NETWORK_HOSTNAME and NETWORK_INTERFACE to the hostname and IP address of your public interface, respectively, in /etc/condor-ce/config.d/99-local.conf directory with the line:


Replacing text with your public interface’s hostname and with your public interface’s IP address.

Limiting or disabling locally running jobs on the CE

If you want to limit or disable jobs running locally on your CE, you will need to configure HTCondor-CE's local and scheduler universes. Local and scheduler universes allow jobs to be run on the CE itself, mainly for remote troubleshooting. Pilot jobs will not run as local/scheduler universe jobs so leaving them enabled does NOT turn your CE into another worker node.

The two universes are effectively the same (scheduler universe launches a starter process for each job), so we will be configuring them in unison.

  • To change the default limit on the number of locally run jobs (the current default is 20), add the following to /etc/condor-ce/config.d/99-local.conf:

    START_LOCAL_UNIVERSE = TotalLocalJobsRunning + TotalSchedulerJobsRunning < <JOB-LIMIT>

    Where <JOB-LIMIT> is the maximum number of jobs allowed to run locally

  • To only allow a specific user to start locally run jobs, add the following to /etc/condor-ce/config.d/99-local.conf:

    START_LOCAL_UNIVERSE = target.Owner =?= "<USERNAME>"

Change <USERNAME> for the username allowed to run jobs locally

  • To disable locally run jobs, add the following to /etc/condor-ce/config.d/99-local.conf:

Enabling the monitoring web interface

The HTCondor-CE View is an optional web interface to the status of your CE. To run the HTCondor-CE View, install the appropriate package and set the relevant configuration.

  1. Begin by installing the htcondor-ce-view package:

    root@host # yum install htcondor-ce-view
  2. Next, uncomment the DAEMON_LIST configuration located at /etc/condor-ce/config.d/05-ce-view.conf:

  3. Restart the condor-ce service

  4. Verify the service by entering your CE's hostname into your web browser

The website is served on port 80 by default. To change this default, edit the value of HTCONDORCE_VIEW_PORT in /etc/condor-ce/config.d/05-ce-view.conf.

Uploading accounting records to APEL

For sites outside of the OSG that need to upload the APEL accounting records, HTCondor-CE supports uploading batch and blah APEL records for HTCondor batch systems:

  1. Install the HTCondor-CE APEL package on your CE host:

    root@host # yum install htcondor-ce-apel
  2. On each worker node, set the appropriate scaling factor in the HTCondor configuration (i.e. /etc/condor/config.d/) and advertise it in the startd ad:

    ApelScaling = <SCALING FACTOR>  # For example, 1.062
  3. Configure the APEL parser, client, and SSM

    • Records are written to APEL_OUTPUT_DIR in the HTCondor-CE configuration (default: /var/lib/condor-ce/apel/)
    • Batch and blah record filenames are prefixed batch- and blah-, respectively
  4. Start and enable the condor-ce-apel services appropriate for your operating system.

Enabling BDII integration

HTCondor-CE supports reporting BDII information for all HTCondor-CE endpoints and batch information for an HTCondor batch system. To make this information available, perform the following instructions on your site BDII host.

  1. Install the HTCondor-CE BDII package:

    root@host # yum install htcondor-ce-bdii
  2. Configure HTCondor (/etc/condor/config.d/) on your site BDII host to point to your central manager:


    Replacing <CENTRAL MANAGER> with the hostname of your HTCondor central manager

  3. Configure BDII static information by modifying /etc/condor/config.d/99-ce-bdii.conf

Additionally, install the HTCondor-CE BDII package on each of your HTCondor-CE hosts:

root@host # yum install htcondor-ce-bdii

Next Steps

At this point, you should have an installation of HTCondor-CE that will forward grid jobs into your site's batch system unchanged. If you need to transform incoming grid jobs (e.g. by setting a partition, queue, or accounting group), configure the HTCondor-CE Job Router. Otherwise, continue to the this document to start the relevant services and verify your installation.

Getting Help

If you have any questions or issues with the installation process, please contact us for assistance,