Job Router Configuration Overview¶

The HTCondor Job Router is at the heart of HTCondor-CE and allows admins to transform and direct jobs to specific batch systems. Customizations are made in the form of job routes where each route corresponds to a separate job transformation: If an incoming job matches a job route's requirements, the route creates a transformed job (referred to as the 'routed job') that is then submitted to the batch system. The CE package comes with default routes located in /etc/condor-ce/config.d/02-ce-*.conf that provide enough basic functionality for a small site.

If you have needs beyond delegating all incoming jobs to your batch system as they are, this document provides an overview of how to configure your HTCondor-CE Job Router

Definitions

Incoming Job: A job which was submitted to HTCondor-CE from an external source.
Routed Job: A job that has been transformed by the Job Router.

Route Syntaxes¶

HTCondor-CE 5 introduced the ability to write job routes using ClassAd transform syntax in addition to the existing configuration syntax. The old route configuration syntax is no longer the default in HTCondor-CE 23.x and one should transition to the new syntax as outlined below.

ClassAd transforms¶

The HTCondor ClassAd transforms were originally introduced to HTCondor to perform in-place transformations of user jobs submitted to an HTCondor pool. In the HTCondor 8.9 series, the Job Router was updated to support transforms and HTCondor-CE 5 adds the configuration necessary to support routes written as ClassAd transforms. If configured to use transform-based routes, HTCondor-CE routes and transforms jobs that by chaining ClassAd transforms in the following order:

Each transform in JOB_ROUTER_PRE_ROUTE_TRANSFORM_NAMES whose requirements are met by the job
The first transform from JOB_ROUTER_ROUTE_NAMES whose requirements are met by the job. See the section on route matching below.
Each transform in JOB_ROUTER_POST_ROUTE_TRANSFORM_NAMES whose requirements are met by the job

Deprecated syntax¶

Planned Removal of Deprecated Syntax

JOB_ROUTER_DEFAULTS, JOB_ROUTER_ENTRIES, JOB_ROUTER_ENTRIES_CMD, and JOB_ROUTER_ENTRIES_FILE are deprecated and will be removed for V24 of the HTCondor Software Suite. New configuration syntax for the job router is defined using JOB_ROUTER_ROUTE_NAMES and JOB_ROUTER_ROUTE_[name].
For a ClassAd transform syntax example vist: HTCondor Documentation - Job Router
Note: The removal will occur during the lifetime of the HTCondor V23 feature series.

Since the inception of HTCondor-CE, job routes have been written as a list of ClassAds. Each job route’s ClassAd is constructed by combining each entry from the JOB_ROUTER_ENTRIES with the JOB_ROUTER_DEFAULTS:

JOB_ROUTER_ENTRIES is a configuration variable whose default is set in /etc/condor-ce/config.d/02-ce-*.conf but may be overriden by the administrator in subsequent files in /etc/condor-ce/config.d/.
JOB_ROUTER_DEFAULTS is a generated configuration variable that sets default job route values that are required for HTCondor-CE's functionality. To view its contents in a readable format, run the following command:
```
user@host $ condor_ce_config_val JOB_ROUTER_DEFAULTS | sed 's/;/;\n/g'
```

Take care when modifying attributes in JOB_ROUTER_DEFAULTS: you may add new attributes and override attributes that are set_* in JOB_ROUTER_DEFAULTS.

The following may break your HTCondor-CE

Do not set the JOB_ROUTER_DEFAULTS configuration variable yourself. This will cause the CE to stop functioning.
If a value is set in JOB_ROUTER_DEFAULTS with eval_set_<variable>, override it by using eval_set_<variable> in the JOB_ROUTER_ENTRIES. Do this at your own risk as it may cause the CE to break.

Choosing a syntax¶

For existing HTCondor-CEs, it's recommended that administrators stop using the deprecated syntax and transition to ClassAd transforms now.

For new HTCondor-CEs, it's recommended that administrators start with ClassAd transforms. The ClassAd transform syntax provides many benefits including:

Statements being evaluated in the order they are written
Use of variables that are not included in the resultant job ad
Use simple case-like logic

Additionally, it is now easier to include job transformations that should be evaluated before or after your routes by including transforms in the lists of JOB_ROUTER_PRE_ROUTE_TRANSFORM_NAMES and JOB_ROUTER_POST_ROUTE_TRANSFORM_NAMES, respectively.

Converting to ClassAd transforms¶

For existing HTCondor-CE's utilizing the deprecated syntax can do the following steps to convert to using the ClassAd transform syntax:

Output the current configuration by running the following:
```
condor_ce_config_val -summary > summary-file
```

Convert the stored configuration by running the following:

condor_transform_ads -convert:file summary-file > 90-converted-job-routes.conf

Place the 90-converted-job-routes.conf from the previous command into the /etc/condor-ce/config.d.

Potential need to rename generated config

The files in /etc/condor-ce/config.d are read in lexicographical order. So if you define your current job router configuration in /etc/condor-ce/config.d in a file that is read later, e.g. 95-local.conf, you will need to rename your generated config file, e.g. 96-generated-job-routes.conf.
Tweak new job routes as needed. For assistance please reach out to htcondor-users@cs.wisc.edu
Set JOB_ROUTER_USE_DEPRECATED_ROUTER_ENTRIES = False (see this documentation in the HTCondor-CE's configuration.
Restart the HTCondor-CE

Not Using Custom Job Routes?

Conversion of job router syntax from the deprecated syntax to ClassAd transform syntax needs to occur if custom job routes have been configured.

How Jobs Match to Routes¶

The Job Router considers incoming jobs in the HTCondor-CE SchedD (i.e., jobs visible in condor_ce_q) that meet the following constraints:

The job has not already been considered by the Job Router
The job's universe is standard or vanilla

If the incoming job meets the above constraints, then the job is matched to the first route in JOB_ROUTER_ROUTE_NAMES whose requirements are satisfied by the job's ClassAd. Additionally:

If you are using the ClassAd transform syntax, transforms in JOB_ROUTER_PRE_ROUTE_TRANSFORM_NAMES and JOB_ROUTER_POST_ROUTE_TRANSFORM_NAMES may also have their own requirements that determine whether or not that transform is applied.
If you are using the deprecated syntax, you may configure the Job Router to evenly distribute jobs across all matching routes (i.e., round-robin matching). To do so, add the following configuration to a file in /etc/condor-ce/config.d/:
```
JOB_ROUTER_ROUND_ROBIN_SELECTION = True
```

Getting Help¶

If you have any questions or issues with configuring job routes, please contact us for assistance.