Stork Wisdom

Jeff's Stork Responsibilities:

Brain dump meeting Monday June 5, 3:30pm notes:
*query Gnats category stork
*fix security bugs first !!!!!!!
query for category==Stork and sub-category==condor-security
*Jaime's nanoHUB bug reports, feature requests
complete, and merge V6_7-storkd-rewrite-branch (module=V6_7_CODE).
This branch  is a complete rewrite of stork server daemon.  The goal of this
effort was to take the original research-quality codebase up to a modern,
production quality daemon.  As of 2006-06-09, this rewrite is incomplete, and
approx 60% done.

	general improvements in V6_7-storkd-rewrite-branch:
	*remove hundreds of fixed length buffers
	*improved fault protection
	*consolidation of many redundant code fragments into new Stork class
	methods.
	*consolidated multiple URL parsers into single new URL parser class.

	new features in V6_7-storkd-rewrite-branch:
	*added requirements and rank to job scheduling.  Old stork scheduling was
	previously a simple FIFO.
	*checks all job attributes in submit file at submit time, not execution time
	*per-job run time sandbox directories
	*added throttle for starting new jobs
	*increased used of [new] ClassAd views [for better or worse ;-) ] to
	factor out many redundant static queries.
	*consolidated all stork job ad attributes into single file stork_job_ad.h
	*consolidated all stork config params into single file stork_config.h

	Gnats bug fixes:
	570 hung job detection uses job submit timestamp
	573 eliminate fixed length strings
	574 convert Stork MyString implementations to std::string
	575 move Stork job queue log to SPOOL directory
	672 user jobs run in LOG directory


	*SC05 stork matchmaker, tag V6_7-SC2005_Stork_Demo_3-branch (See Nick)
*work with Tevfik and grad student Sirish Tummata at LSU to add storage
scheduling
contacts: 	Tevfik Kosar <kosar@cct.lsu.edu>,
Sirish Tummala <stumma1@lsu.edu>
*support stork-announce, stork-discuss lists
*start transfers to FNAL tape, from Business School GLOW storage (old)
contact: Bill Taylor


Installing Stork:
See Gnats PR 602 for a description of this documentation request, and the
required Stork installation steps.

Configuring, Using Stork + HTCondor Credd [Carey's credd, not GreqQ's credd]:

Note: all references below are to the "old" condor_credd, aka the Stork
credential management daemon.  The fact that we now have 2 distinct credential
management daemons is a huge source of confusion.

Configure a personal stork+credd:
# condor_configure --install --make-personal-stork

# condor_config_val DAEMON_LIST
DAEMON_LIST: COLLECTOR, CREDD, MASTER, NEGOTIATOR, SCHEDD, STARTD, STORK

Start HTCondor, including Stork, CredD:
# condor_master

Using Stork:  See tutorial:
http://www.cs.wisc.edu/condor/stork/tutorial/stork-dagman/


Using Stork and [Carey's Stork] Credential Manager daemon:


Store a X.509 credential in the [Stork] Credential Manager:

# stork_store_cred -t x509 -f /tmp/x509up_u19100 -N weber
Credential submitted successfully

list credentials managed by the [Stork] Credential Manager:
# stork_list_cred
Name    Type
-----   -----
weber   X509proxy

Total 1


Direct Stork to use a credential managed by Credential Manager.  Add to stork
submit file:
cred_name = "weber";

[DO NOT use the x509proxy directive in the stork submit file, when using the
cred_name directive.  These are exclusive.  And no, I haven't yet looked up how
a conflict is resolved.

Using [Stork] CredD with MyProxy:
[see http://grid.ncsa.uiuc.edu/myproxy/credmgmt.html for MyProxy credential
mgmt.]

[note: the whole condor_credd/myproxy interaction has a working unit test in
src/condor_credd/test_credd and src/stork/test_harness_common.sh.  run the
-myproxy test with the -nomemory and -noleak options.]

First, start up a MyProxy server.  See
http://grid.ncsa.uiuc.edu/myproxy/man/myproxy-server.8.html.
The best way to install MyProxy is via the VDT.  See Alain.  Here's some VDT
test script to configure/run the MyProxy server, for testing:

--- start myproxy config ---
# first be sure to initialize shell with VDT, GLOBUS shell config!

# this works best as root if credential dirs are restricted to root

export TEMP_DIR=/tmp/myproxy.tmp
mkdir -p $TEMP_DIR

# Start MyProxy
export MYPROXY_PORT=12345
echo "*** Starting MyProxy"
cat > $TEMP_DIR/myproxy.conf <<EOF
accepted_credentials  "*"
authorized_retrievers "*"
default_retrievers    "*"
authorized_renewers   "*"
default_renewers      "none"
EOF

mkdir -p $TEMP_DIR/myproxy_repository
chmod 0700 $TEMP_DIR/myproxy_repository

$VDT_LOCATION/globus/sbin/myproxy-server -p $MYPROXY_PORT -c $TEMP_DIR/myproxy.conf -s $TEMP_DIR/myproxy_repository
--- end   myproxy config ---


store proxy in myproxy server using myproxy-init:
See http://grid.ncsa.uiuc.edu/myproxy/man/myproxy-init.1.html

Optionally: review contents of myproxy server, using myproxy-info
[http://grid.ncsa.uiuc.edu/myproxy/man/myproxy-info.1.html]

Store a credential named 'weber' in the stork credd,
from file /tmp/x509up_u19100
which is automatically refreshed from
the MyProxy Server username@host:port using the MyProxy distinguished name
`grid-cert-info -subject`  :
# stork_store_cred -t x509 -f /tmp/x509up_u19100 -N weber -m username@host:port -D `grid-cert-info -subject`


Now, the condor_credd will check its managed credential every
CRED_CHECK_INTERVAL seconds [default=60] and refresh the credential from the
copy in the MyProxy server, if needed.

The next obvious question is "then why are there _two_ copies of the
credential: in both the MyProxy server, and the condor_credd?".    There are 2
answers:

1)  Carey did it.  ;-)  This code and reasoning predated Jeff.

2). Jeff can infer that the condor_credd was intended to be a HTCondor daemoncore
generic credential management daemon, and was intended to manage a wide variety
of credentials, including X.509 proxies, and using the daemoncore API.  HTCondor
daemoncore clients to the condor_credd only had to retrieve credentials from
the [Stork] condor_credd, and did not have to know or care that the
condor_credd used MyProxy behind the scenes to refresh X.509 credentials.