2.1   General¶

The same person can hold some of these Actor’s roles.

Developer - LSST-funded developer

Science User - LSST affiliate partner

Security - LSST-funded staff member specializing in security

System Administrator - LSST-funded system administration staff member

K8s Cluster Administrator - K8s system software administrator. Responsibilities include K8s software installation and updates, disaster recovery, network overlay deployments, adding and removing K8s nodes.

K8s Service Administrator - Deploys, monitors and maintains LSST services to the production K8s cluster.

5.1   Account Access¶

AA1: Authorized user logs into Jupyter Notebook

1. User navigates to https://lsst-lspdev.ncsa.illinois.edu
2. User clicks on “Notebooks” image.
3. The user is presented with a link to “Sign in to CILogon,” and clicks that button.
4. The user is presented with a request for information which is about to be authorized: “CILogon username” and “Your username and affiliation from your identity provider.” The identity provider is listed as “National Center for Supercomputing Applications,” and a “Log on” link is shown. User clicks that button.
5. The user is presented with the CILogon page, which requires “NCSA Username” and “NCSA Kerberos Password.” An optional selection for “Don’t Remember Login” is also provided. After filling out the Username and Password, the user clicks on the “Login” button.
6. The user is presented with the Duo authentication page. The user authenticates with a push notification to a device or enters a passcode.
7. After successfully authenticating, the user is presented with the “Spawner Options” page. This page allows selection of an image and how much CPU and RAM to dedicate to the notebook. The user makes these selections and clicks “Spawn” button.
8. A new notebook is spawned, and the user is presented with JuypterLab interface.

AA2: Developer wishes to access logs of containers for debugging. (short term)

1. Developer starts an ssh session to cerberus4.ncsa.illinois.edu and authenticates with Duo.
2. From there, Developer starts an ssh session to bastion01.ncsa.illinois.edu.
3. From there, Developer starts an ssh session to kub001
4. Developer runs “kubectl logs” command to retrieve logs for an element in the K8s commons.

AA3: Developer wishes to access logs of containers for debugging. (long-term)

1. Developer starts an ssh session to cerberus4.ncsa.illinois.edu and authenticates with Duo.
2. From there, Developer starts an ssh session to bastion01.ncsa.illinois.edu.
3. From there, Developer starts an ssh session to lsst-headnode.ncsa.illinois.edu (NOTE: CHANGE THIS TO THE CORRECT NODE NAME WHEN THIS IS SET UP).
4. Developer runs “kubectl logs” command to retrieve logs for an element in the K8s commons.

6.1   GPFS filesets¶

Access to “project” (read-write): All users are put into one or more groups(s), and have directory access below the “project” fileset to each group to which they belong. This access is not unrestricted to all of “project.”

Access to “datasets” (read-only): Individuals/groups have different types of access, depending on their standing in the project. Some datasets are restricted for some period to LSST (first tier) collaborators before they become available to other parts of the project.

Access to “scratch” (read-write): All users are put into one or more groups(s), and have directory access below the “scratch” fileset to each to which they belong. This access is not unrestricted to all of scratch.

Access to “jhome” (read-write): LSST Developers and Science Users have access to the jhome fileset. LSST developers have this as a separate mount point named jhome which is accessible from their counts on lsst-dev. When they log in, their home directory is in /home/{user}. Users of lsst-dev also have access to jhome. LSST Science Users can only access the “jhome” fileset through the accounts they access on the K8s commons and have no visibility to /home. An LSST Science User has write access to write to /project and /scratch, and 100GB of disk space.

Access to “software” (read-only): All developers have read-only access to this fileset. This access is currently not available via Jupyter Notebook. This access may be added in the future to access the batch system commands.

6.2   VOSpace/WebDAV¶

Access to “File Workspace” (read-write): File Workspace is a subdirectory under the jhome fileset described above. Anything in the File Workspace is accessible via VOSpace and WebDAV. (Note that because of this exposure, it needs to be a subdirectory, not the $HOME of jhome itself.

Users cannot modify existing containers to add additional software. Any areas in the container that may be writable will be lost when the container is reconstructed. Any other software that the user may wish to use can be stored in their $HOME space in their notebook, project space, or temporarily in scratch space.

Users with approved proposals for larger allocations will be granted more space, based on requirements of their proposals.

7.1   K8s containers and standalone batch¶

Rather than having a static allocation of processing resources, there is a desire to shift how resources are allocated depending on tasks that need them. For example, nightly batch processing resources may be idle on some days, and they could be reallocated to yearly processing tasks. Or we might have some K8s resources that were idle, and that could be temporarily dedicated to nightly batch processing.

Nightly batch processing, yearly batch processing, and L3 resources could co-exist easily as one HTCondor cluster, with nodes given ClassAds to describe which type of processing to which they are dedicated. Jobs submitted to the cluster would use the ClassAd matching mechanism to run on the appropriate systems. Node ClassAds would describe what type of jobs a node could run, along with restrictions on which locations were able to submit jobs. For example, a science user could be prevented from spoofing a job to get more processing resources from other parts of the cluster.

When the HTCondor administrator wants to change how many nodes are dedicated to one type of processing to another, new ClassAds would be updated on those nodes, and the job scheduler would handle the rest. Shifting nodes back and forth would only require some commands to the nodes on the cluster, and will not require downtime.

Things get a bit more complicated when trying to dedicate resources from K8s to a standalone batch system. There are two ways that this might be done. The first would be to issue a command to take the K8s node out of the K8s cluster and rededicating to the batch system. This method would mean some downtime for the system, where neither K8s containers nor HTCondor processing would run.

The second would be to keep the K8s node as is, run a container containing the HTCondor software, and to have it join the batch system.

It’s unclear at this point whether the batch system will run most effectively as a set of containers in the K8s commons or as a standalone traditional batch cluster. There are several things to consider.

We’ve done some preliminary tests and have been able to bring up HTCondor worker nodes in a K8s container, attached to an HTCondor manager which is outside of the K8s cluster. We have not done testing to see what the optimal size (i.e., dedicated memory, core count, etc.) of an HTCondor K8s pod would be. Since an HTCondor node usually is configured to use the total number of cores per node and all available memory, letting the job partition the resources as it needs to, “pre-partitioning” without a complete understanding of all the types of jobs which will run on the batch cluster may be problematic unless the container takes over the whole node. We would need to gauge how much CPU would be required per container for job processing on a node. This may be possible by analyzing the types of Nightly and Yearly processing jobs we will have. Testing still needs to be done to see what other implications of running HTCondor from a container in the K8s commons would be. We expect that HTCondor containers running jobs will have a significant impact on the number of pods that could be run over all if entire nodes were dedicated to HTCondor. These are the same resources which would be used by the LSST science users, and it becomes challenging to gauge the number of resources to dedicating because of the wide variety of jobs Science Users may submit.

K8s assumptions -

1. The HTCondor master node and associate processes run outside of the K8s cluster.
2. HTCondor containers run indefinitely because they provide long-term service.
3. HTCondor pods are already running at the time of Nightly Processing requests.
4. HTCondor batch processing is reserved for its namespace, with appropriate ClassAds for each type of computing to done. This is so that one set of HTCondor batch resources don’t leech resources from each other.
5. HTCondor resources can be brought online by launching new containers and put offline by stopping containers.

We expect that if HTCondor is run from a container that the LSST software stack and HTCondor binaries will be run out of /software, leaving the container itself as small as possible, and allowing it to brought up more quickly.

The following use cases apply whether the batch control system is entirely on K8s or running standalone. All systems are assumed to have HTCondor software installed on them.

BCS 1: Prompt Processing needs more batch resources for processing, and other batch processing services are idle.

1. HTCondor Administrator issues commands to change ClassAds for additional nodes to specify they are part of Prompt Processing.

BCS 2: Prompt Processing has an excess number of batch resources available to it after processing has been caught up, and other batch processing services are below their allocation.

1. HTCondor Administrator issues commands to change ClassAds for Prompt Processing Node(s) to label them as part of the general batch processing services.

BCS 3: Prompt Processing has an excess number of batch resources available to it after processing has been caught up, and other batch processing services are at their designated allocation.

1. K8s Services Administrator deletes these HTCondor pod(s).

BCS 4: Prompt Processing needs more batch resources for processing, and other batch processing services are busy. Assumes K8s resources could be dedicated to batch processing and assumes that HTCondor containers would be used to add resources to batch.

1. K8s Services Administrator deploys new HTCondor pod(s).
2. HTCondor Administrator issues commands to change ClassAds for so those nodes additional are part of Prompt Processing.

BCS 5: Prompt Processing needs more batch resources for processing, and other batch processing services are busy. Assumes K8s resources could be dedicated to batch processing, and the system will not be running HTCondor containers.

1. K8s Services Administrator drains containers from node(s) and waits for the node to become idle.
2. HTCondor Administrator starts HTCondor services on that node.
3. HTCondor Administrator issues commands to change ClassAds for so those nodes additional are part of Prompt Processing.

13.1   Administration¶

During development, a small set of users will need admin access on the K8s cluster to configure resources correctly. Once development has stabilized and we move services to production, the K8s services administrator will deploy services based on instructions devised during development.

13.2   CPU capacity¶

Deployed pods will require between 0.5 and 4 cores per concurrent user.

13.3   Memory¶

Deployed containers will require between 512MB and 8GB per concurrent user.

13.4   Local Storage¶

Local storage per node needs to be about 100GB. As of this writing, containers are about 10GB each, with the expectation that about five different container images will be stored on a node at any given time.

13.5   User Storage¶

User storage in jhome is set to a 100GB quota.

13.6   Container Cache¶

Local container cache size is 250GB total.

13.7   Shared storage¶

This is storage intended for quick prototyping. 10TB total.