Job Launcher

The RStudio Server typically meets our needs, but occasionally running complex models can overload it, affecting user experience. RStudio Job Launcher, a feature of RStudio Server Pro, simplifies running R scripts and Shiny applications on remote resources, efficiently utilizing remote clusters and cloud environments for heavy tasks without impacting the main server.

As we operate entirely on AWS, hosting our Kubernetes cluster on EKS is a logical choice. Our setup will feature a small, permanent node group running the Karpenter autoscaler, while the Job Launcher jobs will be executed on temporary nodes initialized by Karpenter.

We will configure an FSx persistent volume claim, which provides instructions for pods on how to get access to our FSx system. A simple mount command is insufficient; this formal setup is necessary to ensure proper functionality.

Finally, we’ll need to create a Docker image with R and RStudio installed, which will serve as the environment for running jobs. Although RStudio offers default images for various system configurations, our specific setup is not included among them.

The RStudio Job Launcher supports two types of computation clusters: Kubernetes and Slurm. We chose Kubernetes for its popularity and flexibility, having already found it useful for running multiple resource-intensive models in the background. Though we have no immediate plans to try other options, we may consider Slurm in the future.

AWS Fargate, a serverless compute engine for containers compatible with EKS in particular. It appeared ideal, as it enabled effortless creation of a Kubernetes cluster that could automatically scale up from zero and back down. However, the main drawback is that Fargate currently only supports EFS persistent storage. Upon testing EFS, we discovered it was too slow for many of the tasks we commonly perform.

Regarding FSx (which we currently use), it performs adequately but is noticeably slower than EBS. After reviewing various Q&A services and documentation, we determined that although it might be possible to make EBS accessible from the Kubernetes cluster, it is not advised. On the other hand, there are numerous resources available for setting up FSx within a Kubernetes environment.

Before configuring the Job Launcher with Kubernetes, it’s important to understand both Kubernetes and AWS Elastic Kubernetes Service (EKS), and to be familiar with key tools like kubectl and eksctl.

If you’re new to Kubernetes, the official Kubernetes website offers a solid introduction: Kubernetes Basics Tutorial. Focus on understanding essential terms like node, pod, job, namespace, deployment etc and become familiar with the kubectl command line tool, which we will use frequently. Delving into more advanced topics is optional.

Regarding EKS and the corresponding tool, eksctl, AWS offers an EKS Workshop that primarily concentrates on deploying robust web applications using Kubernetes. While our objective differs, as we aim to use Kubernetes for launching R jobs, the initial sections of this workshop will be of use. We’ll detail which ones further.

We’ll be using Cloud9, a cloud-based IDE to manage our cluster. It’s not mandatory for EKS, but it’s a practical and clean choice that’s easy to set up. A Cloud9 environment can also be shared with others, which is particularly useful. If the environment for managing our cluster is deleted or not functioning properly, follow the Setup/In your AWS account section of the EKS workshop to establish a new one.

This chapter outlines the process of launching a Cloud9 environment with a CloudFormation template, which comes with essential tools like kubectl and eksctl pre-configured and all necessary access already established.

The provided aws cloudformation deploy command in the chapter may fail with a known error that is going to be addressed further, and the stack might not be created. However, you can still attempt it; any failed stack will be automatically rolled back. In the CloudFormation console, you can view all stacks and investigate the specific errors. You’ll likely encounter an error indicating the inability to locate a public subnet in our default VPC. To resolve this, explicitly specify a subnet:

aws cloudformation deploy --stack-name {insert an appropriate stack name} \
    --template-file ./eks-workshop-ide-cfn.yaml \
    --parameter-overrides RepositoryRef=stable Cloud9Subnet={insert a subnet ID} \
    --capabilities CAPABILITY_NAMED_IAM

Find the list of all subnets here. Look for any public subnet within our default VPC, identifiable by “RStudio” in its name.

Note that you cannot create a new stack with the same name while a record of a failed stack exists in CloudFormation. You need to delete the failed stack first, which can be done through the CloudFormation console.

Before diving into the setup, you can find useful information about eksctl on this page and in a video provided here.

In the Cloud9 environment, use the eksctl create cluster command with our custom YAML manifest, which you can find at this link, instead of the one provided in the workshop.

eksctl create cluster -f cluster.yaml

Please ensure that you review the subnets IDs in the YAML file. These correspond to two subnets within our RStudio VPC. If any changes have been made, update the YAML file accordingly.

For instance types and the number of nodes, we are open to considering changes to the configuration if necessary, as we are not strictly bound to this particular setup.

In this step, we’ll install the Karpenter autoscaler on our cluster. Follow the instructions in the Migrating from Cluster Autoscaler tutorial. When you reach the “Create default NodePool” step, be sure to use our customized manifest, which can be found here, instead of the default one. Our manifest enforces more stringent instance selection criteria and incorporates a disruption policy to terminate nodes following 2 hours of inactivity. Since we haven’t installed the regular cluster autoscaler, there’s no need to remove it.

Alternatively, you may consider starting with the Getting Started with Karpenter tutorial, which also covers the cluster installation. However, please note that I haven’t personally tested this approach.

To enable communication between the job launcher and our cluster for job execution, we need to:

1. Establish a dedicated Kubernetes namespace named rstudio where the launcher will operate.

2. Create a cluster service account within this namespace, which will be utilized for the job launcher’s interactions with the cluster.

You can find detailed instructions for the steps mentioned above in this chapter of the Workbench Administration Guide.

One aspect not covered in the guide is that in newer Kubernetes versions, secrets are not automatically generated for service accounts. You’ll need to create them manually, as explained in this StackOverflow post. We provide a service account secret manifest in our GitHub repository, which you can find here. To apply this manifest, run the command kubectl apply -f service-account-secret.yaml -n rstudio.

Persistent Volume Claims (PVCs) can be either dynamic or static; in our case, we will use a static PVC because we won’t create new FSx volumes on the fly. However, for the FSx PVC to work, we must first set up the FSx for Lustre Container Storage Interface (CSI) driver.

The Container Storage Interface (CSI) standardizes how storage solutions integrate with Kubernetes. Unlike NFS storage, which typically doesn’t need a separate driver due to its native support, specialized services like FSx require installing a CSI driver for proper Kubernetes integration.

Consult this guide for instructions on installing the driver. Once installed, the guide recommends deploying a storage class and subsequently a PVC. However, in our cluster, we approach this slightly differently by setting up a persistent volume directly instead of a storage class.

Follow the instructions provided in this README for the implementation. Preconfigured manifests with the correct values can be found in our repository here.

Additionally, it’s important to note that while Persistent Volumes (PV) are namespace-agnostic, Persistent Volume Claims (PVC) are tied to specific namespaces. Therefore, when defining a PVC, you should use the following command with a namespace specification:

kubectl apply -f claim.yaml -n rstudio

Next, we’ll configure the RStudio server side. Posit offers a comprehensive guide, which can be found at Posit’s Documentation. It is advisable to familiarize yourself with it, but note that many of the aspects of the guide are not relevant to us.

For convenience, we have created a GitHub repository hosting our standard RStudio server configuration, accessible here. We will provide specific commentaries on the configuration files that contain settings related to the Job Launcher.

Add the Kubernetes cluster as the default:

auncher-adhoc-clusters=Kubernetes
launcher-default-cluster=Kubernetes

In this file, you’ll also notice lines pertaining to Launcher sessions. These are not related to Kubernetes. We run RStudio sessions on the Local cluster, which is not substantially different from the standard way of running sessions. This approach was advised by Posit Support as a resolution for a session-related issue. For more details, refer to this support ticket.

We mount FSx Lustre in our job launcher jobs, not NFS. This file contains the instructions for utilizing the PVC we’ve created on our cluster.

This is the default setup, nothing to comment.

This file outlines both the default and maximum allowable resource requirements for jobs, as well as the specifications for Docker images. For container-images field in launcher.kubernetes.profiles.conf, try searching for a container image that matches our system configuration (Ubuntu version + R version) in RStudio’s Docker registry here. At the time of writing, a matching image was not available, so a custom image had to be created and uploaded to a registry. The process for doing this will be covered in the next chapter. For running a basic test job, you can use any of the images provided by RStudio.

This file has been omitted from the repository due to its inclusion of sensitive information, such as the api-url and certificate-authority details of our cluster. The values for these fields can be found in the “Details” section of the cluster within the EKS console.

The other fields are:

auth-token-path=/etc/rstudio/kubernetes.launcher.token
job-expiry-hours=2

The file must include the secret token for the job-launcher Kubernetes service account. Follow these steps to set it up:

1. In Cloud9, execute the following command to obtain the token:

   kubectl get secret job-launcher-secret -n rstudio -o jsonpath='{.data.token}' | base64 -d
   

2. Transfer this token to /etc/rstudio/kubernetes.launcher.token on the server.

3. Apply these commands to restrict access exclusively to the rstudio-server user:

   chmod 0600 /etc/rstudio/kubernetes.launcher.token
   sudo chown rstudio-server /etc/rstudio/kubernetes.launcher.token
   

RStudio maintains a large repository of pipelines for creating various RStudio-related images, which can be found here. However, it was challenging to determine how to run a pipeline to create an image for our specific system configuration. As a workaround, the necessary Docker files were extracted, modified, and bundled into a separate repository here. The README contains instructions for their use.

You can find the custom r-session-complete image that was created for our system configuration in our ECR registry.

If you need to update the image for a new system configuration, use the git repository as a starting point and adjust the Docker files as needed.

It is recommended to use Amazon’s ECR instead of Github’s registry as ECR works much faster with EKS for obvious reasons. There’s really no need to host our image in a private registry since it won’t contain any sensitive information, and using a public registry will only make things more complicated. You can find the ID of our public ECR registry here. Before pushing the image to ECR, ensure that the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set up correctly, and you should be good to go.

To troubleshoot issues, it can be useful to inspect the Job Launcher logs. You can do this by running the following command: sudo journalctl -u rstudio-launcher.service. If you only want to view the last “n” rows of logs, you can append -n <some number> to the command.

Occasionally, users may encounter job errors that reference *.so libraries and appear as follows:

... error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory

The reason for this error is that the job code requires a specific pre-compiled shared library to run, but this library is not accessible within the job environment. When you execute the Sys.getenv("LD_LIBRARY_PATH") command in your R session, it will display a list of directories where the system searches for the required libraries. Please note that jobs can only directly access files located on the mounted shared drive ("/pfs"). Standard library locations such as “/lib” might contain different files when accessed within the job environment compared to your regular R session environment. The contents of these directories are determined by the Docker image used for running the jobs.

To resolve the issue of a missing shared library, you could consider rebuilding the Docker image. This process would require incorporating a step in the Dockerfile for installing the necessary library. This typically takes the form of an apt-get install command or something similar, rather than manually copying the library file.

Alternatively, there’s a simpler method available. You could copy the library file to a location that’s accessible from the job environment and ensure this location is included in the LD_LIBRARY_PATH. In our existing configuration, the library would be located at “/pfs/lib/ld”. We have augmented the “/etc/rstudio/rsession-profile” file with the following line:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pfs/lib/ld

This line of code appends “/pfs/lib/ld” to the default list of locations. As long as the job environment inherits the value of LD_LIBRARY_PATH, the system will be able to access the library file.