System Configuration

Currently our Workbench setup uses one EC2 server with a mounted FSx Lustre storage. This storage is the place where user home directories and R package installations are located. And, generally, this is the place to store everything that might eventually get big and take a lot of place.

Our initial priority was to facilitate RStudio Project Sharing, which necessitated storage that could support an Access Control List (ACL). Among the services offered by Amazon, our options were limited to an EBS drive or a custom NFS, where the latter is challenging to establish and maintain.

Amazon’s serverless NFS-like service, EFS, lacks ACL support. Regarding FSx storage types, only one of the four available options - FSx for NetApp ONTAP - is compatible with Ubuntu and provides ACL support. However, based on the price calculator, it is quite expensive.

RStudio’s website suggests that FSx for Lustre is a viable option for project sharing. However, when we sought confirmation, RStudio support confirmed that this was a miscommunication.

As a result of these challenges, we opted to use EBS for a certain period.

We subsequently decided to establish a Kubernetes cluster, with the aim of leveraging the RStudio Job Launcher for the execution of computationally intensive tasks. Simultaneously, we realized that Project Sharing was rarely utilized. As such, our focus shifted towards ensuring our storage solution was optimally compatible with Kubernetes, which unfortunately was not the case with EBS. While EFS is compatible with Kubernetes, its performance was subpar for our routine tasks.

FSx Lustre, while not as speedy as EBS, was satisfactory. Not only did it integrate well with Kubernetes, but RStudio also indicated that several of their customers have had success with this storage type. As a result, we ultimately transitioned to FSx Lustre for our storage needs.

Instance type, R5n, was chosen based on RStudio recommendations for EFS. We don’t use EFS, but it seems like a good choice regardless.

Our current setup uses a single r5n.x2large instance. It has an 8-cores CPU and 64 Gb of RAM. For instance storage, I went for 30 Gb. Our instance storage is only going to contain Ubuntu system, base R installation and some additional system library installations. Should be enough.

For the base AMI, select “Ubuntu Server 20.04 LTS (HVM)”. I’ve tried launching Ubuntu 22 and faced RStudio installation problems straight away.

For key pair, either use the one I’ve created, RStudio Workbench, or create a new one. But make sure it won’t get lost.

To prevent accidental instance stopping or termination, enable stop protection and termination protection.

Workbench users need access to specific S3 buckets from their RStudio sessions, as well as read-only access to RDS and Secrets Manager for the g6tr.db package to function correctly. To achieve this securely, we will assign a role to the server instances.

Navigate to the IAM dashboard in the AWS console and select the “Roles” section. Ensure an EC2 role named “workbench_server_role” exists. If not:

1. Click “Create role.”
2. Choose “AWS service” as the entity type and select EC2 for the use case.
3. Under permissions, add the following custom policies:
   • S3SelectedBuckets (read/write access to certain buckets)
   • RDSListAndDescribe (RDS read-only)
   • SecretsManagerRead (Secrets Manager read-only)
4. Name the role and click “Create role.”

Associate this role with the server instance via the EC2 dashboard: select the instance, click “Actions” -> “Security” -> “Modify IAM role,” and pick “workbench_server_role.”

Credentials with the correct policies are securely stored in instance metadata. The functions we use for S3 access rely on the aws.s3 package, which automatically uses the instance metadata when the aws.ec2metadata package is installed. The RDS and Secrets Manager functions also utilize these credentials but through different packages, and they work without additional configuration.

There’s a way to reduce EC2 costs by using the so-called “Reserved Instance” service. But it’s not about reserving specific machines as it may sound. It’s rather a discount plan. Say you purchase one Reserved Instance of a certain type for one year, then this reservation automatically covers expenses for instances of applicable type. If at some point in time you used 2 or more instances of the selected type, only one will be covered. When you have a machine working non-stop 24/7, this brings about 30-40% savings.

Links:

• prices

• aws guide

• current instance reservations

• another guide - see here how to check if current reservations actually cover any expenses.

Be careful when buying a reservation - you need to select the right instance type, platform and tenancy. These values should be equal to the corresponding parameters of your actual instance. For Offering Class I’ve selected “Standard”. I’ve also clicked the “Only show offerings that reserve capacity” switch and selected the availability zone of our instance.

Recently (at the moment of writing) AWS started advertising their other saving program Savings Plans. It is claimed to be easier, more flexible and more advantageous comparing to Reserved Instances. So when our reservation runs out it makes sense to take a look into this direction.

Here’s the full user guide: link.

It covers many aspects, but we only need to do the following:

1. Create a file system.
2. Install Lustre client on the server.
3. Mount the file system to a designated folder.

You can follow the first step as outlined here Here is the configuration I used:

• Storage type: persistent SSD
• Throughput: 125MB/s/TiB
• Storage capacity: 1.2 TiB
• Data compression type: NONE
• Subnet: in the same availability zone as the EC2 server

Keep the remaining settings at their default values. No need to set up backups—it’s done automatically for you.

Before proceeding, be aware that the Lustre client has certain compatibility limitations. For instance, at the time of writing, it was not available for Ubuntu 24. While it is theoretically possible to download the sources and build the client for any system version, this can be very challenging. Please consult the compatibility table here first.

In addition to the OS version, it’s important to note that for Linux-based system there’s such a thing as kernel version. You can check your kernel version with the command uname -r. Multiple kernel versions can be installed on your system, but only one will be active at any given time.

Lustre client installation instructions can be found here. Initially, when setting up the client on a server running Ubuntu 20, it turned out that a simple command was sufficient and no pre-configurations were required:

sudo apt install -y lustre-client-modules-aws

However, when attempting the same on a server running Ubuntu 22, the module was installed correctly but failed to function. The issue arose because the module was automatically installed to the latest supported kernel present on the system instead on the one that was in use:

# Kernel version in use: 6.5.0-1021
sudo modprobe lustre
#> modprobe: FATAL: Module lustre not found in directory /lib/modules/6.5.0-1021-aws

# Lustre was installed to kernel 6.2.0.1018
sudo apt install -y lustre-client-modules-aws
#> ...
#> lustre-client-modules-aws is already the newest version (6.2.0.1018.18~22.04.1).
#> ...

In such situations, the easiest solution is to switch to the supported kernel version, essentially downgrading the kernel. AWS provides instructions on how to do this on this page in the troubleshooting chapter.

Mount the file system to the directory named /pfs. This is a generic name that appears meaningful (e.g., Persistent File Storage), but it doesn’t specifically indicate any particular storage type. This allows for flexibility in case we decide to switch storage type again. You can follow the instructions provided in this link to add the system to the list of automatic mounts. I haven’t been able to get it to work seamlessly. Upon each system reboot, I need to manually start the Lustre client before running sudo mount -a. To start the client, you can reinstall it using sudo apt install -y lustre-client-modules-aws. While this may not be the most optimal solution, it does the job.

The installation itself is super easy. Really, as they describe it here.

By default RStudio listens on port 8787. Makes sense to point RStudio to port 80 straight away. Use this guide. Now you can open instance public DNS in web browser and you should see a login screen. It’s not even close to the end, but it’s good to know that something is working at this point.

We want to achieve a setup where our server can be reached via a nice URL (r.gradientmetrics.com) through SSL-certificated https connection.

When you own a domain, like gradientmetrics.com, it is possible to map a subdomain of it, like r.gradientmetrics.com to any public IP address. It is done in DNS provider settings. However, it doesn’t make sense to map it to IP of an entity like EC2 instance because these IPs are assigned randomly. If you reboot or recreate your setup, you’ll have to edit domain DNS settings once again, and this thing take some time (30 minutes) to take effect. Not very convenient.

Luckily, AWS has this thing called “Elastic IPs”. This is basically a permanent IP you can claim for you company to then attach to an instance or a load balancer. Here’s where all our elastic IPs can be found: aws link. In DNS settings of our domain, subdomain r.gradientmetrics.com is mapped to one of the IPs, marked as “RStudio Workbench”. It is unlikely that we’ll ever have to redo this step, unless the IP gets accidentally removed. But here’s a guide on mapping subdomains for NameCheap DNS provider: link

If you are rebuilding the whole setup from scratch it may make sense to attach this elastic IP to the Workbench EC2 instance and visit “http://r.gradientmetrics.com/" to check that it works at this point. But note “http” at the begging. “https” won’t work like this, it requires some more effort.

We need to set up SSL to be able to access our server through a secure https connection instead of http. If you read the chapter on SSL certificates in RStudio Workbench admin guide, you might get an impression that you simply need to claim some SSL certificates, put them on server and point Workbench to where they are located. This won’t work.

We need a more complicated setup. Check out this article: link. Overall the idea is that we do not obtain some external SSL certificates, instead we put a load balancer in front of our instance and associate an Amazon certificate with this load balancer. The article discusses multiple options for a load balancer. Our server uses http(s) for connection, so we need to setup an application load balancer (ALB).

For security group I’ve selected this one, but this is subject to change. For availability zones, I’ve picked all available, but not sure if it makes sense.

With regards to SSL certificate, we already have one issued for r.gradientmetrics.com. If there’s a need to create a new one, select “Email validation” for validation method. This will send an email with a validation link to our webmaster@gradientmetrics.com address.

Also, we’ll have to set up http -> https redirect on our ALB. Here’s the guide.

Overall, for ALB we’ll have two listeners:

• HTTPS on port 443, accociated with an ACM SSL certificate, forwards traffic to port 80 of our server.

• HTTP on port 80, not associated with any certificates, redirect traffic to HTTPS

ALB has it’s own address which can be found in the description tab when you select the load balancer, see “DNS name” field. If you visit this URL, most likely (depends on your browser) you’ll be shown an error screen. The reason is that SSL certificate associated with this ALB is issued for “r.gradientmetrics.com”, not for the address of the ALB. But if you manage to get through the error screen, confirming that you trust this resource etc, you should be able to see RStudio login page.

An important ALB setting to be aware of is the “Connection idle timeout.” By default, this value is set to 60 seconds. This setting controls how long the load balancer keeps an idle connection open before closing it. For most use cases, this timeout is sufficient, and users will not notice any issues. However, if you’re running Shiny apps, this setting can cause problems as the apps will be idled out after this period of inactivity. Therefore, it is recommended to increase the connection idle timeout to 10 or 20 minutes. Here’s how to do it: link to a paragraph.

Now the problem with ALB is that you cannot associate an elastic IP with it. But you can associate it with a Network Load Balancer (NLB) and put this NLB in front of the ALB. Here’s a guide.

Note: for health check path, don’t set it to “/index.html”, just leave it empty as it is.

Be sure to add targets for two ports:

• TCP:80, forwarding to target group with our ALB which listens at port 80

• TCP:443, forwarding to target group with our ALB which listens at port 443

Both target groups contain the same ALB, but we can only make target group to listen on one port, so we need to create two groups.

Associate the elastic IP address to which we mapped r.gradientmetrics.com to the NLB. guide

Now you should be able to see RStudio login screen at https://r.gradientmetrics.com. A URL with http will be redirected to https.

Here’s how the whole thing works, roughly:

http://r.gradientmetrics.com   https://r.gradientmetrics.com
                     |               |
                     |   DNS name    |
                     |  conversion   |
                     |               |
          <elastic IP>:80         <elastic IP>:443
                      \             /
                       \           /
                        \         /
                         <  NLB  >
                         |       |
        http over TCP    |       |    https over TCP
                         |       |
http -> https redirect   <  ALB  >  SSL certificate
                         |       |
  https://r.gradientmetrics.com  |   
                         |       |
                        ...    <EC2>
                                 |
                             <RStudio>

Note, that if you are creating a new server instance instead of the current one, there’s no need to rebuild the load balancer setup once again, it will be enough to replace the instance in the target group ALB directs traffic to.