R Libraries

This chapter is dedicated to our R library setup. We want to store R libraries along with renv cache on the mounted persistent drive. In the future it may be useful to keep multiple versions of R installed.

We will use R installations on ec-2 internal storage with global shared libraries on the mounted drive. More specifically, global R libraries for all R versions are going to be stored in “/pfs/R/libs”.

Let’s create this directory and a subdirectory for the current R version.

sudo mkdir /pfs/R
sudo mkdir /pfs/R/libs
sudo mkdir /pfs/R/libs/4.2.0  # current R version

We are going to make it writable only for rstudio-admins group. Makes sense to add root to this group and change ownership from root only (root:root) to root as a member of rstudio-admins (root:rstudio-admins).

sudo usermod -a -G rstudio-admins root
sudo chown -R root:rstudio-admins /pfs/R/libs
sudo chmod -R g+w /pfs/R/libs  # writing access for the group 

Check that everything looks right with ll /pfs/R/libs.

Next, set up RStudio settings to make global libraries automatically attached to search paths of all users. The following needs to be added for all R versions in use.

# /etc/rstudio/r-versions
Path: /opt/R/4.2.0/lib/R
Library: /pfs/R/libs/4.2.0

More details on setting up “r-versions” file here.

Renv uses two kinds of global storage: one is called “cache”, the other - “global state storage” or “root”. The first one is a storage of cached package binaries where each package may have multiple versions installed. I really don’t know what role the second thing plays, but the first one can’t be properly used without it.

We are going to make both locations writeable by all rstudio-users. The idea is to make renv cache a collectively built storage of all package versions ever used in our projects. This will make restoring project library states from cache much quicker.

sudo mkdir /pfs/R/renv
sudo mkdir /pfs/R/renv/root
sudo mkdir /pfs/R/renv/cache

sudo usermod -a -G rstudio-users root
sudo chown -R root:rstudio-users /pfs/R/renv
sudo chmod -R 2775 /pfs/R/renv

Note the permissions we give to the renv folder and all its subfolders: 2775. 775 means that owner (root) and group (rstudio-users) get all permissions and all the other users get read and execute rights. The 2 at the beginning means the setgid bit. It makes all the subdirectories to inherit group ownership of the parent directory. In other words, all the new directories inside /pfs/R/renv will be owned by the rstudio-users group and this group will have all rights by default.

This however is not enough, because it does not control file ownership in any way. A file created in a subdirectory of /pfs/R/renv can easily be owned solely by a single user. And this is exactly what happens by default when renv adds a package to cache.

Fortunately there is a way to customize this behavior of renv. More specifically, we can provide a custom function for copying folders by setting the appropriate system option:

options(renv.config.copy.method = function(src, dst) {
  system(paste("cp -R", src, dst))
  system(paste("chgrp -R rstudio-users", dst))
})

First call copies src folder to dst. Second call changes group ownership to rstudio-users for all files and folders inside dst recursively.

Although even this is not enough. renv uses this custom function correctly when you run renv::install() not being in a renv-activated project. But it doesn’t when you are in a renv project. Long story short, we can force renv to always use our custom copying function all the time with:

options(renv.cache.linkable = FALSE)

This option and the one described earlier should always be executed at the session start on Workbench, so we need to put them to the global Rprofile.site file.

In case if it may be useful in the future, the root of the problem is renv:::renv_install_package internal function and the function which is used inside it to choose the method of copying to cache: renv:::renv_cache_linkable. With options(renv.cache.linkable = FALSE) the latter function always returns FALSE.

We should not forget also to make renv know about root and cache directories. For this, we need to set renv environment variables accordingly. And we need to make sure that these variables are set appropriately for all RStudio users for all projects. Let’s set this up in /etc/rstudio/rsession-profile script.

# /etc/rstudio/rsession-profile
export RENV_PATHS_ROOT=/pfs/R/renv/root
export RENV_PATHS_CACHE=/pfs/R/renv/cache

Now start a new session in Workbench and check that it worked

renv::paths$cache()
#> [1] "/pfs/R/renv/v5/R-4.2/x86_64-pc-linux-gnu"
renv::paths$root()
#> [1] "/pfs/R/renv/root"

So, our R libraries are located at /pfs/R/libs and renv storage is at /pfs/R/renv. We need to populate these folders with all the packages we’ve been using. We’ve made both locations writable for rstudio-admins so this can be done from Workbench.

First we need to determine the list of packages that need to be installed. The following is what I did to prepare for installation of all packages we had on our AMI, but it can be reused for other similar cases.

a. open an instance started from AMI

b. Save contents of rownames(installed.packages()) to a txt file. Copy contents of this file.

c. Create a new txt file on the Workbench server. Paste buffer into it.

d. readLines(<file path>) is the list packages to install.

Here is the setup to run before installing packages

options(HTTPUserAgent = paste(
  # system description to get access to RSPM
  paste0(
    "R/", getRversion(), ' R (', 
    getRversion(), " ",
    R.version$platform, " ",
    R.version$arch, " ",
    R.version$os, ")"
  ),
  # access key for internal CRAN
  "127a2ec00989b9f7faf671ed470be7f8"
))

options(repos = c(
  CRAN = "https://cran.rstudio.com/",
  G6TR = "http://g6tr-cran.s3.amazonaws.com",
  CLOUDYR = "http://cloudyr.github.io/drat",
  STAN = "https://mc-stan.org/r-packages"
))

.libPaths("/pfs/R/libs/4.2.0/")

Basically it’s the same set of options as you’ll have if you open a project in the monorepo apart from the CRAN repo URL. In normal user sessions we have “https://packagemanager.rstudio.com/cran/__linux__/focal/latest" for CRAN. This is a binary repository which allows much quicker installations. But it has a drawback that some packages may not be able to locate required system libraries. We are willing to sacrifice speed here to ensure that the shared library is in perfect condition.

We also put the shared library path on top of the search path so that all the newly installed packages will be installed into this folder. Remember, we as an admin have writing rights for this folder.

Let’s proceed with package installation. We will be doing this using renv::install() instead of install.packages() to initialize renv cache along the way. Here’s what I used:

all_pkgs <- readLines(<file path>)

for (i in seq_along(all_pkgs)) {
  message(i, "/", length(all_pkgs), ": ", all_pkgs[i])
  renv::install(all_pkgs[i])
}

This thing will surely encounter a problem at some point, but it will be clearly visible which package caused it. After fixing this intermediate problem, modify for loop to continue from where it stopped.

When it’s done, check that all packages can be loaded:

all_pkgs <- setdiff(rownames(installed.packages()), "conflicted")

for (i in seq_along(all_pkgs)) {
  message(i, "/", length(all_pkgs), ": ", all_pkgs[i])
  library(all_pkgs[[i]], character.only = TRUE)
}