On-Server Monorepo Mirror

We are accustomed to having a separate copy of our monorepo for each project we work on. However, since our repository is quite large, doing git clone every time can be incredibly slow. To address this issue, we’ve set up an on-server monorepo mirror. This is a centralized copy that is regularly synchronized with GitHub. This allows us to quickly create new copies from the mirror instead of cloning the entire repository each time.

The on-server monorepo mirror can be found at /pfs/github/g6tr. To keep it in sync with the GitHub repository, a cron job is used. To view the root user’s cron jobs, run sudo cron -e on the server. Look for the following line:

0 3 * * * /pfs/github/update_monorepo_mirror.sh >> /var/log/update_monorepo_mirror.log 2>&1

This cron job is scheduled to run once a day at 3:00 AM. It executes the script /pfs/github/update_monorepo_mirror.sh and writes the logs to /var/log/update_monorepo_mirror.log.

Let’s take a closer look at the script.

pat=`cat /pfs/github/pat.txt`
pull_command="cd /pfs/github/g6tr && git pull https://idmn:$pat@github.com/g6t/g6tr && git fetch"
eval $pull_command

It reads a Github PAT stored in /pfs/github/pat.txt and uses it to authenticate and pull the latest changes from the GitHub repository.

It is important to note that the PAT file is set to be readable only by the root user for security reasons, and should be regularly updated before it expires.

We’ve simplified the process of creating copies for users to work in by providing a helper function called g6tr_clone_monorepo(). Documentation for this function can be found here. Just in case, we did not hard code the mirror location into the function. Instead we read it from the G6TR_MONOREPO_LOCAL_MIRROR system variable. To ensure that it is set correctly for all users, we define it in /etc/rstudio/rsession-profile:

export G6TR_MONOREPO_LOCAL_MIRROR=/pfs/github/g6tr

Additionally, we provide an RStudio project template that utilizes a similar function.