Internal CRAN

Our Internal CRAN is a repository of all our internal packages, similar to the original CRAN, but exclusive to us and not open to the world. It allows us to easily install our packages using the standard install.packages() command and contains all versions of our packages. While it only contains sources and not binaries, resulting in longer installation times, it is easier to maintain this way. The latest versions of our packages are automatically added to our internal CRAN through a Github build process.

The repository is hosted in an S3 bucket named “g6tr-cran” and can be accessed at the URL http://g6tr-cran.s3.amazonaws.com. This URL is automatically added to the "repos" R session option in all our R sessions. By accessing it, anyone on the internet can view the list of files in the bucket. This is a requirement for the install.packages() function. However, access to the package files themselves is restricted to users with the necessary credentials. To install packages, users must include a secret string in the "HTTPUserAgent" option, which is reflected in the bucket permission scheme’s "UserAgent" field.

{
    "Version": "2008-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
        {
            "Sid": "Public",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::g6tr-cran/*",
            "Condition": {
                "StringLike": {
                    "aws:UserAgent": [
                        "*<secret string 1>*",
                        "*<secret string 2>*",
                        ...
                    ]
                }
            }
        }
    ]
}

Any of the secret strings can be used, and one of them is automatically added to the "HTTPUserAgent" option at the start of our R sessions.

The “src/contrib” folder is where all the important files are stored, including the package archives that contain the source code for our packages. Additionally, the folder contains three files - “PACKAGES”, “PACKAGES.gz”, and “PACKAGES.rds” - all of which contain the same information. These files serve as an index page for the repo and provide a list of all the package versions along with their dependencies. Because of the install.packages() requirements, the PACKAGE files must be publicly accessible.

To add a new package version to the internal CRAN, you need to follow these steps:

1. Build a package .tar.gz source archive. We use devtools::build() for this.

2. Put the generated archive in the “src/contrib” folder of the S3 bucket.

3. Add information about the new version to the PACKAGES files.

While there may be other methods available, we currently only know of one way to properly update the PACKAGES files: directly or indirectly using the tools::update_PACKAGES() function. To use this function, a folder containing all existing source archives and PACKAGE files must be passed to it.

Currently, the implementation we use to update PACKAGES files is rather slow and could likely be improved. The process involves the following steps:

1. Downloading the entire archive.

2. Adding the new version (or versions) of the package(s) to the local copy of the archive.

3. Running the tools::update_PACKAGES() function on the local copy of the archive.

4. Uploading the newly generated PACKAGES files to S3.

New package versions are added to the CRAN as part of the build that updates the baseline Amazon Machine Image (AMI) . This build is triggered manually approximately once a week.

The process of updating CRAN starts in builds/200_internal_cran_check.R script, where we download the archive and determine which packages need updating. Then, in builds/300_build_and_deploy.R , we build new package versions, generate the PACKAGES files, and upload them along with the new archives to the CRAN.

Although it is possible to manually add new packages to the CRAN by running these scripts, it is a rare event.