personal radmin

throw your computer into the ocean with confidence

E. David Aja

Course material

posit-conf-2024/wtf

a brief personal history:

Auditor

• personal laptop
• RStudio in Citrix
• airgapped Posit Workbench
• airgapped Shiny Server Pro
• Special Issue Individual Laptop
• Special Issue Individual Desktop
• Special Issue Team Desktop

Data Scientist

• personal laptop
• work laptop
• RStudio in Citrix
• RStudio Server

Solutions Engineer

• personal laptop
• work laptop
• Posit Workbench Demo Server
• Posit Workbench Internal Server
• misc ephemeral environments

this is intended to provide context on my priorities / habits of thought: - it needs to work across operating systems - You can’t depend on having pro features - avoid admin where possible - as portable as practical - audit context means I need to be able to reproduce exactly - “run this command” preferable to a confluence page of screenshots

How do I…

• upgrade the version of R I’m using for this project?
• track which package versions I’m using for this project?
• move this project from one machine to another?

We’re extending the lessons from Day 1 to think about what it means to work with R projects over time.
That means that in addition to wanting our individual projects to flow smoothly, we also want to strategize about how to manage different versions of the language, different package versions, and different execution contexts. Working on an old project on a new computer is a lot like trying to put an app into production.

warmup

• warmup

• starting R

• installing R packages

• creating reproducible environments

• installing / upgrading R

📦 library

R packages

• the natural unit for distributing R code

base R

• 14 base + 15 recommended packages
• ships with all binary distributions of R

For example, have you used lattice recently? 🤷

• it came with your R installation, can use out of the box
• library(lattice)

Where do packages live locally?

By default, in the default library

.Library

All libraries for the current session

.libPaths()

All installed packages

installed.packages()

explore your package library

usethis::use_course("rstats-wtf/wtf-explore-libraries")
# use_course("rstats-wtf/wtf-explore-libraries", destdir = "my/new/location")

07:30

starting R

• warmup

• starting R

• installing R packages

• creating reproducible environments

• installing / upgrading R

why?

Sometimes we want to change the way our code behaves without having to change the code.

behaves might mean - “what computer do I think I’m on” - “is this dev or prod”?

R startup sequence

R startup sequence (our focus)

⚠️ Use a newline

startup files must end in a newline.

R may silently throw away the last line if you don’t do this.

.Renviron

environment variables

a […] named value that can affect the way running processes will behave on a computer

what goes in .Renviron

✅ R-specific environment variables
✅ API keys or other secrets
❌ R code

Editing .Renviron

usethis::edit_r_environ(scope = ?)

user

~/.Renviron

project

path/to/your/project/.Renviron

example .Renviron

user

PATH="${RTOOLS40_HOME}\usr\bin;${PATH}" # windows users
GITHUB_PAT=ghp_gVa6xK3bwERJ7xpWvoWj2dXMit4z43fWE8R
CONNECT_API_KEY=DaYK2hBUriSBYUEGIAiyXsRJHSjTYJN3
R_HISTSIZE=100000

project

RETICULATE_PYTHON=.venv/bin/python
DB_USER=elephant
DB_PASS=p0stgr3s

read environment variables

Sys.getenv()

edit your user .Renviron

1. usethis::edit_r_environ()
2. add WTF_USER=<your name>_user
3. restart your R session
   CTRL + SHIFT + F10
CMD + SHIFT + 0
4. Retrieve the value of WTF_USER Sys.getenv("WTF_USER")

02:00

edit your project .Renviron

1. usethis::edit_r_environ("project")
2. add WTF_PROJECT=<your name>_project
3. restart your R session
   CTRL + SHIFT + F10
CMD + SHIFT + 0
4. Retrieve the value of WTF_PROJECT Sys.getenv("WTF_PROJECT")

02:00

what’s the value of WTF_USER after you set WTF_PROJECT?

.Renviron recap

project .Renviron “short-circuits”
if it exists, your user .Renviron will not be evaluated

.Rprofile

R code that runs at the start of each session

?interactive()

TRUE

🧑‍💻

• sending individual lines of code to the console

☝️ most .Rprofile customization

FALSE

🤖

• knitting an RMarkdown document
• R -e script.R

what to put in .Rprofile

• set a default CRAN mirror
• customize R prompt

what not to put in .Rprofile

If it matters for code you share, it should not be in .Rprofile

why shouldn’t these go in .Rprofile?

• options(stringsAsFactors = FALSE)
• library(tidyverse)
• f <- dplyr::filter
• theme_set(theme_bw())

discuss with your neighbor

Why might these be safe to put in .Rprofile?

library(usethis)
library(devtools)

02:00

dotfiles

https://github.com/search?q=.Rprofile

activity

• edit your user .Rprofile
• edit your project .Rprofile

1. usethis::edit_r_profile()
2. add print("from my user Rprofile")
3. restart your R session
   CTRL + SHIFT + F10
CMD + SHIFT + 0
4. what value is printed when you restart?

1. usethis::edit_r_profile("project")
2. add print("from my project Rprofile")
3. restart your R session
   CTRL + SHIFT + F10
CMD + SHIFT + 0
4. what value is printed when you restart?

05:00

.Rprofile recap

project .Rprofile “short-circuits”
if it exists, your user .Rprofile will not be evaluated

wrapping up

.Renviron

• secrets
• environment information

.Rprofile

• development dependencies

Environment variables are inherited by child processes, so setting them early in the startup process is valuable Hosting providers trying to deal with secrets will often let you supply environment variables

installing R packages

• warmup

• starting R

• installing R packages

• creating reproducible environments

• installing / upgrading R

why

having a reason to slack off is fun but being able to get your work done quickly is more fun

states of R packages

R Packages – Structure

states of R packages

• show how to set options for different environment managers
• Install package from github
• Develop your own packages

R Packages – Structure

Binary

• compiled ahead of time
• easiest / fastest to install

you may particularly care about installation speed if you’re being billed by the minute for compute time, or if you need computations to finish before a timeout

where to get binaries

	CRAN	Posit Public Package Manager (p3m)
windows	✅	✅
mac OS	✅	✅
linux	❌	✅

are binaries available?

CRAN

may not be the latest

may not be the latest

may not be the latest

> install.packages("parallelly", repos = "https://cran.rstudio.com")

Installing package into ‘C:/Users/WDAGUtilityAccount/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)

  There is a binary version available but the source version is later:
           binary source needs_compilation
parallelly 1.31.0 1.32.1             FALSE

installing the source package ‘parallelly’

trying URL 'https://cran.rstudio.com/src/contrib/parallelly_1.32.1.tar.gz'
Content type 'application/x-gzip' length 124853 bytes (121 KB)
downloaded 121 KB

* installing *source* package 'parallelly' ...
** package 'parallelly' successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'parallelly'
    finding HTML links ... done
    as.cluster                              html  
    autoStopCluster                         html  
    availableConnections                    html  
    availableCores                          html  
    availableWorkers                        html  
    canPortBeUsed                           html  
    cpuLoad                                 html  
    find_rshcmd                             html  
    freeCores                               html  
    freePort                                html  
    getOption2                              html  
    isConnectionValid                       html  
    isForkedChild                           html  
    isForkedNode                            html  
    isLocalhostNode                         html  
    isNodeAlive                             html  
    makeClusterMPI                          html  
    makeClusterPSOCK                        html  
    parallelly.options                      html  
    pid_exists                              html  
    supportsMulticore                       html  
*** copying figures
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
** testing if installed package can be loaded from final location
*** arch - i386
*** arch - x64
** testing if installed package keeps a record of temporary installation path
* DONE (parallelly)

Posit Package Manager

PPM

how do I know I got a binary?

• CRAN
• CRAN
• PPM
• renv

> install.packages("parallelly", repos = "https://cran.rstudio.com")
Installing package into ‘C:/Users/edavi/Documents/R/win-library/4.1’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.1/parallelly_1.32.1.zip'
Content type 'application/zip' length 306137 bytes (298 KB)
downloaded 298 KB

package ‘parallelly’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\edavi\AppData\Local\Temp\Rtmpa2s3e8\downloaded_packages

> install.packages("renv", repos="https://cran.rstudio.com")
Installing package into ‘/Users/edavidaja/Library/R/x86_64/4.1/library’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/renv_0.15.5.tgz'
Content type 'application/x-gzip' length 1866760 bytes (1.8 MB)
==================================================
downloaded 1.8 MB


The downloaded binary packages are in
 /var/folders/b5/fl4ff68d23s148tg1_1gnflc0000gn/T//RtmpMk69B0/downloaded_packages

> install.packages("remotes")
Installing package into ‘C:/Users/WDAGUtilityAccount/AppData/Local/R/win-library/4.2’
(as ‘lib’ is unspecified)
trying URL 'https://packagemanager.rstudio.com/all/latest/bin/windows/contrib/4.2/remotes_2.4.2.zip'
Content type 'binary/octet-stream' length 399930 bytes (390 KB)
downloaded 390 KB

package ‘remotes’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
  C:\Users\WDAGUtilityAccount\AppData\Local\Temp\RtmpA1edRi\downloaded_packages

> renv::install("fs")
Retrieving 'https://packagemanager.rstudio.com/all/latest/bin/windows/contrib/4.2/fs_1.5.2.zip' ...
  OK [downloaded 380.9 Kb in 0.5 secs]
Installing fs [1.5.2] ...
  OK [installed binary]
Moving fs [1.5.2] into the cache ...
  OK [moved to cache in 7.4 milliseconds]

pop quiz, hotshot

Does Posit Package Manager serve a binary of dplyr for R 3.4?

02:00

Source

• compiled in your environment
• requires extra tools when package uses C, C++, Fortran, Rust, etc.

• most common reason to install from source is when you want the development version of a package from github One additional complication for *nix users:
• generally on windows can install and load packages that depend on other lower level libraries quite easily. Explain installing sf, for example.

if you don’t have tools

• make
• g++

Running `R CMD build`...
* checking for file 'C:\Users\WDAGUtilityAccount\AppData\Local\Temp\RtmpkTUH61\remotes192027624804\tidyverse-dplyr-36ef054/DESCRIPTION' ... OK
* preparing 'dplyr':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* installing the package to process help pages
      -----------------------------------
* installing *source* package 'dplyr' ...
** using staged installation
** libs
Warning in system(cmd) : 'make' not found
ERROR: compilation failed for package 'dplyr'
* removing 'C:/Users/WDAGUT~1/AppData/Local/Temp/Rtmp6hYuyV/Rinst1334d1a23d1/dplyr'
      -----------------------------------
ERROR: package installation failed

Warning in untar2(tarfile, files, list, exdir, restore_times) :
  skipping pax global extended headers
* installing *source* package 'dplyr' ...
** using staged installation
** libs
c:/Rtools/mingw_64/bin/g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-36~1.3/include" -DNDEBUG          -O2 -Wall  -mtune=core2 -c filter.cpp -o filter.o
sh: line 1: c:/Rtools/mingw_64/bin/g++: No such file or directory
make: *** [C:/PROGRA~1/R/R-36~1.3/etc/x64/Makeconf:215: filter.o] Error 127
ERROR: compilation failed for package 'dplyr'
* removing 'C:/Users/edavi/DOCUME~1/projects/WTF-PR~1/renv/staging/1/dplyr'
Error: install of package 'dplyr' failed

how to get the tools

• windows
• macOS
• linux

Rtools

XCode

⚠️ don’t run right now

xcode-select --install

install tools via package manager, e.g.

apt install make

got tools?

devtools::has_devel()

## Your system is ready to build packages!

activity

install a package from r-universe

1. open the personal-radmin project

2. install.packages("gitcellar") # not on cran!

3. update your project .Rprofile:
   usethis::edit_r_profile(scope = "project")

   options(repos = c(
     ropensci = "https://ropensci.r-universe.dev",
     CRAN = "https://cloud.r-project.org", # macOS
     RSPM = "https://packagemanager.rstudio.com/cran/latest" # windows
   ))

4. save .Rprofile and restart R

5. install.packages("gitcellar")

05:00

discussion

• did you install gitcellar from binary or source?
  • How do you know?

wrapping up

binaries are usually preferred, but for development versions of packages, you may need to install from source.

creating reproducible environments

• warmup

• starting R

• installing R packages

• creating reproducible environments

• installing / upgrading R

why?

The quick and dirty approach you took then comes right back at you as a boomerang.

Therefore, ban the concept of one-off altogether. No matter what the asker says, it will have to be done again, always!

3/9

— Edwin Thoen (@edwin_thoen) May 16, 2022

You will need to reproduce or update your analysis. You might get audited!

reproducibility strategies

reproducibility strategies

we’re focused in particular on systems where we have both access and responsibility

tools

ppm

renv

renv has a number of tools that are helpful for adopting a project-based workflow: - lockfile records what packages you use and where you instaled them from - renv::install() can handle installation from remote sources, local sources, package versions, bioconductor, rspm - per-project library isolation with global package cache - makes it easy to install experimental versions of packages and then roll back to a stable release if it doesn’t work out

ppm

ppm: latest

ppm: date-based snapshots

activity

time travel!

1. open the personal-radmin project

2. update your project .Rprofile with a date-based snapshot of CRAN

   options(repos = c(RSPM = "https://p3m.dev/cran/2021-03-31"))

3. Restart R

4. install.packages("dplyr")

what version of dplyr did you install?

05:00

renv

shared project environment

• by default all of your packages are installed into your user library
• the unforunate default behavior in macOS is also that this is the system library
• updating a package in the shared environments updates it for all of your projects

.libPaths()

> .libPaths()
[1] "C:/Users/edavi/AppData/Local/R/win-library/4.4" # user
[2] "C:/Program Files/R/R-4.4.0/library" # system

isolated project environments

.libPaths()

- Project '~/sample-project' loaded. [renv 1.0.7]
> .libPaths()
[1] "C:/Users/edavi/Documents/sample-project/renv/library/windows/R-4.4/x86_64-w64-mingw32"        
[2] "C:/Users/edavi/AppData/Local/R/cache/R/renv/sandbox/windows/R-4.4/x86_64-w64-mingw32/88765555"

isolated project environments

• experiment with new packages without breaking other projects
• share exactly which versions you’re using with your team or future you
• re-use packages you’ve already installed

activity

create an isolated project library

1. open your wtf-fix-paths project
2. install.packages("renv")
3. renv::init()
4. renv::status()
5. add library(parallelly) to any file in the wtf-fix-paths/R directory; save file
6. renv::status()
7. install.packages("parallelly")
8. renv::status()
9. renv::snapshot() (y proceed)
10. remove library(parallelly) from the file you added it to; save file
11. renv::status()

10:00

run .libPaths() to see what your default package libraries are initialize an renv–you’ll see that the environment is now local to the project note the .Rprofile that gets placed into your project, which runs an R script in renv/activate.R deactivate the environment and check libPaths() again–it should be back to normal

renv::install()

or install.packages() with shims enabled

renv::install("jsonlite")
renv::install("jsonlite@1.2") # at version
renv::install"jeroen/jsonlite") # latest
renv::install"jeroen/jsonlite@80854359976250f30a86a6992c0d8c3b3d95473d") # at SHA
renv::install("path/to/package") # local source

---

devtools::install("jsonlite")
devtools::install_version("jsonlite@1.2") # at version
devtools::install_github("jeroen/jsonlite") # latest
devtools::install_github("jeroen/jsonlite@80854359976250f30a86a6992c0d8c3b3d95473d") # at SHA
devtools::install("path/to/package") # local source

the junk drawer

for reprexes, tweets, and other ephemera:

~/Documents/scratch

I use renv for everything, and have one junk project into which I install things without regard for stability or reproducibility

installing / upgrading R

• warmup

• starting R

• installing R packages

• creating reproducible environments

• installing / upgrading R

why

why

bigger context

practice makes perfect

• Windows 10+
• macOS
• Linux

Windows Sandbox

tart
utm

Multipass
Digital Ocean
Linode

The best way to become more comfortable with installing things is to practice in an environment you can throw away
- the windows sandbox lets you test in an environment on your machine that won’t affect your computer - enabling it is outside the scope of this course - for linux users, the easiest thing to do is stand up a small VPS - I don’t know of a good safe way to test mac environments

“run this command” >> clickOps

package managers

software whose job it is to install other software
🙉 not the same kinds of packages as PPM

	package manager
windows	scoop, chocolatey, winget
macOS	homebrew
linux	homebrew, apt, yum, zypper

⚠️

Avoid installing your language with a package manager

You want to avoid having the language version your projects are using changed out from you unexpectedly Tools for freezing installations with your package manager are not consistently effective Other software that you install with your package manager may expect that package manager to be running the latest Idiosyncracies of your language will be better understood by a dedicated tool

language managers

using your package manager, install a language manager

• windows
• *nix

scoop install rig

brew install rig

language managers for other languages:

pyenv nvm rustup

• many language managers are often written in some language other than the one they are responsible for managing

switching R versions

CTRL + CLICK

switching R versions

switching R versions

RSwitch

language versions

using your language manager, install a language version:

❯ rig install --help
rig-add

DESCRIPTION:
    Download and install an R version, from the official sources.
    It keeps the already installed R versions, except versions within the
    same minor branch, see below.

EXAMPLES:
    # Add the latest release
    rig add release

    # Install specific version
    rig add 4.1.2

    # Install latest version within a minor branch
    rig add 4.1

if you’re running a distribution of linux that rig doesn’t support: - https://docs.rstudio.com/resources/install-r/

if no pre-compiled binary of rig is available: - https://docs.rstudio.com/resources/install-r-source/

environment managers

using your desired version of the language, install an environment manager:

install.packages("renv")

project

using your environment manager, install your project’s dependencies:

• create a new environment

  renv::init()

• restore an already defined environment

  renv::restore()

activity

• relaxed
• spartan

read the rig FAQ, or the rswitch guide

• install rig
• use rig to install a different major or minor version of R
  ⚠️ avoid installing a patch version
  
• open RStudio with that newly installed R version

10:00