Offline installation on Jura

Installing new software on Jura is complicated because the cluster does not have Internet access. This page covers the installation of R, BioConductor, and Python packages.

R packages

For packages in CRAN

Since Jura cluster is not connected to the Internet, it won't be possible to install R packages directly. A local CRAN mirror has been deployed to ease the installation, it can be used as follows:

source /dcsrsoft/spack/bin/setup_dcsrsoft
module load gcc r
R
>install.packages(c('dplyr','ggplot2','cluster'), repos='http://mirror.dcsr.unil.ch/cran/')
 
For packages not in CRAN

For packages not available in the local CRAN mirror, you will have to go through the following procedure:

STEP 1: download the package on a machine connected to the internet, e.g.:

wget http://cnsgenomics.com/software/gsmr/static/gsmr_1.0.9.tar.gz

STEP 2: transfer the package to Jura using the standard procedure detailed here

STEP 3: log into Jura

STEP 4: launch R

$ source /dcsrsoft/spack/bin/setup_dcsrsoft
$ module load gcc r
$ R

STEP 5: install the package with:

> install.packages("/path/to/gsmr_1.0.9.tar.gz", repos = NULL, type="source")
 
For packages with the source on a Git server

For packages provided on a Git server, you will have to first build a package to transfer to Jura before you can proceed with the installation of the package per se. From a machine connected to the internet:

STEP 1: clone the Git repository in directory hereinafter referred to as "/path/to"

$ git clone https://github.com/jean997/causeSims.git

STEP 2: launch R

$ source /dcsrsoft/spack/bin/setup_dcsrsoft
$ module load gcc r
$ R

STEP 3: build the R package

> require("devtools")
> build("causeSims")

which should output something along these lines:

> build("causeSims")
✔ checking for file ‘/path/to/causeSims/DESCRIPTION’ ...
─ preparing ‘causeSims’:
✔ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ building ‘causeSims_0.1.0.tar.gz’
  Warning: invalid uid value replaced by that for user 'nobody'
  
[1] "/path/to/causeSims_0.1.0.tar.gz"

Then continue with STEPS 2, 3, 4 & 5 of paragraph "For packages not in CRAN" above.

 

BioConductor packages

BioConductor is a package manager that enhances R with bio-informatic packages. A local BioConductor mirror has been deployed to ease the installation.

First, you have to define a `~/.Rprofile` file with, at least, the following content:

options(
    BioC_mirror = "http://mirror.dcsr.unil.ch/bioconductor",
    repos = "http://mirror.dcsr.unil.ch/cran"
)
options(
    BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS = FALSE
)

 

It is very important to have an empty new line at the end of the file!

Then you can launch R:

source /dcsrsoft/spack/bin/setup_dcsrsoft
module load gcc r
R

And install the BioConducter package manager:

> install.packages("http://mirror.dcsr.unil.ch/cran/src/contrib/Archive/BiocManager/BiocManager_1.30.10.tar.gz", repos=NULL, type="source")
> install.packages("http://mirror.dcsr.unil.ch/bioconductor/BiocVersion_3.12.0.tar.gz", repos=NULL, type="source")
> library(BiocManager)

The first step might ask if you want to use a personal library, you can answer yes to both questions.

Finally, you can install the BioConductor packages, for instance edgeR:

> BiocManager::install("edgeR")

At the end of the installation, R might ask you if you want to update BiocManager package. Please don't since newer version is not working with the installed version of R

Python packages

Thankfully Python packages are somewhat easier to deal with - here we use PyTorch as an example

First, on a system that has internet access use pip3 download

$ mkdir torch

$ cd torch

$ pip3 download torch torchvision
Collecting torch
  Downloading https://files.pythonhosted.org/packages/76/58/668ffb25215b3f8231a550a227be7f905f514859c70a65ca59d28f9b7f60/torch-1.5.0-cp37-cp37m-manylinux1_x86_64.whl (752.0MB)
     |████████████████████████████████| 752.0MB 33kB/s 
  Saved ./torch-1.5.0-cp37-cp37m-manylinux1_x86_64.whl
Collecting torchvision
  Downloading https://files.pythonhosted.org/packages/7b/ed/a894f274a7733d6492e438a5831a95b507c5ec777edf6d8c3b97574e08c4/torchvision-0.6.0-cp37-cp37m-manylinux1_x86_64.whl (6.6MB)
     |████████████████████████████████| 6.6MB 15.4MB/s 
  Saved ./torchvision-0.6.0-cp37-cp37m-manylinux1_x86_64.whl
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/1f/df/7988fbbdc8c9b8efb575029498ad84b77e023a3e4623e85068823a102b1d/numpy-1.18.4-cp37-cp37m-manylinux1_x86_64.whl
  Saved ./numpy-1.18.4-cp37-cp37m-manylinux1_x86_64.whl
Collecting future
  Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
     |████████████████████████████████| 829kB 23.1MB/s 
  Saved ./future-0.18.2.tar.gz
Collecting pillow>=4.1.1
  Downloading https://files.pythonhosted.org/packages/ab/f8/d3627cc230270a6a4eedee32974fbc8cb26c5fdb8710dd5ea70133640022/Pillow-7.1.2-cp37-cp37m-manylinux1_x86_64.whl (2.1MB)
     |████████████████████████████████| 2.1MB 15.3MB/s 
  Saved ./Pillow-7.1.2-cp37-cp37m-manylinux1_x86_64.whl
Successfully downloaded torch torchvision numpy future pillow

This will download all the required files which can then be copied to the system without internet access. 

Then we can move `torch` directory to Jura using SFTP server: see 

https://wiki.unil.ch/ci/books/service-de-calcul-haute-performance-%28hpc%29/page/jura

Finally, we can use pip / pip3 to install the package from the downloaded files (torch directory).

$ pip install --user --no-index --find-links=torch torch torchvision

Please be aware that some Python packages take up a lot of space and you may wish to set a non standard installation directory via the --target option of pip - see the Pip documentation for full details.


Revision #35
Created 9 April 2020 13:52:19 by Emmanuel Jeanvoine
Updated 8 September 2021 11:38:00 by Emmanuel Jeanvoine