Parallelization in Seurat with future
Compiled: March 27, 2023
Source:vignettes/future_vignette.Rmdfuture_vignette.RmdIn Seurat, we have chosen to use the future framework for parallelization. In this vignette, we will demonstrate how you can take advantage of the future implementation of certain Seurat functions from a user’s perspective. If you are interested in learning more about the future framework beyond what is described here, please see the package vignettes here for a comprehensive and detailed description.
How to use parallelization in Seurat
To access the parallel version of functions in Seurat, you need to load the future package and set the plan. The plan will specify how the function is executed. The default behavior is to evaluate in a non-parallelized fashion (sequentially). To achieve parallel (asynchronous) behavior, we typically recommend the “multiprocess” strategy. By default, this uses all available cores but you can set the workers parameter to limit the number of concurrently active futures.
## sequential:
## - args: function (..., envir = parent.frame())
## - tweaked: FALSE
## - call: NULL## sequential:
## - args: function (..., workers = 4, envir = parent.frame())
## - tweaked: TRUE
## - call: plan("multiprocess", workers = 4)‘Futurized’ functions in Seurat
The following functions have been written to take advantage of the future framework and will be parallelized if the current plan is set appropriately. Importantly, the way you call the function shouldn’t change.
NormalizeData()ScaleData()JackStraw()FindMarkers()FindIntegrationAnchors()FindClusters()- if clustering over multiple resolutions
For example, to run the parallel version of FindMarkers(), you simply need to set the plan and call the function as usual.
library(Seurat)
pbmc <- readRDS("../data/pbmc3k_final.rds")
# Enable parallelization
plan("multiprocess", workers = 4)
markers <- FindMarkers(pbmc, ident.1 = "NK", verbose = FALSE)Comparison of sequential vs. parallel
Here we’ll perform a brief comparison the runtimes for the same function calls with and without parallelization. Note that while we expect that using a parallelized strategy will decrease the runtimes of the functions listed above, the magnitude of that decrease will depend on many factors (e.g. the size of the dataset, the number of workers, specs of the system, the future strategy, etc). The following benchmarks were performed on a desktop computer running Ubuntu 16.04.5 LTS with an Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz and 96 GB of RAM.Click to see bencharking code
timing.comparisons <- data.frame(fxn = character(), time = numeric(), strategy = character())
plan("sequential")
start <- Sys.time()
pbmc <- ScaleData(pbmc, vars.to.regress = "percent.mt", verbose = FALSE)
end <- Sys.time()
timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "ScaleData", time = as.numeric(end -
start, units = "secs"), strategy = "sequential"))
start <- Sys.time()
markers <- FindMarkers(pbmc, ident.1 = "NK", verbose = FALSE)
end <- Sys.time()
timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "FindMarkers", time = as.numeric(end -
start, units = "secs"), strategy = "sequential"))
plan("multiprocess", workers = 4)
start <- Sys.time()
pbmc <- ScaleData(pbmc, vars.to.regress = "percent.mt", verbose = FALSE)
end <- Sys.time()
timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "ScaleData", time = as.numeric(end -
start, units = "secs"), strategy = "multiprocess"))
start <- Sys.time()
markers <- FindMarkers(pbmc, ident.1 = "NK", verbose = FALSE)
end <- Sys.time()
timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "FindMarkers", time = as.numeric(end -
start, units = "secs"), strategy = "multiprocess"))
library(ggplot2)
library(cowplot)
ggplot(timing.comparisons, aes(fxn, time)) + geom_bar(aes(fill = strategy), stat = "identity", position = "dodge") +
ylab("Time(s)") + xlab("Function") + theme_cowplot()
Frequently asked questions
Where did my progress bar go?
Unfortunately, the when running these functions in any of the parallel plan modes you will lose the progress bar. This is due to some technical limitations in thefutureframework and R generally. If you want to monitor function progress, you’ll need to forgo parallelization and useplan("sequential").What should I do if I keep seeing the following error?
Error in getGlobalsAndPackages(expr, envir = envir, globals = TRUE) :
The total size of the X globals that need to be exported for the future expression ('FUN()') is X GiB.
This exceeds the maximum allowed size of 500.00 MiB (option 'future.globals.maxSize'). The X largest globals are ... For certain functions, each worker needs access to certain global variables. If these are larger than the default limit, you will see this error. To get around this, you can set options(future.globals.maxSize = X), where X is the maximum allowed size in bytes. So to set it to 1GB, you would run options(future.globals.maxSize = 1000 * 1024^2). Note that this will increase your RAM usage so set this number mindfully.Session Info
## R version 4.2.0 (2022-04-22)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] cowplot_1.1.1 ggplot2_3.4.1 SeuratObject_4.1.3 Seurat_4.3.0
## [5] future_1.31.0
##
## loaded via a namespace (and not attached):
## [1] Rtsne_0.16 colorspace_2.1-0 deldir_1.0-6
## [4] ellipsis_0.3.2 ggridges_0.5.4 rprojroot_2.0.3
## [7] fs_1.6.1 spatstat.data_3.0-0 farver_2.1.1
## [10] leiden_0.4.3 listenv_0.9.0 ggrepel_0.9.3
## [13] fansi_1.0.4 codetools_0.2-18 splines_4.2.0
## [16] cachem_1.0.7 knitr_1.42 polyclip_1.10-4
## [19] jsonlite_1.8.4 ica_1.0-3 cluster_2.1.3
## [22] png_0.1-8 uwot_0.1.14 spatstat.sparse_3.0-0
## [25] shiny_1.7.4 sctransform_0.3.5 compiler_4.2.0
## [28] httr_1.4.5 Matrix_1.5-3 fastmap_1.1.1
## [31] lazyeval_0.2.2 limma_3.54.1 cli_3.6.0
## [34] later_1.3.0 formatR_1.14 htmltools_0.5.4
## [37] tools_4.2.0 igraph_1.4.1 gtable_0.3.1
## [40] glue_1.6.2 RANN_2.6.1 reshape2_1.4.4
## [43] dplyr_1.1.0 Rcpp_1.0.10 scattermore_0.8
## [46] jquerylib_0.1.4 pkgdown_2.0.7 vctrs_0.5.2
## [49] nlme_3.1-157 spatstat.explore_3.0-6 progressr_0.13.0
## [52] lmtest_0.9-40 spatstat.random_3.1-3 xfun_0.37
## [55] stringr_1.5.0 globals_0.16.2 mime_0.12
## [58] miniUI_0.1.1.1 lifecycle_1.0.3 irlba_2.3.5.1
## [61] goftest_1.2-3 MASS_7.3-56 zoo_1.8-11
## [64] scales_1.2.1 ragg_1.2.5 promises_1.2.0.1
## [67] spatstat.utils_3.0-1 parallel_4.2.0 RColorBrewer_1.1-3
## [70] yaml_2.3.7 memoise_2.0.1 reticulate_1.28
## [73] pbapply_1.7-0 gridExtra_2.3 sass_0.4.5
## [76] stringi_1.7.12 highr_0.10 desc_1.4.2
## [79] rlang_1.0.6 pkgconfig_2.0.3 systemfonts_1.0.4
## [82] matrixStats_0.63.0 evaluate_0.20 lattice_0.20-45
## [85] tensor_1.5 ROCR_1.0-11 purrr_1.0.1
## [88] labeling_0.4.2 patchwork_1.1.2 htmlwidgets_1.6.1
## [91] tidyselect_1.2.0 parallelly_1.34.0 RcppAnnoy_0.0.20
## [94] plyr_1.8.8 magrittr_2.0.3 R6_2.5.1
## [97] generics_0.1.3 withr_2.5.0 pillar_1.8.1
## [100] fitdistrplus_1.1-8 abind_1.4-5 survival_3.3-1
## [103] sp_1.6-0 tibble_3.1.8 future.apply_1.10.0
## [106] KernSmooth_2.23-20 utf8_1.2.3 spatstat.geom_3.0-6
## [109] plotly_4.10.1 rmarkdown_2.20 grid_4.2.0
## [112] data.table_1.14.8 digest_0.6.31 xtable_1.8-4
## [115] tidyr_1.3.0 httpuv_1.6.9 textshaping_0.3.6
## [118] munsell_0.5.0 viridisLite_0.4.1 bslib_0.4.2