# futurize: Parallelize Common Functions via One Magic Function ![The 'futurize' hexlogo](reference/figures/futurize-logo.png) ## TL;DR The **futurize** package makes it extremely simple to parallelize your existing map-reduce calls, but also a growing set of domain-specific calls. All you need to know is that there is a single function called [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) that will take care of everything, e.g. ``` r y <- lapply(x, fcn) |> futurize() y <- map(x, fcn) |> futurize() b <- boot(city, ratio, R = 999) |> futurize() ``` The [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) function parallelizes via **[futureverse](https://www.futureverse.org)**, meaning your code can take advantage of any **[supported future backends](https://www.futureverse.org/backends.html)**, whether it be parallelization on your local computer, across multiple computers, in the cloud, or on a high-performance compute (HPC) cluster. The **futurize** package has only one hard dependency - the **[future](https://future.futureverse.org)** package. All other dependencies are optional “buy-in” dependencies as shown in the below tables. In addition to getting access to all future-based parallel backends, by using [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) you also get access to all the benefits that come with **futureverse**, including **structured concurrency**. For example, it ensures that remaining parallel tasks are cancelled if there is an error or an interrupt. Also, if the function you parallelize outputs messages and warnings, they will be relayed from the parallel worker to your main R session, just as you get when running sequentially. This is particularly useful when troubleshooting or debugging. Using **futurize** comes with a zero risk buy-in. If there is ever a parallel universe where [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) suddenly stops working, setting `futurize <- identical` avoids rewrites while make all code to run sequentially. ## Supported map-reduce packages The **futurize** package supports transpilation of functions from multiple packages. The tables below summarize the supported map-reduce (Table 1) and domain-specific (Tables 2 and 3) functions, respectively. To programmatically see which packages are currently supported, use: ``` r futurize_supported_packages() ``` To see which functions are supported for a specific package, use: ``` r futurize_supported_functions("caret") ``` | Package | Functions | Requires | |----|----|----| | **base** | [`lapply()`](https://rdrr.io/r/base/lapply.html), [`sapply()`](https://rdrr.io/r/base/lapply.html), [`tapply()`](https://rdrr.io/r/base/tapply.html), [`vapply()`](https://rdrr.io/r/base/lapply.html), [`mapply()`](https://rdrr.io/r/base/mapply.html), [`.mapply()`](https://rdrr.io/r/base/mapply.html), [`Map()`](https://rdrr.io/r/base/funprog.html), [`eapply()`](https://rdrr.io/r/base/eapply.html), [`apply()`](https://rdrr.io/r/base/apply.html), [`by()`](https://rdrr.io/r/base/by.html), [`replicate()`](https://rdrr.io/r/base/lapply.html), [`Filter()`](https://rdrr.io/r/base/funprog.html) | **[future.apply](https://future.apply.futureverse.org)** | | **stats** | [`kernapply()`](https://rdrr.io/r/stats/kernapply.html) | **[future.apply](https://future.apply.futureverse.org)** | | **[purrr](https://cran.r-project.org/package=purrr)** | `map()` and variants, `map2()` and variants, `pmap()` and variants, `imap()` and variants, `modify()`, `modify_if()`, `modify_at()`, `map_if()`, `map_at()` | **[furrr](https://furrr.futureverse.org)** | | **[crossmap](https://cran.r-project.org/package=crossmap)** | `xmap()` and variants, `xwalk()`, `map_vec()`, `map2_vec()`, `pmap_vec()`, `imap_vec()` | \- | | **[foreach](https://cran.r-project.org/package=foreach)** | `%do%`, e.g. `foreach() %do% { }`, `times() %do% { }` | **[doFuture](https://doFuture.futureverse.org)** | | **[plyr](https://cran.r-project.org/package=plyr)** | `aaply()` and variants, `ddply()` and variants, `llply()` and variants, `mlply()` and variants | **[doFuture](https://doFuture.futureverse.org)** | | **[pbapply](https://cran.r-project.org/package=pbapply)** | `pblapply()`, `pbsapply()` and variants, `pbby()`, `pbreplicate()` and `pbwalk()` | **[future.apply](https://future.apply.futureverse.org)** | | **[BiocParallel](https://bioconductor.org/packages/BiocParallel/)** | `bplapply()`, `bpmapply()`, `bpvec()`, `bpiterate()`, `bpaggregate()` | **[doFuture](https://doFuture.futureverse.org)** | *Table 1: Map-reduce functions currently supported by [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) for parallel transpilation.* Here are some examples: ``` r library(futurize) plan(multisession) xs <- 1:10 ys <- lapply(xs, sqrt) |> futurize() xs <- 1:10 ys <- purrr::map(xs, sqrt) |> futurize() xs <- 1:10 ys <- crossmap::xmap_dbl(xs, ~ .y * .x) |> futurize() library(foreach) xs <- 1:10 ys <- foreach(x = xs) %do% { sqrt(x) } |> futurize() xs <- 1:10 ys <- plyr::llply(xs, sqrt) |> futurize() xs <- 1:10 ys <- pbapply::pblapply(xs, sqrt) |> futurize() xs <- 1:10 ys <- BiocParallel::bplapply(xs, sqrt) |> futurize() ``` and ``` r ys <- replicate(3, rnorm(1)) |> futurize() y <- by(warpbreaks, warpbreaks[,"tension"], function(x) lm(breaks ~ wool, data = x)) |> futurize() xs <- EuStockMarkets[, 1:2] k <- kernel("daniell", 50) xs_smooth <- stats::kernapply(xs, k = k) |> futurize() ``` ## Supported domain-specific packages You can also futurize calls from a growing set of domain-specific CRAN and Bioconductor packages that have optional built-in support for parallelization. ### CRAN packages with support for futurize | Package | Functions | Requires | |----|----|----| | **[boot](https://cran.r-project.org/package=boot)** | `boot()`, `censboot()`, `tsboot()` | \- | | **[caret](https://cran.r-project.org/package=caret)** | `bag()`, `gafs()`, `nearZeroVar()`, `rfe()`, `safs()`, `sbf()`, `train()` | **[doFuture](https://doFuture.futureverse.org)** | | **[DiceKriging](https://cran.r-project.org/package=DiceKriging)** | `km()` | **[doFuture](https://doFuture.futureverse.org)** | | **[ez](https://cran.r-project.org/package=ez)** | `ezBoot()`, `ezPerm()`, `ezPlot2()` | **[doFuture](https://doFuture.futureverse.org)** | | **[fwb](https://ngreifer.github.io/fwb/)** | `fwb()`, `vcovFWB()` | \- | | **[gamlss](https://cran.r-project.org/package=gamlss)** | `add1All()`, `add1TGD()`, `drop1All()`, `drop1TGD()`, `gamlssCV()` | \- | | **[glmmTMB](https://cran.r-project.org/package=glmmTMB)** | [`profile()`](https://rdrr.io/r/stats/profile.html) for ‘glmmTMB’ | \- | | **[glmnet](https://cran.r-project.org/package=glmnet)** | `cv.glmnet()` | **[doFuture](https://doFuture.futureverse.org)** | | **[kernelshap](https://cran.r-project.org/package=kernelshap)** | `kernelshap()`, `permshap()` | **[doFuture](https://doFuture.futureverse.org)** | | **[lme4](https://cran.r-project.org/package=lme4)** | `allFit()`, `bootMer()`, [`influence()`](https://rdrr.io/r/stats/lm.influence.html) and [`profile()`](https://rdrr.io/r/stats/profile.html) for ‘merMod’ | \- | | **[metafor](https://cran.r-project.org/package=metafor)** | [`profile()`](https://rdrr.io/r/stats/profile.html), [`rstudent()`](https://rdrr.io/r/stats/influence.measures.html), [`cooks.distance()`](https://rdrr.io/r/stats/influence.measures.html), [`dfbetas()`](https://rdrr.io/r/stats/influence.measures.html) for ‘rma’ | \- | | **[mgcv](https://cran.r-project.org/package=mgcv)** | `bam()`, [`predict()`](https://rdrr.io/r/stats/predict.html) for ‘bam’ | \- | | **[modelsummary](https://cran.r-project.org/package=modelsummary)** | `modelsummary()`, `msummary()`, `modelplot()` | **[future.apply](https://future.apply.futureverse.org)** | | **[parameters](https://cran.r-project.org/package=parameters)** | `bootstrap_model()`, `bootstrap_parameters()` | \- | | **[partykit](https://cran.r-project.org/package=partykit)** | `cforest()`, `ctree_control()`, `mob_control()`, `varimp()` for ‘cforest’ | **[future.apply](https://future.apply.futureverse.org)** | | **[pls](https://cran.r-project.org/package=pls)** | `mvr()`, `plsr()`, `pcr()`, `cppls()`, `crossval()` | \- | | **[pvclust](https://cran.r-project.org/package=pvclust)** | `pvclust()` | \- | | **[riskRegression](https://cran.r-project.org/package=riskRegression)** | `Score()` for ‘list’ | **[doFuture](https://doFuture.futureverse.org)** | | **[rugarch](https://cran.r-project.org/package=rugarch)** | `arfimacv()`, `arfimadistribution()`, `arfimaroll()`, `autoarfima()`, `multifilter()`, `multifit()`, `multiforecast()`, `ugarchboot()`, `ugarchdistribution()`, `ugarchroll()` | \- | | **[sandwich](https://cran.r-project.org/package=sandwich)** | `vcovBS()`, `vcovJK()` | **[future.apply](https://future.apply.futureverse.org)** | | **[seriation](https://cran.r-project.org/package=seriation)** | `seriate_best()`, `seriate_rep()` | **[doFuture](https://doFuture.futureverse.org)** | | **[shapr](https://cran.r-project.org/package=shapr)** | `explain()`, `explain_forecast()` | \- | | **[Sim.DiffProc](https://cran.r-project.org/package=Sim.DiffProc)** | `MCM.sde()` | \- | | **[SimDesign](https://cran.r-project.org/package=SimDesign)** | `runSimulation()`, `runArraySimulation()` | \- | | **[stars](https://cran.r-project.org/package=stars)** | `st_apply()` | **[future.apply](https://future.apply.futureverse.org)** | | **[strucchange](https://cran.r-project.org/package=strucchange)** | `breakpoints()` for ‘formula’ | **[doFuture](https://doFuture.futureverse.org)** | | **[SuperLearner](https://cran.r-project.org/package=SuperLearner)** | `CV.SuperLearner()` | \- | | **[tm](https://cran.r-project.org/package=tm)** | `TermDocumentMatrix()`, `tm_index()`, `tm_map()` | \- | | **[TSP](https://cran.r-project.org/package=TSP)** | `solve_TSP()` | **[doFuture](https://doFuture.futureverse.org)** | | **[vegan](https://cran.r-project.org/package=vegan)** | `adonis()`, `adonis2()`, [`anova()`](https://rdrr.io/r/stats/anova.html) for ‘cca’, `anosim()`, `cascadeKM()`, `estaccumR()`, `mantel()`, `mantel.partial()`, `metaMDSiter()`, `mrpp()`, `oecosimu()`, `ordiareatest()`, `permutest()` for ‘betadisper’, and ‘cca’ | \- | *Table 2: CRAN packages with domain-specific functions currently supported by [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) for parallel transpilation.* Here are some examples: ``` r ratio <- function(d, w) sum(d$x * w)/sum(d$u * w) b <- boot::boot(boot::city, ratio, R = 999) |> futurize() ctrl <- caret::trainControl(method = "cv", number = 10) model <- caret::train(Species ~ ., data = iris, method = "rf", trControl = ctrl) |> futurize() rt <- ez::ezBoot(data = ANT, dv = rt, wid = subnum, within = .(cue, flank), between = group) |> futurize() f <- fwb::fwb(boot::city, ratio, R = 999) |> futurize() m <- DiceKriging::km(~., design = design, response = response, multistart = 8L) |> futurize() cv <- gamlss::gamlssCV(y ~ pb(x), data = abdom, K.fold = 10) |> futurize() cv <- glmnet::cv.glmnet(x, y) |> futurize() ks <- kernelshap::kernelshap(model, X = x_explain, bg_X = bg_X) |> futurize() m <- lme4::allFit(models) |> futurize() fit <- metafor::rma(yi, vi) pr <- profile(fit) |> futurize() b <- mgcv::bam(y ~ s(x0, bs = bs) + s(x1, bs = bs), data = dat) |> futurize() fit <- parameters::bootstrap_model(model, iterations = 1000) |> futurize() cf <- partykit::cforest(dist ~ speed, data = cars) |> futurize() m <- pls::plsr(density ~ NIR, ncomp = 10, data = yarn, validation = "CV") |> futurize() fit <- pvclust::pvclust(mtcars, nboot = 1000) |> futurize() v <- sandwich::vcovBS(fm) |> futurize() sc <- riskRegression::Score(list("CSC" = fit), data = d, formula = Hist(time, event) ~ 1, times = 5, B = 100, split.method = "bootcv") |> futurize() roll <- rugarch::ugarchroll(spec, sp500ret, n.start = 1000, refit.window = "moving", refit.every = 100) |> futurize() result <- shapr::explain(model, x_explain, x_train, approach = "empirical", phi0 = phi0) |> futurize() o <- seriation::seriate_best(d_supreme) |> futurize() res <- Sim.DiffProc::MCM.sde(model, statistic = stat, R = 100) |> futurize() res <- SimDesign::runSimulation(Design, replications = 1000, generate = Generate, analyse = Analyse, summarise = Summarise) |> futurize() s <- stars::st_as_stars(matrix(1:20, nrow = 5, ncol = 4)) res <- stars::st_apply(s, MARGIN = 1, FUN = mean) |> futurize() bp <- strucchange::breakpoints(Nile ~ 1) |> futurize() res <- SuperLearner::CV.SuperLearner(Y, X, SL.library = SL.library) |> futurize() m <- tm::tm_map(crude, content_transformer(tolower)) |> futurize() tour <- TSP::solve_TSP(USCA50, method = "nn", rep = 10) |> futurize() md <- vegan::mrpp(dune, Management) |> futurize() ``` ### Bioconductor packages with support for futurize | Package | Functions | Requires | |----|----|----| | **[DESeq2](https://bioconductor.org/packages/DESeq2/)** | `DESeq()`, `lfcShrink()`, `results()` | **[doFuture](https://doFuture.futureverse.org)** | | **[fgsea](https://bioconductor.org/packages/fgsea/)** | `fgsea()`, `fgseaMultilevel()`, `fgseaSimple()`, `fgseaLabel()`, `geseca()`, `gesecaSimple()`, `collapsePathwaysGeseca()` | **[doFuture](https://doFuture.futureverse.org)** | | **[GenomicAlignments](https://bioconductor.org/packages/GenomicAlignments/)** | `summarizeOverlaps()` | **[doFuture](https://doFuture.futureverse.org)** | | **[GSVA](https://bioconductor.org/packages/GSVA/)** | `gsva()`, `gsvaRanks()`, `gsvaScores()`, `spatCor()` | **[doFuture](https://doFuture.futureverse.org)** | | **[Rsamtools](https://bioconductor.org/packages/Rsamtools/)** | `countBam()`, `scanBam()` | **[doFuture](https://doFuture.futureverse.org)** | | **[scater](https://bioconductor.org/packages/scater/)** | `calculatePCA()`, `calculateTSNE()`, `calculateUMAP()`, `runPCA()`, `runTSNE()`, `runUMAP()`, `runColDataPCA()`, `nexprs()`, `getVarianceExplained()`, `plotRLE()` | **[doFuture](https://doFuture.futureverse.org)** | | **[scuttle](https://bioconductor.org/packages/scuttle/)** | `calculateAverage()`, `logNormCounts()`, `normalizeCounts()`, `perCellQCMetrics()`, `perFeatureQCMetrics()`, `addPerCellQCMetrics()`, `addPerFeatureQCMetrics()`, `addPerCellQC()`, `addPerFeatureQC()`, `numDetectedAcrossCells()`, `numDetectedAcrossFeatures()`, `sumCountsAcrossCells()`, `sumCountsAcrossFeatures()`, `summarizeAssayByGroup()`, `aggregateAcrossCells()`, `aggregateAcrossFeatures()`, `librarySizeFactors()`, `computeLibraryFactors()`, `geometricSizeFactors()`, `computeGeometricFactors()`, `medianSizeFactors()`, `computeMedianFactors()`, `pooledSizeFactors()`, `computePooledFactors()`, `fitLinearModel()` | **[doFuture](https://doFuture.futureverse.org)** | | **[SingleCellExperiment](https://bioconductor.org/packages/SingleCellExperiment/)** | `applySCE()` | **[doFuture](https://doFuture.futureverse.org)** | | **[sva](https://bioconductor.org/packages/sva/)** | `ComBat()`, `read.degradation.matrix()` | **[doFuture](https://doFuture.futureverse.org)** | *Table 3: Bioconductor packages with domain-specific functions currently supported by [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) for parallel transpilation.* Here are some examples: ``` r dds <- DESeq2::DESeq(dds) |> futurize() res <- fgsea::fgsea(pathways, stats) |> futurize() se <- GenomicAlignments::summarizeOverlaps(features, bam_files) |> futurize() es <- GSVA::gsva(GSVA::gsvaParam(expr, geneSets)) |> futurize() counts <- Rsamtools::countBam(bamViews) |> futurize() sce <- scater::runPCA(sce) |> futurize() qc <- scuttle::perFeatureQCMetrics(sce) |> futurize() result <- SingleCellExperiment::applySCE(sce, scuttle::perFeatureQCMetrics) |> futurize() adjusted <- sva::ComBat(dat = dat, batch = batch) |> futurize() ``` # Package index ## All functions - [`futurize()`](https://futurize.futureverse.org/reference/futurize.md) : Turn common R function calls into concurrent calls for parallel evaluation - [`futurize_options()`](https://futurize.futureverse.org/reference/futurize_options.md) : Options controlling resources, scheduling and evaluation of futures - [`futurize_supported_packages()`](https://futurize.futureverse.org/reference/futurize_supported_packages.md) [`futurize_supported_functions()`](https://futurize.futureverse.org/reference/futurize_supported_packages.md) : List packages and functions supporting futurization - [`zzz-futurize.options`](https://futurize.futureverse.org/reference/zzz-futurize.options.md) [`futurize.options`](https://futurize.futureverse.org/reference/zzz-futurize.options.md) [`futurize.debug`](https://futurize.futureverse.org/reference/zzz-futurize.options.md) [`futurize.enable`](https://futurize.futureverse.org/reference/zzz-futurize.options.md) [`R_FUTURIZE_DEBUG`](https://futurize.futureverse.org/reference/zzz-futurize.options.md) [`R_FUTURIZE_ENABLE`](https://futurize.futureverse.org/reference/zzz-futurize.options.md) : Options used by futurize # Articles ### All vignettes - [Parallelize base-R apply functions](https://futurize.futureverse.org/articles/futurize-11-apply.md): - [Parallelize 'purrr' functions](https://futurize.futureverse.org/articles/futurize-21-purrr.md): - [Parallelize 'crossmap' functions](https://futurize.futureverse.org/articles/futurize-22-crossmap.md): - [Parallelize 'plyr' functions](https://futurize.futureverse.org/articles/futurize-31-plyr.md): - [Parallelize 'foreach' functions](https://futurize.futureverse.org/articles/futurize-51-foreach.md): - [Parallelize 'pbapply' functions](https://futurize.futureverse.org/articles/futurize-55-pbapply.md): - [Parallelize 'BiocParallel' functions](https://futurize.futureverse.org/articles/futurize-61-BiocParallel.md): - [Parallelize 'boot' functions](https://futurize.futureverse.org/articles/futurize-81-boot.md): - [Parallelize 'caret' functions](https://futurize.futureverse.org/articles/futurize-81-caret.md): - [Parallelize 'DESeq2' functions](https://futurize.futureverse.org/articles/futurize-81-DESeq2.md): - [Parallelize 'DiceKriging' functions](https://futurize.futureverse.org/articles/futurize-81-DiceKriging.md): - [Parallelize 'ez' functions](https://futurize.futureverse.org/articles/futurize-81-ez.md): - [Parallelize 'fgsea' functions](https://futurize.futureverse.org/articles/futurize-81-fgsea.md): - [Parallelize 'fwb' functions](https://futurize.futureverse.org/articles/futurize-81-fwb.md): - [Parallelize 'gamlss' functions](https://futurize.futureverse.org/articles/futurize-81-gamlss.md): - [Parallelize 'GenomicAlignments' functions](https://futurize.futureverse.org/articles/futurize-81-GenomicAlignments.md): - [Parallelize 'glmmTMB' functions](https://futurize.futureverse.org/articles/futurize-81-glmmTMB.md): - [Parallelize 'glmnet' functions](https://futurize.futureverse.org/articles/futurize-81-glmnet.md): - [Parallelize 'GSVA' functions](https://futurize.futureverse.org/articles/futurize-81-GSVA.md): - [Parallelize 'kernelshap' functions](https://futurize.futureverse.org/articles/futurize-81-kernelshap.md): - [Parallelize 'lme4' functions](https://futurize.futureverse.org/articles/futurize-81-lme4.md): - [Parallelize 'metafor' functions](https://futurize.futureverse.org/articles/futurize-81-metafor.md): - [Parallelize 'mgcv' functions](https://futurize.futureverse.org/articles/futurize-81-mgcv.md): - [Parallelize 'modelsummary' functions](https://futurize.futureverse.org/articles/futurize-81-modelsummary.md): - [Parallelize 'parameters' functions](https://futurize.futureverse.org/articles/futurize-81-parameters.md): - [Parallelize 'partykit' functions](https://futurize.futureverse.org/articles/futurize-81-partykit.md): - [Parallelize 'pls' functions](https://futurize.futureverse.org/articles/futurize-81-pls.md): - [Parallelize 'pvclust' functions](https://futurize.futureverse.org/articles/futurize-81-pvclust.md): - [Parallelize 'riskRegression' functions](https://futurize.futureverse.org/articles/futurize-81-riskRegression.md): - [Parallelize 'Rsamtools' functions](https://futurize.futureverse.org/articles/futurize-81-Rsamtools.md): - [Parallelize 'rugarch' functions](https://futurize.futureverse.org/articles/futurize-81-rugarch.md): - [Parallelize 'sandwich' functions](https://futurize.futureverse.org/articles/futurize-81-sandwich.md): - [Parallelize 'scater' functions](https://futurize.futureverse.org/articles/futurize-81-scater.md): - [Parallelize 'scuttle' functions](https://futurize.futureverse.org/articles/futurize-81-scuttle.md): - [Parallelize 'seriation' functions](https://futurize.futureverse.org/articles/futurize-81-seriation.md): - [Parallelize 'shapr' functions](https://futurize.futureverse.org/articles/futurize-81-shapr.md): - [Parallelize 'Sim.DiffProc' functions](https://futurize.futureverse.org/articles/futurize-81-Sim.DiffProc.md): - [Parallelize 'SimDesign' functions](https://futurize.futureverse.org/articles/futurize-81-SimDesign.md): - [Parallelize 'SingleCellExperiment' functions](https://futurize.futureverse.org/articles/futurize-81-SingleCellExperiment.md): - [Parallelize 'stars' functions](https://futurize.futureverse.org/articles/futurize-81-stars.md): - [Parallelize 'strucchange' functions](https://futurize.futureverse.org/articles/futurize-81-strucchange.md): - [Parallelize 'SuperLearner' functions](https://futurize.futureverse.org/articles/futurize-81-SuperLearner.md): - [Parallelize 'sva' functions](https://futurize.futureverse.org/articles/futurize-81-sva.md): - [Parallelize 'tm' functions](https://futurize.futureverse.org/articles/futurize-81-tm.md): - [Parallelize 'TSP' functions](https://futurize.futureverse.org/articles/futurize-81-TSP.md): - [Parallelize 'vegan' functions](https://futurize.futureverse.org/articles/futurize-81-vegan.md):