Issues with using geotargets and crew

[rsbravoh]

I'm working on a pipeline that incorporates geotargets to perform meteorological data interpolations on Spatraster, and I run multiple iterations on each dataset. If I use crew::crew_controller_local(worker = 4) when running the pipeline, I must set deployment = 'main' following the following; otherwise, the error is: "could not find packages terra, tidyterra in library paths". The same happens if I set deployment = 'worker".
Why does this happen? Use deployment = 'main' following #52. Using deployment = 'main', the pipeline runs and doesn't throw the error, but it doesn't parallelize.
If I don't use crew, there's no problem, but I need to use crew. Please, I need your help and explain.

[brownag]

The difference has to do with deployment = "worker" running each worker isolated in a separate R process.

The error "could not find packages ... in library paths ..." comes from targets:::load_packages(). This indicates that you have set terra and tidyterra as packages, but targets can't find packages with those names in the library path the worker is using.

Have you tried setting the library option via tar_option_set() to a path you know the packages are installed in? e.g.

 tar_option_set(
    controller = crew::crew_controller_local(worker = 4),
    deployment = "worker",
    library = .libPaths(),
    packages = c("terra", "tidyterra")
)

Are you using renv or something that might be affecting the library path? Do you have anything about the library paths changed in a .Rprofile file?

It is not normally required to set library for this case, but my guess is there is some sort of a mismatch in the library paths you have in the main process vs. the worker processes.

[rsbravoh]

Thanks for your reply. I checked the libraries and they are in order in the path ~/R/x86_64-pc-linux-gnu-library/4.5. I followed your advice, but it didn't solve the problem. But it helped me see alternatives. The one that got me moving was removing "terra" and "tidy terra" from tar_option_set(packages = ) and setting it to geotargets::tar_terra_rast(deployment = 'worker', packages = c('terra', 'tidyterra'), library = .libPaths("~/R/x86_64-pc-linux-gnu-library/4.5")). This configuration launches the workers, but before resolving the first one, it gives this error:
target interpolar_tn_5cc5f2c2c41fda5d error: unable to find an inherited method for function 'wrap' for signature 'x = "NULL"'. I understand that this is done by tar_terra_rast(). What's happening here?

[brownag]

The error indicates that a NULL object is being passed to the tar_terra_rast() marshalling method .

I think I'd need to see the the code expression in your tar_terra_rast() target to say exactly what the cause is.

Does your target code return a SpatRaster object? How are you using tidyterra here?

A possibly related issue using tidyterra in targets and ggplot objects: #159

[rsbravoh]

Before answering, I've been running some tests that might shed more light on the problem.

The error indicates that a NULL object is being passed to the tar_terra_rast() serialization method.

Answer: The strange thing is that the error occurs when I use controller <- crew::crew_controller_local(name = 'controleitor', worker = 4, seconds_idle = 180, seconds_timeout = 600).

> > tar_make(interpolar_tn2)
> here() starts at /home/jgatica/Documentos/Proy_R/area_cobertura
> [conflicted] Will prefer dplyr::select over any other package.
> [conflicted] Will prefer dplyr::filter over any other package.
> [conflicted] Will prefer dplyr::lag over any other package.
> [conflicted] Will prefer here::here over any other package.
> [conflicted] Will prefer terra::wrap over any other package.
> ✔ skipping targets (1 so far)...
> ▶ dispatched branch interpolar_tn2_5cc5f2c2c41fda5d
> ▶ dispatched branch interpolar_tn2_7960cde529c8973b
> ▶ dispatched branch interpolar_tn2_1425bd8205663e26
> ▶ dispatched branch interpolar_tn2_efd16a16bffe398d
> ✔ skipped pipeline [13.974 seconds]
> Error:
> ! targets::tar_make() error

If I don't use it, there's no problem.

I think I'd need to look at the code expression in your tar_terra_rast() target to determine the exact cause.

The target is as follows:

tar_terra_rast(interpolate_tn2, stack_interpolation(training_tn, stack, variable = "tn", cl, residuals = 'sr'),
pattern = map(training_tn), deployment = 'worker', cue = tar_cue("always"), packages = c('terra'), library = .libPaths("~/R/x86_64-pc-linux-gnu-library/4.5"))

The custom functions are:

interpolate_t <- function(list_data1, stack, variable, shp, residuals){
month_name <- list_data1 |> as.data.frame() |> distinct(month) |> pull()
iteration_id <- format(Sys.time(), "%m%d_%H%M%OS2") |>
stringr::str_remove("\\.") 
template <- terra::rast(ext(stack[[1]]), resolution= res(stack[[1]]), crs = crs(stack[[1]])) 
df_model <- list_data1 |> as.data.frame() 
formula <- as.formula(paste0(noquote({{variable}}), '~ s(x,y, k =25) + s(elevation, k = 7) + s(continentality, k = 7)')) 
model <- mgcv::gam(formula = formula, data = df_modelo, method = "REML", select = TRUE) 
df_model <- df_model |> mutate(residuals = mgcv::residuals.gam(model, type = 'response')) 
df_predict <- terra::as.data.frame(stack, na.rm = FALSE) |> 
dplyr::select(x, y, elevation, continentality) 
df_predict$predict <- predict(model, newdata = df_predict, type = "response") 
df_predict <- df_predict |> dplyr::select(x, y, predict) 
r_gam <- setValues(template, as.matrix(df_predict[,3])) # "plan C" 
#r_gam <- terra::predict(stack, model, type = 'response') # "plan B" 
#r_gam <- tidyterra::as_spatraster(df_predict, xycols = 1:2, crs = 'EPSG:32719' , digits = 3, ext(stack[[1]])) |> round(2) # original line PLAN A
if(residuals == 'sr'){ 
r <- r_gam 
}else if(residuals == 'idw'){ 
gs <- gstat::gstat(formula = residuals ~ 1, locations = ~x+y, data = list_data1, nmax=1, set=list(idp = 2)) 
idw <- terra::interpolate(template, gs, debug.level=1)[[1]] |> round(2) |> mask(shp) 
r <- r_gam + idw 
} 
names(r) <- month_name 
writeRaster(r, here::here('results', variable, paste0('metric_', residuals), paste0(variable, '_', iteration_id, '_', month_name, '.tif'))) 
return(r)
}

and

stack_interpolation <- function(list_data1, stack, variable, shp, residuals){
stack <- purrr::map(
list_data1, ~interpolate_t(.x, stack, variable, shp, residuals),  .progress = TRUE)
raster_stack <- terra::rast(stack)
return(raster_stack)
}

Does your target code return a SpatRaster object? How are you using tidyterra here?

The Interpolate_t() function produces a SpatRaster, and the stack_interpolation() function produces a SpatRaster stack.
In interpolate() I used
#r_gam <- tidyterra::as_spatraster(df_predict, xycols = 1:2, crs = 'EPSG:32719' , digits = 3, ext(stack[[1]])) |> round(2) # original line PLAN A that returns a Spatraster.

Then I used
#r_gam <- terra::predict(stack, model, type = 'response') # "plan B" that returns a Spatraster.
and finally
r_gam <- setValues(template, as. matrix(df_predict[,3])) # "plan C" that returns a Spatraster.
I stuck with plan C.

However, it's the same every time. It gives the error if I use Crew. If I don't use Crew, there's no problem.
But my interest is using Crew because of the amount I have to process.

A possible issue related to using tidyterra on ggplot targets and objects: #159.

I don't know. The strange thing is that the error only occurs if I use 'worker'.

[Aariq]

Not sure this will help, but I noticed that your function uses the gstat and mgcv packages, which you might try adding to the packages argument if they're not in the main tar_options_set()

You might try a combination of print debugging and saving logs. E.g. somewhere in your function you could check the contents of your assumed spatraster with a print(r_gam) or print(class(r_gam)) or something, then enable worker logging like so:

tar_option_set(
  controller = crew_controller_local(
    workers = 2,
    options_local = crew_options_local(log_directory = "logs")
  )
)

You should then be able to inspect the logs and see what the result of print(r_gam) is at that point in your function.

[Aariq]

@rsbravoh were you able to resolve this? Without a minimal reproducible example it is difficult for me to tell if this is a geotargets issue or something else. My guess is something else because we have passing tests that run tar_terra_rast() targets in crew workers.

geotargets/tests/testthat/test-tar-terra-rast.R

Lines 22 to 41 in 0e50a47

	targets::tar_test("tar_terra_rast() works with multiple workers (tests un/marshaling)", {
	targets::tar_script({
	targets::tar_option_set(
	controller = crew::crew_controller_local(workers = 2)
	)
	list(
	geotargets::tar_terra_rast(
	rast1,
	terra::rast(system.file("ex/elev.tif", package = "terra"))
	),
	geotargets::tar_terra_rast(
	rast2,
	terra::rast(system.file("ex/elev.tif", package = "terra"))
	)
	)
	})
	targets::tar_make(reporter = "silent")
	expect_true(all(is.na(targets::tar_meta()$error)))
	expect_s4_class(targets::tar_read(rast1), "SpatRaster")
	})