generic elpd function

[bnicenboim]

I think I wrote about this some time back. But I find myself using this quite often, when I have my own likelihood values either from k-fold cross validation, or from a validation dataset.

print_dims.default <- function(x){}
## ll is the loglikelihood
elpd <- function(ll) {
  elpd_pointwise <- colLogSumExps(ll) - log(nrow(ll))
  elpd_sum <- sum(elpd_pointwise)
  se_elpd <- sqrt(ncol(ll) * var(elpd_pointwise))
  estimates <- matrix(c(elpd_sum, se_elpd), nrow = 1)
  colnames(estimates) <- c("Estimate", "SE")
  rownames(estimates) <- "elpd"
  pointwise <- matrix(elpd_pointwise, ncol = 1)
  colnames(pointwise) <- "elpd"
  out <- list(estimates = estimates, pointwise = pointwise,
              se_elpd_loo = se_elpd)
  return(structure(out, class = c( "loo")))
}

If it's not wrong, maybe consider adding it? (I could do a pull request and add proper documentation, if this is useful).

[jgabry]

Thanks! Yeah we haven't wrapped this up into a nice little function like this, but I agree that it's something that would be useful for the situations you describe. Other loo devs (@avehtari @paul-buerkner @topipa @MansMeg), any opinion on this?

[avehtari]

Ok for me

[jgabry]

(I could do a pull request and add proper documentation, if this is useful).

@bnicenboim Yeah a PR would be great if you have the time!

[bnicenboim]

yeah, sure I'll do it in the next couple of weeks (when I don't teach), together with a vignette. I actually use this function quite a lot.

How should I call it? elpd? elpd_generic?

[paul-buerkner]

I think the name elpd is good, but it should of course be S3 generic with your method being either .default or .matrix depending on the generality we aim for. Later on, we can then add methods for other objects.

[bnicenboim]

Here is my attempt in #159. If that's fine I can give it a try with elpd.function. And I could write a vignette on how to use this function for handmade kfold CV in rstan

[jgabry]

Thanks Bruno! I'll make some comments over at the PR. A vignette on how to use this would be great!