omit realizing transcripts

[HelenaLC]

Hi there. I noticed that the read_parquet call here realizes transcripts into memory as a data.frame. Instead, I would propose something like the following, which would realize only i) cell IDs and ii) unique FOV entries of the cell IDs present in the data. This also prevents costly dplyr computations (group_by, select, distinct, left_join) on a data.frame that could potentially contain millions of entries, which beats the purpose of having a .parquet file to begin with.

mol <- metadata(xen)$transcripts
mol <- read_parquet(mol, as_data_frame=FALSE) # this is important
idx <- pull(mol, "cell_id")
idx <- match(xen$cell_id, idx)
fov <- pull(mol[idx, ], "fov")

[drighelli]

Hi @HelenaLC,

yes, we know about the as_data_frame=FALSE argument, but as far as I remember we had some trouble working with it in that way.

I'm also tagging @BenedettaBanzi in this issue, which wrote that chunk of code, which I suppose we don't really use at the moment, but I could be wrong.

[BenedettaBanzi]

Hi @HelenaLC ,

thanks for your suggestion! Yes, @drighelli is right, I had some trouble using as_data_frame=FALSE, but this was with an old version of arrow, I think they fixed that. This chunk comes from a very old implementation, I think we can replace arrow::read_parquet with data.table::fread(). Xenium is currently releasing their tx file also in csv.gz and the function has an argument select that allows to import only the desired columns.

[drighelli]

Thanks @BenedettaBanzi,

running through the issues today, I realized that I never replied to this.

Anyway, I guess it would be good to keep the support to the parquet and leaving the user the possibility to load it with the as_data_frame=FALSE by aid of an argument.
So, in the end, it will be the user responsibility to work with is from disk or from memory.

Additionally, we can include the fread behaviour as for some other cases where we offer multiple possibilities to load the data.

In any case, this is not a standard official behaviour (adding FoVs to Xenium data), but this conversation could be useful to implement dedicated functions to store the transcripts into the SPE object.