Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send published files through the plugin into a content-addressed store instead of a traditional results/ directory. The plugin provides a lineage store for lineage.store.location = 'cas://lab', enabling Nextflow's lid:// records to live in the same store. It supports cloud environments (AWS Batch with S3), nf-core pipeline integration, and retention policies. The plugin is currently in beta (0.7.0-beta.1) and is built against Nextflow 26.04.6.

Get Started

Enable the plugin and configure both the lineage store and output directory to point at the same store alias:

plugins {
    id 'nf-blocks@0.7.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias must match ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left out; when unset, it defaults to the alias in lineage.store.location. When set, it must name that alias, or the run aborts. Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

When adding nf-blocks to another pipeline via a -c config file, note that a plugins { } block replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins alongside nf-blocks to avoid Nextflow installing the latest release of each plugin.

Examples

Any pipeline with a workflow output block publishes into the store. With the configuration above, running a pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

To read outputs from another pipeline, configure a read-only member alongside your writable one:

plugins {
    id 'nf-blocks@0.7.0-beta.1'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's own outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only here
    }
    resolve = ['mine', 'lab']
}

Then use fromStore in your workflow:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

For S3-based publishing with AWS Batch:

plugins {
    id 'nf-blocks@0.7.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'
}

To browse a store, use:

nextflow plugin nf-blocks:explore

To manage content retention and cleanup:

nextflow plugin nf-blocks:prune --keep-last 3
nextflow plugin nf-blocks:prune --keep-last 3 --apply true
nextflow plugin nf-blocks:sweep
nextflow plugin nf-blocks:sweep --apply true

License

Apache License 2.0. See the COPYING file for details.

What's new

• Status: beta - This is version 0.7.0-beta.1. Pin the version in plugins { } as breaking changes may occur before 1.0 release.
• Milestone 4 (cloud): Any store member, including the writable one, can be a local directory or S3 location. Runs on AWS Batch publish into writable S3 members with optimized hashing (under Fusion, task nodes hash their own outputs; between S3 buckets, server-side copies compute SHA-256).
• Milestone 5 (nf-core pipelines): Lineage recording from workflow output { } blocks only. Processes using publishDir are warned once; files are stored but counted as unjoined.
• Milestone 6 (retention): Every run's content is a root by default and kept until released. Sweeps only delete released, unpinned content past its grace period.
• Index Snapshot seeding: Large stores can seed their cache from Index Snapshots to avoid reading every block on cold starts.
• Selections: Curate Output Items across runs into named groups via the explorer or command-line tools.

Breaking changes

• The on-disk format (block kinds, the index, the Store Log) may change before version 1.0. Block address changes will include migrations, and existing stores retain their blocks since a block's address is its content.
• In typed scripts (nextflow.enable.types = true), Nextflow 26.04.6 does not reach plugin channel factories through channel.fromStore until nextflow-io/nextflow#7694 is fixed. Call nextflow.Channel.fromStore(..., records: true) instead.
• nf-blocks requires the nf-amazon plugin (>=3.9.2). If installed via local build into NXF_PLUGINS_DIR, manually install the dependency: nextflow plugin install nf-amazon@3.9.2.