back2source: Automate running pipeline on many projects

[pombredanne]

We are running many d2d pipelines in the issues below.
We need to create the scripts to make this a smooth process:

• Create projects from CSV: Given some CSV file as an input, I would like to create all the projects at once, and ensure that they are executing. Add a batch-create management command #1509
  • Use a helper script to create these CSV inputs for instance from a PURL.
• back2source: Report on many d2d projects at once #1524

The related issues are:

• back2source: Run pipeline for 100 Windows C++ projects #1470
• back2source: Run pipeline for 100 macOS projects #1471
• back2source: Run pipeline for 100 .Net projects #1472
• back2source: Run pipeline for Java and native code and Android apk and dex classes #1473
• back2source: Run pipeline for Python and C/C++ source and the python bytecode and native binaries #1474
• back2source: Run pipeline for Rust source and binaries #1475
• back2source: Run pipeline for Ruby source and gem packages #1476
• back2source: Run pipeline for JavaScript #1477

[tdruez]

The new batch-create management command is available to create projects from a CSV input. (#1437)

Documented at https://scancodeio.readthedocs.io/en/latest/command-line-interface.html#scanpipe-batch-create-input-directory-input-directory-input-list-filename-csv

1. Create a 2 columns CSV file:

project_name	input_urls
project-1.0	https://url.com/source-1.0.zip#from, https://url.com/binary-1.0.zip#to
project-2.0	https://url.com/source-2.0.zip#from, https://url.com/binary-2.0.zip#to

2. Run the batch-create management command with the CSV file as input

• Put the CSV file in your current directory

• Run the following replacing <FILENAME.CSV> with the CSV file name and the <PLATFORM_LABEL> with a pertinent tag (e.g.: cpp, Python, Java, Rust, Ruby, JavaScript, ...)

  docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
      --volume ${PWD}:/input-data:ro \
      web scanpipe batch-create \
      --input-list /input-data/<FILENAME.CSV> \
      --pipeline map_deploy_to_develop \
      --label back2source-<PLATFORM_LABEL> \
      --execute --async
  

• Alternatively, if the input CSV file is available online, on the https://github.com/aboutcode-org/back2source-data/ repo for example, you can provide this URL as the --input-list value:

  docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
      web scanpipe batch-create \
      --input-list https://github.com/aboutcode-org/back2source-data/raw/....../<FILENAME.CSV> \
      --pipeline map_deploy_to_develop \
      --label back2source-<PLATFORM_LABEL> \
      --execute --async
  

[tdruez]

Use a helper script to create these CSV inputs for instance from a PURL.

@pombredanne I don't think we need this at the moment, in case we do, it may be better handled as another issue with more details.

I think we have everything we need to run many projects scan with the current implementation of batch-create (see comment above) and the report feature completed in #1524

The scan execution is happening in #1437 (comment)

Let me know if anything else is needed here, otherwise you can close this issue.