Tooling and schemas for the primerschemes index

Installation

Quick Start

Schemas

CLI

pip install primal-page

Each version of a primerscheme has three parts; {schemename}/{ampliconsize}/{version}, which when combined these form the schemes unique identifier.

For a scheme to be added to the repo it requires three essential files.

• primer.bed: Contains the primer information.
• reference.fasta: Contains the reference genomes.
• info.json: Contains key metadata for the scheme.

The primal-page create command will generates the info.json and parses a bed file and a fasta file into primer.bed and reference.fasta

Additional files are copied into the schemes work directory.

For a simple scheme, only fasta file (--reference) and a bed file (--primerbed) are required. Alongside metadata of --schemename, --algorithmversion, --species, --authors.

primal-page create \
    --schemename example-scheme \
    --algorithmversion ps:100 \
    --ampliconsize 500 \
    --species 123  \
    --schemeversion v1.0.0 \
    --primerbed ~/'minimal-scheme/minimal.bed' \
    --reference '~/minimal-scheme/ref.fasta' \
    --authors "me" \
    --authors "you" /
    '~/minimal-scheme' 

create a local fork of https://github.com/quick-lab/primerschemes

Add the files to the forks primerschemes folder

primal-page create primal-page create ... --output ~/primerschemes/primerschemes

Create a pull request

This is the main metadata file for each primerscheme.

• ampliconsize: PositiveInt
• schemeversion: Following format listed below
• schemename: Following format listed below
• primer_bed_md5: MD5 hash of the schemes primer.bed file
• reference_fasta_md5: MD5 hash of the schemes reference.fasta file
• status:
  • withdrawn: Removed due to major issue
  • deprecated: Newer scheme is recommended
  • autogenerated: Scheme has been autogenerated using species-agnostic pipelines
  • draft: Scheme has been inspected in silico
  • tested: Scheme has been tested in the laboratory
  • validated: Scheme has been validated and/or published
• citations: How the scheme should be cited if used. DOI links are recommended, however, tweets/blogs are all allowed
• authors: The person or organization who generated the scheme. It is recommended that only corresponding/primary authors are included, with all other contributors recognized in the citations field
• algorithmversion: The algorithm and the version used to generate the scheme
• species: A list of organisms targeted by this scheme. NCBI TaxIds are recommend
• license: The name of the license the primerscheme is offered under
• primerclass: The class of scheme. Only primerscheme at the moment
• infoschema: The version of the info.json
• articbedversion: The version of the primer.bed (See below)
• description: A free text description to describe the primerscheme
• derivedfrom: To show if this scheme has been based on another primerscheme.
• collections: A collection of vocabulary to provide filtering/grouping of schemes.
  • ARTIC: Developed with the ARTIC network
  • MODJADJI: Developed with MODJADJI
  • QUICK-LAB: Developed with QUICK-LAB
  • COMMUNITY: Developed by the COMMUNITY
  • WASTE-WATER: Scheme capable of recovering genomes from high Ct samples (~30) samples, like wastewater. Typically 400bp schemes
  • CLINICAL-ISOLATES: Scheme capable of recovering genomes from medium Ct samples (~25) samples. Typically ~1000bp schemes
  • WHOLE-GENOME: Scheme that can theoretically recover a full genome
  • PANEL: Scheme that can recover sections of a target genome
  • MULTI-TARGET: Scheme that contains more than one target

Expected primernames (col3) in the primer.bed file

This is the first standard for primernames. It follows the general pattern of {scheme-name|uuid}_{amplicon-number}_{LEFT|RIGHT} and an optional {_alt} to denote spike in primers.

Regex:

V1_PRIMERNAME = r"^[a-zA-Z0-9\-]+_[0-9]+_(LEFT|RIGHT)(_ALT[0-9]*|_alt[0-9]*)*$"

Example:

SARS-CoV-2_10_LEFT
SARS-CoV-2_10_LEFT_alt1

This follows the pattern of {scheme-name|uuid}_{amplicon-number}_{LEFT|RIGHT}_{primer-number}.

• This now enforces that splitting on _ will produce a consistent length, allowing the safe indexing of attributes.
• primer-number is not enforced to be continuous. Therefore, the _0 numbered primer should not be thought of as the original and all other numbers as alts.

Example:

SARS-CoV-2_10_LEFT_0
SARS-CoV-2_10_LEFT_1

Regex:

V2_PRIMERNAME = r"^[a-zA-Z0-9\-]+_[0-9]+_(LEFT|RIGHT)_[0-9]+$"

These are rough guidelines for the format of the primer.bed file. The general format is based on the .bed file and maintains compatibility with other tools.

colnames and indexes:

• 0 - chrom: The name of the reference genome the primers are indexed to
• 1 - chromStart: 0-based inclusive start coordinate
• 2 - chromEnd: 0-based non-inclusive end coordinate
• 3 - primer-name: Name of each primer
• 4 - pool: The pool each primer should be added into. 1 based.
• 5 - strand: Either + (forward) or - (reverse) primer
• 6 - primer-sequence: The 5'-3' sequence of the primer

Depreciated

This was the original 6-col bedfile used very early in PrimalSchemes development, which excluded primer-sequence.

Legacy

This uses the 7 columns described above, alongside V1:primernames.

• Single chrom (reference) is expected
• No header lines

Current

This uses the 7 columns described above, alongside V2:primernames.

• Multiple chrom (references)
• Circular primers are allowed. The start of x_LEFT can be greater than the end of x_RIGHT
• Header lines are used. Denoted with the # character

In the form of v{Major}.{Minor}.{Patch}

• Major: New scheme being generated with different input params
• Minor: Change to primers. Either additional / removal of primers
• Patch: No change to primers. Often used for rebalancing or change in formatting

Regex:

VERSION_PATTERN = r"^v\d+\.\d+\.\d+$"

Must only contain a-z, 0-9, and -. Cannot start or end with -

Regex:

SCHEMENAME_PATTERN = r"^[a-z0-9][a-z0-9-]*[a-z0-9]$"

Usage:

$ primal-page [OPTIONS] COMMAND [ARGS]...

Options:

• --version
• --install-completion: Install completion for the current shell.
• --show-completion: Show completion for the current shell, to copy it or customize the installation.
• --help: Show this message and exit.

Commands:

• aliases: Manage aliases
• build-index: Build an index.json file from all schemes...
• create: Create a new scheme in the required format
• dev: Development commands
• download: Download schemes from the index.json
• modify: Modify an existing scheme's metadata...
• remove: Remove a scheme's version from the repo,...
• validate: Validate a scheme

Manage aliases

Usage:

$ primal-page aliases [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• add: Add an alias:schemename to the alias file
• remove: Remove an alias from the alias file.

Add an alias:schemename to the alias file

Usage:

$ primal-page aliases add [OPTIONS] ALIASES_FILE ALIAS SCHEMENAME

Arguments:

• ALIASES_FILE: The path to the alias file to write to [required]
• ALIAS: The alias to add [required]
• SCHEMENAME: The schemename the alias refers to [required]

Options:

• --help: Show this message and exit.

Remove an alias from the alias file. Does nothing if the alias does not exist

Usage:

$ primal-page aliases remove [OPTIONS] ALIASES_FILE ALIAS

Arguments:

• ALIASES_FILE: The path to the alias file to write to [required]
• ALIAS: The alias to add [required]

Options:

• --help: Show this message and exit.

Build an index.json file from all schemes in the directory

Usage:

$ primal-page build-index [OPTIONS]

Options:

• --gitaccount TEXT: The name of the github account [default: quick-lab]
• --gitserver TEXT: The name of the github server [default: https://github.com/]
• --parentdir PATH: The parent directory [default: .]
• --git-commit-sha TEXT: The git commit
• --force / --no-force: Force the creation of the index.json [default: no-force]
• --help: Show this message and exit.

Create a new scheme in the required format

Usage:

$ primal-page create [OPTIONS]

Options:

• --schemename TEXT: The name of the scheme [required]
• --ampliconsize INTEGER RANGE: Amplicon size [x>=100; required]
• --schemeversion TEXT: Scheme version, default is parsed from config.json [required]
• --species INTEGER: The species this scheme targets. Please use NCBI taxonomy ids [required]
• --authors TEXT: Any authors. To provide multiple, use --authors '1' --authors '2' [required]
• --primerbed PATH: The path to the primer.bed file [required]
• --reference PATH: The path to the reference.fasta file [required]
• --schemestatus [withdrawn|deprecated|autogenerated|draft|tested|validated]: Scheme status [default: draft]
• --citations TEXT: Any associated citations. Please use DOI
• --output PATH: Where to output the scheme [default: primerschemes]
• --configpath PATH: Where the config.json file is located
• --algorithmversion TEXT: The version of primalscheme or other
• --description TEXT: A description of the scheme
• --derivedfrom TEXT: Which scheme has this scheme been derived from
• --primerclass [primerschemes]: The primer class [default: primerschemes]
• --collection [ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP]: The collection tags. To provide multiple, use --collection '1' --collection '2'
• --link-protocol TEXT: Optional link to protocol
• --link-validation TEXT: Optional link to validation data
• --links-homepage TEXT: Optional link to homepage
• --link-vendor TEXT: Optional link to vendors
• --link-misc TEXT: Optional miscellaneous link
• --fix / --no-fix: Attempt to fix the scheme [default: no-fix]
• --contact-info TEXT: Contact information
• --additional-files PATH: Additional files to include in the ./work directory
• --ref-select <TEXT PATH>...: Reference selection file. In the form 'chromosome file'
• --help: Show this message and exit.

Download schemes from the index.json

Usage:

$ primal-page download [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• all: Download all schemes from the index.json
• scheme: Download a scheme from the index.json

Download all schemes from the index.json

Usage:

$ primal-page download all [OPTIONS]

Options:

• --output PATH: The directory the primerschemes dir will be created in [required]
• --index-url TEXT: The URL to the index.json [default: https://raw.githubusercontent.com/quick-lab/primerschemes/main/index.json]
• --help: Show this message and exit.

Download a scheme from the index.json

Usage:

$ primal-page download scheme [OPTIONS] SCHEMENAME AMPLICONSIZE SCHEMEVERSION

Arguments:

• SCHEMENAME: The name of the scheme [required]
• AMPLICONSIZE: Amplicon size [required]
• SCHEMEVERSION: Scheme version [required]

Options:

• --output PATH: The directory the primerschemes dir will be created in [required]
• --index-url TEXT: The URL to the index.json [default: https://raw.githubusercontent.com/quick-lab/primerschemes/main/index.json]
• --help: Show this message and exit.

Modify an existing scheme's metadata (info.json)

Usage:

$ primal-page modify [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• author-add: Append an author to the authors list in...
• author-remove: Remove an author from the authors list in...
• author-reorders: Reorder the authors in the info.json file
• citation-add: Append an citation to the authors list in...
• citation-remove: Remove an citation form the authors list...
• collection-add: Add a Collection to the Collection list in...
• collection-remove: Remove an Collection from the Collection...
• contactinfo: Change the contactinfo field in the info.json
• derivedfrom: Replaces the derivedfrom in the info.json...
• description: Replaces the description in the info.json...
• license: Replaces the license in the info.json file
• link-add: Add a link to the selected link field to...
• link-remove: Add a link to the selected link field to...
• primerclass: Change the primerclass field in the info.json
• refselect-add: Add a refselect file to the info.json
• refselect-remove: Removes a refselect file from the info.json
• regenerate: Validates the info.json and regenerate the...
• status: Change the status field in the info.json

Append an author to the authors list in the info.json file

Usage:

$ primal-page modify author-add [OPTIONS] SCHEMEINFO AUTHOR

Arguments:

• SCHEMEINFO: The path to info.json [required]
• AUTHOR: The author to add [required]

Options:

• --author-index INTEGER: The 0-based index to insert the author at. Default is the end [required]
• --help: Show this message and exit.

Remove an author from the authors list in the info.json file

Usage:

$ primal-page modify author-remove [OPTIONS] SCHEMEINFO AUTHOR

Arguments:

• SCHEMEINFO: The path to info.json [required]
• AUTHOR: The author to remove [required]

Options:

• --help: Show this message and exit.

Reorder the authors in the info.json file

Usage:

$ primal-page modify author-reorders [OPTIONS] SCHEMEINFO [AUTHOR_INDEX]

Arguments:

• SCHEMEINFO: The path to info.json [required]
• [AUTHOR_INDEX]: The indexes in the new order, separated by spaces. e.g. 1 0 2. Any indexes not provided will be appended to the end

Options:

• --help: Show this message and exit.

Append an citation to the authors list in the info.json file

Usage:

$ primal-page modify citation-add [OPTIONS] SCHEMEINFO CITATION

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CITATION: The citation to add [required]

Options:

• --help: Show this message and exit.

Remove an citation form the authors list in the info.json file

Usage:

$ primal-page modify citation-remove [OPTIONS] SCHEMEINFO CITATION

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CITATION: The citation to remove [required]

Options:

• --help: Show this message and exit.

Add a Collection to the Collection list in the info.json file

Usage:

$ primal-page modify collection-add [OPTIONS] SCHEMEINFO COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}

Arguments:

• SCHEMEINFO: The path to info.json [required]
• COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}: The Collection to add [required]

Options:

• --help: Show this message and exit.

Remove an Collection from the Collection list in the info.json file

Usage:

$ primal-page modify collection-remove [OPTIONS] SCHEMEINFO COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}

Arguments:

• SCHEMEINFO: The path to info.json [required]
• COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}: The Collection to remove [required]

Options:

• --help: Show this message and exit.

Change the contactinfo field in the info.json

Usage:

$ primal-page modify contactinfo [OPTIONS] SCHEMEINFO CONTACTINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CONTACTINFO: The contact information for this scheme. Use 'None' to remove the contact info [required]

Options:

• --help: Show this message and exit.

Replaces the derivedfrom in the info.json file

Usage:

$ primal-page modify derivedfrom [OPTIONS] SCHEMEINFO DERIVEDFROM

Arguments:

• SCHEMEINFO: The path to info.json [required]
• DERIVEDFROM: The new derivedfrom. Use 'None' to remove the derivedfrom [required]

Options:

• --help: Show this message and exit.

Replaces the description in the info.json file

Usage:

$ primal-page modify description [OPTIONS] SCHEMEINFO DESCRIPTION

Arguments:

• SCHEMEINFO: The path to info.json [required]
• DESCRIPTION: The new description. Use 'None' to remove the description [required]

Options:

• --help: Show this message and exit.

Replaces the license in the info.json file

Usage:

$ primal-page modify license [OPTIONS] SCHEMEINFO LICENSE

Arguments:

• SCHEMEINFO: The path to info.json [required]
• LICENSE: The new license. Use 'None' show the work is not licensed (Not recommended) [required]

Options:

• --help: Show this message and exit.

Add a link to the selected link field to the info.json

Usage:

$ primal-page modify link-add [OPTIONS] SCHEMEINFO LINKFIELD LINK

Arguments:

• SCHEMEINFO: The path to info.json [required]
• LINKFIELD: The link field to add to. protocols, validation, homepage, vendors, misc [required]
• LINK: The link to add. [required]

Options:

• --help: Show this message and exit.

Add a link to the selected link field to the info.json

Usage:

$ primal-page modify link-remove [OPTIONS] SCHEMEINFO LINKFIELD LINK

Arguments:

• SCHEMEINFO: The path to info.json [required]
• LINKFIELD: The link field to remove from. protocols, validation, homepage, vendors, misc [required]
• LINK: The link to remove. [required]

Options:

• --help: Show this message and exit.

Change the primerclass field in the info.json

Usage:

$ primal-page modify primerclass [OPTIONS] SCHEMEINFO PRIMERCLASS:{primerschemes}

Arguments:

• SCHEMEINFO: The path to info.json [required]
• PRIMERCLASS:{primerschemes}: The primerclass to change to [required]

Options:

• --help: Show this message and exit.

Add a refselect file to the info.json

Usage:

$ primal-page modify refselect-add [OPTIONS] SCHEMEINFO CHROM REFSELECT

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CHROM: The chromosome to add [required]
• REFSELECT: The reference selection file [required]

Options:

• --help: Show this message and exit.

Removes a refselect file from the info.json

Usage:

$ primal-page modify refselect-remove [OPTIONS] SCHEMEINFO CHROM

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CHROM: The chromosome to remove [required]

Options:

• --help: Show this message and exit.

Validates the info.json and regenerate the README.md

Usage:

$ primal-page modify regenerate [OPTIONS] SCHEMEINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]

Options:

• --help: Show this message and exit.

Change the status field in the info.json

Usage:

$ primal-page modify status [OPTIONS] SCHEMEINFO [SCHEMESTATUS]:[withdrawn|deprecated|autogenerated|draft|tested|validated]

Arguments:

• SCHEMEINFO: The path to info.json [required]
• [SCHEMESTATUS]:[withdrawn|deprecated|autogenerated|draft|tested|validated]: The scheme class [default: draft]

Options:

• --help: Show this message and exit.

Remove a scheme's version from the repo, will also remove size and schemename directories if empty

Usage:

$ primal-page remove [OPTIONS] SCHEMEINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]

Options:

• --help: Show this message and exit.

Validate a scheme

Usage:

$ primal-page validate [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• all-schemes: Validate all schemes in a directory.
• scheme: Validate a single scheme

Validate all schemes in a directory. Calls the scheme command for each scheme and checks for final ref-select files.

Usage:

$ primal-page validate all-schemes [OPTIONS] DIRECTORY

Arguments:

• DIRECTORY: The path to the directory [required]

Options:

• --help: Show this message and exit.

Validate a single scheme

Usage:

$ primal-page validate scheme [OPTIONS] SCHEMEINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]

Options:

• --help: Show this message and exit.