Clone this repository

git clone https://github.com/ShakNat/DupyliCate

Mandatory dependencies:

• Tools: BLAST, DIAMOND, MMSeqs2 (latest versions preferred)

• Python libraries: pandas (v2.3.1 or greater), numpy (v2.3.2 or greater), seaborn (v0.13.2 or greater), matplotlib (v3.10.5 or greater), scipy (v1.16.1 or greater), jinja2 (3.1.6), openpyxl (3.1.2 or greater)

Optional dependencies:

• Tools: GeMoMa, BUSCO, AGAT, MAFFT, FastTree (latest versions preferred)

• Python libraries: dendropy (v5.0.8), tqdm (v4.67.1)

docker pull shakunthalan/dupylicate:latest

It is recommended to run the docker image as a user and not root:

 docker run --rm -u $(id -u) -v /path/to/data:/data shakunthalan/dupylicate:latest

Note: If you are using a docker image of the tool, you need not specify the dependencies' full paths while running the tool; The dependencies are built-in in the docker image and that makes it simple to use the tool across systems without the need for local installations.

This method of installation installs all the dependencies in a conda environment using the environment.yml file in this repository

git clone https://github.com/ShakNat/DupyliCate

cd DupyliCate

conda env create -f environment.yml

conda activate dupylicate_v1.0

Usage:

  python3 dupylicate.py --gff <GFF_FILE_OR_DIR>
                        [--fasta <ASSEMBLY_FASTA_FILE_OR_DIR> | --cds <CDS_FASTA_FILE_OR_DIR> | --pep <PEP_FASTA_FILE_OR_DIR>]
                        --out <DIR>


MANDATORY:

	--gff 									STR			Full path to folder containing GFF files

	Choose one of FASTA/PEP/CDS

	[--fasta								STR			Full path to folder containing WGS FASTA files

    --cds									STR			Full path to folder containing CDS FASTA files

    --pep									STR			Full path to folder containing PEP files]

    Output directory

	--out									STR			full path to output folder

OPTIONAL:

	--ref									STR			Name of the organism to be taken as reference [Default is NA and the script runs in reference-free mode]

	--prokaryote							STR			Use the flag if the organisms for analysis are prokaryotes

	--pseudos_gff							STR			yes | no Optional inclusion or exclusion of pseudogenes with coding features in the GFF file [DEFAULT is no]

	--gff_config							STR			Full path to TXT file containing the gff parameters

	--mode									STR			overlap | strict [DEFAULT is overlap] 

	--to_annotate							STR			Full path to TXT file 

	--seq_aligner							STR			blast | diamond | mmseqs2 [DEFAULT is diamond]

	--blast									STR			Full path to BLAST if not already in your PATH environment variable

	--diamond								STR			Full path to diamond if not already in your PATH environment variable

	--mmseqs								STR			Full path to mmseqs2 if not already in your PATH environment variable

	--mafft									STR			Full path to MAFFT if not already in your PATH environment variable

	--evalue								FLOAT		evalue for alignment [DEFAULT is 1e-5]

	--gemoma								STR			Full path to GeMoMa if not already in your PATH environment variable

	--qc									STR			yes | no Quality check with BUSCO [DEFAULT is no]

	--busco									STR			Full path to BUSCO | busco_docker [DEFAULT is busco]

	--busco_lineage							STR			Full path to BUSCO config file [DEFAULT is auto]

	--busco_version							STR			BUSCO version in the format vx.x.x> [DEFAULT is v5.8.2]  

	--container_version						STR			Docker container version of BUSCO [DEFAULT is cv1]

	--docker_host_path						STR			Full host folder path 

	--docker_container_path					STR			Full mount path in the docker container

	--score									STR | FLOAT	auto | float  number between 0 and 1 [DEFAULT is auto]

	--self_simcut							FLOAT		Similarity percentage to remove self alignment hits with low similarity percentage [DEFAULT is 50.0]

    --hits									INT			Number of top hits to be considered for finding suitable orthologs in the reference organism [DEFAULT is 10]

	--ortho_candidates						INT			User specified integer value for listing the potential ortholog candidates [DEFAULT is 3]

	--occupancy								FLOAT		Occupancy cutoff for MAFFT aligned file trimming [DEFAULT is 0.1]

    --scoreratio							FLOAT		Ratio of forward alignment bit score and self alignment bit score of query to assess if the forward hit is valid [DEFAULT is 0.3]

	--fwd_simcut							FLOAT		Similarity percentage to remove forward alignment hits with low similarity percentage [DEFAULT is 40.0]

    --cores									INT			Number of cores needed to run Dupylicate analysis [DEFAULT is 4]

	--proximity								INT			Value for the number of intervening genes to detect proximal duplications [DEFAULT is 10]

    --synteny_score							FLOAT		Value which is used as a cut-off or threshold for synteny analysis [DEFAULT is 0.5]

	--flank									INT			Value specifying the number of flanking genes to be considered to determine the synteny window size in synteny analysis [DEFAULT is 5]

	--side									INT			Value for synteny support from either side of a flanking region of a synteny window [DEFAULT is 1]

	--ka_ks									STR			yes | no  To calculate ka, ks values [DEFAULT is no]

	--ka_ks_method							STR			MYN | NG  Methods for Ka/Ks ratio caclulation [Default is NG]

	--duplicates_analysis					STR			yes | no  For further statistical analysis of identified gene duplicates [DEFAULT is no]

	--specific_duplicates_analysis			STR			yes | no  For further statistical analysis of specified ref genes' gene duplicates> [DEFAULT is no]

	--ref_free_specific_duplicates_analysis	STR			Full path to TXT file containing the names of genes whose expression analysis is to be done in the absence of a reference 

	--dpi									STR			low | moderate | high | very high  Resolution level desired for plots [DEFAULT is moderate]

	--analyse_disperse_duplicates			STR			yes | no  For statistical analysis of dispersed gene duplicates [DEFAULT is no]

	--exp									STR			Full path to the folder with expression counts files

	--avg_exp_cut							FLOAT		Value cutoff for average expression value across samples [Default is 1]

	--genes									INT			Value referring to the number of genes to be taken for random pairing [DEFAULT is 10000] 

	--specific_genes						STR			Full path to TXT file 

	--clean_up								STR			yes | no  Cleans up intermediate files and folders [DEFAULT is yes]

ALLOWED FILE EXTENSIONS:

	<'.gff', '.gff.gz', '.gff3', '.gff3.gz', '.fa', '.fa.gz', '.fna', '.fna.gz','.fasta', '.fasta.gz',
	'.genome.fasta', '.genome.fa', '.genome.fasta.gz', '.genome.fa.gz', '.genome.fna',
	'.genome.fna.gz','.cds.fa', '.cds.fasta','.cds.fa.gz', '.cds.fasta.gz', '.cds.fna',
	'.cds.fna.gz','.pep.fa','.pep.fa.gz','.pep.fasta','.pep.fasta.gz', '.pep.fna',
	'.pep.fna.gz','.tsv','.txt','.txt.gz','.tpms.txt','.tpms.txt.gz'>

--mode In the overlap mode genes repeat among the different duplicates classification. In the strict mode there is no gene repetition and you have a new classification group called mixed duplicates containing the related connected components from the other three duplicate classes

--to_annotate This needs the path to a TXT file. The TXT file must contain names of queries and the corresponding reference organism for GeMoMa annotation separated by comma - Query,Reference -> one pair per line

--gemoma The full path to GeMoMa must also include the name of the GeMoMa jar file

--busco This flag can take the full path to busco. If you have installed busco via conda, activate the environment anf give the full path. If you have a docker-based installation of busco, just give busco_docker as the parameter to this flag

--busco_lineage This flag can take the full path to BUSCO config file that specifies the BUSCO lineage to be used for the different sample organisms in an analysis

--busco_version --container_version --docker_host_path --docker_container_path These flags are needed only if you want to do a BUSCO-based QC or auto scoring and if you have a docker-based busco installation

--score This flag can take in string parameter auto if you want the thresholding for singleton duplicates segregation to be done based on BUSCO; If you want to tune this parameter, instead of auto, you can specify any float number between 0 and 1

--hits These are the number of reference genes that will be picked for tree building in the ortholog finding step from among the list of valid forward hits in the reference organism for each sample organism gene

--ortho_candidates The low confidence orthologs have another output column containing other potential orthologs, since the identified ortholog is of low confidence level. This parameter helps tune the number of such potential ortholgos to be listed

--analyse_disperse_duplicates The number of dispersed duplicates are generally higher, and a routine expression analysis of all idnetifeid dispersed duplicate groups could be time consuming. Hence, this has been made optional. The expression analysis activated with the --exp flag will do the analysis for small scale duplicate groups of tndem and proximal genes. If the dispersed duplicates also need to be subject to expression analysis, then this flag's parameter can be tuned to yes

--genes This flag's parameter gives the number of genes to be considered for random gene pairing to identify a correlation threshold that can be used to assess functional divergence of gene duplicates based on gene expression values

--specific_genes The TXT file pointed by this flag must contain a user given list of specific genes from the reference with one transcript name per line, for analysing gene duplication outputs

--ref_free_specific_duplicates_analysis This flag can be used for analysing expression of gene duplicates in the sample when a reference is not given

1. This is a simple .TXT file containing the gff parameters in different columns

2. The columns can be separated by tabs or spaces

3. There are four columns mandatorily needed in the config file in the following order:

   (i) base file name - same as the base name you use for the gff file | all in case all the gff have the same gff pattern

   (ii) child_attribute: attribute field of the mRNA or transcript feature in the file like ID

   (iii) child_parent_linker: attribute field of the mRNA or transcript, CDS, exon features that link them with their respective parent feature like Parent - Note: base assumption by the tool is that all child levels have the same child-parent linker attribute fields. For eg., if Parent is the child-parent linker in the mRNA feature line, then Parent will be the child-parent linker for all other child-level feature lines in the GFF

   (iv) parent_attribute: attribute field of the gene feature like ID

   (v) By default in the script, child_attribute is ID, child_parent_linker is Parent, and parent_attribute is ID

Sample config file and GFF file example:

If the config looks like this -

all	Name	Parent	ID

And the corresponding GFF file looks as below - 

##gff-version 3

##annot-version Araport11

##species Arabidopsis thaliana columbia

Chr1    	phytozomev12    	gene    	3631    	5899    	.       	+       	.       	ID=AT1G01010.Araport11.447;Name=AT1G01010

Chr1    	phytozomev12    	mRNA    	3631    	5899    	.       	+       	.       	ID=AT1G01010.1.Araport11.447;Name=AT1G01010.1;pacid=37401853;longest=1;geneName=NAC001;Parent=AT1G01010.Araport11.447

Understanding the GFF config file:

• The first column of the file says all. This means all the files in the analysis will have the same GFF file format/ pattern

• The second column that is the child attribute column is Name. So the text following the Name field in the last column of the mRNA feature will be picked which is AT1G01010.1 above

• The third column that is the child-parent linker column is Parent. It is the field Parent in mRNA that links it to its parent gene, and in the above example its corresponding text picked will be AT1G01010.Araport11.447

• The fourth column is the parent attribute that is mentioned as ID. In the above example it is the ID field in the gene feature line which is AT1G01010.Araport11.447

1. If you have a known list of genes in the reference organism whose copy number variation you want to analyze in the sample, the --specific_genes flag can be used.

2. This needs the full path to a simple .TXT file

3. This .TXT file should have one transcript name per line

1. In case, some of your input files lack structural annotation, the --to_annotate flag in the script can be used

2. This needs the full path to a simple .TXT file

3. The name of the organism to be annotated followed by comma and the name of the reference organism to be used for annotation must be mentioned in a line

4. If there are multiple organisms for annotation, specify each of them along with their respective reference in a single line in the format as specified above

   eg.

   Vamurensis,Vvinifera

   Vrotundifolia,Vvinifera

DupyliCate by default uses the --auto-lineage-euk, --auto-lineage-prok to detect the BUSCO lineages of sample organisms. However in case you want to specify a lineage to choose, you can use the --busco_lineage flag and specify the path to this BUSCO config file.

1. This is a simple two column TXT file separated by tab

2. The first column is the sample organism name and the second column is the busco lineage you want to specify

3. Please ensure that this organism name and the input annotation, assembly file names of the sample organism without the extensions are matching.

Usage:

	python3 AGAT_wrapper.py

MANDATORY:

	--gff_in <full path to GFF3 file or folder containing GFF3 annotation files>

	--gff_out <full path to output folder or output file>

OPTIONAL:

	--agat <full path to the agat_convert_sp_gxf2gxf.pl script including the script name>

Usage:

	python3 Fasta_fix.py

MANDATORY:

	--in <full path to a folder containing FASTA files or a single FASTA file>

	--out <full path to output directory>

	--config <full path to config file including the config file name>

OPTIONAL:

	--cores <number of cores for running the script>

1. The config file is a simple .TXT file

2. Specify the organism name i.e. the base name without the extension of your file in the first column

3. If the same set of string manipulation operations are to be performed for all the files in a folder, just specify the word all in the first column and follow it up in the next columns with the desired operations - This single line is enough if the same set of operations are to be performed on all the files

4. Specify the various string manipulation operations to be performed on the FASTA header of this particular organism's file in the subsequent columns separated by white space or tab

5. IMPORTANT: The operations will be performed in the same order as you specify them in the config wise i.e. column wise order of the different operations you specify will be followed by the script; Hence operation ORDER is IMPORTANT

6. Possible operations and the manner in which they are to be specified are as follows

   i. extract: example- extract:ID=:;

   ii. split: example- split:_

   iii. take: example- take:1,3

   iv. join: example- join:- <This will join a list of strings into a single string separated by - >

   v. remove: example- remove:v2

   vi. add_prefix: add_suffix: example: add_suffix:ath

   vii. replace: <to replace a specfic character with another character or pattern; Specify the pattern to be replaced first and the pattern to be added in its place - separate these two with a comma> example- replace:phytozome,phyt

   viii. uppercase

       lowercase

• md5sum.tsv: Contains the md5sums of every input file used in a particular run of the tool

• run_parameters.json: json file that keeps a record of all the parameters used for a particular run of the tool

• Duplication landscape plots: Histogram plots of normalized bit scores of a gene's second best hit and self hit in self alignment of the sample; Depicts the genome level gene duplication status

• Duplication frequency plots: Bar plots depicting the actual and relative percentage of genes in the different classification categories

• BUSCO QC Summary file: TSV file with information about BUSCO completeness, duplication, Feff (pseudo-ploidy number - approximate indication of ploidy)

• Tandem duplicates, Proximal duplicates, Dispersed duplicates: Folders containing organism-wise tandem, proximal and dispersed gene duplicates output TSV files

• Mixed duplicates: Mixed duplicates folder containing organism-wise mixed duplicates output TSV files (output in strict mode)

• Duplicates_relationships: Duplicates relationships folder containing organism-wise duplicates relationships across duplicate gruops (output in overlap mode)

• The different gene duplicates classification output files have a number of columns. In all the duplicates results files, both in the presence or absence of a reference organism, the following columns are present:

  (i) Pairwise gene distance (bp) - Genetic distance in bp between two neighbouring genes in an identified duplicates group

  (ii) Actual intervening gene number: The tool considers only protein coding genes for identifying and classifying gene duplicates. But it also important to know the actual number of protein coding and non-coding genes in-between gene duplicates, and the actual intervening gene number gives this information

  (iii) Apparent intervening gene number: The number of protein coding genes between neighbouring genes in an identified duplicates group

  (iv) Group confidence: The gene duplicates are clustered into groups. This column gives a confidence score for the reliability of the identified groups -

   confidence_score <= 0.3 -> Low confidence gene duplicates group

   confidence_score <= 0.5 -> Moderate confidence gene duplicates group

   confidence_score > 0.5 -> High confidence gene duplicates group

  (v) Nature of duplicate group: This column is present only in the small scale duplicate groups - tandems and proximals output files in the presence of a reference organism;

   Small scale duplicates are mainly responsible for the evolutionary innovations seen in stress response mechanisms and biosynthetic pathways;

   Understanding their nature like whether they lead to gene expansion, conservation or de novo duplication can lead to crucial biological insights;

   The nature of duplicate group provides this information as detailed in the image below

• Singletons: Singletons folder containing organism-wise singleton gene output TSV files

• In the presence of a reference organism, the small scale gene duplicates (tandems, proximals) files and the singletons files have a column called Synteny information

  (vi) Synteny information: This column evaluates the synteny between the identified gene duplicates and their corresponding orthoolog genes in the reference organism

The ortholog confidence scoring is more stringent than the gene duplicates confidence scoring system because of the species specific variations and distance considerations between the reference and sample organisms

• Summary.tsv: Summary file consolidating the number of different gene duplicates in each analysed sample organism; It also lists the number of gene duplicate groups classified as low/ moderate/ high confidence groups

• Ka_Ks_analysis: Folder containing organism-wise Ka/Ks computation TSV files of gene duplicates on an individual gene basis along with statistical significance and nature of selection pressure

  • Each file in this folder is a tsv file. Every individual gene in a gene duplicate group is assigned a Ka/Ks value in this file. This gene-level Ka/Ks value is obtained by a series of statistical analyses, and normalization approaches in the script

• Ka/Ks plot: Distribution plot of statistically significant Ka/Ks values (adjusted pval<=0.05) of gene duplicates to assess the trend of selection pressures on gene duplicates; This is a filtered Ka/Ks plot taht removes extreme Ka/Ks values

• Duplicates_analysis: Folder containing organism wise gene expression output folders; Each organism folder shows Plots folder, Stats folder, the kernel density estimation plot of correlation coefficients used for determining the divergent expression threshold, and TXT file of perceived pseudogenes (genes that have low expression across the different RNASeq samples used for generating the counts table file) in that organism;

   It is important to note that some gene duplicate groups can have statistics folder present, but might not have corresponding plots folder due to the possibility of all or most of them being perceived pseudogenes, disabling their correlation analysis and gene expression plotting; The stats.tsv file for each duplicate group lists the iterative gene pairs analyzed in a duplicate group, the Bonferroni corrected p-value for every pair, the Spearman correlation value and finally the strength of correlation or expression divergence.

• Specific_duplicates_analysis: Folder similar to the Duplicates_analysis folder, except that contains the respective output folders, and files for specific gene duplicates

• Specific_ref_free_duplicates_analysis: Folder similar to the Duplicates_analysis folder, except that contains the respective output folders, and files for specified genes if presen as duplicates in the sample organism

• The errors and warnings output by DupyliCate are stored in the Tmp folder created within the output folder you specify /output_folder/Tmp/Errors_warnings

• By defualt if the run completes successfully, the Tmp folder is removed automatically; If you want to retain the intermediate files specify no in the --clean_up flag

• Please remember to add / at the end of the folder path when giving a folder of input files to DupyliCate

• Please ensure that the files have the allowed extensions as listed above in the main script usage

• The base names of the files excluding the extensions must be the same across all input files, the config files, and the annotation TXT file for GeMoMa annotation

• Gene duplications classification needs the GFF file(s) as the inputs. Since GFF files have wide varying formats, the analysis can be interrupted at the validation step due to GFF file formatting issues

• Some GFF files have mRNA features denoted as mRNA, while some have them denoted as transcript. In cases of merged annotation files, both the terms can denote an mRNA feature. But the script looks for the keyword mRNA first, and if not found, then searches for the word transcript. In case you have a merged annotation, please make the mRNA feature indicator uniform throughout the GFF file - either mRNA throughout or transcript throughout

• Next, the FASTA headers need to match specific GFF attribute fields. Otherwise, the analysis would stop after the validation and PEP file generation step

• Please look into the GFF config file preparation instructions to avoid such errors during analysis

• Altschul, S.F., Gish, W., Miller, W., Myers, E.W., Lipman, D.J. (1990) “Basic local alignment search tool.” J. Mol. Biol. 215:403-410. PubMed

• Buchfink B, Reuter K, Drost HG, "Sensitive protein alignments at tree-of-life scale using DIAMOND", Nature Methods 18, 366–368 (2021). doi:10.1038/s41592-021-01101-x

• Steinegger M and Soeding J. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, doi: 10.1038/nbt.3988 (2017).

• Price, M.N., Dehal, P.S., and Arkin, A.P. (2010) FastTree 2 -- Approximately Maximum-Likelihood Trees for Large Alignments. PLoS ONE, 5(3):e9490. doi:10.1371/journal.pone.0009490

• Kazutaka Katoh, Daron M. Standley, MAFFT Multiple Sequence Alignment Software Version 7: Improvements in Performance and Usability, Molecular Biology and Evolution, Volume 30, Issue 4, April 2013, Pages 772–780, https://doi.org/10.1093/molbev/mst010

• J. Keilwagen, F. Hartung, M. Paulini, S. O. Twardziok, and J. Grau Combining RNA-seq data and homology-based gene prediction for plants, animals and fungi. BMC Bioinformatics, 2018. doi: 10.1186/s12859-018-2203-5

• Dainat J. AGAT: Another Gff Analysis Toolkit to handle annotations in any GTF/GFF format.
  (Version v1.5.1). Zenodo. https://www.doi.org/10.5281/zenodo.3552717

• Fredrik Tegenfeldt, Dmitry Kuznetsov, Mosè Manni, Matthew Berkeley, Evgeny M Zdobnov, Evgenia V Kriventseva, OrthoDB and BUSCO update: annotation of orthologs with wider sampling of genomes, Nucleic Acids Research, Volume 53, Issue D1, 6 January 2025, Pages D516–D522, https://doi.org/10.1093/nar/gkae987

• Jeet Sukumaran, Mark T. Holder, DendroPy: a Python library for phylogenetic computing, Bioinformatics, Volume 26, Issue 12, June 2010, Pages 1569–1571, https://doi.org/10.1093/bioinformatics/btq228

• Cheng, Chia-Yi, Vivek Krishnakumar, Agnes P. Chan, Françoise Thibaud-Nissen, Seth Schobel, and Christopher D. Town. 2017. “Araport11: A Complete Reannotation of the Arabidopsis Thaliana Reference Genome.” The Plant Journal 89(4):789–804. doi:10.1111/tpj.13415.

• Pucker B, Holtgräwe D, Stadermann KB, Frey K, Huettel B, Reinhardt R, et al. (2019) A chromosome-level sequence assembly reveals the structure of the Arabidopsis thaliana Nd-1 genome and its gene set. PLoS ONE 14(5): e0216233. https://doi.org/10.1371/journal.pone.0216233

Natarajan, S. & Pucker, B. DupyliCate: mining, classifying, and characterizing gene duplications. 2025.10.10.681656 Preprint at https://doi.org/10.1101/2025.10.10.681656 (2025).