groupschoof · asishallab · Jan 30, 2012 · Feb 27, 2012 · Feb 27, 2012 · Feb 27, 2012
diff --git a/.classpath b/.classpath
@@ -21,5 +21,6 @@
 	<classpathentry kind="lib" path="lib/xom-1.2.6.jar"/>
 	<classpathentry kind="lib" path="lib/yamlbeans-1.06.jar"/>
 	<classpathentry kind="lib" path="lib/mysql-connector-java-5.1.35-bin.jar"/>
+	<classpathentry kind="lib" path="lib/je-7.0.6.jar"/>
 	<classpathentry kind="output" path="classes"/>
 </classpath>
diff --git a/.gitignore b/.gitignore
@@ -37,3 +37,4 @@ trainer_batch_ymls
 .externalToolBuilders/
 .project
 /classes/
+AHRD_DB
diff --git a/README.textile b/README.textile
@@ -11,6 +11,7 @@ h2. Table of contents
 ### "Build the executable jar":#122-build-the-executable-jar
 # "Usage":#2-usage
 ## "AHRD example usages":#21-ahrd-example-usages
+### "AHRD's Database":#211-ahrds-database
 ## "Input":#22-input
 ### "Required input data":#221-required-input-data
 ### "Optional input data":#222-optional-input-data
@@ -53,9 +54,9 @@ h4. 1.2.1 Get AHRD
 Copy (clone) AHRD to your computer using git via command-line, then change into AHRD's directory, and finally use the latest stable version:
 <pre>git clone https://github.com/groupschoof/AHRD.git
 cd AHRD
-git checkout tags/v3.3.3</pre>
+git checkout tags/v3.4</pre>
 
-Alternativelly without using @git@, you can download AHRD version @v3.3.3@ ("zip":https://github.com/groupschoof/AHRD/archive/v3.3.3.zip or "tar.gz":https://github.com/groupschoof/AHRD/archive/v3.3.3.tar.gz) and extract it.
+Alternativelly without using @git@, you can download AHRD version @v3.4@ ("zip":https://github.com/groupschoof/AHRD/archive/v3.4.zip or "tar.gz":https://github.com/groupschoof/AHRD/archive/v3.4.tar.gz) and extract it.
 
 h4. 1.2.2 Build the executable jar
 
@@ -75,6 +76,8 @@ All parameters can be set manually, or the default ones can be used as given in
 
 In order to parallelize the protein function annotation processes,  AHRD can be run on batches of recommended size between 1,000 to 2,000 proteins.  If you want to annotate very large protein sets or have low memory capacities use the included Batcher to split your input-data into Batches of appropriate size (see section "2.3":#23-batcher). _Note:_ As of Java 7 or higher AHRD is quite fast and batching might no longer be necessary.
 
+AHRD extracts some information about the reference proteins into a persitent database. As in tabular sequence similarity search outputs (Blast, Blat, Diamand etc) the description lines and the reference (Hit) lengths are not stored, this information has to be extracted from the original reference protein databases (in Fasta Format). Furthermore the user can provide a _single_ Gene Ontology Annotation (GOA) file for _all_ reference proteins, which will be parsed, too. If GOAs are available for the reference proteins AHRD will also annotate the query proteins with GO Terms (see section "3.3.1":#331-parameters-controlling-the-parsing-of-tabular-sequence-similarity-search-result-tables-legacy-blast-blast-and-blat).
+
 h3. 2.1 AHRD example usages
 
 There are _two_ template AHRD input files provided that you should use according to your use case. All example input files are stored in @./test/resources@ and are named @ahrd_example_input*.yml@. You can run AHRD on any of these use cases with <pre>java -Xmx2g -jar ./dist/ahrd.jar your_use_case_file.yml</pre>
@@ -83,6 +86,18 @@ There are _two_ template AHRD input files provided that you should use according
 | Annotate your Query proteins with Human Readable Descriptions (HRD) | @./test/resources/ahrd_example_input.yml@ |
 | Annotate your Query proteins with HRD _and_ Gene Ontology (GO) terms | @./test/resources/ahrd_example_input_go_prediction.yml@ |
 
+h4. 2.1.1 AHRD's Database
+
+_Do not worry_: You do not have to take care of or setup AHRD's database - it does that automatically for you. If AHRD does not find a database when run, it will extract all required information from the provided reference sequence databases (Fasta-Format) and optionally the provided Gene Ontology Annotation (GOA) file. By default, extracted information will be stored in the _current working directory_ in the database-directory @AHRD_DB@. You can use an alternative database-directory by providing the optional parameter @ahrd_db: path/2/your_db_dir@. If AHRD is started and finds an existing database it will not parse the reference fasta databases nor the GOA files and will expect to find the required information in the AHRD-Database instead. _Note_, that AHRD will terminate with an error if Blast-Hits are found that are not in its database!
+
+If you want or need to, you can _just_ setup an AHRD-Database without running the analysis. In order to do so use the following command:
+
+<pre>
+java -Xmx20g -cp ./dist/ahrd.jar ahrd.controller.DatabaseSetup ./ahrd_example_database_setup_input.yml
+</pre>
+
+See file @ahrd_example_database_setup_input.yml@ for more details.
+
 h3. 2.2 Input
 
 Example files for all input files can be found under @./test/resources/@. _NOTE:_ Only files containing @example@ in their filename should be used as template input files. Other YAML files are used for testing purposes.
@@ -320,7 +335,7 @@ To have AHRD annotate your proteins with GO terms, you just need to provide the
 
 h5. 3.3.2.0 Prefer reference proteins as candidates that have GO Term annotations
 
-The parameter @prefer_reference_with_go_annos: true@ is highly recommended when aiming to annotate your query proteins with GO Terms. If this parameter is set to true only those candidate references are considered that also have GO Term annotations. However, if you put more emphasis on Human Readable Descriptions and are prepared to accept a couple of your queries to not get any GO Term predictions you can switch this off with @prefer_reference_with_go_annos: false@ or just omit the parameter as by default it is set to @false@.  
+The parameter @prefer_reference_with_go_annos: true@ can be used when aiming to annotate your query proteins with GO Terms. If this parameter is set to true only those candidate references are considered that also have GO Term annotations. However, if you put more emphasis on Human Readable Descriptions and are prepared to accept a couple of your queries to not get any GO Term predictions you can switch this off with @prefer_reference_with_go_annos: false@ or just omit the parameter as by default it is set to @false@. We recommend the default behaviour.
 
 h5. 3.3.2.1 Custom reference Gene Ontology annotations (non UniprotKB GOA)
 

diff --git a/ahrd_example_database_setup_input.yml b/ahrd_example_database_setup_input.yml
@@ -0,0 +1 @@
+./test/resources/ahrd_example_database_setup_input.yml
diff --git a/build.xml b/build.xml
@@ -81,6 +81,7 @@
 		</junit>
 		<delete file="test/ahrd_output.csv" />
 		<delete file="test/sim_anneal_path_log.csv" />
+		<delete dir="AHRD_DB" />
 	</target>
 
 	<target name="test.run" depends="compile.test">
@@ -89,6 +90,7 @@
 			<formatter type="plain" usefile="false" />
 			<test name="ahrd.test.AhrdTestRun" />
 		</junit>
+		<delete dir="AHRD_DB" />
 	</target>
 
 	<target name="test.regexs" depends="compile.test">

diff --git a/lib/je-7.0.6.jar b/lib/je-7.0.6.jar
diff --git a/src/ahrd/controller/AHRD.java b/src/ahrd/controller/AHRD.java
@@ -1,8 +1,9 @@
 package ahrd.controller;
 
+import static ahrd.controller.DatabaseSetup.setupOrUseExistingDatabase;
 import static ahrd.controller.Settings.getSettings;
 import static ahrd.controller.Settings.setSettings;
-import static ahrd.model.ReferenceGoAnnotations.parseReferenceGoAnnotations;
+import static ahrd.model.AhrdDb.closeDb;
 
 import java.io.IOException;
 import java.sql.SQLException;
@@ -19,7 +20,6 @@
 import ahrd.exception.MissingInterproResultException;
 import ahrd.exception.MissingProteinException;
 import ahrd.model.BlastResult;
-import ahrd.model.GOterm;
 import ahrd.model.InterproResult;
 import ahrd.model.Protein;
 import ahrd.view.FastaOutputWriter;
@@ -29,15 +29,12 @@
 
 public class AHRD {
 
-	public static final String VERSION = "3.11";
+	public static final String VERSION = "3.4";
 
 	private Map<String, Protein> proteins;
 	private Map<String, Double> descriptionScoreBitScoreWeights = new HashMap<String, Double>();
-	private Map<String, Set<String>> referenceGoAnnotations;
-	private Set<String> uniqueBlastResultShortAccessions;
 	private long timestamp;
 	private long memorystamp;
-	private Map<String, GOterm> goDB;
 
 	protected long takeTime() {
 		// Measure time:
@@ -55,14 +52,12 @@ protected long takeMemoryUsage() {
 	}
 
 	public static void main(String[] args) {
-		System.out.println("Usage:\njava -Xmx2g -jar ahrd.jar input.yml\n");
+		System.out.println("Usage:\njava -Xmx30g -jar ahrd.jar input.yml\n");
 
 		try {
 			AHRD ahrd = new AHRD(args[0]);
 			// Load and parse all inputs
 			ahrd.setup(true);
-			// After the setup the unique short accessions are no longer needed:
-			ahrd.setUniqueBlastResultShortAccessions(null);
 
 			// Iterate over all Proteins and assign the best scoring Human
 			// Readable Description
@@ -77,11 +72,13 @@ public static void main(String[] args) {
 			// Log
 			System.out.println("Wrote output in " + ahrd.takeTime() + "sec, currently occupying "
 					+ ahrd.takeMemoryUsage() + " MB");
-			
+
 			System.out.println("\n\nDONE");
 		} catch (Exception e) {
 			System.err.println("We are sorry, an un-expected ERROR occurred:");
 			e.printStackTrace(System.err);
+		} finally {
+			closeDb();
 		}
 	}
 
@@ -107,24 +104,20 @@ public static IOutputWriter initializeOutputWriter(Collection<Protein> proteins)
 	public AHRD(String pathToYmlInput) throws IOException {
 		super();
 		setSettings(new Settings(pathToYmlInput));
-		// The following fields are only used if AHRD is requested to generate
-		// Gene Ontology term annotations:
-		if (getSettings().hasGeneOntologyAnnotations()) {
-			this.setUniqueBlastResultShortAccessions(new HashSet<String>());
-			this.setReferenceGoAnnotations(new HashMap<String, Set<String>>());
-		}
 	}
 
 	public void initializeProteins() throws IOException, MissingAccessionException {
 		setProteins(Protein.initializeProteins(getSettings().getProteinsFasta()));
 	}
 
-	public void parseBlastResults() throws IOException, MissingProteinException, SAXException {
+	public void parseBlastResults()
+			throws IOException, MissingProteinException, SAXException, MissingAccessionException {
 		for (String blastDatabase : getSettings().getBlastDatabases()) {
-			BlastResult.readBlastResults(getProteins(), blastDatabase, getUniqueBlastResultShortAccessions());
+			BlastResult.readBlastResults(getProteins(), blastDatabase);
 		}
 	}
 
+	@Deprecated
 	public void parseInterproResult() throws IOException {
 		if (getSettings().hasInterproAnnotations()) {
 			Set<String> missingProteinAccessions = new HashSet<String>();
@@ -142,18 +135,6 @@ public void parseInterproResult() throws IOException {
 		}
 	}
 
-	/**
-	 * Method finds GO term annotations for Proteins in the searched Blast
-	 * databases and stores them in a Map.
-	 * 
-	 * @throws IOException
-	 */
-	public void setUpReferenceGoAnnotations() throws IOException {
-		if (getSettings().hasGeneOntologyAnnotations()) {
-			setReferenceGoAnnotations(parseReferenceGoAnnotations(getUniqueBlastResultShortAccessions()));
-		}
-	}
-
 	public void filterBestScoringBlastResults(Protein prot) {
 		for (String blastDatabaseName : prot.getBlastResults().keySet()) {
 			prot.getBlastResults().put(blastDatabaseName,
@@ -163,7 +144,7 @@ public void filterBestScoringBlastResults(Protein prot) {
 
 	/**
 	 * Method initializes the AHRD-run: 1. Loads Proteins 2. Parses BlastResults
-	 * 3. Parses InterproResults 4. Parses Gene-Ontology-Results
+	 * 3. Parses InterproResults. Step 3 is deprecated!
 	 * 
 	 * @throws IOException
 	 * @throws MissingAccessionException
@@ -173,6 +154,8 @@ public void filterBestScoringBlastResults(Protein prot) {
 	 */
 	public void setup(boolean writeLogMsgs)
 			throws IOException, MissingAccessionException, MissingProteinException, SAXException, ParsingException {
+		setupOrUseExistingDatabase(writeLogMsgs);
+
 		if (writeLogMsgs)
 			System.out.println("Started AHRD...\n");
 
@@ -189,15 +172,7 @@ public void setup(boolean writeLogMsgs)
 			System.out.println("...parsed blast results in " + takeTime() + "sec, currently occupying "
 					+ takeMemoryUsage() + " MB");
 
-		// Reference GO Annotations (for Proteins in the searched Blast
-		// Databases)
-		setUpReferenceGoAnnotations();
-		if (writeLogMsgs) {
-			System.out.println("...parsed reference Gene Ontology Annotations (GOA) in " + takeTime()
-					+ "sec, currently occupying " + takeMemoryUsage() + " MB");
-		}
-
-		// one single InterproResult-File
+		// one single InterproResult-File (DEPRECATED)
 		if (getSettings().hasValidInterproDatabaseAndResultFile()) {
 			InterproResult.initialiseInterproDb();
 			parseInterproResult();
@@ -229,19 +204,11 @@ public void assignHumanReadableDescriptions() throws MissingInterproResultExcept
 			// currentScore - (Token-High-Score / 2)
 			prot.getTokenScoreCalculator().filterTokenScores();
 			// Find the highest scoring Blast-Result:
-			prot.getDescriptionScoreCalculator().findHighestScoringBlastResult(this.getReferenceGoAnnotations());
-			// If AHRD is requested to annotate Gene Ontology Terms, do so:
-			if (getSettings().hasGeneOntologyAnnotations()
-					&& prot.getDescriptionScoreCalculator().getHighestScoringBlastResult() != null
-					&& getReferenceGoAnnotations().containsKey(
-							prot.getDescriptionScoreCalculator().getHighestScoringBlastResult().getShortAccession())) {
-				prot.setGoResults(getReferenceGoAnnotations()
-						.get(prot.getDescriptionScoreCalculator().getHighestScoringBlastResult().getShortAccession()));
-			}
+			prot.getDescriptionScoreCalculator().findHighestScoringBlastResult();
 			// filter for each protein's most-informative
-			// interpro-results
+			// interpro-results (DEPRECATED)
 			InterproResult.filterForMostInforming(prot);
-		}		
+		}
 	}
 
 	public Map<String, Protein> getProteins() {
@@ -259,29 +226,4 @@ public Map<String, Double> getDescriptionScoreBitScoreWeights() {
 	public void setDescriptionScoreBitScoreWeights(Map<String, Double> descriptionScoreBitScoreWeights) {
 		this.descriptionScoreBitScoreWeights = descriptionScoreBitScoreWeights;
 	}
-
-	public Map<String, Set<String>> getReferenceGoAnnotations() {
-		return referenceGoAnnotations;
-	}
-
-	public void setReferenceGoAnnotations(Map<String, Set<String>> referenceGoAnnotations) {
-		this.referenceGoAnnotations = referenceGoAnnotations;
-	}
-
-	public Set<String> getUniqueBlastResultShortAccessions() {
-		return uniqueBlastResultShortAccessions;
-	}
-
-	public void setUniqueBlastResultShortAccessions(Set<String> uniqueBlastResultShortAccessions) {
-		this.uniqueBlastResultShortAccessions = uniqueBlastResultShortAccessions;
-	}
-
-	public Map<String, GOterm> getGoDB() {
-		return goDB;
-	}
-
-	public void setGoDB(Map<String, GOterm> goDB) {
-		this.goDB = goDB;
-	}
-
 }
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		./test/resources/ahrd_example_database_setup_input.yml