Cleanup main loop

workflow/README.md

@@ -200,9 +200,10 @@ This module:
 
 TODO 
 
-* Copy over source image directories from robot - are these alse named by the ExpJobs name?
+* Copy over source image directories from robot
-* Suggest renaming ExpJobs to something like "scans" or "images"
+* MasterPlate_ file **should not be an xlsx file**, no portability
-* MasterPlate_ file **should not** be an xlsx file, no portability
+* We can keep the existing xlsx code for old style fallback
+* But moving forward should switch to csv or something open
 
 NOTES
 
@@ -437,6 +438,70 @@ INSTRUCTIONS
 
 ![Drug Media file](docs/imgs/easy/24-dm-file.png "Drug Media file")
 
+
+
+To analyze a new Q-HTCP experiment:
+
+* Open the EASY Software.
+* Open 'EstartConsole.m' with MATLAB
+* Click the Run icon (play button)
+* When prompted, click "Change Folder" (do not select "Add to Path").
+* In the pop-up display, select from the 'File' dropdown: 'New Experiment'. 
+* From the pop-up, choose where to save the new file. 
+* Navigate to the relevant job in the ExpJobs folder, name the file accordingly, and click 'save'.
+* The newly created .mat file in the newly created Results folder will automatically be loaded. 
+* The file name will then be automatically appended by the code with the current date information (e.g. 'A1.mat' will become 'Results2023-07-19A1)
+* If the experiment has already been created, it can be reloaded by clicking 'Load Experiment' instead of 'New Experiment' and selecting the relevant results 
+* In the pop-up display, click on the 'Run' dropdown menu and select 'Image CurveFit ComboAnalysis'.
+* In the updated pop-up, choose/highlight all desired image folders for analysis (this is generally all of the folders, since only the ones that need analysis should be there) and then click on 'continue'. 
+* As the program is running, updates will periodically appear in the Command Window; there will be an initial pause at "Before call to NIscanIntens…..". 
+* When the curve fitting is finished, the EASY console will pop back up.
+* Check to see the completed analysis results in the newly created 'PrintResults' Folder, inside of the 'Results' Folder.
+* Other folders ('CFfigs', 'figs', 'Fotos') are created for later optional use and will be empty. 
+* **NOTE:** The image analysis is completed independent of labeling the data (strains, media type, etc. Labeling happens next with the 'GenReports' function).
+* Click on the 'GenReports' dropdown and select 'DrugMediaMP Generate .mat'
+* **NOTE:** The 'MasterPlate' and 'DrugMedia' files have very specific formats and should be completed from a template. 
+* The Masterplate file must be exact (it must contain all and only the strains that were actually tested). 
+* For example, if only part of a library is tested, the complete library file must be modified to remove irrelevant strains.
+* You will be prompted to first select the 'MasterPlate' file. You will need to navigate away from the working directory to get to it. 
+* It is fine for the 'MasterPlate_' file to be .xlsx (or .xls), and if you don't see it in the popup window, then change the file type from '.xls' to "all files" and then select it. 
+* Once it is selected, a report of the number of master plates in the file will pop up; when the report appears, assuming it is correct, click on 'OK'.
+* You will then be prompted to select the 'DrugMedia' file from the relevant job folder. You will automatically return to the correct prior directory location. 
+* Choose it and click 'OK'. You may see a warning about column headers being modified, but that's ok.
+* This will create an additional file in the 'MasterPlatesFiles' folder named 'MPDMmat.mat'
+* Click on the 'GenReports' dropdown and select 'Results_Generate.' 
+* You will first see '!!ResultsElr_.txt' generated in the 'PrintResults' folder. 
+* Refreshing will reveal an increasing file size until you see the '!!ResultsStd_.txt' being generated. 
+* When finished, the '!!ResultsStd_.txt' will be about the same file size and it should be used in the following StudiesQHTCP analysis.  
+* 'NoGrowth_.txt', and 'GrowthOnly_.txt' files will be generated in the 'PrintResults' folder. 
+
+
+
+
+Issues:
+* We need full documentation for all of the current workflow. There are different documents that need to be integrated. This will need to be updated as we make improvements to the system.
+* MasterPlate_ file must have ydl227c in orf column, or else it Z_interaction.R will fail, because it can't calculate shift values.
+* Make sure there are no special characters; e.g., (), “, ', ?, etc.;  dash and underscore are ok as delimiters
+* Drug_Media_ file must have letter character to be read as 'text'.
+* MasterPlate_ file and DrugMedia_ are .xlsx or .xls, but !!Results_ is .txt.
+* In Z_interactions.R, does it require a zero concentration/perturbation (should we use zero for the low conc, even if it's not zero), e.g., in order to do the shift correctly.
+* Need to enable all file types (not only .xls) as the default for GenerateResults (to select MP and DM files as .xlsx).
+* Explore differences between the ELR and STD files - 24_0414; John R modified Z script to format ELR file for Z_interactions.R analysis.
+* To keep time stamps when transferring with FileZilla, go to the transfer drop down and turn it on, see https://filezillapro.com/docs/v3/advanced/preserve-timestamps/ 
+* Could we change the 'MasterPlateFiles' folder label in EASY to 'MasterPlate_DrugMedia' (since there should be only one MP and there is also a DM file required?
+* I was also thinking of adding a 'MasterPlateFilesOnly' folder to the QHTCP directory template where one could house different MPFiles (e.g., with and without damps, with and without Refs on all MPs, etc; other custom MPFiles, updated versions, etc)
+* Currently updated files are in '23_1011_NewUpdatedMasterPlate_Files' on Mac (yeast strains/23_0914…/)
+* For EASY to report cell array positions (plate_row_column) to facilitate analyzing plate artifacts. The MP File in Col 3 is called 'LibraryLocation' and is reported after 'Specifics' in the !!Results.
+* Can EASY/StudiesQ-HTCP be updated at any time by rerunning with updated MP file (new information for gene, desc, etc)- or maybe better to always start with a new template?
+* Need to be aware of file formatting to avoid dates (e.g., with gene names like MAY24, OCT1, etc, and with plate locations 1E1, 1E2, etc)- this has been less of a problem. 
+* In StudiesQHTCP folders, remember to annotate Exp1, Exp2, in the StudyInfo.csv file.
+* Where are gene names called from for labeling REMc heatmaps, TSHeatmaps, Z-interaction graphs, etc? Is this file in the QHTCP 'code' folder, or is it in the the results file (and thus ultimately the MP file)?
+* Is it ok for a MasterPlate_ file to have multiple sheets (e.g., readme tab- is only the first tab read in)?
+* What are the rules for pulling information from the MasterPlateFile to the !!Results_ (e.g., is it the column or the Header Name, etc that is searched? Particular cells in the DrugMedia file?).
+* Modifier, Conc are from DM sheet, and refer to the agar media arrays. OrfRep is from MasterPlate_ File. 'Specifics' (Last Column) is experiment specific and accommodate designs involving differences across the multi-well liquid arrays. 'StrainBkGrd' (now 'Library location') is in the 3rd column and reported after 'Specifics' at the last col of the '!!Results..' file.
+* Do we have / could we make an indicator- work in progress or idle/complete with MP/DM and after gen-report. Now, we can check for the MPDMmat.mat file, or we can look in PrintResults, but would be nice to know without looking there. 
+* File>>Load Experiment wasn't working (no popup to redirect). Check this again.
+
 ### ezview
 
 TODO WIP
@@ -610,7 +675,7 @@ This submodule:
 
 * **$1** (string): First Exp# name
 * **$2** (string): Second Exp# name
-* **$3** (string): StudyInfo.txt file
+* **$3** (string): StudyInfo.csv file
 * **$4** (string): output directory
 
 ### r_gta_heatmaps
@@ -663,7 +728,11 @@ NOTES
 
 #### Arguments
 
-* **$1** (string): The current working directory
+* **$1** (string): The input directory
+* **$2** (string): The study info file
+* **$3** (string): The zscores directory
+* **$4** (string): SGD_features.tab
+* **$5** (integer): delta SD background value (default: 5)
 
 ### r_join_interactions
 
@@ -699,7 +768,16 @@ NOTE
 
 #### Arguments
 
-* **$1** (string): The output directory
+* **$1** (string): GeneByGOAttributeMatrix_nofiltering-2009Dec07.tab
+* **$2** (string): ORF_List_Without_DAmPs.txt
+* **$3** (string): REMcRdy_lm_only.csv
+* **$4** (string): The output directory
+* **$5** (string): The output file
+
+#### Exit codes
+
+* **0**: if expected output file exists
+* **1**: if expected output file does not exist
 
 ### r_add_shift_values
 
@@ -806,7 +884,6 @@ TODO
 #### Variables set
 
 * **STUDIES_NUMS** (array): Contains Exp numbers
-* **NUM_STUDIES** (int): Number of existing studies
 
 #### Exit codes
 

workflow/qhtcp-workflow

@@ -738,10 +738,11 @@ module init_project
 #
 # TODO 
 #
-# * Copy over source image directories from robot - are these alse named by the ExpJobs name?
+# * Copy over source image directories from robot
-# * Suggest renaming ExpJobs to something like "scans" or "images"
+# * MasterPlate_ file **should not be an xlsx file**, no portability
-# * MasterPlate_ file **should not** be an xlsx file, no portability
+# * We can keep the existing xlsx code for old style fallback
-#  
+# * But moving forward should switch to csv or something open
+# 
 # NOTES
 #
 # * Copy over the images from the robot and then DO NOT TOUCH that directory except to copy from it
@@ -1026,10 +1027,6 @@ module easy
 #
 #
 #
-#
-#
-3
-#
 # To analyze a new Q-HTCP experiment:
 #
 #   * Open the EASY Software.
@@ -2266,12 +2263,19 @@ main() {
     done
 
   done
+
+  cat <<-EOF
+		Successfully ran module(s): ${MODULES[*]}
+		And submodule(s): ${SUBMODULES[*]}
+		On project(s): ${PROJECTS[*]}
+	EOF
   unset PROJECTS MODULE SUBMODULES EXCLUDE_MODULES
 }
 
 # (Safe) main loop
 for ((i=1; i<2; i++)); do
-  main "$@" && i=0 # on successful run, reset the counter
+  main "$@" &&
+  i=0
 done
 
 exit $?