MicroBooNE Single Particle Generation Example MCC8¶
These instructions are for the generation of single particle samples going into the MicroBooNE Cincinnati Workshop 2017. They do not cover all of the options that are possible within project.py and most importantly do not cover information about how to generate custom samples (through either custom fcl or modules). There are some general comments about grid submission snuck into the instructions, so enjoy and learn!
Configuring your project.py xml file¶
There is a template file to base you project xml file off of located here:
Copy this file into your app volume area:
cp /uboone/app/users/kirby/single_particle_mcc8_template/test_single_particle_template.xml /uboone/app/users/$USER/<name_your_working_area>/<name_your_file>.xml
Note that the /uboone/app/users/ area should be used to store configuration files, uboonecode builds, but should never be used to store data files. At the top of the xml file you will find this section while must be completed by the analyzer:
<!DOCTYPE project [ <!ENTITY user_id "put you user name here"> <!-- e.g. kirby, mrmooney, etc. this is the same as your kerberos principal UID and FNAL email --> <!ENTITY number_of_jobs "number of jobs to run"> <!-- you have to put in the number of jobs which should be num events/50 but you may actually have to think--> <!ENTITY number_of_events "number of events you want"> <!-- you have to put in the total number of events you want to generate (the total, not the number per file or jobs) --> <!ENTITY gen_fcl_file "fcl file name"> <!-- this is the name of the fcl file that you want to use to generate the events, it MUST be part of the tagged release relsim--> <!ENTITY name "name of the sample"> <!-- name of the samplethat you are generating, replace the "prod" from the fcl file name with test and remove the .xml"> <!-- Note that the name will be used for the name of output files so please use something reasonable --> <!-- Examples are here: <!ENTITY user_id "kirby"> <!ENTITY number_of_jobs "200"> ratio of number_of_events/number_of_jobs should be about 50 <!ENTITY number_of_events "10000"> <!ENTITY gen_fcl_file "prod_muminus_0.1-2.0GeV_isotropic_uboone.fcl"> <!ENTITY name "test_muminus_0.1-2.0GeV_isotropic_uboone"> -->
You can find the fcl files available for generation at this location:
NOTE!!!! You have to check the spacecharge fcl file for GEANT4 stage, and the DDR/Noise fcl files for the detector simulation stage of the xml file. That's important. The default is with spacecharge(on), DDR (on), and Noise(on), but if you want something different, you have to change this. DO NOT IGNORE THIS and keep track of these settings for your sample!!!
Once you have edited your file, then you need to make sure that the staging and output directories are ready. You credit those files with these commands:
mkdir -p /pnfs/uboone/scratch/users/$USER #This creates a directory in the dCache scratch area mkdir -p /uboone/data/users/$USER #this creates a directory in the BlueArc data volume
Note that the dCache scratch area does not have a quota, but files have a limited lifetime before they are flushed from the volume (usually ~ 30 days). But there is NO guarantee that files will be stored permanently. While on the BlueArc data volume, there is a 1.5 TiB quota for each user but the storage is permanent. For full details look here: https://redmine.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes
Now you're ready to submit jobs. First setup the environment with a current version of uboonecode so that you get a current version of the larbatch UPS product.
source /cvmfs/uboone.opensciencegrid.org/products/setup_uboone.sh setup uboonecode v06_26_01 -q e10:prof cd /uboone/app/users/$USER/<name_your_working_area>/ project.py --xml `pwd`/prod_muminus_0.1-2.0GeV_isotropic_uboone.xml --stage sim --submit #this submits the "sim" stage of the project which is gen+geant4+detsim jobsub_q --user=$USER
The last command checks to make sure that the jobs have been submitted. You should see a list of jobs with status "I" equal to the xml variable number_of_jobs. Once all the jobs are complete, then you'll run this command:
project.py --xml `pwd`/prod_muminus_0.1-2.0GeV_isotropic_uboone.xml --stage sim --check #this checks the "sim" stage files to make sure they were produced successfully
If there are no error, then you can continue to reco and mergeana stage.
project.py --xml `pwd`/prod_muminus_0.1-2.0GeV_isotropic_uboone.xml --stage reco --submit #this submits the "reco" stage of the project which is reco1+reco2 project.py --xml `pwd`/prod_muminus_0.1-2.0GeV_isotropic_uboone.xml --stage reco --check #this checks the "reco" stage files to make sure they were produced successfully project.py --xml `pwd`/prod_muminus_0.1-2.0GeV_isotropic_uboone.xml --stage mergeana --submit #this submits the "mergeana" stage of the project which produces AnaTree files project.py --xml `pwd`/prod_muminus_0.1-2.0GeV_isotropic_uboone.xml --stage mergeana --checkana #NOTE CHECKANA!!!!! this checks the "mergeana" stage files to make sure they were produced successfully
You should now have files in three locations within /pnfs/uboone/scratch (again these files have a finite lifetime since they are in a scratch area!!!!). Note that these are the xml variables (e.g. "&tag;"), and so you will have to translate them:
/pnfs/uboone/scratch/users/&user_id;/&tag;/&relsim;/sim/&name; /pnfs/uboone/scratch/users/&user_id;/&tag;/&relreco1;/reco/&name; /pnfs/uboone/scratch/users/&user_id;/&tag;/&relreco2;/mergeana/&name;
And we need to move them to permanent storage. We will do this using SAM4Users utilities. http://microboone-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=6896 These commands will have to be translated slightly but hopefully you understand what is being done. First we will make text files that contain the full paths to the files we've generated:
###ls /pnfs/uboone/scratch/users/&user_id;/&tag;/&relsim/sim/&name;/*/&name;*.root >& /uboone/app/users/$USER/&name;_sim_filelist.txt #this has to be translated and using the example above would become ls /pnfs/uboone/scratch/users/kirby/mcc8/v06_26_01/sim/test_muminus_0.1-2.0GeV_isotropic_uboone/*/test_muminus_0.1-2.0GeV_isotropic_uboone*.root >& /uboone/app/users/$USER/test_muminus_0.1-2.0GeV_isotropic_uboone_sim_filelist.txt ls /pnfs/uboone/scratch/users/kirby/mcc8/v06_26_01/reco/test_muminus_0.1-2.0GeV_isotropic_uboone/*/test_muminus_0.1-2.0GeV_isotropic_uboone*.root >& /uboone/app/users/$USER/test_muminus_0.1-2.0GeV_isotropic_uboone_reco_filelist.txt ls /pnfs/uboone/scratch/users/kirby/mcc8/v06_26_01/mergeana/test_muminus_0.1-2.0GeV_isotropic_uboone/*/ana*.root >& /uboone/app/users/$USER/test_muminus_0.1-2.0GeV_isotropic_uboone_mergeana_filelist.txt
Now we're going to setup FIFE utils UPS product, declare those files to SAM4Users, and then copy them to tape backed archive. But you much first come up with a dataset name. I recommend "$USER_mcc8_&name;_sim_v1" and if you regenerate, to increment the "v1" on the end. So this looks like:
source /cvmfs/uboone.opensciencegrid.org/products/setup_uboone.sh setup uboonecode v06_26_01 -q e10:prof setup fife_utils ##sam_add_dataset -n $USER_mcc8_&name;_v1 -f /uboone/app/users/$USERS/&name;_sim_filelist.txt sam_add_dataset -n $USER_mcc8_test_muminus_0.1-2.0GeV_isotropic_uboone_sim_v1 -f /uboone/app/users/$USERS/test_muminus_0.1-2.0GeV_isotropic_uboone_sim_filelist.txt sam_move2archive_dataset -n $USER_mcc8_test_muminus_0.1-2.0GeV_isotropic_uboone_sim_v1 sam_add_dataset -n$USER_mcc8_test_muminus_0.1-2.0GeV_isotropic_uboone_reco_v1 -f /uboone/app/users/$USERS/test_muminus_0.1-2.0GeV_isotropic_uboone_reco_filelist.txt sam_move2archive_dataset -n $USER_mcc8_test_muminus_0.1-2.0GeV_isotropic_uboone_reco_v1 sam_add_dataset -n$USER_mcc8_test_muminus_0.1-2.0GeV_isotropic_uboone_mergeana_v1 -f /uboone/app/users/$USERS/test_muminus_0.1-2.0GeV_isotropic_uboone_mergeana_filelist.txt sam_move2archive_dataset -n $USER_mcc8_test_muminus_0.1-2.0GeV_isotropic_uboone_mergeana_v1
At that point, the files are removed from dCache scratch space and moved to tape-backed,permanent storage. You will now need to access them through the SAM datasets definitions.