vendredi 28 avril 2017

Using if else and xargs to pipe into gnu parallel

Using if else and xargs to pipe into gnu parallel (open to other suggestions, find…)

In short am trying to write a script to check for existing folders and run a command (script for the non-existing folders/files) in gnu parallel (open to other suggestions)

I will be running the script for a list of subjects/files and they will be distributed over several compute nodes on a server, so it is important that the command does not write over the existing folders again as the same script will be run on different nodes and all of them point to the same directories. So far, my idea is the following:

For simplicity

SUBJ_LIST=(a text file with list of subjects/files to be executed .txt)
SUBJ_EXIST=(folder that the program outputs to)
SUBJ_tp_do= (hopefully only subjects/files that have not been done)

cat ${SUBJ_LIST} | xargs -I{} -n 1 if [ -d ${SUBJ_EXIST}/{} ]
then
   echo "folder exists"
else
   SUBJ_tp_do={}
fi

parallel -j8 command {}  ::: ${SUBJ_to_do}

as you can tell this script does not work

Apologies in advance for my rudimentary knowledge in scripting, any help/input is really appreciated.

Aucun commentaire:

Enregistrer un commentaire