GEN_BE stage 4 using dmpar and multiple processors

Issues with running 3DVAR.

GEN_BE stage 4 using dmpar and multiple processors

Postby sureshzig » Sun Feb 05, 2017 1:06 pm

Hi !!

I am trying to generate BE matrix for own domain which is of the order 1000(E-W) x1000 (N-S) grid points

I compiled the WRFDA code on HPC (IBM) with INTEL compilers using dmpar option and compilation was successful.

when trying to run gen_be stage0 to stage3 are running fine, since domain is big I want to run stage4 on multiple processors

I am using following command inside script gen_be_stage4_regional.ksh to run it on multiple processors

cat > job_be_submit.sh << EOF1
#!/bin/ksh
#BSUB -J BE_WRF
#BSUB -W 24:00
#BSUB -n 512
#BSUB -R "span[ptile=16]"
#BSUB -q HPC_OPR
#BSUB -e hybrid.error.%J
#BSUB -o hybrid.output.%J
#BSUB -x

rm -f hostfile
cat $LSB_DJOB_HOSTFILE > hostfile

export SAVE_ALL_TASKS=no
export I_MPI_FABRICS=shm:ofa
export I_MPI_FALLBACK=1
export I_MPI_FABRICS_LIST=ofa,dapl
export KMP_AFFINITY=verbose,scatter
export FORT_BUFFERED=yes
####/usr/bin/time -p mpiexec.hydra -f ./hostfile -perhost 16 -np 416 ${TMP_DIR1}/gen_be_stage4_regional.exe > gen_be_stage4_regional_${VARIABLE}_${VINDEX}.out 2>&1
/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64/bin/mpirun -np 512 ${TMP_DIR1}/gen_be_stage4_regional.exe > gen_be_stage4_regional_${VARIABLE}_${VINDEX}.out 2>&1

EOF1
chmod 777 job_be_submit.sh

bsub -K job_be_submit.sh

Even though i am using 512 processors it is taking same time as when i am using a single processors (24 hrs for one level for one control variable..... total time for completion of stage4 will be 200 days)

please suggest me the possible solution why it taking so much of time
sureshzig
 
Posts: 1
Joined: Sun Feb 05, 2017 12:45 pm

Return to Runtime Problems

Who is online

Users browsing this forum: No registered users and 1 guest