WRF 4.3.x SIGTERM Error After First Time Step

Any issues with the actual running of the WRF.

WRF 4.3.x SIGTERM Error After First Time Step

Postby jbellino » Tue Jan 11, 2022 6:26 pm

Hi all, I'm new to WRF so please assume I know little to nothing about it aside from what I've read in various forum posts and the online WRF users guide. I also know just enough about computers to be dangerous so please excuse any misuse of terminology. I'm encountering the issue below with both WRF 4.3.1 and 4.3.2.

BLUF: wrf.exe runs, completes the first time step, and then bails with the following error:
Code: Select all
taskid: 0 hostname: nid00218
 module_io_quilt_old.F        2931 F
Quilting with  10 groups of   8 I/O tasks.
 Ntasks in X           16 , ntasks in Y           20
  Domain # 1: dx =  9000.000 m
WRF V4.3.2 MODEL
git commit 57f8f5508dbfff90bee4647192e98338870a4656
 *************************************
 Parent domain
 ids,ide,jds,jde            1         792           1         382
 ims,ime,jms,jme           -4          57          -4          27
 ips,ipe,jps,jpe            1          50           1          20
 *************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
   alloc_space_field: domain            1 ,               82899032  bytes allocated
  med_initialdata_input: calling input_input
   Input data is acceptable to use: wrfinput_d01
 CURRENT DATE          = 2021-12-08_12:00:00
 SIMULATION START DATE = 2021-12-08_12:00:00
Timing for processing wrfinput file (stream 0) for domain        1:   14.01548 elapsed seconds
  med_initialdata_input: calling input_aux_model_input7
Max map factor in domain 1 =  1.10. Scale the dt in the model accordingly.
 D01: Time step                              =    5.000000      (s)
 D01: Grid Distance                          =    9.000000      (km)
 D01: Grid Distance Ratio dt/dx              =   0.5555556      (s/km)
 D01: Ratio Including Maximum Map Factor     =   0.6138647      (s/km)
 D01: NML defined reasonable_time_step_ratio =    6.000000
INPUT LandUse = "MODIFIED_IGBP_MODIS_NOAH"
 LANDUSE TYPE = "MODIFIED_IGBP_MODIS_NOAH" FOUND          41  CATEGORIES           2  SEASONS WATER CATEGORY =           17  SNOW CATEGORY =           15
 SOIL TEXTURE CLASSIFICATION = STAS FOUND          19  CATEGORIES
ThompMP: computing qr_acr_qg
Writing qr_acr_qgV3.dat in Thompson MP init
ThompMP: computing qr_acr_qs
Writing qr_acr_qsV2.dat in Thompson MP init
ThompMP: computing freezeH2O
Writing freezeH2O.dat in Thompson MP init
 mediation_integrate.G        1944 DATASET=HISTORY
 mediation_integrate.G        1945  grid%id            1  grid%oid            1
Timing for Writing wrfout_d01_2021-12-08_12:00:00 for domain        1:   14.04243 elapsed seconds
d01 2021-12-08_12:00:00  Input data is acceptable to use: wrfbdy_d01
Timing for processing lateral boundary for domain        1:   21.71185 elapsed seconds
 Tile Strategy is not specified. Assuming 1D-Y
WRF TILE   1 IS      1 IE     50 JS      1 JE     20
WRF NUMBER OF TILES =   1
d01 2021-12-08_12:00:00  ----------------------------------------
d01 2021-12-08_12:00:00  W-DAMPING  BEGINS AT W-COURANT NUMBER =    1.000000
d01 2021-12-08_12:00:00  ----------------------------------------
Timing for main: time 2021-12-08_12:00:05 on domain   1:   44.89969 elapsed seconds
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
wrf.exe            0000000023CFD93B  Unknown               Unknown  Unknown
libpthread-2.26.s  00002AAAAD38A360  Unknown               Unknown  Unknown
libmpich_intel.so  00002AAAACFCACC0  Unknown               Unknown  Unknown
libmpich_intel.so  00002AAAACFCF598  MPIDI_CH3I_Progre     Unknown  Unknown
libmpich_intel.so  00002AAAACED97F5  MPIR_Wait_impl        Unknown  Unknown
libmpich_intel.so  00002AAAACED9CC8  MPI_Wait              Unknown  Unknown
wrf.exe            0000000023C4B083  Unknown               Unknown  Unknown
wrf.exe            00000000213A2C76  module_comm_dm_0_        8124  module_comm_dm_0.f90
wrf.exe            0000000021B51BEA  module_first_rk_s         374  module_first_rk_step_part2.f90
wrf.exe            0000000021279AE3  solve_em_                 964  solve_em.f90
wrf.exe            0000000021068300  solve_interface_          122  solve_interface.f90
wrf.exe            000000002017E757  module_integrate_         329  module_integrate.f90
wrf.exe            00000000200147E8  module_wrf_top_mp         326  module_wrf_top.f90
wrf.exe            0000000020013FA5  MAIN__                     29  wrf.f90
wrf.exe            0000000020013F52  Unknown               Unknown  Unknown
libc-2.26.so       00002AAAAD7BAF8A  __libc_start_main     Unknown  Unknown
wrf.exe            0000000020013E6A  Unknown               Unknown  Unknown

I am running a model with 3 nested domains, each fully contained by its parent. I have run all WPS pre-processing as well as real.exe to create input from the ERA5 reanalysis dataset. I'm running the model on the USGS Denali supercomputer using 400 cpus (40 cpus on each of 10 nodes, --ntasks-per-node set to 40) for this problem which should be reasonable according to an NCAR FAQ topic I found. The lines of code listed in the error above don't seem to be anything of particular note, it just seems to be where the computation was at when memory/stack was exhausted. Ulimit is set to unlimited.

Here are some of the solutions I've tried so far with no change in the results:

  • running with fewer and more cpus,
  • set max_dom to 1,
  • adding the "heap-arrays" option during compile,
  • tried various optimization settings during compile (e.g. O2, O1),
  • manually increasing stacksize in my slurm script using KMP_STACKSIZE=640000000, and
  • recompiling with additional environment variables listed in forum post https://forum.wrfforum.com/viewtopic.ph ... 531#p19497

Needless to say, I am open to any suggestions or troubleshooting steps you all might have for me.

I can't seem to attach any kind of files to this post so I'm pasting the contents of my namelist.input file below:
Code: Select all
    &time_control
     run_days                            = 0,
     run_hours                           = 42,
     run_minutes                         = 0,
     run_seconds                         = 0,
     start_year                          = 2021, 2021, 2021,
     start_month                         = 12, 12, 12,
     start_day                           = 08, 09, 09,
     start_hour                          = 12, 06, 06,
     start_minute                        = 00,   00,   00,
     start_second                        = 00,   00,   00,
     end_year                            = 2021, 2021, 2021,
     end_month                           = 12, 12, 12,
     end_day                             = 10, 10, 10,
     end_hour                            = 06, 06, 06,
     end_minute                          = 00,   00,   00,
     end_second                          = 00,   00,   00,
     interval_seconds                    = 3600
     input_from_file                     = .true.,.true.,.true.,
     history_interval                    = 60,  60,   60,
     frames_per_outfile                  = 1, 1000, 1000,
     restart                             = .false.,
     io_form_history                     = 2
     io_form_restart                     = 2
     io_form_input                       = 2
     io_form_boundary                    = 2
     debug_level                         = 0
    /
   
    &domains
     time_step                           = 30,
     time_step_fract_num                 = 0,
     time_step_fract_den                 = 1,
     max_dom                             = 1,
     e_we                                = 792, 1138, 1045,
     e_sn                                = 382,  850, 1336,
     e_vert                              = 50,    50,   50,
     p_top_requested                     = 5000,
     num_metgrid_levels                  = 38,
     num_metgrid_soil_levels             = 4,
     dx                                  = 9000, 3000,  1000,
     dy                                  = 9000, 3000,  1000,
     grid_id                             = 1,     2,     3,
     parent_id                           = 1,     1,     2,
     i_parent_start                      = 1,   206,   383,
     j_parent_start                      = 1,    54,   178,
     parent_grid_ratio                   = 1,     3,     3,
     parent_time_step_ratio              = 1,     3,     3,
     feedback                            = 1,
     smooth_option                       = 0,
    /
   
    &physics
     mp_physics                          = 8,     3,     3,
     ra_lw_physics                       = 4,     4,     4,
     ra_sw_physics                       = 4,     4,     4,
     radt                                = 5,    30,    30,
     sf_sfclay_physics                   = 1,     1,     1,
     sf_surface_physics                  = 4,     4,     4,
     bl_pbl_physics                      = 1,     1,     1,
     bldt                                = 0,     0,     0,
     cu_physics                          = 0,     0,     0,
     cudt                                = 5,     5,     5,
     cu_rad_feedback                     = .true.,
     shcu_physics                        = 0,
     isfflx                              = 1,
     ifsnow                              = 1,
     icloud                              = 1,
     surface_input_source                = 1,
     num_soil_layers                     = 4,
     num_land_cat                        = 21,
     sf_urban_physics                    = 0,     0,     0,
     rdmaxalb                            = .false.,
     mp_zero_out                         = 0,
     iz0tlnd                             = 1,
     sst_update                          = 0,
     tmn_update                          = 1,
     lagday                              = 150,
     sst_skin                            = 1,
     slope_rad                           = 1,
     topo_shading                        = 0,
     topo_wind                           = 1,
     bucket_mm                           = 100.0,
     bucket_J                            = 1.e9,
     prec_acc_dt                         = 60.,
     do_radar_ref                        = 1,
     fractional_seaice                   = 1,
     seaice_threshold                    = 0.
     aer_opt                             = 1
    /
   
    &noah_mp
     dveg                                = 9,
     opt_crs                             = 1,
     opt_btr                             = 2,
     opt_run                             = 5,
     opt_sfc                             = 1,
     opt_frz                             = 1,
     opt_inf                             = 1,
     opt_rad                             = 3,
     opt_alb                             = 2,
     opt_snf                             = 4,
     opt_tbot                            = 1,
     opt_stc                             = 3,
     opt_gla                             = 1,
     opt_rsf                             = 1
    /
   
    &dynamics
     w_damping                           = 1,
     diff_opt                            = 2,      2,      2,
     km_opt                              = 4,      4,      4,
     diff_6th_opt                        = 0,      0,      0,
     diff_6th_factor                     = 0.12,   0.12,   0.12,
     base_temp                           = 290.
     damp_opt                            = 3,
     zdamp                               = 5000.,  5000.,  5000.,
     dampcoef                            = 0.2,    0.2,    0.2
     khdif                               = 0,      0,      0,
     kvdif                               = 0,      0,      0,
     non_hydrostatic                     = .true., .true., .true.,
     moist_adv_opt                       = 1,      1,      1,     
     scalar_adv_opt                      = 1,      1,      1,     
    /
   
    &namelist_quilt
     nio_tasks_per_group = 8,
     nio_groups = 10,
    /
   
    &bdy_control
     spec_bdy_width                      = 10,
     spec_zone                           = 1,
     relax_zone                          = 9,
     spec_exp                            = 0.33
     specified                           = .true., .false.,.false.,
     nested                              = .false., .true., .true.,
    /
jbellino
 
Posts: 3
Joined: Tue Jan 11, 2022 5:39 pm

Re: WRF 4.3.x SIGTERM Error After First Time Step

Postby jbellino » Thu Jan 13, 2022 4:28 pm

For whatever it's worth, I'm able to run this model using WRF v3.9.1.1. During testing of runtimes I've just encountered a SIGTERM error after completion of the first timestep by changing io_history from 2 to 11 for parallel netCDF writing. The routines listed in the error output are different, however, and since I'm using option 2 in the v4.3.x runs I'm sure that's not the problem. BUT, since the error occurs at the same point would it be fair to assume it has something to do with I/O even if it's not scheduled to write at that time?
jbellino
 
Posts: 3
Joined: Tue Jan 11, 2022 5:39 pm

Re: WRF 4.3.x SIGTERM Error After First Time Step

Postby dcvz » Tue Jan 18, 2022 11:12 am

There could be several sources for the problem. I notice that your namelist has dt = 30, while the rsl printout says DT = 5. Not sure why that is the case. I would see if you can get any 4.3 job to run. Try setting sf_surface_physics to 2 (to see if the problem is in noah-mp). I would change the run_minutes to 30 and run_hours to 0 since you just want to see if you can get the model to go past the first few timesteps. If it still fails after one timestep, set debug_level to 1000 just to see if you get more information (set it back to 0 for your production runs).

Since 3.9.1.1 is able to run, then that suggests the computer environment is OK. (As long as WRF V3.9.1.1 was compiled in the same way as V4.3.2).

Make sure that real is run using the same namelist.input as is used by wrf.

Your tiling seems OK. The only questionable value I see in the namelist is radt=5 may be too short which will be expensive, but it shouldn't be fatal.
dcvz
 
Posts: 205
Joined: Tue Apr 15, 2008 12:02 am

Re: WRF 4.3.x SIGTERM Error After First Time Step

Postby jbellino » Tue Jan 18, 2022 2:25 pm

dcvz wrote:There could be several sources for the problem. I notice that your namelist has dt = 30, while the rsl printout says DT = 5. Not sure why that is the case.

I must have copied an old rsl file that was run with a shorter time-step; regardless of the dt used I still get the same error.

dcvz wrote:Try setting sf_surface_physics to 2 (to see if the problem is in noah-mp). I would change the run_minutes to 30 and run_hours to 0 since you just want to see if you can get the model to go past the first few timesteps. If it still fails after one timestep, set debug_level to 1000 just to see if you get more information (set it back to 0 for your production runs).

I set surface physics to "2" and reran real.exe and am still getting the error. I also separately tested increasing "radt" to "30" for all domains. I'm still unable to attach any kind of files here so pasting the last few hundred lines of the rsl error files generated with debug level set to "1000".

Code: Select all
d01 2021-12-08_12:00:00 module_io.F: in wrf_read_field
 inc/wrf_bdyin.inc ext_read_field PC_BTYS memorder YS Status =            0
inc/wrf_bdyin.inc ext_read_field PC_BTYE memorder YE
 date 2021-12-08_12:00:00
 ds            1           1           1
 de          791          10           1
 ps            1           1           1
 pe           57          10           1
 ms           -4           1           1
 me           57          10           1
d01 2021-12-08_12:00:00 module_io.F: in wrf_read_field
 inc/wrf_bdyin.inc ext_read_field PC_BTYE memorder YE Status =            0
d01 2021-12-08_12:00:00  input_wrf: end, fid =            1
Timing for processing lateral boundary for domain        1:    3.58452 elapsed seconds
d01 2021-12-08_12:00:00 module_integrate: calling solve interface
 Tile Strategy is not specified. Assuming 1D-Y
WRF TILE   1 IS      1 IE     50 JS      1 JE     20
WRF NUMBER OF TILES =   1
d01 2021-12-08_12:00:00  grid spacing, dt, time_step_sound=   9000.000       30.00000               4
d01 2021-12-08_12:00:00 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2021-12-08_12:00:00  call rk_step_prep
d01 2021-12-08_12:00:00 calling inc/HALO_EM_A_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2021-12-08_12:00:00  call rk_phys_bc_dry_1
d01 2021-12-08_12:00:00  call init_zero_tendency
d01 2021-12-08_12:00:00 calling inc/HALO_EM_PHYS_A_inline.inc
d01 2021-12-08_12:00:00  call phy_prep
d01 2021-12-08_12:00:00  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:00]
d01 2021-12-08_12:00:00  call radiation_driver
d01 2021-12-08_12:00:00 Top of Radiation Driver
d01 2021-12-08_12:00:00 CALL cldfra1
d01 2021-12-08_12:00:00 CALL rrtmg_lw
d01 2021-12-08_12:00:00 CALL rrtmg_sw
d01 2021-12-08_12:00:00 calling inc/HALO_PWP_inline.inc
d01 2021-12-08_12:00:00  call surface_driver
d01 2021-12-08_12:00:00 in SST_SKIN_UPDATE
d01 2021-12-08_12:00:00 in TMN_UPDATE
d01 2021-12-08_12:00:00 in SFCLAY
d01 2021-12-08_12:00:00 in NOAH DRV
d01 2021-12-08_12:00:00  call pbl_driver
d01 2021-12-08_12:00:00 in YSU PBL
d01 2021-12-08_12:00:00  call cumulus_driver
d01 2021-12-08_12:00:00  call shallow_cumulus_driver
d01 2021-12-08_12:00:00 calling inc/HALO_EM_FDDA_SFC_inline.inc
d01 2021-12-08_12:00:00  call fddagd_driver
d01 2021-12-08_12:00:00  call calculate_phy_tend
d01 2021-12-08_12:00:00  call compute_diff_metrics
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TKE_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_A1_inline.inc
d01 2021-12-08_12:00:00  call bc for diffusion_metrics
d01 2021-12-08_12:00:00  call cal_deform_and_div
d01 2021-12-08_12:00:00 calling inc/HALO_EM_HELICITY_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TKE_D_inline.inc
d01 2021-12-08_12:00:00  call calculate_km_kh
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TKE_E_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_PHY_BC_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_CHEM_inline.inc
d01 2021-12-08_12:00:00  call phy_bc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_PHYS_PBL_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_PHYS_DIFFUSION_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TKE_5_inline.inc
d01 2021-12-08_12:00:00  call update_phy_ten
d01 2021-12-08_12:00:00  call horizontal_diffusion_2
d01 2021-12-08_12:00:00  call rk_tendency
d01 2021-12-08_12:00:00  ----------------------------------------
d01 2021-12-08_12:00:00  W-DAMPING  BEGINS AT W-COURANT NUMBER =    1.000000
d01 2021-12-08_12:00:00  ----------------------------------------
d01 2021-12-08_12:00:00  call small_step_prep
d01 2021-12-08_12:00:00 calling inc/HALO_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00  call rk_small_finish
d01 2021-12-08_12:00:00 calling inc/HALO_EM_D_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_EM_DA_inline.inc
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00 calling inc/HALO_EM_D2_5_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_D_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_MOIST2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_CHEM2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_TRACER2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_SCALAR2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_TKE_inline.inc
d01 2021-12-08_12:00:00  call rk_phys_bc_dry_2
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TKE_5_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_MOIST_E_5_inline.inc
d01 2021-12-08_12:00:00  call rk_step_prep
d01 2021-12-08_12:00:00 calling inc/HALO_EM_A_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2021-12-08_12:00:00  call rk_phys_bc_dry_1
d01 2021-12-08_12:00:00  call rk_tendency
d01 2021-12-08_12:00:00  call small_step_prep
d01 2021-12-08_12:00:00 calling inc/HALO_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00  call rk_small_finish
d01 2021-12-08_12:00:00 calling inc/HALO_EM_D_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_EM_DA_inline.inc
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00 calling inc/HALO_EM_D2_5_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_D_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_MOIST2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_CHEM2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_TRACER2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_SCALAR2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_TKE_inline.inc
d01 2021-12-08_12:00:00  call rk_phys_bc_dry_2
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TKE_7_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_MOIST_E_7_inline.inc
d01 2021-12-08_12:00:00  call rk_step_prep
d01 2021-12-08_12:00:00 calling inc/HALO_EM_A_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2021-12-08_12:00:00  call rk_phys_bc_dry_1
d01 2021-12-08_12:00:00  call rk_tendency
d01 2021-12-08_12:00:00  call small_step_prep
d01 2021-12-08_12:00:00 calling inc/HALO_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C2_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2021-12-08_12:00:00  call rk_small_finish
d01 2021-12-08_12:00:00  call rk_update_scalar_pd
d01 2021-12-08_12:00:00 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2021-12-08_12:00:00  call rk_update_scalar_pd
d01 2021-12-08_12:00:00 calling inc/HALO_EM_SCALAR_OLD_E_7_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_SCALAR_OLD_inline.inc
d01 2021-12-08_12:00:00  call rk_update_scalar_pd
d01 2021-12-08_12:00:00 calling inc/HALO_EM_CHEM_OLD_E_7_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_CHEM_OLD_inline.inc
d01 2021-12-08_12:00:00  call rk_update_scalar_pd
d01 2021-12-08_12:00:00 calling inc/HALO_EM_TRACER_OLD_E_7_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_TRACER_OLD_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_D_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_EM_DA_inline.inc
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call rk_scalar_tend
d01 2021-12-08_12:00:00  call rk_update_scalar
d01 2021-12-08_12:00:00  call advance_ppt
d01 2021-12-08_12:00:00  call phy_prep_part2
d01 2021-12-08_12:00:00  call moist_physics_prep
d01 2021-12-08_12:00:00  call microphysics_driver
d01 2021-12-08_12:00:00 calling inc/HALO_EM_SBM_inline.inc
d01 2021-12-08_12:00:00 microphysics_driver: calling wsm3
d01 2021-12-08_12:00:00 microphysics_driver: returning from
d01 2021-12-08_12:00:00  call moist_physics_finish
d01 2021-12-08_12:00:00 calling inc/HALO_EM_THETAM_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_EM_THETAM_inline.inc
d01 2021-12-08_12:00:00  call chem polar filter
d01 2021-12-08_12:00:00 calling inc/HALO_EM_D3_5_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_D3_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_MOIST_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_CHEM_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_TRACER_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_SCALAR_inline.inc
d01 2021-12-08_12:00:00  call set_phys_bc_dry_2
d01 2021-12-08_12:00:00  call spec_bdy_final
d01 2021-12-08_12:00:00 calling inc/HALO_EM_C_inline.inc
d01 2021-12-08_12:00:00 calling inc/PERIOD_BDY_EM_E_inline.inc
d01 2021-12-08_12:00:00  call set_w_surface
d01 2021-12-08_12:00:00 calling inc/HALO_EM_PHYS_W_inline.inc
d01 2021-12-08_12:00:00 calling inc/HALO_EM_PHYS_HCW_inline.inc
d01 2021-12-08_12:00:00 --> TOP OF AFTER ALL RK STEPS
d01 2021-12-08_12:00:00 --> CALLING DIAGNOSTICS DRIVER
d01 2021-12-08_12:00:00 --> TOP OF DIAGNOSTICS PACKAGE
d01 2021-12-08_12:00:00 --> CALL DIAGNOSTICS PACKAGE: ACCUMULATED AND BUCKET DIAGNOSTICS
d01 2021-12-08_12:00:00  call HALO_RK_E
d01 2021-12-08_12:00:00 calling inc/HALO_EM_E_5_inline.inc
d01 2021-12-08_12:00:00  call HALO_RK_MOIST
d01 2021-12-08_12:00:00 calling inc/HALO_EM_MOIST_E_5_inline.inc
d01 2021-12-08_12:00:00  call end of solve_em
d01 2021-12-08_12:00:00  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:00]
d01 2021-12-08_12:00:00  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:00]
d01 2021-12-08_12:00:00  DEBUG wrf_timetoa():  returning with str = [2021-12-10_06:00:00]
d01 2021-12-08_12:00:00  DEBUG wrf_timeinttoa():  returning with str = [0000000000_000:000:030]
 DEBUG domain_clockadvance():  before WRFU_ClockAdvance,  clock start time = 2021-12-08_12:00:00
 DEBUG domain_clockadvance():  before WRFU_ClockAdvance,  clock current time = 2021-12-08_12:00:00
 DEBUG domain_clockadvance():  before WRFU_ClockAdvance,  clock stop time = 2021-12-10_06:00:00
 DEBUG domain_clockadvance():  before WRFU_ClockAdvance,  clock time step = 0000000000_000:000:030
d01 2021-12-08_12:00:30  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:30]
d01 2021-12-08_12:00:30  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:00]
d01 2021-12-08_12:00:30  DEBUG wrf_timetoa():  returning with str = [2021-12-10_06:00:00]
d01 2021-12-08_12:00:30  DEBUG wrf_timeinttoa():  returning with str = [0000000000_000:000:030]
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock start time = 2021-12-08_12:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock current time = 2021-12-08_12:00:30
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock stop time = 2021-12-10_06:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock time step = 0000000000_000:000:030
d01 2021-12-08_12:00:30 module_integrate: back from solve interface
d01 2021-12-08_12:00:30  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:30]
Timing for main: time 2021-12-08_12:00:30 on domain   1:   20.28235 elapsed seconds
d01 2021-12-08_12:00:30 in med_latbound_in
d01 2021-12-08_12:00:30 module_integrate: calling solve interface
d01 2021-12-08_12:00:30  grid spacing, dt, time_step_sound=   9000.000       30.00000               4
d01 2021-12-08_12:00:30 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2021-12-08_12:00:30 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2021-12-08_12:00:30  call rk_step_prep
d01 2021-12-08_12:00:30 calling inc/HALO_EM_A_inline.inc
d01 2021-12-08_12:00:30 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2021-12-08_12:00:30  call rk_phys_bc_dry_1
d01 2021-12-08_12:00:30  call init_zero_tendency
d01 2021-12-08_12:00:30 calling inc/HALO_EM_PHYS_A_inline.inc
d01 2021-12-08_12:00:30  call phy_prep
d01 2021-12-08_12:00:30  DEBUG wrf_timetoa():  returning with str = [2021-12-08_12:00:30]
d01 2021-12-08_12:00:30  call radiation_driver
d01 2021-12-08_12:00:30 Top of Radiation Driver
d01 2021-12-08_12:00:30 calling inc/HALO_PWP_inline.inc
d01 2021-12-08_12:00:30  call surface_driver
d01 2021-12-08_12:00:30 in SST_SKIN_UPDATE
d01 2021-12-08_12:00:30 in TMN_UPDATE
d01 2021-12-08_12:00:30 in SFCLAY
d01 2021-12-08_12:00:30 in NOAH DRV
d01 2021-12-08_12:00:30  call pbl_driver
d01 2021-12-08_12:00:30 in YSU PBL
d01 2021-12-08_12:00:30  call cumulus_driver
d01 2021-12-08_12:00:30  call shallow_cumulus_driver
d01 2021-12-08_12:00:30 calling inc/HALO_EM_FDDA_SFC_inline.inc
d01 2021-12-08_12:00:30  call fddagd_driver
d01 2021-12-08_12:00:30  call calculate_phy_tend
d01 2021-12-08_12:00:30  call compute_diff_metrics
d01 2021-12-08_12:00:30 calling inc/HALO_EM_TKE_C_inline.inc
d01 2021-12-08_12:00:30 calling inc/PERIOD_BDY_EM_A1_inline.inc
d01 2021-12-08_12:00:30  call bc for diffusion_metrics
d01 2021-12-08_12:00:30  call cal_deform_and_div
d01 2021-12-08_12:00:30 calling inc/HALO_EM_HELICITY_inline.inc
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
wrf.exe            0000000023CFD93B  Unknown               Unknown  Unknown
libpthread-2.26.s  00002AAAAD38A360  Unknown               Unknown  Unknown
libmpich_intel.so  00002AAAACFF127B  Unknown               Unknown  Unknown
libmpich_intel.so  00002AAAACFF1DCA  MPID_nem_gni_poll     Unknown  Unknown
libmpich_intel.so  00002AAAACFCFB46  MPIDI_CH3I_Progre     Unknown  Unknown
libmpich_intel.so  00002AAAACED97F5  MPIR_Wait_impl        Unknown  Unknown
libmpich_intel.so  00002AAAACED9CC8  MPI_Wait              Unknown  Unknown
wrf.exe            0000000023C4B083  Unknown               Unknown  Unknown
wrf.exe            00000000213BBC09  module_comm_dm_1_         608  module_comm_dm_1.f90
wrf.exe            0000000021B5454A  module_first_rk_s         489  module_first_rk_step_part2.f90
wrf.exe            0000000021279AE3  solve_em_                 964  solve_em.f90
wrf.exe            0000000021068300  solve_interface_          122  solve_interface.f90
wrf.exe            000000002017E757  module_integrate_         329  module_integrate.f90
wrf.exe            00000000200147E8  module_wrf_top_mp         326  module_wrf_top.f90
wrf.exe            0000000020013FA5  MAIN__                     29  wrf.f90
wrf.exe            0000000020013F52  Unknown               Unknown  Unknown
libc-2.26.so       00002AAAAD7BAF8A  __libc_start_main     Unknown  Unknown
wrf.exe            0000000020013E6A  Unknown               Unknown  Unknown
jbellino
 
Posts: 3
Joined: Tue Jan 11, 2022 5:39 pm

Re: WRF 4.3.x SIGTERM Error After First Time Step

Postby dcvz » Tue Jan 18, 2022 10:30 pm

Well, it's dying in a halo routine which is outside of my experience. I suggest you sign up at https://forum.mmm.ucar.edu/ and post your question along with your namelist and rsl debug output. Maybe they'll have a better idea of what is going on.

One other thing to try, though, is to look at your wrfinput files using ncview. I guess it's possible there is some weird data in your initial conditions.
dcvz
 
Posts: 205
Joined: Tue Apr 15, 2008 12:02 am


Return to Runtime Problems

Who is online

Users browsing this forum: No registered users and 5 guests