WRF Installation Automation

All things involving the setup (compiling, NetCDF, etc.) of the WRF Model itself.

WRF Installation Automation

Postby Joshua Lipstone » Sat Sep 05, 2015 3:23 pm

All,

I developed a few scripts that are designed to make setting up and running WRF installations significantly faster/easier, and I wanted to post them here for people to use. The scripts work on Debian Linux (e.g. Ubuntu) and Mac OSX. The full source code, download links, and instructions are available (for free, for use, etc.) here: https://github.com/Toberumono/WRF-Setup-Script and https://github.com/Toberumono/WRF-Runner.

The setup script automatically installs the required libraries, sets the necessary environment variables (it turns out that it is possible to detect the correct values for all of them), and runs the configure and compile scripts for WRF and WPS.
The runner script can automatically run simulations on the resulting installation (serial, SMPAR, and DMPAR are all supported). This includes automatically downloading data, creating temporary working directories, and storing the output in timestamped directories. Additionally, creating a quick cron job (instructions included) allows it to perform rolling simulations.

More detailed explanations, including full usage tutorials are available on the repository homepages. Those are, again (autoscrolling to the instructions this time): https://github.com/Toberumono/WRF-Setup-Script#wrf-setup-script and https://github.com/Toberumono/WRF-Runner#wrf-runner.

Thank you very much for your time.

If you have any comments, questions, etc, please post them here or send them via the issues tab on the respective GitHub repository.

--Joshua Lipstone
Joshua Lipstone
 
Posts: 24
Joined: Wed Sep 02, 2015 5:41 pm

Re: WRF Installation Automation

Postby brunota2003 » Sun Sep 06, 2015 9:02 am

Good morning!

I am a brand new beginner to Linux, but have learned a decent amount binging over the last couple nights, trying to set up the WRF. Reading through several user guides/tutorials yields different things you need to download to get the model to actually compile/work...and it seems you need literally ALL those things...how frustrating! Some of the guides break it down pretty well, others obviously expect you to know you need to download and install gcc/fortran/whatever else and never even bother to mention getting those! It's like trying to put together a big puzzle with only half the pieces hah

I tried out your script and ran into some serious errors. Just for reference, I am running Ubuntu 14.04 LTS. I followed the guide, downloaded the script into a directory, along with the various pieces of the WRF/geodata.

When I attempted to run the script, it came back with this:

modeldesktop-ET1331G:~/Model/WRF-Setup-Script-master> sudo ./WRFSetup.sh
[sudo] password for modeldesktop:
wget: unrecognized option '--show-progress'
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
./WRFSetup.sh: line 32: bash_upgrade: command not found
wget: unrecognized option '--show-progress'
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Using apt.
No sudo. Skipping installation.
Failed to install: pv ncl doxygen mpich netcdf.
Please install these items manually or try running this again with sudo privileges.


I tried it several times, and I know I put in the password correctly...Any ideas?

Thanks!
brunota2003
 
Posts: 3
Joined: Sun Sep 06, 2015 8:28 am

Re: WRF Installation Automation

Postby winash12 » Sun Sep 06, 2015 9:52 pm

Very nicely done. How can this be modified for ifort ?
winash12
 
Posts: 37
Joined: Thu May 29, 2014 2:56 am

Re: WRF Installation Automation

Postby Joshua Lipstone » Wed Sep 09, 2015 7:00 pm

For brunota2003:

That's an interesting issue. It would appear that your version of wget does not recognize the --show-progress option, so that would explain the sudo-detection breaking, which would in turn break the sub-module installation (netcdf, ncl, etc.). Can you let me know the version of wget that you are using so I can test it?

In the mean time, you can open WRFSetup.sh and change line 27 to read:
Code: Select all
[ "$(which wget)" == "" ] && pull_command="curl -#fSL" || pull_command="wget -qO -"

(Remove '--show-progress' from it) as a temporary fix.

For winash12:

While I cannot directly test the ifort compilers on my system, you should be able to select the ifort/icc compilers in the configuration step (While the libraries (e.g. netcdf) might not build with ifort, they should still work). Unfortunately, the only way to build the libraries with ifort would be to build them manually, and, while that is an option, I'm also curious if the ifort compiler will work without going through all of that. Can you let me know if that works? (While I cannot directly test the ifort/icc compilers, I should be able to throw together a script that at least automatically builds the libraries from source using a user-defined compiler if needed; however, it won't be ready for a few days at least)

If the ifort compiler does not show up during the configuration step, then wrf likely does not support using ifort on your system.
Joshua Lipstone
Joshua Lipstone
 
Posts: 24
Joined: Wed Sep 02, 2015 5:41 pm

Re: WRF Installation Automation

Postby brunota2003 » Fri Sep 11, 2015 4:36 am

Just wanted to let you know: I replaced the whole script with the one from just prior to the Added Progress bars version and it worked like a charm! Not sure what version I was using, it shows I have Version 1.15 currently.

Now I'm having issues using NCL to view the data produced by the model. I can get the terrain data to work just fine with sample scripts, but anything requiring WRFUserARW.ncl gives me a bunch of "syntax error: possibly an undefined procedure"

For example, when the file paths are properly edited, this script http://www2.mmm.ucar.edu/wrf/OnLineTuto ... s_ter2.ncl works fine...but when I try this script http://www2.mmm.ucar.edu/wrf/OnLineTuto ... rface1.ncl it errors out. Note: Both scripts are samples from here: http://www2.mmm.ucar.edu/wrf/OnLineTuto ... amples.htm

I'm not sure if there is something I have to change I just don't realize, or what...but it gives me a ton of errors like this:
Code: Select all
modeldesktop-ET1331G:~/Model/NCAR> ncl samplescript.sh
 Copyright (C) 1995-2013 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.1.2
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
fatal:syntax error: line 269 in file /home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl before or near \n
     copy_VarAtts(var3d,var3)
-----------------------------^

fatal:syntax error: possibly an undefined procedure
fatal:syntax error: line 271 in file /home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl before or near \n
       delete_VarAtts(var3,(/"description"/))
---------------------------------------------^

fatal:syntax error: possibly an undefined procedure
fatal:syntax error: line 274 in file /home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl before or near \n
       delete_VarAtts(var3,(/"units"/))
---------------------------------------^

fatal:syntax error: possibly an undefined procedure
fatal:syntax error: line 277 in file /home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl before or near \n
       delete_VarAtts(var3,(/"MemoryOrder"/))
---------------------------------------------^


The very last error it gives me is this:
Code: Select all
fatal:syntax error: possibly an undefined procedure
fatal:Syntax Error in block, block not executed
fatal:error at line 2106 in file /home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl

fatal:Undefined identifier: (wrf_user_getvar) is undefined, can't continue
fatal:["Execute.c":8128]:Execute: Error occurred at or near line 33 in file samplescript.sh


Line 32, 33, and 34 on my text document reads:
Code: Select all
; What times and how many time steps are in the data set?
  times = wrf_user_getvar(a,"times",-1)  ; get all times in the file
  ntimes = dimsizes(times)         ; number of times in the file


It states that wrf_user_getvar is undefined...is that possibly the source for what is causing the massive failures? How do I fix it so it is "defined"?

Thanks!
brunota2003
 
Posts: 3
Joined: Sun Sep 06, 2015 8:28 am

Re: WRF Installation Automation

Postby Joshua Lipstone » Fri Sep 11, 2015 3:30 pm

Hi brunota,

Thank you for letting me know. I'm fairly sure worked out the issue with the scripts. The last error that you saw occurs when it cannot find the file. (Being unable to open the file actually results in undefined functions. Go figure.) The filename line that I used is:
Code: Select all
  a = addfile("./Working/2015-09-07_06_00_00/wrfout_d01_2015-09-07_06:00:00.nc","r")


As for the wget problem, I highly recommend updating wget if you can.
Joshua Lipstone
Joshua Lipstone
 
Posts: 24
Joined: Wed Sep 02, 2015 5:41 pm

Re: WRF Installation Automation

Postby brunota2003 » Sat Sep 12, 2015 2:30 pm

Well, here is the script as edited for my system:

Code: Select all
;   Example script to produce plots for a WRF real-data run,
;   with the ARW coordinate dynamics option.

load "/home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "/home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
;load "/home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"

begin
;
; The WRF ARW input file. 
; This needs to have a ".nc" appended, so just do it.
  a = addfile("/home/modeldesktop/Model/WRF-Setup-Script-master/WRFV3/test/em_real/wrfout_d01_2015-09-08_10:00:00.nc","r")

; We generate plots, but what kind do we prefer?
  type = "x11"
; type = "pdf"
; type = "ps"
; type = "ncgm"
  wks = gsn_open_wks(type,"plt_Surface1")

; Set some basic resources
  res = True
  res@MainTitle                   = "REAL-TIME WRF"

  pltres = True
  mpres = True


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; What times and how many time steps are the data set?
  times = wrf_user_getvar(a,"times",-1)  ; get all times in the file
  ntimes = dimsizes(times)         ; number of times in the file

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

  do it = 0,ntimes-1,2             ; TIME LOOP

    print("Working on time: " + times(it) )
    res@TimeLabel = times(it)   ; Set Valid time to use on plots


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; First get the variables we will need       

    slp = wrf_user_getvar(a,"slp",it)    ; slp
      wrf_smooth_2d( slp, 3 )            ; smooth slp
    tc = wrf_user_getvar(a,"tc",it)      ; 3D tc
    td = wrf_user_getvar(a,"td",it)      ; 3D td
    u  = wrf_user_getvar(a,"ua",it)      ; 3D U at mass points
    v  = wrf_user_getvar(a,"va",it)      ; 3D V at mass points
    td2 =  wrf_user_getvar(a,"td2",it)   ; Td2 in C
    tc2 = wrf_user_getvar(a,"T2",it)     ; T2 in Kelvin
       tc2 = tc2-273.16                  ; T2 in C
    u10 = wrf_user_getvar(a,"U10",it)    ; u at 10 m, mass point
    v10 = wrf_user_getvar(a,"V10",it)    ; v at 10 m, mass point

    tf2 = 1.8*tc2+32.                    ; Turn temperature into Fahrenheit
      tf2@description = "Surface Temperature"
      tf2@units = "F"
    td_f = 1.8*td2+32.                   ; Turn temperature into Fahrenheit
      td_f@description = "Surface Dew Point Temp"
      td_f@units = "F"
    u10 = u10*1.94386                    ; Turn wind into knots
    v10 = v10*1.94386
      u10@units = "kts"
      v10@units = "kts"

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

    ; Plotting options for T               
      opts = res                         
      opts@cnFillOn = True 
      opts@ContourParameters = (/ -20., 90., 5./)
      opts@gsnSpreadColorEnd = -3  ; End third from the last color in color map
      contour_tc = wrf_contour(a,wks,tf2,opts)
      delete(opts)


    ; Plotting options for Td
      opts = res         
      opts@cnFillOn = True
      opts@cnLinesOn = True
      opts@cnLineLabelsOn = True
      opts@ContourParameters = (/ -20., 90., 5./)
      opts@cnLineLabelBackgroundColor = -1
      opts@gsnSpreadColorEnd = -3  ; End third from the last color in color map
      contour_td = wrf_contour(a,wks,td_f,opts)
      delete(opts)


    ; Plotting options for SLP                     
      opts = res         
      opts@cnLineColor = "Blue"
      opts@cnHighLabelsOn = True
      opts@cnLowLabelsOn = True
      opts@ContourParameters = (/ 900., 1100., 4. /)
      opts@cnLineLabelBackgroundColor = -1
      opts@gsnContourLineThicknessesScale = 2.0
      contour_psl = wrf_contour(a,wks,slp,opts)
      delete(opts)

    ; Plotting options for Wind Vectors                 
      opts = res         
      opts@FieldTitle = "Wind"       ; overwrite Field Title
      opts@NumVectors = 47           ; density of wind barbs
      vector = wrf_vector(a,wks,u10,v10,opts)
      delete(opts)
 

    ; MAKE PLOTS                                       
      plot = wrf_map_overlays(a,wks,(/contour_tc,contour_psl,vector/),pltres,mpres)
      plot = wrf_map_overlays(a,wks,(/contour_td,vector/),pltres,mpres)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

  end do        ; END OF TIME LOOP

end


I'm pretty sure I have changed everything I need to change? But it still comes out as multiple errors.

When I change line 33 to this:
Code: Select all
times = addfile("/home/modeldesktop/Model/WRF-Setup-Script-master/WRFV3/test/em_real/wrfout_d01_2015-09-08_10:00:00.nc","r")  ; get all times in the file


It stops giving a line 33 error, but now gives a line 40 error (as well as all the undefined errors):

Code: Select all
fatal:syntax error: possibly an undefined procedure
fatal:Syntax Error in block, block not executed
fatal:error at line 2106 in file /home/modeldesktop/Model/NCAR/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl

fatal:Could not coerce values for operation
fatal:["Execute.c":8128]:Execute: Error occurred at or near line 40 in file samplescript.sh


Line 38, 40, and 41 all say:
Code: Select all
do it = 0,ntimes-1,2             ; TIME LOOP

    print("Working on time: " + times(it) )
    res@TimeLabel = times(it)   ; Set Valid time to use on plots


Frustrating! Hah
brunota2003
 
Posts: 3
Joined: Sun Sep 06, 2015 8:28 am

Re: WRF Installation Automation

Postby Joshua Lipstone » Sat Sep 12, 2015 4:11 pm

You need to add the '.nc' suffix to the file manually - WRF doesn't output the files with that extension by default.

That should (I think) work.
Joshua Lipstone
Joshua Lipstone
 
Posts: 24
Joined: Wed Sep 02, 2015 5:41 pm

Re: WRF Installation Automation

Postby cmor » Mon Sep 14, 2015 11:47 pm

Thanks so much for the effort put into this script.

I've worked through a few errors during install but am now stuck on this:

Code: Select all
$ sudo ./WRFSetup.sh
Password:
./WRFSetup.sh: line 16: bash_upgrade: command not found
./WRFSetup.sh: line 32: bash_upgrade: command not found
Using brew.
Checking for upgradeable packages.
-                                     0%[                                                                   ]       0  --.-KB/s   in 0s     
-                                     0%[                                                                   ]       0  --.-KB/s   in 0s     
Already up-to-date.
Found git
Found wget
Warning: caskroom/cask/brew-cask-0.56.0 already installed
Warning: A Cask for ncar-ncl is already installed. Add the "--force" option to force re-install.
-                                   100%[==================================================================>]   1.67K  --.-KB/s   in 0s     
/dev/fd/63: line 20: update_rc: command not found
./WRFSetup.sh: line 190: : No such file or directory
Found m4
-                                     0%[                                                                   ]       0  --.-KB/s   in 0s     
-                                     0%[                                                                   ]       0  --.-KB/s   in 0s     
Warning: pv-1.6.0 already installed
Warning: homebrew/dupes/ncurses-6.0 already installed
Warning: cairo-1.14.2_1 already installed
Warning: libpng-1.6.18 already installed
Warning: szip-2.1 already installed
Warning: lzlib-1.7 already installed
Warning: pixman-0.32.6 already installed
Warning: doxygen-1.8.10 already installed
Warning: mpich-3.1.4_1 already installed
Warning: tcsh-6.19.00 already installed
Warning: homebrew/science/hdf5-1.8.14 already installed
Warning: jasper-1.900.1 already installed
Warning: homebrew/science/netcdf-4.3.3.1 already installed
Failed to install: ncl.
Please install these items manually or try running this again with sudo privileges.


How can I get through the NCL bit at the end?
cmor
 
Posts: 9
Joined: Mon Sep 14, 2015 11:44 pm

Re: WRF Installation Automation

Postby Joshua Lipstone » Wed Sep 16, 2015 12:21 am

Hi cmor,

Can you let me know what version of bash you are using? (bash --version)

If you are on a Mac, the default version of bash is actually incredibly old (version 3.2.x, I believe), and if you still have that version, the script should automatically use homebrew to get an updated version and then run itself using that version. (It doesn't change the version that your system uses)

The error that you are getting indicates that the section of the script that is supposed to handle getting the correct version of bash might not be working properly. I just pushed an update that should fix it. Just re-run step 4 in the preparation section to get the new version.

Let me know if that fixes it.
Joshua Lipstone
Joshua Lipstone
 
Posts: 24
Joined: Wed Sep 02, 2015 5:41 pm

Next

Return to Installation

Who is online

Users browsing this forum: Google [Bot] and 4 guests