Using the GFDL CMIP5 Datasets

Dataset specific topics involving WPS.

Using the GFDL CMIP5 Datasets

Postby newbiewrfuser » Mon May 05, 2014 4:46 am

Hi all,

Sorry if this question comes up in the wrong place or if it's already answered (in that case, please help me to show the link).

Anyway. It's been a long time since last time using WRF. I am new in this matter and not a frequent user. Now, i want to try use the GFDL available datasets for future projection simulation. I also wonder if this is possible.

Code: Select all
GRIB1| Level| From |  To  | metgrid  |  metgrid | metgrid                                  |
Param| Type |Level1|Level2| Name     |  Units   | Description                              |
-----+------+------+------+----------+----------+------------------------------------------+
  11 | 100  |   *  |      | TT       | K        | Temperature                              |
  33 | 100  |   *  |      | UU       | m s-1    | U                                        |
  34 | 100  |   *  |      | VV       | m s-1    | V                                        |
  52 | 100  |   *  |      | RH       |          | Relative Humidity                        |
  7  | 100  |   *  |      | HGT      | m        | Height                                   |
  11 | 105  |   2  |      | TT       | %        | Temperature                              | At 2 m
  52 | 105  |   2  |      | RH       | %        | Relative Humidity                        | At 2 m
  33 | 105  |  35  |      | UU       | m s-1    | U                                        | At 10 m
  34 | 105  |  35  |      | VV       | m s-1    | V                                        | At 10 m
   1 |  1   |   0  |      | PSFC     | Pa       | Surface Pressure                         |
   2 | 102  |   0  |      | PMSL     | Pa       | Sea-level Pressure                       |
  81 |  1   |   0  |      | LANDSEA  |          | Land/Sea flag                            |
   8 |  1   |   0  |      | SOILHGT  | m        | Terrain field of source analysis         |
  11 |  1   |   0  |      | SKINTEMP | K        | Skin temperature (can use for SST also)  |
  11 | 111  |  50  |      | SOILT050 | K        | Temperature at 50 cm below               |
-----+------+------+------+----------+----------+------------------------------------------+


So, i already retrieve from what i assume to be compatible input data for WRF, which are GFDL RCP4.5 data, and since my reference is the Vtable.GFDL, what i download are:

- 6 hour pressure level data (Plev): ua, va, psl, ta
- Monthly land field data : tsl
- Fix surface data : orog, sftlf, sftof
- Daily data : hur, rhs, tas, uas, vas, zg, tos
- 3 hour data : ps

And so i convert those .nc data to .grb using a simple CDO command, with respective to the level type and grib parameter number for each parameter. And then normally continue the process (ungribbing, metgrid, etc.)

Also, i done some modification in namelist.input, but with no accurate basics. After the wrfbdy and wrfinput are created and wrf ran, it crashed and produce just one wrfout with a very bad result.

What im thinking is, there is seems to be a problem with the vertical level of the data. Actually, i dont know how to set the data with a correct vertical level so it could be ingested into the process, and i didnt make a modification / convertion to the data, just leave it as it is (since i assume the data use the pressure level already).

Could somebody who has experience in this case be willing to help me? Or i want to know if this kind of process is unable to do at all?

Thanks before.
newbiewrfuser
 
Posts: 16
Joined: Tue Feb 12, 2013 9:08 pm

Re: Using the GFDL CMIP5 Datasets

Postby newbiewrfuser » Mon May 19, 2014 10:35 am

Still no response :(

I'm working back on this again, and so far, with the data ive mentioned before, generally these are what ive done:

- The 3 hourly ps variable is used for the PSFC
- The 6 hourly ua,va, and ta are used for the UU, VV and TT each, and already the pressure coordinates with 9 level
- The 6 hourly psl used for PMSL
- The daily zg and hur are used for 3-dimensional HGT and RH, with the same 9 pressure level (10000 - 100)
- The daily uas, vas, tas, and rhs used for surface UU, VV, TT and RHS
- The monthly ts is for SKINTEMP and the tsl is for SOIL0TO50
- For the constants, the orog is for SOILHGT and sftlf for LANDSEA, and as you already know ungribbed in another step for the constants_name
- In additional, i use the daily tso as SST, but also ungribbed in another step with Vtable.SST and produced different intermediate file SST

Here are some facts that i know nothing about, and may be worth noticed:
- The namelist.wps for all of this (except for fix/invariant variable) is one year with 86400 interval seconds
- At the prompt of the ungrib process, the PSFC is placed on the separate line with the pressure level, it's on the 2001 line, as well as PMSL in the 2031 line.
- At first, the ungrib cannot find the time of the daily and monthly data, so i adjust the time to 00 both for the daily and monthly. The daily data originally set at 12:00:00 each day, meanwhile the monthly data originally has 00:00:00 and 12:00:00 in the middle of the month (15 or 16). The ungrib can be done afterwards.
- The 2 dimensional variables nc file originally contain some additional field (average_dT), so i have to remove it and select the necessary variable first before converting.
- At the converting process, the message prompt something like "missing areacella". I dont have any idea what this means. I have a separate areacella nc file but dont know what to do or how it would be related.

I know converting with CDO is way too risky and possible to. Still, i do want to know rather this process correct or not, and if not, in which step it flawed and why.

For now all those datasets ar ungribbed succesfully (the cross are filled at the column for each necessary parameters) as well as the metgrid. But the produced met_em files are a mess. For example the TT file are not distribute correctly in, let's say, the 100000 level. The figure are just like plain image with straight gradation. I not even sure in which process the mistake were made. Is it at the convert process, the defining parameter process, defining level type, the ungrib, the metgrid, the vtable requirement, the units of each variables, etc?

I will gladly deliver more information in case someone actually interested in helping me for this. Thank you.
newbiewrfuser
 
Posts: 16
Joined: Tue Feb 12, 2013 9:08 pm

Re: Using the GFDL CMIP5 Datasets

Postby sdn128 » Thu Nov 20, 2014 5:42 pm

Hello,
I have to thank you for writing this post, I was driving myself crazy thinking I was the only one to have trouble with GFDL-ESM2M and its usage as input for the WRF model. I hope you have not given up before now, but I only recently came upon the need to assimilate it into WRF for downscaled climate simualtions. After spending nearly a month grinding away at this problem I found a workable solution where I have now successfully ingested and ran simulations with GFDL-ESM2M RCP60 data that was first processed by CDO. I should note here that my solution involved the application of Python, although you should be able to use other languages as well.

Okay, so let's start with the data. I downloaded all of my data from the ESGF website (http://pcmdi9.llnl.gov/esgf-web-fe/) and for my research I downloaded both the historical and rcp60 experimental data. I noticed from your original post that you may need some additional data given the lack fo surface variables. I downloaded the following:

3 hourly: uas, vas, tas, huss
6 hourly: ua, va, ta, hus, psl
daily: tos
monthly: tsice, ts, mrlsl, tsl
fixed: sftlf

Once these data were downloaded, I generated a combined surface temperature file (i.e., ts + tsice + tos). First I interpolated tsice to the same grid as ts and combined these data. I then carried out a linear interpolation of the combined ts + tsice data to make these data daily and finally combined it with the tos data and saved the combined data output as netcdf files.

Next, I read in each netcdf file, resorted the variable order, and removed the excess data. I found the GFDL output is not entirely CFL compliant. To fix it you need resort the variables and dimensions to be in the same order. Specifically, time, lev, lev_bnds, lat, lat_bnds, lon, lon_bnds in the variables and time, lev, lat, lon in the dimensions. The order of the remaining variables I found did not matter. You will also need to removed the "bnds" attribute from the a and b variables (else CDO will not read the vertical coordinate correctly). If you need a reference, I would suggest downloading a file or two from MIROC5 and trying to match its style. I found this format served me well with MIROC5, but also with GFDL-ESM2M.

At this stage you are ready for CDO. Going through the whole procedure here would take a while, so I will refer you to the following webpage which discusses the technique applied for CORDEX to read in MIROC5 data. Unfortunately, the forum will not let me upload attachments, so I have copied by bash script for generating CDO-based grb files for GFDL below based upon the CORDEX technique. Once you have generated the 5 grb files (ATM, PSFC, NSFC, SOIL, and TSK) you will need to make one more change. There is a data conflict between the GFDL data and CDO which leads to a situtation where the data is written correctly, but two grb variable headers are not. This error causes the data to view fine in data viewers (Python, Matlab, etc) because these programs read the data and the lat/lon information, but the WRF WPS only reads the data and what is in the grb file headers. I do not know the cause, but CDO incorrectly calculates the header attributes iDirectionIncrement and jDirectionIncrement. To fix it, you will need to modify each grb file and change the values of these attributes to 1000X's the values of iDirectionIncrementInDegrees, jDirectionIncrementInDegrees, respectively. To demonstrate how to do this, I have copied my python script below.

Finally, you will need to make changes to your Vtable file and run ungrib appropriately. I have copied my Vtable at the bottom of the post. If you use only the variables I mentioned above not other modifications should be necessary, just copy it to your ungrib/Variable_Tables directory. Once in place you will need to run ungrib once for each timescale in your data. It is for this reason I have 5 grb files because I had to run ungrib 5 times to have WPS process the data correctly. Once you have generated all the intermediate files, run geogrid and metgrid and you should get GFDL data that both looks correct and can be used to run WRF.

Whew!! I hope this helps and that I have not been too long winded. The proceedure is long and tedious, but I promise you it does work.

Regards,
--sdn128


##############################
#CDO bash script:

#!/bin/bash
#
#set -x

# This program carries out the pre-processing of CMIP5 data available at ESGF. It is currently working for GFDL-ESM2M data.
# Written by Amin Dezfuli, Fall 2013, Johns Hopkins University, using the procedure described at
# http://www.meteo.unican.es/wiki/cordexw ... eprocessor

# ################################################## CHANGES HERE #####################################################################
#exper='historical' # set the experiment name: 'historical', 'rcp26', 'rcp45', 'rcp60', 'rcp85'
#yr=2002 # years, for now let's do just one year. I will later add a loop for multiple years.
#mon=(1 2 3 4 5 6 7 8 9 10 11 12) # month index. choose from (1 2 3 4 5 6 7 8 9 10 11 12), e.g., mon=(1 2 3) for JAN-FEB-MAR

model='GFDL-ESM2M' #Name of model (all caps) -- used for input and output filenames
exper='rcp60' # set the experiment name: 'historical', 'rcp26', 'rcp45', 'rcp60', 'rcp85'
sdate='20871001' #First date in file (yyyymmdd)
edate='20881001' #Late date in file (yyyymmdd)

sfc='yes' # set 'yes' to create near surface and skin temperature files. Because these files are available at once-a-year interval, they only need to be generated once for each year. If the number of months is greater than one, the program will automatically set this to 'no' after it finishes the calculations for the first month.
soil='yes' # set 'yes' to create a soil temperature & moisture file.
soilmoist='yes' # Do you want to include soil moisture? Set 'yes' or 'no'.

tlevdir='/lou/s2k/snicholl/CMIP5' # Top level directory where this script is located
dirin=$tlevdir'/data' # path to CMIP5 NetCDF data
dirout=$tlevdir'/gribs' # where to store the grib outputs.
dirutil=$tlevdir'/util' # directory including all *.dat files (e.g., myzaxis10m.dat)

#################################################### END OF CHANGES ###################################################################
cd util
ln -sf $model"_zaxisinvert.dat" myzaxisinvert.dat
cd ..
mkdir -p $tlevdir'/tmp/'${sdate}
cd $tlevdir'/tmp/'${sdate}
rm -f $tlevdir'/tmp/'${sdate}/*
########################## some calculations! #########################################################################################
# ######################## 6 hourly pressure level variables (ua, va, ta, hus) ATM.grb files ########################################################

var_plev=('ua' 'va' 'ta' 'hus')
grb_par=(33 34 11 52) # GRIB1 parameters used in Vtable.CMIP5 for ua, va, ta, hus, respectively.

for ivar in {0..3}; do
##>
echo ' '
sfc_fn=${dirin}/${var_plev[ivar]}_6hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
echo 'Now interpolating '$sfc_fn' to '${var_plev[ivar]}'_Plev.grb'
cdo invertlev $sfc_fn "temp1.nc"
ncdump -v p0 $sfc_fn | grep "p0 ="
cdo outputf,%10.3f,2 -mulc,100000.0 -selname,a_bnds temp1.nc
cdo outputf,%10.3f,2 -selname,b_bnds temp1.nc
cdo setzaxis,${dirutil}/${model}_zaxisinvert.dat temp1.nc temp2.nc
cdo delname,a_bnds,b_bnds, temp2.nc temp3.nc
export EXTRAPOLATE=1
cdo ml2pl,100000,97500,95000,92500,90000,87500,85000,82500,80000,77500,75000,70000,65000,60000,55000,50000,45000,40000,35000,30000,25000,22500,20000,17500,15000,12500,10000,7000,5000,3000 temp3.nc ${var_plev[ivar]}_Plev.nc
cdo delname,ps ${var_plev[ivar]}_Plev.nc temp1.nc
cdo -f grb setltype,100 -chparam,-1,${grb_par[ivar]} temp1.nc ${var_plev[ivar]}_Plev.grb
#<
done

# ######################## Surface Pressure ###########################################################################################
cdo -f grb setltype,1 -chparam,-1,1 -selname,ps $sfc_fn ps.grb # We don't really need this as it is contained in 3D fields.

if [ $sfc = 'yes' ]; then # To create NSCF.grb, TSK.grb files, only when needed. =====================================================
# ######################## Mean Sea Level Pressure ####################################################################################
echo ' '
echo 'Now interpolating '${dirin}/psl_6hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc to PSL-"$model"_"$exper"-"$sdate"-"$edate".grb"
cdo -f grb setltype,102 -chparam,-1,2 ${dirin}/psl_6hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" PSL-$model"_"$exper"-"$sdate"-"$edate".grb"
mv PSL-$model"_"$exper"-"$sdate"-"$edate".grb" ${dirout}/

# ######################## 3 hourly near-surface variables (uas, vas, tas, thuss) #####################################################
echo ' '
echo 'Now interpolating '${dirin}/uas_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
cdo -f grb setzaxis,${dirutil}/myzaxis10m.dat -setltype,105 -chparam,-1,33 ${dirin}/uas_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" uass.grb # for u-wind
echo ' '
echo 'Now interpolating '${dirin}/vas_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
cdo -f grb setzaxis,${dirutil}/myzaxis10m.dat -setltype,105 -chparam,-1,34 ${dirin}/vas_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" vass.grb # for v-wind
echo ' '
echo 'Now interpolating '${dirin}/tas_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
cdo -f grb setzaxis,${dirutil}/myzaxis2m.dat -setltype,105 -chparam,-1,11 ${dirin}/tas_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" tass.grb # for temperature
echo ' '
echo 'Now interpolating '${dirin}/huss_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
cdo -f grb setzaxis,${dirutil}/myzaxis2m.dat -setltype,105 -chparam,-1,52 ${dirin}/huss_3hr_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" huss.grb # for specific humidity
echo ' '
echo 'Now merging tas.grb, vass.grb, tass.grb, and huss,grib to NSFC-'$model'_'$exper'-'$sdate'-'$edate'.grb'
cdo merge *ss.grb NSFC-$model"_"$exper"-"$sdate"-"$edate".grb" # near-surface fields + mean sea level pressure
mv NSFC-$model"_"$exper"-"$sdate"-"$edate".grb" ${dirout}/

# ######################## skin temperature (combine land and sea surface temperatures ################################################

echo ' '
echo 'Now obtaining SST and land/ice temperatures from files'
#echo ${dirin}/tslsi_day_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
echo ${dirin}/tso_day_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
# cdo min ${dirin}/tslsi_day_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" ${dirin}/tso_day_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" my_skin.nc
# cdo -f grb -setltype,1 -chparam,-1,11 my_skin.nc TSK-$model"_"$exper"-"$sdate"-"$edate".grb"
# mv TSK-$model"_"$exper"-"$sdate"-"$edate".grb" ${dirout}/


cdo -f grb setltype,1 -chparam,-1,11 ${dirin}/tso_day_$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" TSK-$model"_"$exper"-"$sdate"-"$edate".grb"
mv TSK-$model"_"$exper"-"$sdate"-"$edate".grb" ${dirout}/

fi #===================================================================================================================================

if [ $soil = 'yes' ]; then # To create SOIL.grb file, only when needed. **************************************************************
# ######################## split soil temperature into levels described in the Vtable #################################################
echo ' '
echo 'Opening '${dirin}/"tsl_mon_"$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
echo 'Spliting soil temperature into its levels into files starting with soilt'
cdo splitlevel ${dirin}/"tsl_mon_"$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" soilt
# Mid-level thickness are combined to get soil temperature grib files
cdo -f grb setzaxis,${dirutil}/myzaxis0_5.dat -setltype,112 -chparam,-1,11 soilt000.01.nc soilt01.grb
cdo -f grb setzaxis,${dirutil}/myzaxis5_25.dat -setltype,112 -chparam,-1,11 soilt00.175.nc soilt02.grb
cdo -f grb setzaxis,${dirutil}/myzaxis25_100.dat -setltype,112 -chparam,-1,11 soilt0000.7.nc soilt03.grb
cdo -f grb setzaxis,${dirutil}/myzaxis100_200.dat -setltype,112 -chparam,-1,11 soilt0001.6.nc soilt04.grb


## ######################## split soil moisture into levels described in the Vtable ####################################################
if [ $soilmoist = 'yes' ]; then
echo ' '
echo 'Now opening '${dirin}/"mrlsl_mon_"$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc"
echo 'Now interpolating soil moisture information to set levels of file name soilmois'
cdo splitlevel ${dirin}/"mrlsl_mon_"$model"_"$exper"_r1i1p1_"$sdate"-"$edate".nc" soilmois
cdo -f grb setzaxis,${dirutil}/myzaxis0_5.dat -setltype,112 -chparam,-1,144 soilmois000.01.nc soilmois01.grb
cdo -f grb setzaxis,${dirutil}/myzaxis5_25.dat -setltype,112 -chparam,-1,144 soilmois00.175.nc soilmois02.grb
cdo -f grb setzaxis,${dirutil}/myzaxis25_100.dat -setltype,112 -chparam,-1,144 soilmois0000.7.nc soilmois03.grb
cdo -f grb setzaxis,${dirutil}/myzaxis100_200.dat -setltype,112 -chparam,-1,144 soilmois0001.6.nc soilmois04.grb

# convert GFDL-ESM2M water content (kg m-2) to WRF soil moisture fraction (m3 m-3)
cdo mulc,0.02 soilmois01.grb soilmois01_frac.grb
cdo mulc,0.005 soilmois02.grb soilmois02_frac.grb
cdo mulc,0.0013 soilmois03.grb soilmois03_frac.grb
cdo mulc,0.001 soilmois04.grb soilmois04_frac.grb
fi

if [ $soilmoist = 'yes' ]; then cdo merge soilmois*_frac* soilt0*.grb SOIL-$model"_"$exper"-"$sdate"-"$edate".grb"; else cdo merge soilt0*.grb SOIL-$model"_"$exper"-"$sdate"-"$edate".grb"; fi # Create soil temperature and moisture file.
mv SOIL-$model"_"$exper"-"$sdate"-"$edate".grb" ${dirout}/
fi #**********************************************************************************************************************************

########################### generate grib files for 3D atmospheric variables: one file for each month ##################################
cdo merge *_Plev.grb ps.grb ATM-$model"_"$exper"-"$sdate"-"$edate".grb" #atmospheric variables at different pressure levels above surface + surface #pressure
mv ATM-$model"_"$exper"-"$sdate"-"$edate".grb" ${dirout}/
rm -f $tlevdir'/tmp/'${sdate}/*

cd $tlevdir
echo "Checking for errors in grib file headers"
<Checks for header consistency in dx and dy spacings and replaces it if incorrect>
sed -e "s/sdate/$sdate/g" \
-e "s/edate/$edate/g" \
-e "s/model_run/$model/g" \
-e "s/rtype/$exper/g"<template.grib_modify.py>grib_modify.py

/nobackupp8/snicholl/anaconda/bin/python grib_modify.py

echo ' '
echo '*****************'
echo 'SUCCESS prep_grib has finished running'
echo '*****************'



##############################
#Python grb file modification script

from matplotlib import rcParams #Loads a script for changing font sizes of color bar and plot axes
from matplotlib.colors import LinearSegmentedColormap #Loads a module allowing for colors in colormaps to be obtained
from mpl_toolkits.basemap import Basemap, addcyclic #Loads mapping data
import math #Imports math functions
import matplotlib.pyplot as mpl #Loads map plotting library
import matplotlib.ticker as mpltick #Load a script that allows for adjustments to colorbar ticks
import numpy as np #Loads functions for array, linear algebra, array operations
import os #Enables Operating System functions
import pylab as plt #Imports all functions from pylab plotting package
import pygrib
import scipy.interpolate as sp_int #Allows grid interpolation functions
import scipy.ndimage as ndimage #Calls ndimage which contains a gaussian smoothing program
import scipy.stats as spstat #Import the scipy statistics package
import scipy as sp #Enhances NumPY with science operations (interpolate, signal, etc)
import time #Loads time function

#SLP
flevs=np.arange(972,1040,4)
fst_cmap_clr=50 #Color out of 255 where you want colorbar to start
make_new_grib = 'Y'
plot_data = 'N'
model = 'GFDL-ESM2M'
run_type = 'rcp60'
var_list = ['PSL','TSK','SOIL','NSFC','ATM']
date_str = '20871001-20881001'

for var in var_list:
#print var
file_in = 'gribs/'+var+'-'+model+'_'+run_type+'-'+date_str+'.grb'
file_out = 'gribs/'+var+'-'+model+'_'+run_type+'-'+date_str+'_test.grb'

print file_in
print file_out

grbs = pygrib.open(file_in)
grbs_out = open(file_out,'wb')

grb_count = 1

for grb in grbs:
#print grb,'****count = ',str(grb_count)

for f in grb.keys():
if f == 'parametersVersion':
pass
else:
#print f
#exec('print grb.'+f)
if f == 'iDirectionIncrement':
i_incr = grb.iDirectionIncrement
if f == 'jDirectionIncrement':
j_incr = grb.jDirectionIncrement
if f == 'iDirectionIncrementInDegrees':
i_incr_deg = grb.iDirectionIncrementInDegrees
if f == 'jDirectionIncrementInDegrees':
j_incr_deg = grb.jDirectionIncrementInDegrees

#print i_incr,j_incr,i_incr_deg,j_incr_deg
if i_incr != 1000.00 * i_incr_deg:
#print 'blah1'
grb.iDirectionIncrement = 1000.00 * i_incr_deg
if j_incr != 1000.00 * j_incr_deg:
#print 'blah2'
grb.jDirectionIncrement = 1000.00 * j_incr_deg

#print grb.iDirectionIncrement,grb.jDirectionIncrement

msg = grb.tostring()

grbs_out.write(msg)

grb_count=grb_count+1

grbs.close()
grbs_out.close()
del grbs,grbs_out,msg

exec('os.system('+"'"+'mv gribs/'+var+'-'+model+'_'+run_type+'-'+date_str+'_test.grb'+' gribs/'+var+'-'+model+'_'+run_type+'-'+date_str+'.grb'+"'"+')')


##############################
Vtable.CMIP5

GRIB1| Level| From | To | metgrid | metgrid | metgrid |
Param| Type |Level1|Level2| Name | Units | Description |
-----+------+------+------+----------+----------+------------------------------------------+
11 | 100 | * | | TT | K | Temperature |
33 | 100 | * | | UU | m s-1 | U |
34 | 100 | * | | VV | m s-1 | V |
52 | 100 | * | | SPECHUMD | kg kg-1 | |
| 100 | * | | RH | % | Relative Humidity |
11 | 105 | 2 | | TT | K | Temperature | At 2 m
52 | 105 | 2 | | SPECHUMD | kg kg-1 | | At 2 m
| 105 | 2 | | RH | % | Relative Humidity at 2 m | At 2 m
33 | 105 | 10 | | UU | m s-1 | U | At 10 m
34 | 105 | 10 | | VV | m s-1 | V | At 10 m
81 | 1 | 0 | | LANDSEA | | Land/Sea flag |
1 | 1 | 0 | | PSFC | Pa | Surface Pressure |
2 | 102 | 0 | | PMSL | Pa | Sea-level Pressure |
11 | 1 | 0 | | SKINTEMP | K | Skin Temperature (and SST) |
144 | 112 | 0 | 5 | SM000010 | fraction | Soil Moist 0-5 cm below grn layer (Up) |
144 | 112 | 5 | 25 | SM010040 | fraction | Soil Moist 5-25 cm below grn layer |
144 | 112 | 25 | 100 | SM040100 | fraction | Soil Moist 25-100 cm below grn layer |
144 | 112 | 100 | 200 | SM100200 | fraction | Soil Moist 100-200 cm below gr layer |
11 | 112 | 0 | 5 | ST000010 | K | T 0-5 cm below ground layer (Upper) |
11 | 112 | 5 | 25 | ST010040 | K | T 5-25 cm below ground layer (Upper) |
11 | 112 | 25 | 100 | ST040100 | K | T 25-100 cm below ground layer (Upper) |
11 | 112 | 100 | 200 | ST100200 | K | T 100-200 cm below ground layer (Bottom) |
-----+------+------+------+----------+----------+------------------------------------------+
sdn128
 
Posts: 1
Joined: Fri Oct 24, 2014 8:10 pm

Re: Using the GFDL CMIP5 Datasets

Postby parichat » Fri Jan 30, 2015 3:23 am

Hi, sdn128

I followed your step but I still can't run wrf by using GFDJ data.
I found that ungrib.exe calculated excessive RH, min=1% and max=448.2% at surface level.

Do you have any idea?
Any help could be appriciated.


Best regards,
Parichat
parichat
 
Posts: 2
Joined: Thu Mar 27, 2014 10:18 am

Re: Using the GFDL CMIP5 Datasets

Postby easy » Sun Jun 07, 2015 10:31 pm

Thank you very much for a so good post. My question is that what's the resolution u used about the variables, C360? How about the performance? I am just wondering if you could give your email address to me to keep contact which would be very appreciate. Thank you very much~~~
easy
 
Posts: 10
Joined: Sun Jun 07, 2015 10:26 pm

Re: Using the GFDL CMIP5 Datasets

Postby jaganagmet » Thu Dec 31, 2015 7:39 am

This has reference to the post of sdn128 for the conversion of GFDL CMIP5 data for ingesting to WRF. He used this website http://www.meteo.unican.es/wiki/cordexw ... eprocessor which was done as an example for MIROC5. There are some issues I want to know. First the p0 for MIROC5 is 100000 while it is p0 = 101325 for GFDL. I don not know whether any body has successfully done the conversion.

They can post to help persons like me
jaganagmet
 
Posts: 1
Joined: Thu Dec 31, 2015 7:20 am

Re: Using the GFDL CMIP5 Datasets

Postby Orome » Wed Jan 20, 2016 7:30 am

I am following the step from the CORDEX WRF website:

http://www.meteo.unican.es/wiki/cordexw ... eprocessor

But when I get to the following section:

Code: Select all
cdo ml2pl,100000,97500,95000,92500,90000,87500,85000,82500,80000,77500,75000,70000,65000,60000,55000,50000,45000,40000,35000,30000,25000,22500,20000,17500,15000,12500,10000,7000,5000,3000 temp3.nc ta_Plev.nc


I get the following error message:

Code: Select all
cdo ml2pl: Extrapolation of missing values enabled!

cdo ml2pl (Abort): Surface pressure not found!


Has anyone else encountered this or know of a fix? What's wrong?

Thanks. :D
Orome
 
Posts: 35
Joined: Mon Dec 05, 2011 6:09 am

Re: Using the GFDL CMIP5 Datasets

Postby basantasamal » Thu Feb 02, 2017 10:16 am

Hi,
Don't delete ps before ml2pl, do after ml2pl. I mean
cdo delname,a_bnds,b_bnds temp2.nc temp3.nc
then try the step of ml2pl,it will work. then remove ps
cdo ml2pl,100000,97500,95000,92500,90000,87500,85000,82500,80000,77500,75000,70000,65000,60000,55000,50000,45000,40000,35000,30000,25000,22500,20000,17500,15000,12500,10000,7000,5000,3000 temp3.nc temp4.nc
cdo delname,ps temp4.nc ta_Plev.nc
Hope it helps.
Thanks
Basanta
basantasamal
 
Posts: 1
Joined: Mon Jan 30, 2017 10:10 am

pre-process GCM data to run WRF in climatic mode for climate

Postby migaltar » Tue Feb 21, 2017 8:29 pm

Hi everyone,

I'm trying to run WRF for Mexico with GCM data: HadGem2, IPSL-CM5A-LR, MPI-ESM-MR or GFDL-ESM2M.

1) Has someone had successful in pre-process GCM data and to fix options in namelist.input to run WRF for several months and years?? to share experiences, ideas and files about configuration in namelist.input

2) I had success to pre-process MIROC5 data, after a lot of work, and i have ran WPS, real.exe and, partially, wrf.exe in Linux Centos 7, X84, netcdf-4.4.1.1, netcdf-fortran-4.4.4, mpich2-1.5.

3) As a token of my seriousness, I share my code that works in a csh script that i have modified taking as reference http://www.meteo.unican.es/wiki/cordexw ... ools/ClWrf

##################
#!/bin/csh -f

# Modelo a procesar
#
set Modelo = "MIROC5"
rm *grb

##################### ATMOSFERICAS #################################
#
# Lista los archivos (ta, hus, va, ua) a los que se les cambiara de coordenadas hibridas (del CMIP5) a niveles de presion (para que sean leidos en WRF)

ls ta_6*$Modelo* hus_6*$Modelo* ua_6*$Modelo* va_6*$Modelo* > zList

# Ciclo 1 para leer con Foreach el listado de archivos en zList
foreach file (`cat zList`)
set filIn=`echo $file | awk -F" " '{print $1}'`
rm temp1.nc temp2.nc
# Instruccion para extraer caracteres de la variable. substr(s,c,n). This returns the substring from string s starting from character position c up to a maximum length of n characters
set fraccion = `echo $filIn | awk '{print substr($0,0,3)}'`
set fraccion2 = `echo $filIn | awk '{print substr($0,0,2)}'`
echo $fraccion $fraccion2

# Compara el archivo en procesamiento para indicar nombres de salida
if($fraccion == hus) then
set filO1="hus_Plev.nc"
set filO2="hus_Plev.grb"
set Param = 52
else if($fraccion == ua_) then
set filO1="ua_Plev.nc"
set filO2="ua_Plev.grb"
set Param = 33
else if($fraccion == va_) then
set filO1="va_Plev.nc"
set filO2="va_Plev.grb"
set Param = 34
else if($fraccion == ta_) then
set filO1="ta_Plev.nc"
set filO2="ta_Plev.grb"
set Param = 11
set filIn2 = $filIn
endif
echo "$filO1","$filO2","$Param"
#
# 2.1. Invert the height levels, from bottom-top to top-bottom. E.g.:
echo '# 2.1. Invert the height levels, from bottom-top to top-bottom. E.g.:'
cdo invertlev $filIn temp1.nc

# 2.2. Obtain the hybrid levels coefficients.
echo '### 2.2. Obtain the hybrid levels coefficients.'
ncdump -v p0 $filIn | grep "p0 ="
cdo outputf,%10.3f,2 -mulc,100000.0 -selname,a_bnds temp1.nc # Despliega contenido coeficientes a_bnds
cdo outputf,%10.3f,2 -selname,b_bnds temp1.nc # Despliega contenido coeficientes b_bnds

# 2.3 Set the z-axis description in the netdcf file
echo '# 2.3 Set the z-axis description in the netdcf file'
##/home/Migue/Programas/bin/ncrename -v a,ap temp1.nc
set PID1=$!
wait $PID1
cdo setzaxis,myzaxisinvert.dat temp1.nc temp2.nc # Asigna las caracteristicas del eje vertical y guarda en temp2.nc

# 2.4 Remove unused variables
echo '# 2.4 Remove unused variables'
##cdo delname,ap_bnds,b_bnds temp2.nc temp3.nc
cdo delname,a_bnds,a,b_bnds_2 temp2.nc temp3.nc
##ncks -x -v a_bnds,a,b_bnds_2 temp2.nc temp3.nc

# 2.5 Interpolate to pressure levels
echo '## 2.5 cdo ml2pl ',"$filO1"
setenv EXTRAPOLATE 1
cdo ml2pl,100000,97500,95000,92500,90000,87500,85000,82500,80000,77500,75000,70000,65000,60000,55000,50000,45000,40000,35000,30000,25000,22500,20000,17500,15000,12500,10000,7000,5000,3000 temp3.nc $filO1

# 3. Creating grib files for pressure-levels variables
echo '## 3. delname ',"$filO2"
cdo delname,ps $filO1 temp1.nc
#cdo -f grb setltype,100 -chparam,-1,$Param temp1.nc $filO2 # Instruccion original de la pagina para procesar MIROC

cdo -f grb -setmissval,0 -chparam,-1,$Param temp1.nc $filO2 ## your input NetCDF file doesn't provide all the required meta-data for the GRIB (such as level type, table number etc, although they are not required for CDO to operate). You could try -setmissval,0 ie. https://code.zmaw.de/boards/1/topics/2907 En caso de que sea un tamaño muy grande del archivo o de la malla, recomiendan utilizar *** cdo -b 16 -f grb setmissval,0 file.nc file.grb ***

end # Fin del ciclo 1 Foreach ****

##################### PRESION EN SUPERFICIE #################################
#
echo '### 4. Surface and mean sea level pressures'
##cdo -f grb -setmissval,0 setltype,1 -chparam,-1,1 -selname,ps $filIn2 ps.grb # Instruccion original de la pagina para procesar

cdo delname,$fraccion2 $filO1 temp1.nc # Estas dos lineas de instruccion las introduje para asegurar que se haga la referencia a la presion, tal como lo indican los valores en Vtable, ya que la instruccion original no lo asignaba
cdo -f grb setltype,1 -chparam,-1,1 temp1.nc ps.grb

set PID1=$!
wait $PID1

### Debido a que tiene datos para 365 dias (1460 tiempos, 4 tiempos de 6 hrs por dia) el archivo "psl_6hrPlev_MIROC5_historical_r4i1p1_2008010100-2008123118.nc" es que se extraen los tiempos para el mes a procesar, en este caso 124 (31 dias) para enero, para que coincida con los pasos del archivo de presion en superficie ""
#cdo seltimestep,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124 psl_6hrPlev_MIROC5_historical_r4i1p1_2008010100-2008123118.nc PsNivelMarEne.nc
#cdo -f grb setltype,102 -chparam,-1,2 PsNivelMarEne.nc surface.grb # psl.grb

##cdo -f grb setltype,102 -chparam,-1,2 psl_6hrPlev_MIROC5_historical_r4i1p1_2008010100-2008123118.nc psl.grb
cdo -f grb setltype,102 -chparam,-1,2 psl_6hrPlev_MIROC5_historical_r4i1p1_2008010100-2008123118.nc surface.grb

# Concatena los archivos &&&&&&&&&&&&&&
## cdo merge ps.grb psl.grb surface.grb

# Concatena los archivos &&&&&&&&&&&&&&
echo '### 3. MERGE atmosfericos + Presion superficie'
##cdo merge *_Plev.grb atmos.grb
cdo merge *_Plev.grb ps.grb atmos.grb

##################### VARIABLES CERCA SUPERFICIE #################################
#
# Lista los archivos (tas, huss, uas, vas) para convertir a formato GRIB
ls tas_*"$Modelo"* huss_*"$Modelo"* uas_*"$Modelo"* vas_*"$Modelo"* > zList

# Ciclo 1 para leer con Foreach el listado de archivos en zList
#
foreach file (`cat zList`)
set filIn=`echo $file | awk -F" " '{print $1}'`
echo '### 5. Near surface variables. Archivo ',$file
# Instruccion para extraer caracteres de la variable. substr(s,c,n). This returns the substring from string s starting from character position c up to a maximum length of n characters
set fraccion = `echo $filIn | awk '{print substr($0,0,4)}'` # = "NGC 2345"
echo $fraccion

# Compara el archivo en procesamiento para indicar nombres de salida
if($fraccion == tas_) then
set filO1="ttas.grb"
set Param = 11
set myzaxisX = "myzaxis2m.dat"
else if($fraccion == huss) then
set filO1="thuss.grb"
set Param = 52
set myzaxisX = "myzaxis2m.dat"
else if($fraccion == uas_) then
set filO1="tuas.grb"
set Param = 33
set myzaxisX = "myzaxis10m.dat"
else if($fraccion == vas_) then
set filO1="tvas.grb"
set Param = 34
set myzaxisX = "myzaxis10m.dat"
endif
#
#cdo seltimestep,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127,129,131,133,135,137,139,141,143,145,147,149,151,153,155,157,159,161,163,165,167,169,171,173,175,177,179,181,183,185,187,189,191,193,195,197,199,201,203,205,207,209,211,213,215,217,219,221,223,225,227,229,231,233,235,237,239,241,243,245,247 $filIn Todo_NearSurface.nc
#cdo -f grb setzaxis,$myzaxisX -setltype,105 -chparam,-1,$Param Todo_NearSurface.nc $filO1

cdo -f grb setzaxis,$myzaxisX -setltype,105 -chparam,-1,$Param $filIn $filO1
#
end

# Concatena los archivos &&&&&&&&&&&&&&
echo '### 5. MERGE cerca superficie'
cdo merge ttas.grb thuss.grb tuas.grb tvas.grb near_surface.grb

##################### SKIN #################################
#
echo '### 6. Proceso Skin Temperatura'
# Las siguientes 3 lineas son para procesar solo datos especificos de la serie
#cdo seltimestep,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127,129,131,133,135,137,139,141,143,145,147,149,151,153,155,157,159,161,163,165,167,169,171,173,175,177,179,181,183,185,187,189,191,193,195,197,199,201,203,205,207,209,211,213,215,217,219,221,223,225,227,229,231,233,235,237,239,241,243,245,247 tslsi_3hr_MIROC5_historical_r1i1p1_200801010000-200812312100.nc TmpMar.nc
#cdo seltimestep,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127,129,131,133,135,137,139,141,143,145,147,149,151,153,155,157,159,161,163,165,167,169,171,173,175,177,179,181,183,185,187,189,191,193,195,197,199,201,203,205,207,209,211,213,215,217,219,221,223,225,227,229,231,233,235,237,239,241,243,245,247 tso_3hr_MIROC5_historical_r1i1p1_200801010000-200812312100.nc TmpContinente.nc
#cdo min TmpMar.nc TmpContinente.nc my_skin.nc

cdo min tslsi_3hr_MIROC5_historical_r1i1p1_200801010000-200812312100.nc tso_3hr_MIROC5_historical_r1i1p1_200801010000-200812312100.nc my_skin.nc
cdo -f grb -setltype,1 -chparam,-1,11 my_skin.nc my_skin.grb

# De GDF en Internet
# cdo -f grb setltype,1 -chparam,-1,11 tso_3hr_MIROC5_historical_r1i1p1_200801010000-200812312100.nc my_skin.grb

################# SUELO ########################
#
echo '### 7. Soil temperature and moisture'
# ****!!!!!!!!Make sure that units of these variables match with WRF expected inputs (defined in the Registry). For example, to convert MIROC5 water content (kg m-2) to WRF soil moisture fraction (m3 m-3): !!!!!!!!!!************
# La siguientes lineas son para procesar solo datos especificos de la serie. Son datos mensuales desde 1850.
#cdo seltimestep,1873 tsl_Lmon_MIROC5_historical_r1i1p1_185001-201212.nc TmpSuelo.nc
#cdo splitlevel TmpSuelo.nc soilt

cdo splitlevel tsl_Lmon_MIROC5_historical_r1i1p1_185001-201212.nc soilt
cdo -f grb setzaxis,myzaxis0_5.dat -setltype,112 -chparam,-1,11 soilt00.025.nc soilt01.grb
cdo -f grb setzaxis,myzaxis5_25.dat -setltype,112 -chparam,-1,11 soilt00.025.nc soilt02.grb
cdo -f grb setzaxis,myzaxis25_100.dat -setltype,112 -chparam,-1,11 soilt00.625.nc soilt03.grb
cdo -f grb setzaxis,myzaxis100_200.dat -setltype,112 -chparam,-1,11 soilt0001.5.nc soilt04.grb

# La siguientes lineas son para procesar solo datos especificos de la serie
#cdo seltimestep,1873 mrlsl_Lmon_MIROC5_historical_r1i1p1_185001-201212.nc HumedadSuelo.tc
#cdo splitlevel HumedadSuelo.tc soilmois

cdo splitlevel mrlsl_Lmon_MIROC5_historical_r1i1p1_185001-201212.nc soilmois
cdo -f grb setzaxis,myzaxis0_5.dat -setltype,112 -chparam,-1,144 soilmois00.025.nc soilmois01.grb
cdo -f grb setzaxis,myzaxis5_25.dat -setltype,112 -chparam,-1,144 soilmois00.025.nc soilmois02.grb
cdo -f grb setzaxis,myzaxis25_100.dat -setltype,112 -chparam,-1,144 soilmois00.625.nc soilmois03.grb
cdo -f grb setzaxis,myzaxis100_200.dat -setltype,112 -chparam,-1,144 soilmois0001.5.nc soilmois04.grb

# Cambia de unidades en caso de ser necesario, water content (kg m-2) to WRF soil moisture fraction (m3 m-3): %%%%%% OJO CAMBIAR DE ACUERDO AL MODELO %%%%%%

cdo mulc,0.02 soilmois01.grb soilmois01_frac.grb
cdo mulc,0.005 soilmois02.grb soilmois02_frac.grb
cdo mulc,0.0013 soilmois03.grb soilmois03_frac.grb
cdo mulc,0.001 soilmois04.grb soilmois04_frac.grb

echo '### 7. MERGE skin'
cdo merge soilmois*_frac* soilt0*grb soillev.grb

# 8. Final remarks %%%%%% OJO CAMBIAR DE ACUERDO AL MODELO %%%%%%

# When CDO converts netcdf files to grib files, it does not copy the projection information related to the increment in latitude/longitude of the corresponding grid if it cannot be stored correctly (it has been modified in the last version of CDO). For example, the iDirectionIncrement for T85 is 1.40625, but due to limitations in resolution, only the 1406 millidegree is stored. This could give slightly wrong grid coordinates. To solve this problem with older versions of CDO, we can modify ungrib or we can use the grib tools. For example, for a gaussian grid with a longitude grid spacing of 1.406 deg., the following instruction could be used (keep in mind that the small error in longitude is still present):

echo '### 8. MERGE todos'
cdo merge atmos.grb surface.grb near_surface.grb my_skin.grb soillev.grb miroc_input.grb

grib_set -s iDirectionIncrement=1406 miroc_input.grb miroc_input_fixed.grb

###########

My e-mail is mgaac@yahoo.com

Best regards
migaltar
 
Posts: 1
Joined: Tue Feb 21, 2017 8:00 pm


Return to Working with Various Datasets

Who is online

Users browsing this forum: No registered users and 7 guests