.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/qpf_manila.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_qpf_manila.py: QPF (Manila) ======================================================== This example demonstrates how to perform operational deterministic QPF up to three hours using raingauge data from Manila and radar data from Subic. .. GENERATED FROM PYTHON SOURCE LINES 10-13 Definitions -------------------------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 13-38 .. code-block:: default import os import numpy as np import pandas as pd import xarray as xr import cartopy.feature as cfeature import matplotlib.pyplot as plt from matplotlib.colors import BoundaryNorm, ListedColormap from pyresample import utils from swirlspy.rad.uf_ph import read_uf_ph from swirlspy.qpe.utils import locate_file, timestamps_ending from swirlspy.qpf import rover from swirlspy.qpf import sla from swirlspy.utils import standardize_attr, FrameType from swirlspy.utils.conversion import to_rainfall_depth, acc_rainfall_depth from swirlspy.core.resample import grid_resample plt.switch_backend('agg') THIS_DIR = os.getcwd() os.chdir(THIS_DIR) start_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 39-47 Initialising --------------------------------------------------- This section demonstrates extracting radar reflectivity data. Step 1: Define a basetime. .. GENERATED FROM PYTHON SOURCE LINES 47-51 .. code-block:: default # Supply basetime basetime = pd.Timestamp('20180811112000').floor('min') .. GENERATED FROM PYTHON SOURCE LINES 52-54 Step 2: Using basetime, generate timestamps of desired radar files timestamps_ending() and locate files using locate_file(). .. GENERATED FROM PYTHON SOURCE LINES 54-67 .. code-block:: default # Obtain radar files dir = THIS_DIR + '/../tests/samples/uf_ph/sub/' located_files = [] radar_ts = timestamps_ending( basetime, duration=pd.Timedelta(60, 'm'), interval=pd.Timedelta(10, 'm') ) for timestamp in radar_ts: located_files.append(locate_file(dir, timestamp)) .. GENERATED FROM PYTHON SOURCE LINES 68-69 Step 3: Define the target grid as a pyresample AreaDefinition. .. GENERATED FROM PYTHON SOURCE LINES 69-87 .. code-block:: default area_id = "epsg3123_240km" description = ("A 240 m resolution rectangular grid " "centred at Subic RADAR and extending to 240 km " "in each direction") proj_id = 'epsg3123' projection = ('+proj=tmerc +lat_0=0 ' '+lon_0=121 +k=0.99995 +x_0=500000 ' '+y_0=0 +ellps=clrk66 +towgs84=-127.62,-67.24,' '-47.04,-3.068,4.903,1.578,-1.06 +units=m ' '+no_defs') x_size = 500 y_size = 500 area_extent = (191376.04113, 1399386.68659, 671376.04113, 1879386.68659) area_def = utils.get_area_def( area_id, description, proj_id, projection, x_size, y_size, area_extent ) .. rst-class:: sphx-glr-script-out .. code-block:: none /tmp/build/docs/swirlspy/swirlspy/examples/qpf_manila.py:84: UserWarning: 'get_area_def' has moved, import it with 'from pyresample import get_area_def' area_id, description, proj_id, projection, x_size, y_size, area_extent /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() .. GENERATED FROM PYTHON SOURCE LINES 88-91 Step 4: Read data from radar files into xarray.DataArray using read_uf_ph(). .. GENERATED FROM PYTHON SOURCE LINES 91-101 .. code-block:: default reflectivity_list = [] # stores reflec from read_iris() for filename in located_files: reflec = read_uf_ph( filename, area_def=area_def, coord_label=['easting', 'northing'], indicator='cappi', elevation=2 ) reflectivity_list.append(reflec) .. rst-class:: sphx-glr-script-out .. code-block:: none /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 return _prepare_from_string(" ".join(pjargs)) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 projstring = _prepare_from_string(" ".join((projstring, projkwargs))) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 return _prepare_from_string(" ".join(pjargs)) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 projstring = _prepare_from_string(" ".join((projstring, projkwargs))) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 return _prepare_from_string(" ".join(pjargs)) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 projstring = _prepare_from_string(" ".join((projstring, projkwargs))) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 return _prepare_from_string(" ".join(pjargs)) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 projstring = _prepare_from_string(" ".join((projstring, projkwargs))) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 return _prepare_from_string(" ".join(pjargs)) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 projstring = _prepare_from_string(" ".join((projstring, projkwargs))) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 return _prepare_from_string(" ".join(pjargs)) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 projstring = _prepare_from_string(" ".join((projstring, projkwargs))) /opt/conda/envs/swirlspy/lib/python3.6/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() .. GENERATED FROM PYTHON SOURCE LINES 102-104 Step 5: Assigning reflectivity xarrays at the last two timestamps to variables for use during ROVER QPF. .. GENERATED FROM PYTHON SOURCE LINES 104-107 .. code-block:: default initialising_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 108-114 Running ROVER and Semi-Lagrangian Advection ------------------------------------------- 1. Concatenate two reflectivity xarrays along time dimension. 2. Run ROVER, with the concatenated xarray as the input. 3. Perform Semi-Lagrangian Advection using the motion fields from rover. .. GENERATED FROM PYTHON SOURCE LINES 114-131 .. code-block:: default # Combining the two reflectivity DataArrays # the order of the coordinate keys is now ['y', 'x', 'time'] # as opposed to ['time', 'x', 'y'] reflec_concat = xr.concat(reflectivity_list, dim='time') standardize_attr(reflec_concat, frame_type=FrameType.dBZ, zero_value=9999.) # Rover motion = rover(reflec_concat) rover_time = pd.Timestamp.now() # Semi Lagrangian Advection reflectivity = sla(reflec_concat, motion, nowcast_steps=30) sla_time = pd.Timestamp.now() .. rst-class:: sphx-glr-script-out .. code-block:: pytb Traceback (most recent call last): File "/tmp/build/docs/swirlspy/swirlspy/examples/qpf_manila.py", line 122, in motion = rover(reflec_concat) File "/tmp/build/docs/swirlspy/swirlspy/qpf/_mf/rover.py", line 67, in rover from rover.rover import rover as rover_api ImportError: libopencv_core.so.3.4: cannot open shared object file: No such file or directory .. GENERATED FROM PYTHON SOURCE LINES 132-138 Concatenating observed and forecasted reflectivities ------------------------------------------------------- 1. Add forecasted reflectivity to reproj_reflectivity_list. 2. Concatenate observed and forecasted reflectivity xarray.DataArrays along the time dimension. .. GENERATED FROM PYTHON SOURCE LINES 138-145 .. code-block:: default reflectivity = xr.concat([reflec_concat[:-1, ...], reflectivity], dim='time') reflectivity.attrs['long_name'] = 'Reflectivity 2km CAPPI' standardize_attr(reflectivity) concat_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 146-154 Generating radar reflectivity maps ----------------------------------- Define the color scale and format of the plots and plot using xarray.plot(). In this example, only hourly images will be plotted. .. GENERATED FROM PYTHON SOURCE LINES 154-233 .. code-block:: default # Defining colour scale and format levels = [ -32768, 10, 15, 20, 24, 28, 32, 34, 38, 41, 44, 47, 50, 53, 56, 58, 60, 62 ] cmap = ListedColormap([ '#FFFFFF', '#08C5F5', '#0091F3', '#3898FF', '#008243', '#00A433', '#00D100', '#01F508', '#77FF00', '#E0D100', '#FFDC01', '#EEB200', '#F08100', '#F00101', '#E20200', '#B40466', '#ED02F0' ]) norm = BoundaryNorm(levels, ncolors=cmap.N, clip=True) # Defining the crs crs = area_def.to_cartopy_crs() # Generating a timelist for every hour timelist = [ (basetime + pd.Timedelta(minutes=60*i-10)) for i in range(4) ] # Obtaining the slice of the xarray to be plotted da_plot = reflectivity.sel(time=timelist) # Defining motion quivers qx = motion.coords['easting'].values[::5] qy = motion.coords['northing'].values[::5] qu = motion.values[0, ::5, ::5] qv = motion.values[1, ::5, ::5] # Defining coastlines hires = cfeature.GSHHSFeature( levels=[1], scale='h', edgecolor='k' ) # Plotting p = da_plot.plot( col='time', col_wrap=2, subplot_kws={'projection': crs}, cbar_kwargs={ 'extend': 'max', 'ticks': levels[1:], 'format': '%.3g' }, cmap=cmap, norm=norm ) for idx, ax in enumerate(p.axes.flat): ax.quiver(qx, qy, qu, qv, pivot='mid', regrid_shape=20) ax.add_feature(hires) # coastlines ax.gridlines() ax.set_title( "Reflectivity\n" f"Based @ {basetime.strftime('%H:%MH')}", loc='left', fontsize=9 ) ax.set_title( '' ) ax.set_title( f"{basetime.strftime('%Y-%m-%d')} \n" f"Valid @ {timelist[idx].strftime('%H:%MH')} ", loc='right', fontsize=9 ) plt.savefig( THIS_DIR + f"/../tests/outputs/rover-output-map-mn.png", dpi=300 ) radar_image_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 234-242 Accumulating hourly rainfall for 3-hour forecast ------------------------------------------------ Hourly accumulated rainfall is calculated every 30 minutes, the first endtime is the basetime i.e. T+0min. #. Convert rainrates to rainfalls in 10 mins with to_rainfall_depth(). #. Accumulate hourly rainfall every 30 minutes using acc_rainfall_depth(). .. GENERATED FROM PYTHON SOURCE LINES 242-263 .. code-block:: default # Convert reflectivity to rainrates rainfalls = to_rainfall_depth(reflectivity, a=300, b=1.4) # Converting the coordinates of xarray from start to endtime rainfalls.coords['time'] = [ pd.Timestamp(t) + pd.Timedelta(10, 'm') for t in rainfalls.coords['time'].values ] rainfalls.attrs['step_size'] = pd.Timedelta(10, 'm') # Accumulate hourly rainfall every 30 minutes acc_rf = acc_rainfall_depth( rainfalls, basetime, basetime+pd.Timedelta(hours=3) ) acc_rf.attrs['long_name'] = 'Rainfall accumulated over the past 60 minutes' acc_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 264-271 Plotting rainfall maps --------------------------------------- Define the colour scheme and format and plot using xarray.plot(). In this example, only hourly images will be plotted. .. GENERATED FROM PYTHON SOURCE LINES 271-346 .. code-block:: default # Defining the colour scheme levels = [ 0, 0.5, 2, 5, 10, 20, 30, 40, 50, 70, 100, 150, 200, 300, 400, 500, 600, 700 ] cmap = ListedColormap([ '#ffffff', '#9bf7f7', '#00ffff', '#00d5cc', '#00bd3d', '#2fd646', '#9de843', '#ffdd41', '#ffac33', '#ff621e', '#d23211', '#9d0063', '#e300ae', '#ff00ce', '#ff57da', '#ff8de6', '#ffe4fd' ]) norm = BoundaryNorm(levels, ncolors=cmap.N, clip=True) # Defining projection crs = area_def.to_cartopy_crs() # Defining zoom extent r = 64000 proj_site = acc_rf.proj_site zoom = ( proj_site[0]-r, proj_site[0]+r, proj_site[1]-r, proj_site[1]+r ) # (x0, x1, y0, y1) # Defining times for plotting timelist = [basetime + pd.Timedelta(i, 'h') for i in range(4)] # Obtaining xarray slice to be plotted da_plot = acc_rf.sel( easting=slice(zoom[0], zoom[1]), northing=slice(zoom[3], zoom[2]), time=timelist ) # Plotting p = da_plot.plot( col='time', col_wrap=2, subplot_kws={'projection': crs}, cbar_kwargs={ 'extend': 'max', 'ticks': levels, 'format': '%.3g' }, cmap=cmap, norm=norm ) for idx, ax in enumerate(p.axes.flat): ax.add_feature(hires) # using GSHHS coastlines defined previously ax.gridlines() ax.set_xlim(zoom[0], zoom[1]) ax.set_ylim(zoom[2], zoom[3]) ax.set_title( "Past Hour Rainfall\n" f"Based @ {basetime.strftime('%H:%MH')}", loc='left', fontsize=8 ) ax.set_title( '' ) ax.set_title( f"{basetime.strftime('%Y-%m-%d')} \n" f"Valid @ {timelist[idx].strftime('%H:%MH')} ", loc='right', fontsize=8 ) plt.savefig( THIS_DIR + f"/../tests/outputs/rainfall_mn.png", dpi=300 ) rf_image_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 347-359 Extract the rainfall values at a specified location ------------------------------------------------------------------ In this example, the rainfall values at the location is assumed to be the same as the nearest gridpoint. 1. Read information regarding the rain gauge stations into a pandas.DataFrame. 2. Extract the rainfall values at the nearest gridpoint to location for given timesteps (in this example, 30 minute intervals). 3. Store rainfall values over time in a pandas.DataFrame. 4. Plot the time series of rainfall at different stations. .. GENERATED FROM PYTHON SOURCE LINES 359-413 .. code-block:: default # Getting rain gauge station coordinates df = pd.read_csv( os.path.join(THIS_DIR, "../tests/samples/manila_rg_list.csv"), delim_whitespace=True, usecols=[0, 3, 4] ) # Extract rainfall values at gridpoint closest to the # location specified for given timesteps and storing it # in pandas.DataFrame. rf_time = [] for time in acc_rf.coords['time'].values: rf = [] for index, row in df.iterrows(): rf.append(acc_rf.sel( time=time, northing=row[2], easting=row[1], method='nearest' ).values) rf_time.append(rf) rf_time = np.array(rf_time) station_rf = pd.DataFrame( data=rf_time, columns=df.iloc[:, 0], index=pd.Index( acc_rf.coords['time'].values, name='time' ) ) print(station_rf) loc_stn = \ ['BAA', 'BUM', 'PAF', 'QUL', 'ZAP', 'ZAA'] loc_stn_drop = [ stn for stn in station_rf.columns.to_list() if stn not in loc_stn ] df_loc = station_rf.drop(loc_stn_drop, axis=1) print(df_loc) # Plotting time series graph for selected stations ax = df_loc.plot(title="Time Series of Hourly Accumulated Rainfall", grid=True) ax.set_ylabel("Hourly Accumulated Rainfall (mm)") plt.savefig(THIS_DIR+"/../tests/outputs/qpf_time_series.png") extract_time = pd.Timestamp.now() .. GENERATED FROM PYTHON SOURCE LINES 414-417 Checking run time of each component -------------------------------------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 417-436 .. code-block:: default print(f"Start time: {start_time}") print(f"Initialising time: {initialising_time}") print(f"Rover time: {rover_time}") print(f"SLA time: {sla_time}") print(f"Plotting radar image time: {radar_image_time}") print(f"Accumulating rainfall time: {acc_time}") print(f"Concatenating time: {concat_time}") print(f"Plotting rainfall map time: {rf_image_time}") print(f"Extracting and plotting time series time: {extract_time}") print(f"Time to initialise: {initialising_time-start_time}") print(f"Time to run rover: {rover_time-initialising_time}") print(f"Time to perform SLA: {sla_time-rover_time}") print(f"Time to concatenate xarrays: {concat_time - sla_time}") print(f"Time to plot radar image: {radar_image_time - concat_time}") print(f"Time to accumulate rainfall: {acc_time - radar_image_time}") print(f"Time to plot rainfall maps: {rf_image_time-acc_time}") print(f"Time to extract and plot time series: {extract_time-rf_image_time}") .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 59.745 seconds) .. _sphx_glr_download_auto_examples_qpf_manila.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: qpf_manila.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: qpf_manila.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_