swirlspy.qpf.dl package

Subpackages

Submodules

swirlspy.qpf.dl.config module

swirlspy.qpf.dl.config.cfg_from_file(file_name, target)

Load a config file and merge it into the default options.

Parameters:

  • file_name (str) – config file path

  • target (.helpers.ordered_easydict.OrderedEasyDict) – default config

swirlspy.qpf.dl.config.load_latest_cfg(dir_path, target)

Load latest configuration

Parameters:

  • dir_path (str) – path to load config

  • target (.helpers.ordered_easydict.OrderedEasyDict) – default configuration

swirlspy.qpf.dl.config.ordered_dump(data, stream=None, Dumper=None, **kwds)
swirlspy.qpf.dl.config.save_cfg(dir_path, source)

Save config file

Parameters:

  • dir_path (str) – path to save the config file

  • source (.helpers.ordered_easydict.OrderedEasyDict) – model configuration

swirlspy.qpf.dl.encoder_forecaster module

swirlspy.qpf.dl.hko_benchmark module

class swirlspy.qpf.dl.hko_benchmark.HKOBenchmarkEnv(pd_path, save_dir='hko7_benchmark', mode='fixed')

Bases: object

The Benchmark environment for the HKO7 Dataset

There are two settings for the Benchmark, the “fixed” setting and the “online” setting. In the “fixed” setting, pre-defined input sequences that have the same length will be

fed into the model for prediction.

This setting tests the model’s ability to use the instant past to predict the future.

In the “online” setting, M frames will be given each time and the forecasting model
is required to predict the next K frames every stride steps.
If the begin_new_episode flag is turned on, a new episode has begun, which means that

the current received images have no relationship with the previous images.

If the need_upload_prediction flag is turned on, the model is required to predict the This setting tests both the model’s ability to adapt in an online fashion and

the ability to capture the long-term dependency.

The input frame will be missing in some timestamps.

To run the benchmark in the fixed setting:

env = HKOBenchmarkEnv(…) while not env.done:

# Get the observation in_frame_dat, in_mask_dat, in_datetime_clips, out_datetime_clips, begin_new_episode =

env.get_observation(batch_size=batch_size)

# Running your algorithm to get the prediction prediction = … # Upload prediction to the environment env.upload_prediction(prediction)

property done
get_observation(batch_size=1)
Parameters:

batch_size (int)

Returns:

  • in_frame_dat (np.ndarray) – Will be between 0 and 1

  • in_datetime_clips (list)

  • out_datetime_clips (list)

  • begin_new_episode (bool)

  • need_upload_prediction (bool)

print_stat_readable()
reset()
save_eval(print_out=False)
upload_prediction(prediction)
Parameters:

prediction (np.ndarray)

swirlspy.qpf.dl.hko_evaluation module

class swirlspy.qpf.dl.hko_evaluation.HKOEvaluation(seq_len, use_central, no_ssim=True, threholds=None, central_region=None)

Bases: object

begin()
calculate_stat()

The following measurements will be used to measure the score of the forecaster

See Also [Weather and Forecasting 2010] Equitability Revisited: Why the “Equitable Threat Score” Is Not Equitable http://www.wxonline.info/topics/verif2.html

We will denote (a b (hits false alarms

c d) = misses correct negatives)

We will report the POD = a / (a + c) FAR = b / (a + b) CSI = a / (a + b + c) Heidke Skill Score (HSS) = 2(ad - bc) / ((a+c) (c+d) + (a+b)(b+d)) Gilbert Skill Score (GSS) = HSS / (2 - HSS), also known as the Equitable Threat Score

HSS = 2 * GSS / (GSS + 1)

MSE = mask * (pred - gt) **2 MAE = mask * abs(pred - gt) GDL = valid_mask_h * abs(gd_h(pred) - gd_h(gt)) + valid_mask_w * abs(gd_w(pred) - gd_w(gt))

clear_all()
print_stat_readable(prefix='')
save(prefix, print_out=False)
save_pkl(path)
save_txt_readable(path, print_out)
update(gt, pred, mask, start_datetimes=None)
Parameters:

  • gt (np.ndarray)

  • pred (np.ndarray)

  • mask (np.ndarray) – 0 indicates not use and 1 indicates that the location will be taken into account

  • start_datetimes (list) – The starting datetimes of all the testing instances

swirlspy.qpf.dl.hko_evaluation.dBZ_to_pixel(dBZ_img)
Parameters:

dBZ_img (np.ndarray)

swirlspy.qpf.dl.hko_evaluation.get_GDL(prediction, truth, mask, sum_batch=False)

Calculate the masked gradient difference loss

Parameters:

  • prediction (np.ndarray) – Shape: (seq_len, batch_size, 1, height, width)

  • truth (np.ndarray) – Shape: (seq_len, batch_size, 1, height, width)

  • mask (np.ndarray or None) – Shape: (seq_len, batch_size, 1, height, width) 0 –> not use 1 –> use

Returns:

gdl – Shape: (seq_len,) or (seq_len, batch_size)

Return type:

np.ndarray

swirlspy.qpf.dl.hko_evaluation.get_PSNR(prediction, truth)

Peak Signal Noise Ratio

Parameters:

  • prediction (np.ndarray)

  • truth (np.ndarray)

Returns:

ret

Return type:

np.ndarray

swirlspy.qpf.dl.hko_evaluation.get_SSIM(prediction, truth)

Calculate the SSIM score following [TIP2004] Image Quality Assessment: From Error Visibility to Structural Similarity

Same functionality as https://github.com/coupriec/VideoPredictionICLR2016/blob/master/image_error_measures.lua#L50-L75

We use nowcasting.helpers.msssim, which is borrowed from Tensorflow to do the evaluation

Parameters:

  • prediction (np.ndarray)

  • truth (np.ndarray)

Returns:

ret

Return type:

np.ndarray

swirlspy.qpf.dl.hko_evaluation.get_balancing_weights(data, mask, base_balancing_weights=None, thresholds=None)
swirlspy.qpf.dl.hko_evaluation.get_correlation(prediction, truth)
Parameters:

  • prediction (np.ndarray)

  • truth (np.ndarray)

swirlspy.qpf.dl.hko_evaluation.get_hit_miss_counts(prediction, truth, mask=None, thresholds=None, sum_batch=False)

This function calculates the overall hits and misses for the prediction, which could be used to get the skill scores and threat scores:

This function assumes the input, i.e, prediction and truth are 3-dim tensors, (timestep, row, col) and all inputs should be between 0~1

Parameters:

  • prediction (np.ndarray) – Shape: (seq_len, batch_size, 1, height, width)

  • truth (np.ndarray) – Shape: (seq_len, batch_size, 1, height, width)

  • mask (np.ndarray or None) – Shape: (seq_len, batch_size, 1, height, width) 0 –> not use 1 –> use

  • thresholds (list or tuple)

Returns:

  • hits (np.ndarray) – (seq_len, len(thresholds)) or (seq_len, batch_size, len(thresholds)) TP

  • misses (np.ndarray) – (seq_len, len(thresholds)) or (seq_len, batch_size, len(thresholds)) FN

  • false_alarms (np.ndarray) – (seq_len, len(thresholds)) or (seq_len, batch_size, len(thresholds)) FP

  • correct_negatives (np.ndarray) – (seq_len, len(thresholds)) or (seq_len, batch_size, len(thresholds)) TN

swirlspy.qpf.dl.hko_evaluation.get_rainfall_mse(prediction, truth)
swirlspy.qpf.dl.hko_evaluation.pixel_to_dBZ(img)
Parameters:

img (np.ndarray or float)

swirlspy.qpf.dl.hko_evaluation.pixel_to_rainfall(img, a=None, b=None)

Convert the pixel values to real rainfall intensity

Parameters:

  • img (np.ndarray)

  • a (float32, optional)

  • b (float32, optional)

Returns:

rainfall_intensity

Return type:

np.ndarray

swirlspy.qpf.dl.hko_evaluation.rainfall_to_pixel(rainfall_intensity, a=None, b=None)

Convert the rainfall intensity to pixel values

Parameters:

  • rainfall_intensity (np.ndarray)

  • a (float32, optional)

  • b (float32, optional)

Returns:

pixel_vals

Return type:

np.ndarray

swirlspy.qpf.dl.hko_factory module

swirlspy.qpf.dl.hko_iterator module

class swirlspy.qpf.dl.hko_iterator.HKOIterator(pd_path, sample_mode, seq_len=30, max_consecutive_missing=2, begin_ind=None, end_ind=None, stride=None, width=None, height=None, base_freq='6min')

Bases: object

The iterator for HKO-7 dataset

property begin_time
check_new_start()
property end_time
get_airflow_terms(frames, batch_size)

Retrieve airflow terms: u and v

Parameters:

  • frames (np.ndarray) – Shape: (out_len+2, valid_batch_size, 1, height, width)

  • batch_size (int) – Batch size

Returns:

  • u (np.ndarray) – Shape: (out_len+1, valid_batch_size, 1, height, width)

  • v (np.ndarray) – Shape: (out_len+1, valid_batch_size, 1, height, width)

random_reset()
reset(begin_ind=None, end_ind=None)
sample(batch_size, only_return_datetime=False)

Sample a minibatch from the hko7 dataset based on the given type and pd_file

Parameters:

  • batch_size (int) – Batch size

  • only_return_datetime (bool) – Whether to only return the datetimes

Returns:

  • frame_dat (np.ndarray) – Shape: (seq_len, valid_batch_size, 1, height, width)

  • mask_dat (np.ndarray) – Shape: (seq_len, valid_batch_size, 1, height, width)

  • datetime_clips (list) – length should be valid_batch_size

  • new_start (bool)

sample_physical_params(frames, batch_size)

Sample u, v and source for physical loss function

Parameters:

  • frames (np.ndarray) – Shape: (out_len+2, valid_batch_size, 1, height, width)

  • batch_size (int) – Batch size

Returns:

  • u (np.ndarray) – Shape: (seq_len, valid_batch_size, 1, height, width)

  • v (np.ndarray) – Shape: (seq_len, valid_batch_size, 1, height, width)

  • source (np.ndarray) – Shape: (seq_len, valid_batch_size, 1, height, width)

set_begin_end(begin_ind=None, end_ind=None)
property total_frame_num
property use_up
class swirlspy.qpf.dl.hko_iterator.HKOSimpleBuffer(df, max_buffer_length, width, height)

Bases: object

get(timestamps)

timestamps must be sorted

Parameters:

timestamps

reset()
swirlspy.qpf.dl.hko_iterator.convert_datetime_to_filepath(date_time)

Convert datetime to the filepath

Parameters:

date_time (datetime.datetime)

Returns:

ret

Return type:

str

swirlspy.qpf.dl.hko_iterator.convert_datetime_to_maskpath(date_time)

Convert datetime to path of the mask

Parameters:

date_time (datetime.datetime)

Returns:

ret

Return type:

str

swirlspy.qpf.dl.hko_iterator.decode_month(code)

Decode the month code back to the month value

Parameters:

code (np.ndarray) – (…, 2) float

Returns:

month – (…,) int

Return type:

np.ndarray

swirlspy.qpf.dl.hko_iterator.encode_month(month)

Encode the month into a vector

Parameters:

month (np.ndarray) – (…,) int, between 1 and 12

Returns:

ret – (…, 2) float

Return type:

np.ndarray

swirlspy.qpf.dl.hko_iterator.get_exclude_mask()
swirlspy.qpf.dl.hko_iterator.get_valid_datetime_set()
swirlspy.qpf.dl.hko_iterator.pad_hko_dat(frame_dat, mask_dat, batch_size)
swirlspy.qpf.dl.hko_iterator.precompute_mask(img)

swirlspy.qpf.dl.image module

exception swirlspy.qpf.dl.image.UnknownImageFormat

Bases: Exception

swirlspy.qpf.dl.image.cv2_read_img(path, read_storage, grayscale)
swirlspy.qpf.dl.image.cv2_read_img_resize(path, read_storage, resize_storage, frame_size, grayscale)
swirlspy.qpf.dl.image.quick_imsize(file_path)

Return (width, height) for a given img file content - no external dependencies except the os and struct modules from core

Parameters:

file_path

swirlspy.qpf.dl.image.quick_read_frames(path_list, im_w=None, im_h=None, resize=False, frame_size=None, grayscale=True)

Multi-thread Frame Loader

Parameters:

  • path_list (list)

  • resize (bool, optional)

  • frame_size (None or tuple)

swirlspy.qpf.dl.mask module

swirlspy.qpf.dl.mask.quick_read_masks(path_list)
swirlspy.qpf.dl.mask.read_mask_file(filepath, out)

Load mask file to numpy array

Parameters:

  • filepath (str)

  • out (np.ndarray)

swirlspy.qpf.dl.mask.save_mask_file(npy_mask, filepath)

swirlspy.qpf.dl.movingmnist_iterator module

class swirlspy.qpf.dl.movingmnist_iterator.MovingMNISTAdvancedIterator(digit_num=None, distractor_num=None, img_size=None, distractor_size=5, max_velocity_scale=3.6, initial_velocity_range=(0.0, 3.6), acceleration_range=(0.0, 0.0), scale_variation_range=(0.9090909090909091, 1.1), rotation_angle_range=(-30, 30), global_rotation_angle_range=(-30, 30), illumination_factor_range=(0.6, 1.0), period=5, global_rotation_prob=0.5, index_range=(0, 40000))

Bases: object

draw_distractors(canvas_img, distractor_seeds)
Parameters:

canvas_img

draw_imgs(base_img, affine_transforms, prev_affine_transforms=None)
Parameters:

  • base_img (list) – Inner Shape: (H, W)

  • affine_transforms (np.ndarray) – Shape: (digit_num, 2, 3)

  • prev_affine_transforms (np.ndarray) – Shape: (digit_num, 2, 3)

load(file)

Initialize to draw samples from pre-computed parameters.

Parameters:

  • file (Either the file name (string) or an open file (file-like)

  • loaded. (object) from which the data will be)

sample(batch_size, seqlen, random=True)
Parameters:

  • batch_size (int)

  • seqlen (int)

  • random (take random samples from loaded parameters. Ignored if no parameters are loaded.)

Returns:

  • seq (np.ndarray) – Shape: (seqlen, batch_size, 1, H, W)

  • motion_vectors (np.ndarray) – Shape: (seqlen, batch_size, 2, H, W)

save(seqlen, num_samples=10000, file=None)

Draw random numbers for num_samples sequences and save them.

This initializes the state of MovingMNISTAdvancedIterator to generate sequences based on the hereby drawn parameters.

Note that each call to sample(batch_size, seqlen) will use batch_size of the num_samples parameters.

Parameters:

  • num_samples (Number of unique MovingMNISTAdvanced sequences to draw) – parameters for

  • file (Either the file name (string) or an open file (file-like) – object) where the data will be saved. If file is a string or a Path, the .npz extension will be appended to the file name if it is not already there.

class swirlspy.qpf.dl.movingmnist_iterator.MovingMNISTIterator

Bases: object

sample(digitnum=3, width=64, height=64, seqlen=20, lower=3.0, upper=5.0, batch_size=32, index_range=(0, 50000))
Parameters:

  • digitnum

  • width

  • height

  • seqlen

  • batch_size

  • index_range

Returns:

seq – Shape: (seqlen, batch_size, 1, width, height)

Return type:

np.NDArray

swirlspy.qpf.dl.movingmnist_iterator.crop_mnist_digit(digit_img, tol=5)

Return the cropped version of the mnist digit

Parameters:

digit_img (np.ndarray) – Shape: ()

swirlspy.qpf.dl.movingmnist_iterator.load_mnist(training_num=50000)

Load the mnist dataset

Parameters:

training_num

swirlspy.qpf.dl.movingmnist_iterator.move_step(v0, p0, bounding_box)

swirlspy.qpf.dl.my_module module

swirlspy.qpf.dl.numba_accelerated module

swirlspy.qpf.dl.numba_accelerated.get_GDL_numba(prediction, truth, mask)

Accelerated version of get_GDL using numba(http://numba.pydata.org/)

Parameters:

  • prediction

  • truth

  • mask

Return type:

gdl

swirlspy.qpf.dl.numba_accelerated.get_balancing_weights_numba(data, mask, base_balancing_weights=None, thresholds=None)

Get the balancing weights

Parameters:

  • data

  • mask

  • base_balancing_weights

  • thresholds

swirlspy.qpf.dl.numba_accelerated.get_hit_miss_counts_numba(prediction, truth, mask, thresholds=None)

This function calculates the overall hits and misses for the prediction, which could be used to get the skill scores and threat scores:

This function assumes the input, i.e, prediction and truth are 3-dim tensors, (timestep, row, col) and all inputs should be between 0~1

Parameters:

  • prediction (np.ndarray) – Shape: (seq_len, batch_size, 1, height, width)

  • truth (np.ndarray) – Shape: (seq_len, batch_size, 1, height, width)

  • mask (np.ndarray or None) – Shape: (seq_len, batch_size, 1, height, width) 0 –> not use 1 –> use

  • thresholds (list or tuple)

Returns:

  • hits (np.ndarray) – (seq_len, batch_size, len(thresholds)) TP

  • misses (np.ndarray) – (seq_len, batch_size, len(thresholds)) FN

  • false_alarms (np.ndarray) – (seq_len, batch_size, len(thresholds)) FP

  • correct_negatives (np.ndarray) – (seq_len, batch_size, len(thresholds)) TN

swirlspy.qpf.dl.ops module

class swirlspy.qpf.dl.ops.ParamsReg

Bases: object

get(name, **kwargs)
get_inner()
reset()
swirlspy.qpf.dl.ops.activation(data, act_type, name=None)
swirlspy.qpf.dl.ops.conv2d(data, num_filter, kernel=(1, 1), stride=(1, 1), pad=(0, 0), dilate=(1, 1), no_bias=False, name=None, **kwargs)
swirlspy.qpf.dl.ops.conv2d_act(data, num_filter, kernel=(1, 1), stride=(1, 1), pad=(0, 0), dilate=(1, 1), no_bias=False, act_type='relu', name=None, **kwargs)
swirlspy.qpf.dl.ops.conv2d_bn_act(data, num_filter, kernel=(1, 1), stride=(1, 1), pad=(0, 0), dilate=(1, 1), no_bias=False, act_type='relu', momentum=0.9, eps=1.0000001e-05, fix_gamma=True, name=None, use_global_stats=False, **kwargs)
swirlspy.qpf.dl.ops.conv3d(data, num_filter, kernel=(1, 1, 1), stride=(1, 1, 1), pad=(0, 0, 0), dilate=(1, 1, 1), no_bias=False, name=None, **kwargs)
swirlspy.qpf.dl.ops.conv3d_act(data, num_filter, kernel=(1, 1, 1), stride=(1, 1, 1), pad=(0, 0, 0), dilate=(1, 1, 1), no_bias=False, act_type='relu', name=None, **kwargs)
swirlspy.qpf.dl.ops.conv3d_bn_act(data, num_filter, height, width, kernel=(1, 1, 1), stride=(1, 1, 1), pad=(0, 0, 0), dilate=(1, 1, 1), no_bias=False, act_type='relu', momentum=0.9, eps=1.0000001e-05, fix_gamma=True, name=None, use_global_stats=False, **kwargs)
swirlspy.qpf.dl.ops.deconv2d(data, num_filter, kernel=(1, 1), stride=(1, 1), pad=(0, 0), adj=(0, 0), no_bias=True, target_shape=None, name='deconv2d', **kwargs)
swirlspy.qpf.dl.ops.deconv2d_act(data, num_filter, kernel=(1, 1), stride=(1, 1), pad=(0, 0), adj=(0, 0), no_bias=True, target_shape=None, act_type='relu', name='deconv2d', **kwargs)
swirlspy.qpf.dl.ops.deconv2d_bn_act(data, num_filter, kernel=(1, 1), stride=(1, 1), pad=(0, 0), adj=(0, 0), no_bias=True, target_shape=None, act_type='relu', momentum=0.9, eps=1.0000001e-05, fix_gamma=True, name='deconv2d', use_global_stats=False, **kwargs)
swirlspy.qpf.dl.ops.deconv3d(data, num_filter, kernel=(1, 1, 1), stride=(1, 1, 1), pad=(0, 0, 0), adj=(0, 0, 0), no_bias=True, target_shape=None, name=None, **kwargs)
swirlspy.qpf.dl.ops.deconv3d_act(data, num_filter, kernel=(1, 1, 1), stride=(1, 1, 1), pad=(0, 0, 0), adj=(0, 0, 0), no_bias=True, target_shape=None, act_type='relu', name=None, **kwargs)
swirlspy.qpf.dl.ops.deconv3d_bn_act(data, num_filter, height, width, kernel=(1, 1, 1), stride=(1, 1, 1), pad=(0, 0, 0), adj=(0, 0, 0), no_bias=True, target_shape=None, act_type='relu', momentum=0.9, eps=1.0000001e-05, fix_gamma=True, name=None, use_global_stats=False, **kwargs)
swirlspy.qpf.dl.ops.downsample_module(data, num_filter, kernel, stride, pad, b_h_w, name, aggre_type=None)
swirlspy.qpf.dl.ops.fc_layer(data, num_hidden, no_bias=False, name='fc', **kwargs)
swirlspy.qpf.dl.ops.fc_layer_act(data, num_hidden, no_bias=False, act_type='relu', name='fc', **kwargs)
swirlspy.qpf.dl.ops.fc_layer_bn_act(data, num_hidden, no_bias=False, act_type='relu', momentum=0.9, eps=1.0000001e-05, fix_gamma=True, name=None, use_global_stats=False, **kwargs)
swirlspy.qpf.dl.ops.reset_regs()
swirlspy.qpf.dl.ops.upsample_module(data, num_filter, kernel, stride, pad, b_h_w, name, aggre_type=None)

swirlspy.qpf.dl.prediction_base_factory module

swirlspy.qpf.dl.utils module

class swirlspy.qpf.dl.utils.ConvResize2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1)

Bases: Module

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

swirlspy.qpf.dl.utils.cross_entropy_npy(a, b)
swirlspy.qpf.dl.utils.load_params(prefix, epoch)
Parameters:

  • prefix (str)

  • epoch (int)

Returns:

  • arg_params (dict)

  • aux_params (dict)

swirlspy.qpf.dl.utils.logging_config(folder=None, name=None, level=20, console_level=10)
Parameters:

  • folder (str or None)

  • name (str or None)

  • level (int)

  • console_level

swirlspy.qpf.dl.utils.make_layers(block)
swirlspy.qpf.dl.utils.parse_ctx(ctx_args)
swirlspy.qpf.dl.utils.safe_eval(expr)

Module contents