HydroCNHS User Manual

The Hydrological model for Coupled Natural-Human Systems (HydroCNHS) is an open-source Python package supporting four Application Programming Interfaces (APIs) that enable users to integrate their human decision models, which can be programmed with the agent-based modeling (ABM) concept, into the HydroCNHS.

Manual contributors: Chung-Yi Lin & Ali Ghaffari

Installation

Install HydroCNHS by pip.

pip install hydrocnhs

To install the latest version (beta version) of HydroCNHS, users can (1) install HydroCNHS by git.

pip install git+https://github.com/philip928lin/HydroCNHS.git

Or, (2) download the HydroCNHS package directly from the HydroCNHS GitHub repository. Then, install HydroCNHS from the setup.py.

# Need to move to the folder containing setup.py first.
python setup.py install

If you fail to install HydroCNHS due to the DEAP package, first downgrade setuptools to 57 and try to install HydroCNHS again.

pip install setuptools==57

101 classroom

Watershed/Basin

A land area that drains water to a particular water body like lakes, rivers, and oceans. Ridges, mountains, and hills usually define the boundary of a watershed/basin.

Subbasin

Subbasin is a sub-drainage land unit inside a watershed. HydroCNHS uses subbasins as the base unit for hydrological modeling. Each subbasin has a corresponding outlet. It is similar to the concept of the hydrologic response unit used in the SWAT model. However, users can freely determine the subbasins’ resolution to balance the computational cost and required resolution of the output information.

Semi-distributed hydrological model

HydroCNHS is a semi-distributed hydrological model with resolution/complexity between a lumped hydrological model and a distributed hydrological model, as shown in Fig. 1. A lumped model describes the water balance of the entire watershed as a water bucket (Fig. 1a). A distributed model simulates water responses in each grid and routes them to the downstream outlets (Fig. 1c). A semi-distributed hydrological model is a hybrid approach for the two mentioned methods. The streamflow at a downstream outlet is calculated by routing the subbasins’ (not grid) runoffs simulated by lumped models (Fig. 1b).

Comparison of three hydrological modeling schema: a) Lumped model, b) Semi-distributed model, and c) Distributed model.

Rainfall-runoff process modeling

The rainfall-runoff process is a series of mechanisms describing how a raindrop or snowfall becomes a runoff. This process often includes evapotranspiration, interception, infiltration, snow melting, and groundwater recession. HydroCNHS supports two process-based modeling options: (1) the General Water Loading Function (GWLF; Haith et al., 1987) with nine parameters and (2) the ABCD model (Thomas, 1981) with five parameters. Detailed calculations can be found in the supplementary material in Lin et al. (2022). Users can also pre-calculate subbasins’ runoffs using their preferred rainfall-runoff models and input them into HydroCNHS.

Routing & Routing outlets

Routing captures the delay in water movement from upstream to downstream. HydroCNHS adopts the Lohmann routing model (Lohmann et al., 1998) and the unit hydrograph parameterization described in Wi et al. (2015), with the former tracing the runoff from subbasins through the river channel (i.e., inter-subbasin routing) and the latter accounting for the within-subbasin routing process (Fig. 2a). A gamma distribution is adopted to represent the unit hydrograph of within-subbasin routing.

In HydroCNHS, we define routing outlets as those subbasin outlets with routed streamflow information. In other words, for subbasin outlets that are not defined as routing outlets, HydroCNHS will not route the streamflow, and only the runoff information is stored. However, we do not encourage users to define too many routing outlets if the streamflow information is not necessary at those outlets. Minimizing the number of routing outlets will reduce the model complexity and usually lead to a better model performance.

A water system in node-link structure

HydroCNHS represents a water system in a node-link structure. Therefore, users must learn how to abstract a water system with a node-link representation and understand the routing logic behind a given node-link structure used by HydroCNHS. We introduce three typical cases, shown in Fig. 2.

Routing schema using in HydroCNHS. a) single routing outlet, b) multiple routing outlets, and c) with a reservoir agent.

Single outlet with no human components (Fig. 2a)

In Fig. 2a, we are only interested in the streamflow at the basin outlet, assigned as the only routing outlet. In this case, HydroCNHS will route each runoff generated in each subbasin to the basin outlet (white dot). Namely, each subbasin contains both within-subbasin (represented by a unit hydrologic response curve) and inter-subbasin routing.

Multiple routing outlets (Fig. 2b)

In Fig. 2b, we are interested in the streamflow at multiple (in here two) outlets. In this case, calibration is possible if streamflow data is available for the two outlets. In Figure 2b, the three most upstream outlets (black dots) are routed to the upstream routing outlet and will NOT be routed again for the downstream routing outlet. Instead, the routed streamflow at the upstream routing outlet will be further routed to the downstream outlet with only river routing (i.e., only inter-subbasin and no within-subbasin routing).

Routing with human components using Dam API (Fig. 2c)

Fig. 2c introduces human components (e.g., a reservoir; ResAgt) integrated with Dam API to the node-link structure. A reservoir is considered an in-stream object that completely redefines the downstream flow according to its water releasing rules. Therefore, such an in-stream object (e.g., a reservoir) is defined as a pseudo routing outlet used for the downstream routing. Note that the upstream node of a reservoir has to be a routing outlet to simulate the inflow. Or, we can view that ResAgt takes water from its upstream routing outlet and release water based on the reservoir’s operational rules. More details about other APIs are presented in the following section.

Design a water system with the supporting APIs

A generic example of HydroCNHS coupling APIs and water system description.

The four APIs in the HydroCNHS (Fig. 3) are (1) Dam API, (2) RiverDiv API, (3) Conveying API, and (4) InSitu API.

Dam API is designed for integrating in-stream agents like reservoirs (e.g., R1 and R2 in Fig. 3) that could significantly alter the streamflow regime. Agents with Dam API will be considered pseudo routing outlets (no routing is needed) involved in the routing scheme. Namely, streamflow is directly defined by agents’ water releases decision.

RiverDiv API is created for agents that divert water from rivers and may have return flows to other outlets, e.g., diversion agent D1 diverts water from N3 and returns water to N1 in Fig. 3. This API ensures the diverted outlet is routed before agents’ diversions. At the outlet receiving return flow, the subbasin runoff and returned flow join and enter the within-subbasin routing process since return flows often have no explicit return location.

Conveying API is designed for transferring water to another outlet from a routing outlet where the routing process has already been executed. The transferred water has no within-subbasin routing (no within-subbasin delay like runoff). Therefore, they will be routed separately from the subbasin’s runoffs. If an agent wants to convey water from the downstream outlet to the upstream outlet (e.g., pump stations), the water will be delivered with delays (e.g., C2 diverts water from N3 first and delivers it to S2 at a later time step).

InSitu API is developed for agents that directly affect runoffs via “within subbasin activities” (e.g., I1 in Fig. 3). For example, those runoff changes may come from land-use changes due to urbanization or exploiting groundwater through wells. Such adjustments will be made before any routing process at each time step.

We summarize the supported connections of each API in Table 1.

The supported connections of each API.

APIs	Minus (divert/take) from		Plus (return/release) to
	Routing outlets	Outlets	Routing outlets	Outlets
Dam API*	V	X	–	–
RiverDiv API	V	X	V	V
Conveying API	V	X	V	V
InSitu API**	V	V	V	V
*Human components using Dam API will serve as a pseudo routing outlet for the downstream routing. **InSitu API only works on a single outlet (i.e., subbasin).

Input/Output

Inputs

HydroCNHS has the following three main inputs:

1. Daily climate data including precipitation, temperature, and (optional) potential evapotranspiration (python dictionary object),

2. A model configuration file (.yaml; settings for the HydroCNHS and ABM models), and

3. ABM modules (.py; customized human models).

Outputs

The outputs of HydroCNHS are stored in a data collector object, an attribute of HydroCNHS (e.g., model.dc). The main output is the daily streamflow at routing outlets. However, this data collector object will also contain other user-specified agent outputs as long as users use this data collector object in their ABM modules. See Integrate an ABM and Data collector for more information.

Daily climate data

Temperature (temp; degC) and precipitation (prec; cm) are two required weather inputs for a simulation. Potential evapotranspiration (pet; cm) can be automatically calculated by HydroCNHS using the Hamon method. However, we encourage users to pre-calculate pet and provide it to HydroCNHS as an input to avoid repeated calculation, especially when running calibration. The daily climate inputs are in a python dictionary format, in which the subbasin outlet name is the key, and a daily time series (list) is the value, as shown below.

temp = {'subbasin1': [7.7, 7.0, 6.6, 6.3, .......],
        'subbasin2': [8.5, 4.0, 5.3, 6.2, .......]}

Model configuration file

A model configuration file is a YAML file that contains settings for the entire water system. The main sections in a model configuration file include Path, WaterSystem, LSM, Routing, and ABM. Please see the following description for each setting. Parameter and inputs definitions for rainfall-runoff and routing models are summarized in Table 2 and Table 3, respectively. HydroCNHS has an embedded model builder. Therefore, users do not need to manually create the model configuration file (see the example in the following section).

Path:
  WD:       <working directory>
  Modules:  <directory to the folder containing ABM modules>

WaterSystem:
  StartDate:    <simulation start date, e.g., 1981/1/1>
  EndDate:      <simulation end date, e.g., 2013/12/31>
  DataLength:   <total simulation length/days, e.g., 12053>
  NumSubbasins: <number of subbasins>
  Outlets:      <a list of subbasins'/outlets' names, e.g., ["outlet1", "outlet2"]>
  NodeGroups:   <a list of node group lists, e.g., [["outlet1", "outlet2"], []]>
  LSM:          <selected rainfall-runoff models, e.g., 'GWLF' or 'ABCD' or 'Other'>
  Routing:      <selected routing model, e.g., 'Lohmann'>
  ABM:
    Modules:    <a list of ABM modules, e.g., ['TRB_ABM_Instit.py']>
    InstitDMClasses:
        <a dict of {InstitDMClass: a list of institutional decision-making objects}>
    DMClasses:      <a list of decision-making classes, e.g., ['ReleaseDM', 'TransferDM']>
    DamAPI:         <a list of agent type classes using DamAPI, e.g., ['Reservoir_AgtType']>
    RiverDivAPI:    <a list of agent type classes using RiverDivAPI, e.g., ['Diversion_AgtType']>
    InsituAPI:      <a list of agent type classes using InsituAPI, e.g., ['Drain_AgtType']>
    ConveyingAPI:   <a list of agent type classes using ConveyingAPI, e.g., ['Pipe_AgtType']>
    Institutions:
        <a dict of {an institutional decision-making agent: a list of agent members}>

RainfallRunoff:
  <an outlet name, e.g., 'outlet1'>:
    Inputs: <input dict for selected rainfall-runoff model at outlet1>
    Pars:   <parameter dict for selected rainfall-runoff model at outlet1>
  <an outlet name, e.g., 'outlet2'>:
    Inputs: <input dict for selected rainfall-runoff model at outlet2>
    Pars:   <parameter dict for selected rainfall-runoff model at outlet2>

Routing:
  <a routing outlet name, e.g., 'outlet1'>:
    <an upstream outlet name, e.g., 'outlet1'>:
      Inputs:   <input dict of Lohmann routing model for link between outlet1 and the routing outlet>
      Pars:     <parameter dict of Lohmann routing model for link between outlet1 and the routing outlet>
    <an upstream outlet name, e.g., 'outlet2'>:
      Inputs:   <input dict of Lohmann routing model for link between outlet2 and the routing outlet>
      Pars:     <parameter dict of Lohmann routing model for link between outlet2 and the routing outlet>

ABM:
  <an agent type class name, e.g., 'Reservoir_AgtType'>:
    <an agent name belongs to this class, e.g., 'ResAgt'>:
      Attributes: "agent's attributes dict, e.g., {}"
      Inputs:
        Priority:   <exercution piority is conflict occurs, e.g., 0>
        Links:      <linkage dict, e.g., divert from 'outlet1' and return to 'outlet2,' {'outlet1': -1, 'outlet2': 1}>
        DMClass:    <assigned decision-making class or institution or none, e.g., 'ReleaseDM'>
      Pars:     <parameter dict of the agent, e.g., {}>

Hydrological model parameters and suggested bounds.

Module	Parameter name	Unit	Parameter	Bound
GWLF	Curve number	–	\(CN2\)	[25, 100]
	Interception coefficient	–	\(IS\)	[0, 0.5]
	Recession coefficient	–	\(Res\)	[10-3, 0.5]
	Deep seepage coefficient	–	\(Sep\)	[0, 0.5]
	Baseflow coefficient	–	\(\alpha\)	[0, 1]
	Percolation coefficient	–	\(\beta\)	[0, 1]
	Available/soil water capacity	cm	\(U_r\)	[1, 15]
	Degree-day coefficient for snowmelt	cm/°C	\(D_f\)	[0, 1]
	Land cover coefficient	–	\(K_c\)	[0.5, 1.5]
ABCD	Controls the amount of runoff and recharge during unsaturated soil	–	\(a\)	[0, 1]
	Control of the saturation level of the soils	–	\(b\)	[0, 400]
	Ratio of groundwater recharge to runoff	–	\(c\)	[0, 1]
	Control of groundwater discharge rate	–	\(d\)	[0, 1]
	Degree-day coefficient for snowmelt	cm/°C	\(D_f\)	[0, 1]
Lohmann routing	Subbasin unit hydrograph shape parameter	–	\(G_{shape}\)	[1, 100]
	Subbasin unit hydrograph rate parameter	–	\(G_{scale}\)	[10-2, 150]
	Wave velocity in the linearized Saint- Venant equation	m/s	\(Velo\)	[0.5, 100]
	Diffusivity in the linearized Saint- Venant equation	m2/s	\(Diff\)	[200, 5000]

Hydrological model inputs and default values.

Module	Parameter name	Unit	Parameter	Default
GWLF	Subbasin’s drainage area	ha	\(Area\)	–
	Latitude	deg	\(Latitude\)	–
	Initial shallow saturated soil water content	cm	\(S0\)	2
	Initial unsaturated soil water content	cm	\(U0\)	10
	Initial snow storage	cm	\(SnowS\)	5
ABCD	Subbasin’s drainage area	–	\(Area\)	–
	Latitude	deg	\(Latitude\)	–
	Initial saturated soil water content	cm	\(XL\)	2
	Initial snow storage	cm	\(SnowS\)	5
Lohmann routing	Flow length between two outlets	m	\(FlowLength\)	–
	An instream control object, e.g., a reservoir	–	\(InstreamControl\)	False

ABM modules

ABM modules are customized python scripts in which human components are designed through programming agent type classes and decision-making classes. HydroCNHS will load those user-specified classes and use them to initialize agents. More details are provided in the Integrate an ABM section.

Build a hydrological model

To get familiar with the HydroCNHS model construction process, let’s start with a hydrological model without human components. We will go through the following steps:

1. Create a model configuration file (.yaml) using a model builder.

2. Complete a model configuration file (.yaml)

3. Run a calibration

4. Run a simulation

We will adopt the following delineated watershed (Fig. 4) as a demonstration, in which we want to build a semi-distributed hydrological model to simulate the streamflow at the basin outlet, WSLO. The corresponding subbasins’ information is provided in Table 4.

Example watershed.

Subbasins’ information.

Subbasin/outlet	Drainage area [ha]	Latitude [deg]	Flow length to WSLO [m]
Hagg	10034.2408	45.469	101469.139
DLLO	22568.2404	45.475	91813.075
TRGC	24044.6363	45.502	80064.864
DAIRY	59822.7546	45.520	70988.164
RCTV	19682.6046	45.502	60398.680
WSLO	47646.8477	45.350	0

Step 1: Create a model configuration file

HydroCNHS comes with a model builder to help users create an initial model configuration file.

First, initiate a model builder object with a working directory (your working folder directory).

import os
import HydroCNHS
prj_path, this_filename = os.path.split(__file__)

Second, set up the water system with the simulation period (i.e., start date and end date).

wd = prj_path
mb = HydroCNHS.ModelBuilder(wd)
mb.set_water_system(start_date="1981/1/1", end_date="2013/12/31")

Third, we can add subbasins (i.e., outlet_list) into the model. We adopt the GWLF model as the rainfall-runoff model (i.e., runoff_model). We also assign the corresponding subbasins’ areas (i.e., area_list) and latitude (i.e., lat_list) using the information from Table 4. Note that area_list and lat_list are optional arguments. Users can manually enter that information to the model configuration file (.yaml).

outlet_list = ['Hagg', 'DLLO', 'TRGC', 'DAIRY', 'RCTV', 'WSLO']
area_list = [10034.2408, 22568.2404, 24044.6363, 59822.7546, 19682.6046,
             47646.8477]
lat_list = [45.469, 45.475, 45.502, 45.520, 45.502, 45.350]
mb.set_rainfall_runoff(outlet_list=outlet_list,area_list=area_list,
                       lat_list=lat_list, runoff_model="GWLF")

Fourth, we want to add the routing setting. Here, we only have one routing outlet, WSLO, hence, we only need to run the following code once, and all the outlets are considered upstream outlets of WSLO.

flow_length_list = [101469.139, 91813.075, 80064.864, 70988.164, 60398.680, 0]
mb.set_routing_outlet(routing_outlet="WSLO",
                      upstream_outlet_list=outlet_list,
                      flow_length_list=flow_length_list)

Finally, we can print out the model for quick view and then write it to .yaml file with the given name.

### Print the model in the console
mb.print_model()

### Output initial model configuration file (.yaml)
mb.write_model_to_yaml(filename="HydroModel.yaml")

Now, we should have the initial model configuration file (i.e., HydroModel.yaml) in the corresponding folder.

Step 2: Complete a model configuration file

Before using the created HydroModel.yaml for the later experiment, we need to open it and ensure the information is correct. The “Inputs” in LSM and Routing sections should be all populated. Some default values may be used (e.g., \(S0\), \(U0\), and \(SnowS\)). They should be updated according to users’ needs. For this example, we don’t need to modify anything. You might notice that all the parameters have a value of -99. This is the default value in HydroCNHS, meaning that those values need to be calibrated if not provided by users.

Step 3: Run a calibration

First, we load the climate data, observed flow data at WSLO, and the model configuration file. Here, we have calculated the evapotranspiration using the Hamon method. Therefore, we load it as the inputs as well.

import matplotlib.pyplot as plt
import pandas as pd
import HydroCNHS.calibration as cali
from copy import deepcopy

# Load climate data
temp = pd.read_csv(os.path.join(wd,"Data","Temp_degC.csv"),
                index_col=["Date"]).to_dict(orient="list")
prec = pd.read_csv(os.path.join(wd,"Data","Prec_cm.csv"),
                index_col=["Date"]).to_dict(orient="list")
pet = pd.read_csv(os.path.join(wd,"Data","Pet_cm.csv"),
                index_col=["Date"]).to_dict(orient="list")

# Load flow gauge monthly data at WSLO
obv_flow_WSLO = pd.read_csv(os.path.join(wd,"Data","WSLO_M_cms.csv"),
                            index_col=["Date"], parse_dates=["Date"])

# Load model
model_dict = HydroCNHS.load_model(os.path.join(wd, "HydroModel.yaml"))

Second, we generate default parameter bounds and create a convertor for calibration. Details about the converter are provided in the Calibration section.

# Generate default parameter bounds
df_list, df_name = HydroCNHS.write_model_to_df(model_dict)
par_bound_df_list, df_name = HydroCNHS.gen_default_bounds(model_dict)

# Create convertor for calibration
converter = cali.Convertor()
cali_inputs = converter.gen_cali_inputs(wd, df_list, par_bound_df_list)
formatter = converter.formatter

Third, we program the evaluation function for a genetic algorithm (GA). Kling-Gupta efficiency (KGE; Gupta et al., 2009) is adopted to represent the model performance of simulated monthly streamflow at the WSLO routing outlet.

# Code evaluation function for GA algorthm
def evaluation(individual, info):
    cali_wd, current_generation, ith_individual, formatter, _ = info
    name = "{}-{}".format(current_generation, ith_individual)

    ##### individual -> model
    # Convert 1D array to a list of dataframes.
    df_list = cali.Convertor.to_df_list(individual, formatter)
    # Feed dataframes in df_list to model dictionary.
    model = deepcopy(model_dict)
    for i, df in enumerate(df_list):
        s = df_name[i].split("_")[0]
        model = HydroCNHS.load_df_to_model_dict(model, df, s, "Pars")

    ##### Run simuluation
    model = HydroCNHS.Model(model, name)
    Q = model.run(temp, prec, pet)

    ##### Get simulation data
    # Streamflow of routing outlets.
    cali_target = ["WSLO"]
    cali_period = ("1981-1-1", "2005-12-31")
    sim_Q_D = pd.DataFrame(Q, index=model.pd_date_index)[cali_target]
    # Resample the daily simulation output to monthly outputs.
    sim_Q_M = sim_Q_D[cali_target].resample("MS").mean()

    KGE = HydroCNHS.Indicator().KGE(
        x_obv=obv_flow_WSLO[cali_period[0]:cali_period[1]][cali_target],
        y_sim=sim_Q_M[cali_period[0]:cali_period[1]][cali_target])

    fitness = KGE
    return (fitness,)

Forth, we set up a GA for calibration. Again, we will explain calibration in more detail in the Calibration section. Here, the code is demonstrated. Note that calibration might take some time to run, depending on your system specifications. Users can lower down ‘pop_size ‘ and ‘max_gen’ if they just want to experience the process instead of seeking convergence. Note that to debug your code, please set ‘paral_cores’ to 1 to show the error messages.

config = {'min_or_max': 'max',
     'pop_size': 100,
     'num_ellite': 1,
     'prob_cross': 0.5,
     'prob_mut': 0.15,
     'stochastic': False,
     'max_gen': 100,
     'sampling_method': 'LHC',
     'drop_record': False,
     'paral_cores': -1,
     'paral_verbose': 1,
     'auto_save': True,
     'print_level': 1,
     'plot': True}

seed = 5
rn_gen = HydroCNHS.create_rn_gen(seed)
ga = cali.GA_DEAP(evaluation, rn_gen)
ga.set(cali_inputs, config, formatter, name="Cali_HydroModel_gwlf_KGE")
ga.run()
summary = ga.summary
individual = ga.solution

Finally, we export the calibrated model (i.e., Best_HydroModel_gwlf_KGE.yaml).

##### Output the calibrated model.
df_list = cali.Convertor.to_df_list(individual, formatter)
model_best = deepcopy(model_dict)
for i, df in enumerate(df_list):
    s = df_name[i].split("_")[0]
    model = HydroCNHS.load_df_to_model_dict(model_best, df, s, "Pars")
HydroCNHS.write_model(model_best, os.path.join(ga.cali_wd, "Best_HydroModel_gwlf_KGE.yaml"))

Step 4: Run a simulation

After obtaining a calibrated model, users can now use it for any simulation-based experiment (e.g., streamflow uncertainty under climate change). The calibrated model configuration file (e.g., Best_HydroModel_gwlf_KGE.yaml) can be directly loaded into HydroCNHS to run a simulation.

### Run a simulation.
model = HydroCNHS.Model(os.path.join(wd, "Cali_HydroModel_gwlf_KGE",
                                    "Best_HydroModel_gwlf_KGE.yaml"))
Q = model.run(temp, prec, pet)
result = pd.DataFrame(Q, index=model.pd_date_index).resample("MS").mean()

### Plot
fig, ax = plt.subplots()
ax.plot(obv_flow_WSLO.index, obv_flow_WSLO.loc[:, "WSLO"], label="Obv")
ax.plot(obv_flow_WSLO.index, result["WSLO"], ls="--", label="Sim")
ax.legend()

The complete script can be found at ./tutorials/Hydro_example/Main_HydroModel.py.

The following section will further introduce how to incorporate user-defined ABM modules into the model.

Integrate an ABM

After getting familiar with HydroCNHS from the hydrological model example in the previous section, we want to build a water system with human components (i.e., ABM). We will go through a similar process in the last example and focus on adding the human components.

1. Create a model configuration file (.yaml) using a model builder.

2. Complete a model configuration file (.yaml)

3. Program an ABM module (.py)

4. Run a calibration

5. Run a simulation

We adopt the Tualatin River Basin (TRB; Fig. 5; Table 5) as the tutorial example. The corresponding subbasins’ information is shown in Table 5. In this example, we consider three human components (Table 6), including (1) a reservoir (ResAgt), (2) an irrigation diversion (DivAgt), and (3) a trans-basin water transfer (PipeAgt), to demonstrate the HydroCNHS’s functionalities. Also, we model each human agent with different levels of behavioral complexities to provide users with a sense of how an agent can be modeled (Table 6). We will calibrate the model using the streamflow at DLLO and WSLO, reservoir releases from ResAgt, and water diversions from DivAgt on a monthly scale. More details about TRB can be found in Lin et al. (2022). Here, we focus on the coding part.

The node-link structure of the Tualatin River Basin with human components.

Subbasins’ information for the Tualatin River Basin.

Subbasin/outlet	Drainage area [ha]	Latitude [deg]	Flow length [m]
HaggIn	10034.2408	45.469	0 (to HaggIn)
TRTR	329.8013	45.458	30899.4048 (to DLLO)
ResAgt*	–	–	9656.064 (to DLLO)
DLLO	22238.4391	45.475	0 (to DLLO) 11748.211 (to TRGC)
TRGC	24044.6363	45.502	0 (to TRGC) 80064.864 (to WSLO)
DAIRY	59822.7546	45.520	70988.164 (to WSLO)
RCTV	19682.6046	45.502	60398.680 (to WSLO)
WSLO	47646.8477	45.350	0 (to WSLO)
*ResAgt is a reservoir agent integrated with Dam API. It is considered a pseudo routing outlet.

Summary of human components’ setting.

Item	Agent name	API	Behavioral design	From	To
Reservoir	ResAgt	Dam API	Fixed operational rules	HaggIn	–
Irrigation diversion	DivAgt	RiverDiv API	Calibrated adaptive behavioral rules	TRGC	WSLO
Trans-basin water transfer	PipeAgt	Conveying API	External input data	–	TRTR

Step 1: Create a model configuration file

Creating a node-link structure of a modeled water system is a vital step before using HydroCNHS. Subbasin outlets are determined based on the major tributaries in the TRB. However, the subbasin outlet, TRTR, is given because we have an inlet there for trans-basin water transfer. For the routing outlet assignment, DLLO and WSLO are selected because the streamflow at these two is part of the calibration targets, TRGC is also chosen since an agent integrated using RiverDiv API can only divert water from a routing outlet, and HaggIn is picked because it is the inflow of ResAgt (i.e., ResAgt takes water from HaggIn).

With a node-link structure of the TRB water system, we can follow the same process shown in the “Build a hydrological model” to initialize a model builder, set up the water system with the simulation period, add subbasins, and add four routing outlets. Note that ResAgt is considered a pseudo routing outlet that needs to be assigned to one of the upstream outlets of a routing outlet.

import os
import HydroCNHS
prj_path, this_filename = os.path.split(__file__)

### Initialize a model builder object.
wd = prj_path
mb = HydroCNHS.ModelBuilder(wd)

### Setup a water system simulation information
mb.set_water_system(start_date="1981/1/1", end_date="2013/12/31")

### Setup land surface model (rainfall-runoff model)
# Here we have seven subbasins and we select GWLF as the rainfall-runoff model.
outlet_list = ['HaggIn', 'TRTR', 'DLLO', 'TRGC', 'DAIRY', 'RCTV', 'WSLO']
area_list = [10034.2408, 329.8013, 22238.4391, 24044.6363, 59822.7546,
            19682.6046, 47646.8477]
lat_list = [45.469, 45.458, 45.475, 45.502, 45.520, 45.502, 45.350]
mb.set_rainfall_runoff(outlet_list=outlet_list,area_list=area_list,
                    lat_list=lat_list, runoff_model="GWLF")

### Setup routing outlets
# Add WSLO
mb.set_routing_outlet(routing_outlet="WSLO",
                    upstream_outlet_list=["TRGC", "DAIRY", "RCTV", "WSLO"],
                    flow_length_list=[80064.864, 70988.164, 60398.680, 0])
# Add TRGC
mb.set_routing_outlet(routing_outlet="TRGC",
                    upstream_outlet_list=["DLLO", "TRGC"],
                    flow_length_list=[11748.211, 0])
# Add DLLO
# Specify that ResAgt is an instream object.
mb.set_routing_outlet(routing_outlet="DLLO",
                    upstream_outlet_list=["ResAgt", "TRTR", "DLLO"],
                    flow_length_list=[9656.064, 30899.4048, 0],
                    instream_objects=["ResAgt"])
# Add HaggIn
mb.set_routing_outlet(routing_outlet="HaggIn",
                    upstream_outlet_list=["HaggIn"],
                    flow_length_list=[0])

Initialize ABM setting

To add human components, we need to first initialize the ABM setting block by assigning an ABM module folder’s directory and planned ABM module filename. If they are not given, default values will be applied, namely, working directory and “ABM_module.py, “respectively. abm_module_name will be used as the filename for the ABM module template if users choose to generate one using the model builder.

mb.set_ABM(abm_module_folder_path=wd, abm_module_name="TRB_ABM.py")

Add agents

Next, we add human components (i.e., agents) to the model builder. We first add a reservoir agent (ResAgt), in which its corresponding agent type class, agent name, api, link dictionary, and decision-making class can be assigned at this stage. Although not all information has to be provided now (i.e., it can be manually added to the model configuration file later), we encourage users to provide complete details here.

mb.add_agent(agt_type_class="Reservoir_AgtType", agt_name="ResAgt",
             api=mb.api.Dam,
             link_dict={"HaggIn": -1, "ResAgt": 1},
             dm_class="ReleaseDM")

The setting shown above means that ResAgt (an agent object) will be created from Reservoir_AgtType (an agent type class) and integrated into HydroCNHS using the Dam API. A decision-making object will be created from ReleaseDM (a decision-making class) and assigned to ResAgt as its attribute. This agent, ResAgt, will take water (factor = -1) from HaggIn routing outlet and release (factor = 1) water to ResAgt. Remember that ResAgt itself is a pseudo routing outlet.

Following a similar procedure, we add a water diversion agent (DivAgt). However, we have parameters, including ReturnFactor, a, and b, involved in this agent. Hence, a dictionary is provided to the par_dict argument. The format of the parameter dictionary is that keys are parameter names, and values are parameter values (-99 means waiting to be calibrated).

However, if the parameter is the factor used in the link_dict, users need to follow the format shown here. For example, we want to calibrate a return factor (ReturnFactor) to determine the portion of diverted water returned to the WSLO subbasin. To do that, a list, [“ReturnFactor”, 0, “Plus”], is given to link_dict at WSLO. HydroCNHS will interpret it as taking the factor value from parameter ReturnFactor with a list index of 0. “Plus” tells HydroCNHS we add water to WSLO. If water is taken from WSLO, then “Minus” should be assigned.

mb.add_agent(agt_type_class="Diversion_AgType", agt_name="DivAgt",
             api=mb.api.RiverDiv,
             link_dict={"TRGC": -1, "WSLO": ["ReturnFactor", 0, "Plus"]},
             dm_class="DivertDM",
             par_dict={"ReturnFactor": [-99], "a": -99, "b":-99})

Finally, we add a trans-basin water transfer agent (PipeAgt).

mb.add_agent(agt_type_class="Pipe_AgType", agt_name="PipeAgt",
             api=mb.api.Conveying,
             link_dict={"TRTR": 1},
             dm_class="TransferDM")

Add institution

We did not include an institution in this TRB example; however if users want to assign an institution (e.g., “ResDivInstitution”) to ResAgt and DivAgt, they should do so by assuming that there is a cooperation between water release decisions and water diversion decisions. Namely, release decisions from ResAgt and diversion decisions from DivAgt are made simultaneously using a single decision-making object (Fig. 6). Users can do the following.

mb.add_institution(institution="ResDivInstitution",
                   instit_dm_class=" ResDivDMClass",
                   agent_list=[" ResAgt ", "DivAgt"])

Note that ResDivInstitution will overwrite the originally assigned DM classes (if any) of ResAgt and DivAgt. The above command means a single ResDivInstitution decision-making object initialized from ResDivDMClass will be assigned to ResAgt and DivAgt’s attributes (e.g., self.dm). Users can utilize this property to design their agents.

Generate ABM module template & output model configuration file

In addition to outputting a model configuration file (.yaml), the model builder can generate an ABM module template (.py) for users, in which the model builder will create the outline of agent type classes and decision-making classes, and users can concentrate on programming the calculation for each class given in the template.

### Output initial model configuration file (.yaml) and ABM module template.
mb.write_model_to_yaml(filename="HydroABMModel.yaml")
mb.gen_ABM_module_template()

Step 2: Complete a model configuration file

After the model configuration file (.yaml) is created, users should open the file to complete and correct any missing or misinterpreted values. For this example, again, we will keep the default values.

Step 3: Program ABM module (.py)

In the generated ABM module (.py), users can find mainly two types of classes, including agent type classes (AgtType) and decision-making classes (DMClass/Institutional DMClass). Agent type classes are used to define agents’ actions and store up-to-date information (e.g., current date and current time step) in agents’ attributes. Decision-making classes are used to program a specific decision-making process. Decision-making classes can be further separated into DMClass and Institutional DMClass.

The ABM design logic is illustrated in Fig. 6. A “class” is a template for objects that can be initiated with object-specific attributes and settings. For example, Agent1 and Agent2 are initiated from the same AgtType1 class. Agent 2, Agent 4, and Agent 5 are initiated from the AgtType2 class. Each agent will be assigned a DM object or Institution object as one of its attributes. DM objects initiated from DMClass are NOT shared with other agents; Namely, agents with DM objects will only have one unique DM object (e.g., Agent 1 and Agent 2 in Fig. 6). In contrast, an Institution object can be shared with multiple agents, in which those agents can make decisions together. For example, multiple irrigation districts make diversion decisions together to share the water shortage during a drought period. We will not implement the Institutional DMClass in this TRB example; however, we will show how to add an institution through a model builder.

ABM design logic. An agent is a combination of an AgtType class and an (Institutional) DM class. An Institution object can be shared among a group of agent objects (i.e., make decisions together), while a DM object can only be assigned to one agent object.

Agent type class (AgtType):

• self.name = agent’s name

• self.config = agent’s configuration dictionary, {‘Attributes’: …, ‘Inputs’: …, ‘Pars’: …}.

• self.start_date = start date (datetime object).

• self.current_date = current date (datetime object).

• self.data_length = data/simulation length.

• self.t = current timestep.

• self.dc = data collector object containing data. Routed streamflow (Q_routed) is also collected here.

• self.rn_gen = NumPy random number generator.

• self.agents = a dictionary of all initialized agents, {agt_name: agt object}.

• self.dm = (institutional) decision-making object if DMClass or institution is assigned to the agent, else None.

# AgtType
class XXX_AgtType(Base):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # The AgtType inherited attributes are applied.
        # See the note at top.

    def act(self, outlet):
        # Read corresponding factor of the given outlet
        factor = read_factor(self.config, outlet)

        # Common usage:
        # Get streamflow of outlet at timestep t
        Q = self.dc.Q_routed[outlet][self.t]

        # Make decision from (Institutional) decision-making
        # object if self.dm is not None.
        #decision = self.dm.make_dm(your_arguments)

        if factor <= 0:     # Divert from the outlet
            action = 0
        elif factor > 0:    # Add to the outlet
            action = 0

        return action

(Institutional) decision-making classes (DMClass):

• self.name = name of the agent or institute.

• self.dc = data collector object containing data. Routed streamflow (Q_routed) is also collected here.

• self.rn_gen = NumPy random number generator.

# DMClass
class XXX_DM(Base):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # The (Institutional) DMClass inherited attributes are applied.
        # See the note at top.

    def make_dm(self, your_arguments):
        # Decision-making calculation.
        decision = None
        return decision

To keep the manual concise, we provide a complete ABM module for the TRB example at ./tutorials/HydroABM_example/TRB_ABM_complete.py. Theoretical details can be found in Lin et al. (2022), and more coding tips are available at Advanced ABM coding tips.

Step 4: Run a calibration

First, we load the model configuration file, the climate data, and the observed monthly flow data for DLLO and WSLO, reservoir releases of ResAgt, and water diversions of DivAgt. Here, we have calculated the evapotranspiration using the Hamon method. Therefore, PET data is input along with other data. Note that we manually change the ABM module from “TRB_ABM.py” to “TRB_ABM_complete.py.”

import matplotlib.pyplot as plt
import pandas as pd
import HydroCNHS.calibration as cali
from copy import deepcopy

# Load climate data
temp = pd.read_csv(os.path.join(wd,"Data","Temp_degC.csv"),
                index_col=["Date"]).to_dict(orient="list")
prec = pd.read_csv(os.path.join(wd,"Data","Prec_cm.csv"),
                index_col=["Date"]).to_dict(orient="list")
pet = pd.read_csv(os.path.join(wd,"Data","Pet_cm.csv"),
                index_col=["Date"]).to_dict(orient="list")

# Load flow gauge monthly data at WSLO
obv_flow_data = pd.read_csv(os.path.join(wd,"Data","Cali_M_cms.csv"),
                            index_col=["Date"], parse_dates=["Date"])

# Load model
model_dict = HydroCNHS.load_model(os.path.join(wd, "HydroABMModel.yaml"))
# Change the ABM module to the complete one.
model_dict["WaterSystem"]["ABM"]["Modules"] = ["TRB_ABM_complete.py"]

Second, we generate default parameter bounds and create a convertor for calibration. Note that we manually change the default ABM parameter bounds as shown in the code. Details about the Converter are provided in the Calibration section.

# Generate default parameter bounds
df_list, df_name = HydroCNHS.write_model_to_df(model_dict)
par_bound_df_list, df_name = HydroCNHS.gen_default_bounds(model_dict)

# Modify the default bounds of ABM
df_abm_bound = par_bound_df_list[2]
df_abm_bound.loc["ReturnFactor.0", [('DivAgt', 'Diversion_AgType')]] = "[0, 0.5]"
df_abm_bound.loc["a", [('DivAgt', 'Diversion_AgType')]] = "[-1, 1]"
df_abm_bound.loc["b", [('DivAgt', 'Diversion_AgType')]] = "[-1, 1]"

# Create convertor for calibration
converter = cali.Convertor()
cali_inputs = converter.gen_cali_inputs(wd, df_list, par_bound_df_list)
formatter = converter.formatter

Third, we program the evaluation function for a genetic algorithm (GA). The four calibration targets’ mean Kling-Gupta efficiency (KGE; Gupta et al., 2009) is adopted to represent the model performance.

# Code evaluation function for GA algorthm
def evaluation(individual, info):
    cali_wd, current_generation, ith_individual, formatter, _ = info
    name = "{}-{}".format(current_generation, ith_individual)

    ##### individual -> model
    # Convert 1D array to a list of dataframes.
    df_list = cali.Convertor.to_df_list(individual, formatter)
    # Feed dataframes in df_list to model dictionary.
    model = deepcopy(model_dict)
    for i, df in enumerate(df_list):
        s = df_name[i].split("_")[0]
        model = HydroCNHS.load_df_to_model_dict(model, df, s, "Pars")

    ##### Run simuluation
    model = HydroCNHS.Model(model, name)
    Q = model.run(temp, prec, pet)

    ##### Get simulation data
    # Streamflow of routing outlets.
    cali_target = ["WSLO","DLLO","ResAgt","DivAgt"]
    cali_period = ("1981-1-1", "2005-12-31")
    sim_Q_D = pd.DataFrame(Q, index=model.pd_date_index)[["WSLO","DLLO"]]
    sim_Q_D["ResAgt"] = model.dc.ResAgt["Release"]
    sim_Q_D["DivAgt"] = model.dc.DivAgt["Diversion"]
    # Resample the daily simulation output to monthly outputs.
    sim_Q_M = sim_Q_D[cali_target].resample("MS").mean()

    KGEs = []
    for target in cali_target:
        KGEs.append(HydroCNHS.Indicator().KGE(
            x_obv=obv_flow_data[cali_period[0]:cali_period[1]][[target]],
            y_sim=sim_Q_M[cali_period[0]:cali_period[1]][[target]]))

    fitness = sum(KGEs)/4
    return (fitness,)

Fourth, we set up a GA for calibration. Again, we will explain calibration in more detail in the Calibration section. Here, only the code is demonstrated. Note that calibration might take some time to run, depending on your system specifications. Users can lower down ‘pop_size’ and ‘max_gen’ if they want to experience the process instead of seeking convergence. In order to debug your code, set ‘paral_cores’ to 1 to show the error message.

config = {'min_or_max': 'max',
        'pop_size': 100,
        'num_ellite': 1,
        'prob_cross': 0.5,
        'prob_mut': 0.15,
        'stochastic': False,
        'max_gen': 100,
        'sampling_method': 'LHC',
        'drop_record': False,
        'paral_cores': -1,
        'paral_verbose': 1,
        'auto_save': True,
        'print_level': 1,
        'plot': True}

seed = 5
rn_gen = HydroCNHS.create_rn_gen(seed)
ga = cali.GA_DEAP(evaluation, rn_gen)
ga.set(cali_inputs, config, formatter, name="Cali_HydroABMModel_gwlf_KGE")
ga.run()
summary = ga.summary
individual = ga.solution

Finally, we export the calibrated model (i.e., Best_HydroABMModel_gwlf_KGE.yaml).

##### Output the calibrated model.
df_list = cali.Convertor.to_df_list(individual, formatter)
model_best = deepcopy(model_dict)
for i, df in enumerate(df_list):
    s = df_name[i].split("_")[0]
    model = HydroCNHS.load_df_to_model_dict(model_best, df, s, "Pars")
HydroCNHS.write_model(model_best, os.path.join(ga.cali_wd, "Best_HydroABMModel_gwlf_KGE.yaml"))

Step 5: Run a simulation

After obtaining a calibrated model, users can now use it for any simulation-based experiment (e.g., streamflow uncertainty under climate change). The calibrated model configuration file (i.e., Best_HydroABMModel_gwlf_KGE.yaml) can be directly loaded into HydroCNHS to run a simulation.

### Run a simulation.
model = HydroCNHS.Model(os.path.join(ga.cali_wd, "Best_HydroABMModel_gwlf_KGE.yaml"))
Q = model.run(temp, prec, pet)
sim_Q_D = pd.DataFrame(Q, index=model.pd_date_index)[["WSLO","DLLO"]]
sim_Q_D["ResAgt"] = model.dc.ResAgt["Release"]
sim_Q_D["DivAgt"] = model.dc.DivAgt["Diversion"]
sim_Q_M = sim_Q_D[["WSLO","DLLO","ResAgt","DivAgt"]].resample("MS").mean()
### Plot
fig, axes = plt.subplots(nrows=4, sharex=True)
axes = axes.flatten()
x = sim_Q_M.index
axes[0].plot(x, sim_Q_M["DLLO"], label="$M_{gwlf}$")
axes[1].plot(x, sim_Q_M["WSLO"], label="$M_{gwlf}$")
axes[2].plot(x, sim_Q_M["ResAgt"], label="$M_{gwlf}$")
axes[3].plot(x, sim_Q_M["DivAgt"], label="$M_{gwlf}$")

axes[0].plot(x, obv_flow_data["DLLO"], ls="--", lw=1, color="black", label="Obv")
axes[1].plot(x, obv_flow_data["WSLO"], ls="--", lw=1, color="black", label="Obv")
axes[2].plot(x, obv_flow_data["ResAgt"], ls="--", lw=1, color="black", label="Obv")
axes[3].plot(x, obv_flow_data["DivAgt"], ls="--", lw=1, color="black", label="Obv")

axes[0].set_ylim([0,75])
axes[1].set_ylim([0,230])
axes[2].set_ylim([0,23])
axes[3].set_ylim([0,2])

axes[0].set_ylabel("DLLO\n($m^3/s$)")
axes[1].set_ylabel("WSLO\n($m^3/s$)")
axes[2].set_ylabel("Release\n($m^3/s$)")
axes[3].set_ylabel("Diversion\n($m^3/s$)")

axes[0].axvline(pd.to_datetime("2006-1-1"), color="grey", ls="-", lw=1)
axes[1].axvline(pd.to_datetime("2006-1-1"), color="grey", ls="-", lw=1)
axes[2].axvline(pd.to_datetime("2006-1-1"), color="grey", ls="-", lw=1)
axes[3].axvline(pd.to_datetime("2006-1-1"), color="grey", ls="-", lw=1)

axes[0].legend(ncol=3, bbox_to_anchor=(1, 1.5), fontsize=9)

fig.align_ylabels(axes)

Data collector

A data collector is a container object created by HydroCNHS in each simulation that can be passed around HydroCNHS and user-defined ABM modules. A data collector object can store dictionaries and lists. Each of the items is associated with a unique field name. Each field has properties including data, data type (e.g., a dictionary or a list), description, and unit. We provide some usage examples below.

First, we manually create a data collector object for demonstration.

import HydroCNHS

### Create a data collector object
dc = HydroCNHS.Data_collector()

Then, we add two fields, “field_1” and “field 2”, with corresponding field information to the collector. Spaces are not allowed here, and the code will convert “field 2” to “field_2”.

To read the data in a data collector, e.g., reading field_2, we may do the following:

### Read data
dc.field_2
# Out[0]: [1, 2, 3]

We can also create a shortcut for accessing a field by the following command, in which any modifications on the shortcut will be passed into the data collector object.

### Get a field shortcut
shortcut = dc.get_field("field_1", copy=False)
shortcut["new_key"] = "new value"
dc.field_1
# Out[0]: {'new_key': 'new value'}

If we want to get a copy of a field (not a shortcut), we must assign “True” to the “copy” argument.

### Get a copy of a field
copied = dc.get_field("field_1", copy=True)
copied["new_key2"] = "new value2"
dc.field_1
# Out[0]: {'new_key': 'new value'}
print(copied)
# {'new_key': 'new value', 'new_key2': 'new value2'}

We can also delete a field using the following commands.

### Delete a field
dc.del_field("field_1")
dc.list_fields()
# field_2
#   type
#     <class 'list'>
#   desc
#     None
#   unit
#     cm

Finally, users can export the entire data collector to a dictionary.

### Export the entire data collector to a dictionary
dictionary = dc.get_dict(copy=True)
print(dictionary)
# {'field_info': {'field_2': {'type': <class 'list'>, 'desc': None, 'unit': 'cm'}},
#  'field_2': [1, 2, 3]}

Calibration

HydroCNHS is equipped with a genetic algorithm package powered by Distributed Evolutionary Algorithms in Python (DEAP), which can be used for calibration in parallel. HydroCNHS is unique in that aspect that both the parameters of the hydrological model and user-defined ABM can be calibrated simultaneously as long as they are defined in the model configuration file (.yaml). Furthermore, users are allowed to assign initial guesses to the algorithm.

Calibration with the genetic algorithm (GA)

An evaluation function, a calibration input dictionary, and a GA configuration dictionary are required to use the GA calibration module supported by the HydroCNHS.

Evaluation function

Assuming we want to maximize \(y=-x_1^2+5x_1-x_2\) with \(x_1,x_2 \in [-5,5]\), we can design the evaluation function as:

import os
import HydroCNHS
import HydroCNHS.calibration as cali
prj_path, this_filename = os.path.split(__file__)

def evaluation(individual, info):
    x = individual
    fitness = -x[0]**2 + 5*x[0] - x[1]
    return (fitness,)

    cali_wd, current_generation, ith_individual, formatter, _ = info
    name = "{}-{}".format(current_generation, ith_individual)

    ##### individual -> model
    # Convert 1D array to a list of dataframes.
    df_list = cali.Convertor.to_df_list(individual, formatter)
    # Feed dataframes in df_list to model dictionary.

The evaluation must have “individual” and “info” arguments. Also, the return fitness value has to be of tuple format, e.g., (fitness,). The “info” contains additional information for users to design a more complex evaluation, as shown in “Build a hydrological model” and “Integrate an ABM” sections. This “info” variable contains information, including 1. working directory to the folder of this calibration experiment (cali_wd), 2. current generation of the GA run (current_generation), 3. index of the individual in the current generation (ith_individual), 4. formatter (formatter) for a converter, and 5. NumPy random number generator (rn_gen).

cali_wd, current_generation, ith_individual, formatter, rn_gen = info

cali_wd, current_generation, and ith_individual can be used to create sub-folders or output files for each evaluation. This is especially useful for calibrating a large model that requires a long simulation time. The formatter contains information to convert 1-D array individuals back to the original parameter format (e.g., a list of DataFrames) through a converter object. We will introduce the Converter in the next section. rn_gen is recommended to be used to generate random numbers (e.g., input to a HydroCNHS object like model = HydroCNHS.Model(model, name, rn_gen)). By applying rn_gen, HydroCNHS can guarantee the reproducibility of the numerical experiment with a given random seed.

Calibration input dictionary

A calibration input dictionary contains three keys, including par_name, par_bound, and wd. par_name is a list of the parameter names, par_bound is a list of parameter bounds, and wd is the working directory for the calibration experiment. The calibration input dictionary for this example is shown below.

# Calibration inputs
cali.get_inputs_template() # print an input template.

inputs = {'par_name': ['x1', 'x2'],
          'par_bound': [[-5, 5], [-5, 5]],
          'wd': 'working directory'}
inputs["wd"] = prj_path

Note that users can use a converter to generate the calibration input dictionary automatically.

GA configuration dictionary

A GA configuration dictionary contains many control options. Please see the following code for the explanation of each option.

# GA configuration
cali.get_config_template()

config = {'min_or_max': 'max',      # maximize or minimize the evaluation function.
        'pop_size': 100,            # Size of the population.
        'num_ellite': 1,            # Number of ellites.
        'prob_cross': 0.5,          # Crossover probability for uniform crossover operator.
        'prob_mut': 0.15,           # Mutation probability of each parameter.
        'stochastic': False,        # Is the evaluation stochastic?
        'max_gen': 100,             # Maximum generation number.
        'sampling_method': 'LHC',   # Sampling method for the initial population.
        'drop_record': False,       # Whether to drop historical records to save space.
        'paral_cores': -1,          # Number of parallel cores. -1 means all available cores.
        'paral_verbose': 1,         # Higher value will output more console messages.
        'auto_save': True,          # If true, users may continue the run later on by loading the auto-save file.
        'print_level': 1,           # Control the number of generations before the printing summary of GA run.
        'plot': True}               # Plot to time series of the best fitnesses over a generation.

Finally, we can run the GA calibration.

# Run GA
rn_gen = HydroCNHS.create_rn_gen(seed=3)
ga = cali.GA_DEAP(evaluation, rn_gen)
ga.set(inputs, config, formatter=None, name="Cali_example")
ga.run()
ga.solution
# Out[0]: array([ 2.47745344, -4.96991833])

After the GA terminated, we got the solution \(x_1=2.4775\) and \(x_2=-4.9699\), in which the theoretical values are 2.5 and -5 for \(x_1\) and \(x_2\), respectively. The fitness plot (Fig. 7) and auto-saved file (GA_auto_save.pickle) will be stored in the GA working directory. This GA_auto_save.pickle can be loaded into the GA object and continue the calibration when algorithms encounter a sudden breakdown or continue the run with a larger “max_gen,” as shown below.

# Continue the run with larger "max_gen"
ga = cali.GA_DEAP(evaluation, rn_gen)
ga.load(os.path.join(prj_path, "Cali_example", "GA_auto_save.pickle"),
        max_gen=120)
ga.run()
# =====Generation 120=====
#   Elapsed time 00:00:05
#   Min -6.69464
#   Max 11.21948
#   Avg 10.99626
#   Std 1.77931

# GA done!

The fitness and within-population standard deviation plot.

Converter

The Converter is designed to convert a list of parameter DataFrames into a 1-D array for GA calibration and return a 1-D array to the original list of DataFrames. It can automatically exclude NaN values in DataFrames and fix parameters not intended to be calibrated. Also, a converter can generate the calibration input dictionary.

We use the following synthetic list of DataFrames as an example.

import numpy as np
import pandas as pd
import HydroCNHS
import HydroCNHS.calibration as cali

### Prepare testing data.
par_df1 = pd.DataFrame({"Subbasin1": [1000,1000,3], "Subbasin2": [4,5,6]},
                       index=["a", "b", "c"])
par_df2 = pd.DataFrame({"Agent1": [9,8,7], "Agent2": [6,5,None]},
                       index=["Par1", "Par2", "Par3"])

bound_df1 = pd.DataFrame({"Subbasin1": [[0,1000],[0,1000],[0,10]], "Subbasin2": [[0,10],[0,10],[0,10]]},
                         index=["a", "b", "c"])
bound_df2 = pd.DataFrame({"Agent1": [[0,10],[0,10],[0,10]], "Agent2": [[0,10],[0,10],None]},
                         index=["Par1", "Par2", "Par3"])

df_list = [par_df1, par_df2]
par_bound_df_list = [bound_df1, bound_df2]

Now, we want to generate the calibration input dictionary with fixed “a” and “b” parameters for “Subbasin1.” We can do the following:

### Create an object called Converter.
converter = cali.Convertor()

### Generate GA inputs with fixed a & b parameters for Subbasin1.
fixed_par_list = [[(["a","b"], ["Subbasin1"])], []]
cali_inputs = converter.gen_cali_inputs(
    "working directory", df_list, par_bound_df_list, fixed_par_list)
### Get formatter
formatter = converter.formatter

### Show cali_inputs
print(cali_inputs)
r"""
{'wd': 'working directory',
'par_name': ['a|Subbasin2', 'b|Subbasin2', 'c|Subbasin1', 'c|Subbasin2',
            'Par1|Agent1', 'Par1|Agent2', 'Par2|Agent1', 'Par2|Agent2',
            'Par3|Agent1'],
'par_bound': [[0, 10], [0, 10], [0, 10], [0, 10], [0, 10], [0, 10], [0, 10],
            [0, 10], [0, 10]]}
"""

We can see the ‘par_name’ in cali_inputs does not contain ‘a|Subbasin1’ and ‘b|Subbasin1.’ The outputted a formatter contains the relationship between a 1-D array and a list of DataFrames.

Convert from a list of DataFrames to a 1D array

### to 1D array
converter.to_1D_array(df_list, formatter)
r"""
# Out[31]: array([4., 5., 3., 6., 9., 6., 8., 5., 7.])
# Note the order of the array corresponds to "par_name" in the cali_inputs.
"""

Convert from a 1D array to the original list of DataFrames

### to df_list
var_array = np.array([5]*9)
converter.to_df_list(var_array, formatter)
r"""
Out[46]:
[   Subbasin1  Subbasin2
a     1000.0        5.0
b     1000.0        5.0
c        5.0        5.0,
    Agent1  Agent2
Par1     5.0     5.0
Par2     5.0     5.0
Par3     5.0     NaN]
"""

Advanced ABM coding tips

In this section, we provide some coding tips for ABM module designs.

Enhancing computational speed

Avoid extensive calls to DataFrame

Reading data to a DataFrame (e.g., df.loc[ , ]) tends to be slow. We suggest users use NumPy array, list, or dictionary for calculations or data storage.

Avoid repeated loading of external data

If there is common data among multiple agent-type classes, we suggest loading the data once to a global variable at the top of the ABM module and using the variable across classes. This might save some time from repeated loading of external data inside each class (e.g., at def __init__(self)).

Avoid extensive deepcopy

deepcopy is a function to create a copy with a different storage address (not just copy a pointer that points to the same storage address). Therefore, it will take a longer time to complete the task. We suggest using deepcopy only when it is necessary.

Avoid storing redundant data in a data collector

A data collector is a data container object to store model outputs. We encourage users to utilize it to store agents’ results; however, users will need to consider the storage capacity of their computing devices, especially with a considerable number of agents. Overusing the computer storage might also slow down the computational speed.

Logging

Logging is a python package to organize model output messages. We encourage users to adopt it in their ABM module design. This will help you to integrate your agent output messages into HydroCNHS.

import logging
logger = logging.getLogger("ABM")
logger.info(<general information message>)
logger.error(<error message>)

HydroCNHS package

HydroCNHS.abm module

class HydroCNHS.abm.Base(**kwargs)

Bases: object

Agent_type class’s available items: * name: agent’s name. * config: agent’s configuration dictionary the model file (.yaml). * start_date: datetime object. * data_length: length of the simulation. * data_collector: a container to store simulated data. * rn_gen: random number generator to ensure reproducibility (e.g., * self.rn_gen.random()). Note that do NOT set a global random seed in * this module! All type of random number should be created by “rn_gen.” * dm: decision making object if assigned in the model file (.yaml).

Decision-making class’s available items:

• start_date: datetime object.

• data_length: length of the simulation.

• abm: the ABM configuration dictionary from the model file (.yaml).

• data_collector: a container to store simulated data.

• rn_gen: random number generator to ensure reproducibility (e.g.,

• self.rn_gen.random()). Note that do NOT set a global random seed in

• this module! All type of random number should be created by “rn_gen.

class HydroCNHS.abm.agent

Bases: object

HydroCNHS.abm.read_factor(ag_config, outlet)

Read factor from agent’s ag_config at a given outlet.

The function will automatically parse the list-like value assignment. e.g., [ReturnFactor, 0, Plus].

Parameters

ag_configdict

Agent’s configuration.

outletstr

Outlet name.

HydroCNHS.abm_script module

HydroCNHS.abm_script.add_agt_class(agt_type)

HydroCNHS.abm_script.add_dm_class(dm_type, is_institution=False)

HydroCNHS.data_collector module

class HydroCNHS.data_collector.Data_collector

Bases: object

add_field(field, data_type={}, desc=None, unit=None, check_exist=True)

Add a field to the data collector.

A field can be a dictionary or a list. A Data_collector object cannot have duplicated field name.

Parameters

fieldstr

Field name. Cannot have space in a field name.

data_typedict, optional

Data type of the field (e.g., {} and []), by default {}. User can also populate the field by directly assigning data here.

descstr

Field description.

unit

Unit of the field.

check_existbool, optional

If Ture, check the given field name is not existed before adding, by default True.

del_field(field)

Delete a field from the data collector.

Parameters

fieldstr

Field name.

get_dict(copy=False)

Get data collector in dictionary format.

Note that if copy=False, any modification on a variable assigned with the returned dictionary will also affect the data stored in the data collector object.

Parameters

copystr

If true, a copy of dictionary will be returned, else a pointer will be returned. Default False.

Returns

dict

A dictionary contains all fields.

get_field(field, copy=False)

Get a field.

This function create a shortcut to access a field. Namely, changes of a local variable of get_field() will be accumulated back to the original data_collector. copy=Ture if a copy of a field is needed.

Parameters

fieldstr

Field name.

copybool

If true, create a copy of a field, which has seperate storage

pointer than the original data_collector. Otherwise, return a shortcut of a field to the original data_collector. Returns ——- Assigned field type.

list_fields()

Print available fields in the data collector.

HydroCNHS.hydrocnhs module

class HydroCNHS.hydrocnhs.Model(model, name=None, rn_gen=None, checked=False, parsed=False, log_filename=None)

Bases: object

get_model_object()

Get the model object in a dictionary form.

Returns

dict

model object dictionary.

load_weather_data(temp, prec, pet=None, outlets=[])

Load temperature, precipitation, and otential evapotranpiration data.

Parameters

tempdict

[degC] Daily mean temperature time series data (value) for each subbasin named by its outlet. E.g., {“subbasin1”:[…], “subbasin2”:[…]}

precdict

[cm] Daily precipitation time series data (value) for each subbasin named by its outlet. E.g., {“subbasin1”:[…], “subbasin2”:[…]}

petdict, optional

[cm] Daily potential evapotranpiration time series data (value) for each subbasin named by its outlet, by default None. E.g., {“subbasin1”:[…], “subbasin2”:[…]}

run(temp, prec, pet=None, assigned_Q={}, assigned_UH={}, disable=False)

Run HydroCNHS simulation.

Parameters

tempdict

[degC] Daily mean temperature.

precdict

[cm] Daily precipitation.

petdict, optional

[cm] Potential evapotranspiration, by default None. If none, pet is calculted by Hamon’s method.

assigned_Qdict, optional

[cms] If user want to manually assign Q for certain outlets {“outlet”: array}, by default {}.

assigned_UHdict, optional

If user want to manually assign UH (Lohmann) for certain outlet {“outlet”: array}, by default {}.

disablebool, optional

Disable display progress bar, by default False.

Returns

dict

A dictionary of flow time series.

HydroCNHS.indicators module

class HydroCNHS.indicators.Indicator

Bases: object

static CP(x_obv, y_sim, r_na=False)

Correlation of persistence.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

Correlation of persistence.

static KGE(x_obv, y_sim, r_na=True)

Kling–Gupta efficiency.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

Kling–Gupta efficiency.

static NSE(x_obv, y_sim, r_na=False)

Nash–Sutcliffe efficiency.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

Nash–Sutcliffe efficiency.

static RSR(x_obv, y_sim, r_na=False)

RMSE-observations standard deviation ratio.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

RMSE-observations standard deviation ratio.

static cal_indicator_df(x_obv, y_sim, index_name='value', indicators_list=None, r_na=True)

static iKGE(x_obv, y_sim, r_na=True)

Inverse Kling–Gupta efficiency.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

Inverse Kling–Gupta efficiency.

static iNSE(x_obv, y_sim, r_na=False)

Inverse Nash–Sutcliffe efficiency.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

Inverse Nash–Sutcliffe efficiency.

static r(x_obv, y_sim, r_na=True)

Correlation.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

r coefficient.

static r2(x_obv, y_sim, r_na=True)

Coefficient of determination.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

r2 coefficient.

static remove_na(x_obv, y_sim)

Remove nan in x_obv and y_sim.

This function makes sure there is no nan involves in the indicator calculation. If nan is detected, data points will be remove from x_obv and y_sim simultaneously.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

Returns

tuple

Updated (x_obv, y_sim)

static rmse(x_obv, y_sim, r_na=False)

Root mean square error.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

r_nabool, optional

Remove nan, by default True

Returns

float

Root mean square error.

HydroCNHS.model_builder module

class HydroCNHS.model_builder.ModelBuilder(wd)

Bases: object

add_agent(agt_type_class, agt_name, api, priority=1, link_dict={}, dm_class=None, par_dict={}, attr_dict={})

Add agent.

Parameters

agt_type_classstr

Assigned agent type class.

agt_namestr

Agent name.

apistr

The API to integrate the agent to the HydroCNHS. e.g., mb.Dam.

priorityint, optional

Priority of the agent if conflicts occur, by default 1.

link_dictdict, optional

Linkage dictionary, by default {}.

dm_classstr, optional

Assigned decision-making class, by default None

par_dictdict, optional

Parameter dictionary, by default {}

attr_dictdict, optional

Attribution dictionary, by default {}

add_institution(institution, instit_dm_class, agent_list)

Add a institution.

Parameters

institutionstr

Institution name.

instit_dm_classstr

Assigned institutional decision-making class.

agent_listlist

Agent member list of the institute.

gen_ABM_module_template()

Generate ABM module template based on the ABM setting.

help()

print_model(indentor='  ', level=1)

Print model to the console

Parameters

indentorstr, optional

Indentor, by default ” “.

levelint, optional

Print out level of a nested dictionary, by default 1.

set_ABM(abm_module_folder_path=None, abm_module_name='ABM_module.py')

Set up ABM

Parameters

abm_module_folder_pathstr, optional

Folder directory of ABM modules. It it is not given, working directory will be assigned, by default None.

abm_module_namestr, optional

The ABM module name, by default “ABM_module.py”

set_rainfall_runoff(outlet_list, area_list=None, lat_list=None, runoff_model='GWLF')

Set up RainfallRunoff.

Parameters

outlet_listlist

A list of subbasin outlet names.

area_listlist, optional

Area [ha] list corresponding to outlet_list, by default None.

lat_liststr, optional

Latitude [deg] list corresponding to outlet_list, by default None.

runoff_modelstr, optional

“GWLF” or “ABCD” or “Other”, by default None.

Note

If “Other” is selected for runoff_model, users must provide precalculated runoffs for each subbasin as an input to HydroCNHS.

set_routing_outlet(routing_outlet, upstream_outlet_list, instream_objects=[], flow_length_list=None, routing_model='Lohmann')

Set up a routing outlet.

Parameters

routing_outletstr

Name of routing outlet. routing_outlet should be one of outlets in RainfallRunoff.

upstream_outlet_listlist

A list of outlets or dam agents that contribute to the streamflow at routing_outlet.

instream_objectslist, optional

A list of instream objects’ names (i.e., dam agents), by default [].

flow_length_listlist, optional

A list of flow lengths. The order has to consist to the upstream_outlet_list.

routing_modellist, optional

Routing model, by default “Lohmann”.

set_water_system(start_date, end_date)

Set up WaterSystem.

Parameters

start_datestr

“yyyy/mm/dd”

end_datestr

“yyyy/mm/dd”

write_model_to_yaml(filename)

Output model configuration file (.yaml)

Parameters

filenamestr

Filename

HydroCNHS.routing module

HydroCNHS.routing.form_UH_Lohmann(inputs, routing_pars, force_ingrid_off=False)

Derive HRU’s UH at the (watershed) outlet.

Parameters

inputsdict

Inputs dictionary containing FlowLength [m] Travel distence of flow between two outlets [float] and InstreamControl [bool].

routing_parsdict

Four parameters for routing: GShape, GScale, Velo, Diff [float]

force_ingrid_offbool, optional

If True, then within subbasin routing will be forced to turn off, by default False.

HydroCNHS.routing.run_step_Lohmann(routing_outlet, routing, UH_Lohmann, Q, Q_runoff, t)

Calculate a single time step routing for a given routing_outlet at time t.

Parameters

routing_outletstr

routing outlet.

routingdict

Routing setting dictionary from model.yaml file.

UH_Lohmanndict

A dictionary containing pre-formed UHs.

Qdict

A dictionary containing newest routed flows.

Q_runoffdict

A dictionary containing newest unrouted flows without.

tint

Index of current time step [day].

Returns

float

Routed flow of routing_outlet at time t.

HydroCNHS.routing.run_step_Lohmann_convey(routing_outlet, routing, UH_Lohmann_convey, Q_convey, t)

Calculate a single time step conveying water routing for a given routing_outlet at time t.

Parameters

routing_outletstr

routing outlet.

routingdict

Routing setting dictionary from model.yaml file.

UH_Lohmann_conveydict

A dictionary containing pre-formed conveying UHs (i.e., no within subbasin routing).

Q_conveydict

A dictionary containing conveying water.

tint

Index of current time step [day].

Returns

float

Routed conveying flow of routing_outlet at time t.

HydroCNHS.util module

HydroCNHS.util.check_RainfallRunoff(model_dict)

HydroCNHS.util.check_WS(model_dict)

HydroCNHS.util.check_agent_in_routing(model_dict)

HydroCNHS.util.check_model(model_dict)

Check the model dictionary

Parameters

model_dictdict

Model dictionary

Returns

bool

True if pass the check.

HydroCNHS.util.create_rn_gen(seed)

Create random number generator based on numpy module.

Parameters

seedint

Random seed.

Returns

object

Random number generator.

HydroCNHS.util.dict_to_string(dictionary, indentor='  ', level=1)

Ture a dictionary into a printable string.

Parameters

dictionarydict

A dictionary.

indentorstr, optional

Indentor, by default ” “

Returns

str

String.

HydroCNHS.util.form_sim_seq(node_list, back_tracking_dict)

HydroCNHS.util.gen_default_bounds(model_dict, key_option=['Pars'])

Generate default parameter bounds in the format of a list of DataFrames.

Parameters

model_dictdict

Model dictionary.

key_optionlist, optional

Output sections (e.g., Pars, Inputs, Attributes), by default [“Pars”].

Returns

tuple

A list of parameter bound DataFrames, A list of DataFrame names.

HydroCNHS.util.list_callable_public_object(obj)

HydroCNHS.util.load_customized_module_to_class(Class, module_name, path)

Load classes and functions in a user defined module (.py) into a given Class.

Parameters

Classclass

A class to collect classes and functions in a given module.

module_namestr

filename.py or filename.

pathstr

Directory of filename.py.

HydroCNHS.util.load_df_to_model_dict(model_dict, df, section, key)

Load dataframe to model dictionary.

Note that the column and index names have to be identical as outputted from write_model_to_df() function.

Parameters

model_dictdict

Model dictionary.

dfDataFrame

Dataframe.

sectionstr

RainfallRunoff or Routing or ABM.

keystr

nputs or Pars or Attributes.

Returns

dict

Updated model_dict.

HydroCNHS.util.load_model(model, checked=False, parsed=False, print_summary=True)

Load model.yaml or model dictionary.

Parameters

modelstr/dict

Filename (e.g., model.yaml) or model dictionary.

checkedbool, optional

If True, no pre-check will be implemented, by default False.

parsedbool, optional

If True, model will not be re-parses, by default False.

print_summarybool, optional

Print the summary of loaded model, by default True.

Returns

dict

Model dictionary.

HydroCNHS.util.parse_model(model_dict, print_summary=True)

Parse model dictionary. Populate SystemParsedData.

Args:

model_dict (dict): Load from model.yaml.

Returns:

dict: model_dict

HydroCNHS.util.parse_sim_seq(model_dict, print_summary=True)

HydroCNHS.util.set_logging_config(log_filename=None)

Set up logging config.

Parameters

log_filenamestr, optional

Log filename, by default None

HydroCNHS.util.update_sim_seq_with_group(sim_seq, group, back_tracking_dict)

HydroCNHS.util.write_model(model_dict, filename, org_model=None)

Write model dictionary to .yaml file

If org_model is given, comments in the original file will be kept in the output model file.

Parameters

model_dictdict

Model dictionary.

filenamestr

Model configuration filename (e.g. model.yaml).

org_modelstr, optional

Original model name (e.g. org_model.yaml), by default None.

Note

Common error :

1. value type is numpy.float64 => convert it to float.

HydroCNHS.util.write_model_to_csv(folder_path, model_dict, key_option=['Pars'], prefix='')

Write model (dictionary) to csv files.

Parameters

folder_pathstr

Output folder directory.

model_dictdict

Model dictionary.

key_optionlist, optional

Output sections (e.g., Pars, Inputs, Attributes), by default [“Pars”].

prefixstr, optional

Prefix for the filenames, by default “”.

Returns

list

A list of filenames.

HydroCNHS.util.write_model_to_df(model_dict, key_option=['Pars'], prefix='')

Write model dictionary to dataframes given key options.

This function will convert model dictionary to dataframes according to user’s specified sections (key_option).

Parameters

model_dictdict

Model dictionary.

key_optionlist, optional

Output sections (e.g., Pars, Inputs, Attributes), by default [“Pars”].

prefixstr, optional

Prefix for the filenames, by default “”.

HydroCNHS.visual module

class HydroCNHS.visual.Visual

Bases: object

Collection of some plotting functions.

static plot_reg(x_obv, y_sim, title=None, xy_labal=None, same_xy_limit=True, return_reg_par=False, save_fig_path=None, show=True)

Plot regression.

Parameters

x_obvarray

Observation data.

y_simarray

Simulation data.

titlestr, optional

Title, by default None.

xy_laballist, optional

List of x and y labels, by default None.

same_xy_limitbool, optional

If True same limit will be applied to x and y axis, by default True.

return_reg_parbool, optional

If True, slope and interception will be return, by default False.

save_fig_pathstr, optional

If given, plot will be save as .png, by default None.

showbool, optional

If True, the plot will be shown in the console, by default True.

Returns

ax or list

axis object or [slope, intercept].

static plot_simple_ts(df, title=None, xy_labal=None, data_dots=True, save_fig_path=None, **kwargs)

Plot timeseries.

Parameters

dfDataFrame

Dataframe.

titlestr, optional

Title, by default None.

xy_laballist, optional

List of x and y labels, by default None.

data_dotsbool, optional

If Ture, show data marker, by default True.

save_fig_pathstr, optional

If given, plot will be save as .png, by default None.

Returns

object

axis object.

static plot_timeseries(x_obv, y_sim, xticks=None, title=None, xy_labal=None, save_fig_path=None, legend=True, show=True, **kwargs)

Plot timeseries.

This function can plot two DataFrames with same column names.

Parameters

x_obvarray/DataFrame

Observation data.

y_simarray/DataFrame

Simulation data.

xtickslist, optional

Ticks for x-axis, by default None.

titlestr, optional

Title, by default None.

xy_laballist, optional

List of x and y labels, by default None.

save_fig_pathstr, optional

If given, plot will be save as .png, by default None.

legendbool, optional

If True, plot legend, by default None.

showbool, optional

If True, the plot will be shown in the console, by default True.

kwargsoptional

Other keywords for matplotlib.

Returns

object

axis object.

Module contents

Subpackages

HydroCNHS.calibration package

Submodules

HydroCNHS.calibration.convertor module

class HydroCNHS.calibration.convertor.Convertor

Bases: object

gen_cali_inputs(wd, df_list, par_bound_df_list, fixed_par_list=None)

Generate inputs dictionary required for calibration.

Parameters

wdstr

Working directory.

df_listlist

A list of parameter dataframes. Dataframe index is parameter names.

par_bound_df_listlist

A list of parameter bound dataframes.

fixed_par_listlist, optional

A list contains a list of tuples of fixed parameter location like [([“CN2”], [“outlet1”, “outlet2”]), ([“b”], [“outlet1”])], by default None. The number of tuples in the list is corresponding to the number of dataframes.

Returns

dict

Input for calibration.

gen_formatter(df_list, fixed_par_list=None)

Generate formatter for given list of dataframe objects. This is already included in gen_cali_inputs().

Parameters

df_listlist

A list of parameter dataframes. Dataframe index is parameter names.

fixed_par_listlist, optional

A list contains a list of tuples of fixed parameter location. For example, we want to fix par1 of outlet1 & 2 and all par2 in dataframe 1 and not to fix df2. We can input the following list, [[([“par1”], [“outlet1”, “outlet2”]), ([“par2”], [“:”])], []]. fixed_par_list is by default None. The number of lists in the list is corresponding to the number of dataframes.

Returns

dict

formatter.

static to_1D_array(df_list, formatter)

Convert a list of dataframes to a 1D array.

Parameters

df_listlist

A list of parameter dataframes. Dataframe index is parameter names.

formatterdict

Formatter, generated by gen_formatter or gen_cali_inputs. It is stored in attributes of the Convertor object.

Returns

array

1D array.

static to_df_list(var_array, formatter)

Convert 1D array back to a list of dataframes.

Parameters

var_arrayarray

1D array.

formatterdict

Formatter, generated by gen_formatter or gen_cali_inputs. It is stored in attributes of the Convertor object.

Returns

list

A list of dataframes.

HydroCNHS.calibration.ga_deap module

class HydroCNHS.calibration.ga_deap.GA_DEAP(evaluation_func, rn_gen=None)

Bases: object

auto_save()

find_best_and_record(pop)

gen_rn_gens(seed, size)

load(GA_auto_save_file, max_gen=None)

Load save pickle file (i.e., continue previous run).

Parameters

GA_auto_save_filestr

Filename.

max_genint, optional

This allow user to increase max_gen for continuing calibration for a longer searching, by default None.

run(guess_pop=None)

Run calibration.

Parameters

guess_pop2darray, optional

Assigned initial guesses, by default None. guess_pop has the size = [number of guesses, number of parameters]

run_individual(individual='best', name='best')

Run the evaluation for a given individual.

Warning! run_individual() does not generantee the same rn_gen will be assign to the evaluation, but the same one will be used for run_individual()

Parameters

individual1darray, optional

Individual or solution, by default “best”.

namestr, optional

This will be sent to the evaluation function through info = (cali_wd, name, name, formatter, rn_gen), by default “best”.

set(inputs, config, formatter=None, name='Calibration')

Setup the GA calibration.

Parameters

inputsdict

Calibration input dictionary generated by Convertor. Or, get the template by calling get_inputs_template().

configdict

Calibration configuration dictionary. Get the template by calling get_config_template().

formatterdict, optional

Formatter generated by Convertor, by default None.

namestr, optional

Name of the calibration, by default “Calibration”.

HydroCNHS.calibration.ga_deap.cxUniform(ind1, ind2, indpb, rn_gen_gen)

HydroCNHS.calibration.ga_deap.descale(individual, bound_scale, lower_bound)

individual is 1d ndarray.

HydroCNHS.calibration.ga_deap.gen_init_pop(creator, size, method='LHC', guess_pop=None, rn_gen_gen=None)

HydroCNHS.calibration.ga_deap.logger = <Logger HydroCNHS.GA (INFO)>

inputs = {“par_name”: [“a”,”b”,”c”],

“par_bound”: [[0,1],[0,1],[0,1]], “wd”: r””}

config = {“min_or_max”: “min”,

“pop_size”: 100, “num_ellite”: 1, “prob_cross”: 0.5, “prob_mut”: 0.1, “stochastic”: False, “max_gen”: 100, “sampling_method”: “LHC”, “drop_record”: False, “paral_cores”: -1, “paral_verbose”: 1, “auto_save”: True, “print_level”: 10, “plot”: True }

HydroCNHS.calibration.ga_deap.mut_middle(individual, p1, p2, prob_mut, rn_gen_gen)

HydroCNHS.calibration.ga_deap.mut_uniform(individual, prob_mut, rn_gen_gen)

HydroCNHS.calibration.ga_deap.sample_by_LHC(size, rn_gen_gen)

HydroCNHS.calibration.ga_deap.sample_by_MC(size, rn_gen_gen)

HydroCNHS.calibration.ga_deap.scale(individual, bound_scale, lower_bound)

individual is 1d ndarray.

HydroCNHS.calibration.ga_deap.selRoulette(individuals, k, rn_gen_gen, fit_attr='fitness')

Module contents

HydroCNHS.calibration.get_config_template()

HydroCNHS.calibration.get_inputs_template()

HydroCNHS.calibration.helper()

HydroCNHS.rainfall_runoff_model package

Submodules

HydroCNHS.rainfall_runoff_model.abcd module

HydroCNHS.rainfall_runoff_model.abcd.run_ABCD(pars, inputs, temp, prec, pet, data_length, **kwargs)

ABCD rainfall-runoff model.

Inputs:

Area: [ha] Subbasin area. XL: [cm] Initial saturated soil water content. Latitude: [deg] SnowS: [cm] Snow storage.

Pars:

a: Controls the amount of runoff and recharge during unsaturated soil. b: Controls Saturation level of the soils. c: Ratio of groundwater recharge to runoff. d: Controls groundwater discharge rate. Df: [cm/degC] Degree-day coefficient

Parameters

parsdict

Parameter dictionary containing 5 parameters: a, b, c, d, Df.

inputsdict

Input dictionary containing 4 inputs: Area, Latitude, XL, SnowS.

temparray

[degC] Daily mean temperature.

precarray

[cm] Daily precipitation.

petarray

[cm] Daily potential evaportranspiration.

data_lengthint

Total data length (i.e., simulation period).

Returns

array

[cms] Discharge

HydroCNHS.rainfall_runoff_model.gwlf module

HydroCNHS.rainfall_runoff_model.gwlf.run_GWLF(pars, inputs, temp, prec, pet, start_date, data_length, **kwargs)

GWLF rainfall-runoff model.

Inputs:

Area: [ha] Subbasin area. Latitude: [deg] S0: [cm] Shallow saturated soil water content. U0: [cm] Unsaturated soil water content. SnowS: [cm] Snow storage.

Pars:

CN2: Curve number. IS: Interception coefficient. Res: Recession coefficient. Sep: Deep seepage coefficient. Alpha: Baseflow coefficient. Beta: Percolation coefficient. Ur: [cm] Available/Soil water capacity. Kc: Land cover coefficient. Df: [cm/degC] Degree-day coefficient.

Note that the simulation period has to be longer than a month.

Parameters

parsdict

Parameter dictionary containing 9 parameters: CN2, IS, Res, Sep, Alpha, Beta, Ur, Kc, Df.

inputsdict

Input dictionary containing 5 inputs: Area, Latitude, S0, U0, SnowS.

temparray

[degC] Daily mean temperature.

precarray

[cm] Daily precipitation.

petarray

[cm] Daily potential evaportranspiration.

start_date: str

Start date “yyyy/mm/dd”.

data_lengthint

Total data length (i.e., simulation period).

Returns

array

[cms] Discharge

HydroCNHS.rainfall_runoff_model.pet_hamon module

HydroCNHS.rainfall_runoff_model.pet_hamon.cal_pet_Hamon(temp, Lat, start_date, dz=None)

Calculate potential evapotranspiration (pet) with Hamon (1961) equation.

Parameters

temparray

[degC] Daily mean temperature.

Latfloat

[deg] Latitude.

start_datestr

Start date “yyyy/mm/dd”.

dzfloat, optional

[m] Altitude temperature adjustment, by default None.

Returns

array

[cm/day] Potential evapotranspiration

Module contents

How to cite?

Lin, C. Y., Yang, Y. C. E., & Wi S. (2022). HydroCNHS: A Python Package of Hydrological Model for Coupled Natural Human Systems, Journal of Water Resources Planning and Management, 148(12), 6022005.

Related studies

Lin, C. Y., & Yang, Y. E. (2022). The effects of model complexity on model output uncertainty in co-evolved coupled natural-human systems. Earth’s Future, e2021EF002403.

Indices and tables

• Index

• Module Index

• Search Page