The International Stormwater Best Management Practices (BMP) Database project website includes more than 600 BMP performance studies. This is a tremendous free resource for engineers, scientists, policymakers, and others who seek to characterize the pollutant removal and hydrologic performance of various storm water treatment systems; however, it is only as reliable as the information that goes into it. To improve reliability, the website also includes monitoring program design guidance and data input tools that allow a standardized format and content for summary reports. Whether looking at an individual test of a single treatment system or a database with hundreds of results, it is important to focus not only on what the data tells us, but also on what stories we may be missing.
Monitoring Matters
You have probably seen a presentation or read a field monitoring paper that includes this familiar storyline: A project moves from the active construction phase to the field monitoring stage only to discover there were obvious errors made during design or construction that needed to be fixed prior to monitoring. Once corrected, data is collected and shows less-than-ideal performance, and, as a result, changes are made or extra maintenance is performed until information consistent with the project goals is obtained. After this shakedown period, the “real” data collection period begins.
Monitored projects face a much higher level of scrutiny than non-monitored projects, and they often include extra repairs, renovations and maintenance. The incentive to get it right is powerful. It is easier to get a presentation accepted or to secure additional funding if there is a positive story to tell.
It is important to recognize that this storyline includes several forms of bias that compound to create storm water management program vulnerability. Reporting bias occurs when positive results are viewed more favorably or promoted more actively than negative results. Selection bias occurs when an exemplary BMP is chosen as a test subject rather than a typical BMP, when poor initial results are omitted, or when any extra BMP TLC given by the research team introduces bias, however well-intentioned.
Broadening Perspectives
At the American Society of Civil Engineers Environmental & Water Resources Institute conference in November 2017, along with the usual academics, consulting engineers and exhibitors in the crowd, there were many municipal storm water maintenance program managers. Their stories painted a different picture than the usual idealized story about how green infrastructure and low impact development BMPs are almost universally applicable, are easier and cheaper to construct and maintain, and offer improvements in storm water infrastructure robustness. After hearing so many stories of pesky gophers, floating mulch, invasive weeds, inferior material substitutions, construction phase failures, standing stagnant water and excessive litter control costs, I began to wonder if I had somehow ended up on the wrong side of the looking glass.
The truth spans the entire range from cost-effective and perfectly designed, constructed and operating BMPs to those that are abject failures. Because we are using BMP performance data to make decisions at the programmatic level, it seems that we need to broaden our perspective when we ask questions about BMP performance.
It is not enough to look at water quality or hydrologic performance of a particular BMP type without also asking how often these results are likely to be replicated by systems not being actively monitored and coaxed into providing optimal results. Given the myriad potential failure points in the project steps, we need to focus as much research effort on systematic performance as we do on individual performance. After all, if an on-specification BMP provides 80% load reduction but in practice is only on-spec 50% of the time, from a programmatic perspective, it is probably no better than a BMP that achieves only 50% load reduction but does it reliably.
