top of page

Paper Summary
Impact Evaluation with Time Series

Paper Title

Evaluating Impact Using Time-Series Data

Year

2021

Author Team

Hannah S. Wauchope, Tatsuya Amano, Jonas Geldmann, Alison Johnston, Benno I. Simmons, William J. Sutherland, Julia P. G. Jones

Causal inference is becoming an increasingly hot topic in conservation

TT_1.png

This is a case of us getting entirely the wrong idea about a population trend from a data subset. If we had only surveyed the population in these 5 years we’d think it was doing brilliantly!

To try and figure out the chances of this happening, we took >27,000 long term waterbird population trends, and then took small chunks out of each one. We looked to see how often the chunks matched the longer term trend, like so:

TT2.gif

In this case the long term trend was negative. If a three year sample matched that, it was correct (tick), if it showed the opposite it was incorrect (cross) and if it showed an insignificant trend, then it was a “missed negative”, as in, we missed that a negative trend was occurring.

In this example, we were taking 3 year chunks from a 30 year trend. But we ran this with every possible iteration within that, e.g. a 10 year chunk from a 29 year trend, a 4 year chunk from a 10 year trend, a 7 year chunk from an 18 year trend etc etc, and found the number of times in each case that the chunk was likely to represent the full trend. And these are the results:

TT_Consecutive.jpg

The top right box shows cases where the sample and the long-term trend are “matching” (correct). As expected, the closer the sample and long-term trend are in length, the more likely the sample is to be correct. But even when the sample is quite close to the length of the long-term trend, it can still often be incorrect. E.g. if we have a 24 year sample of a 30 year long-term trend, it will only be correct 70-80% of the time. And a 10 year sample of a 30 year long-term trend will only be correct 50-60% of the time!

But that doesn’t tell the whole story, as the bottom left panel shows the amount that we get the opposite trend, and this almost never happens (<10% of the time). The middle two panels show what happens when the long-term trend is insignificant (not really going up or down), but the sample gives a significant positive or negative trend. We can see that this is pretty uncommon too. The final two panels show the situation where the long-term trend is positive or negative, but our sample is insignificant, and this happens a little more often.

The take home from all of this is that the when we have a significant trend from a sample, it is unlikely to be the opposite of the long-term trend, and also unlikely to be erroneously picking up a trend when the long-term trend is insignificant. So a significant sample trend is likely to be correct!

This has big implications, as it means that a significant trend from a sample of even just a few years is likely to be correct. So if we detect that a population is declining, we should act now!

In the rest of the paper we investigate a bunch of other scenarios that provide some useful info (for instance, what about if we compared the magnitude of the slope, or take samples in intervals etc). If you're interested, all relevant code is here.

Wild Landscape
bottom of page