GivingTuesday and the Association of Fundraising Professionals came together in 2020 to improve upon the Fundraising Effectiveness Project, a quarterly benchmark report based on the world’s largest nonprofit fundraising dataset, the Growth in Giving Database. Through extensive research, we made several noteworthy discoveries and found a number of ways we can improve upon our quarterly benchmarks.
The volatility in 2020 charitable giving resulted in a highly anomalous year. While total giving was up, some cause areas, organizations, and donor categories saw significant gains while others suffered reductions. The sector-wide results were not the best indication of what was typical. In 2020, more than any other year, the average results for the sector, however estimated, fail to tell the real story. A deeper evaluation was necessary to understand the winners and losers within the trends and the micro-trends for causes, organizations, and donor behavior in 2020.
We discovered that we needed new methodology to better explain the volatility in 2020. We found that our previous methodology:
- Didn’t account for our data over-representing human services organizations (the one sector that dramatically grew)
- Lacked visibility to small organization growth/contraction.
- Represented contraction for other organizations conservatively, by selecting for highly stable orgs (with at least 5 consecutive years of data).
- Used sampling that resulted in a smaller, less representative dataset and therefore less robust to outliers.
- Lacked outlier detection.
The modifications of our methodology led to these changes in top-line metrics:
- Reduced the dollar growth (10.3% would change to 7.2%)
- Reduction in donor growth (8.1% would change to 4.9%)
- No change in donor retention (-3.3% would stay at -3.3%)
Most of the impact on reported results is from handling outliers and overly limiting our dataset size; the other issues mostly contribute to data variability. We have begun to handle data variability in Q1 by reporting estimates of dollars, donors, and retention, as some organizations are slow to report Q1 data.
Applying our new methodology to 2021 data, as well as all historical data appearing in 2021 reports, gives us a broader representation of the sector as a whole, as well as more visibility into the “typical” organization fundraising experience. This new approach also allows us to contextualize results by organization size and cause. Finally, these changes will make ongoing reporting more resilient to volatility in the future.
For nonprofit leaders, our changes ensure greater visibility into how key factors of your organization – size and cause – impact the top-line dollar results of published reports. These new approaches also allow us to share median results, indicating the difference between the overall sector and the typical organization.
Methods Changes for 2021
In 2021, we’ve implemented a number of changes to address the key issues we identified and provide a better representation of the ‘typical’ experience of organizations. These changes include:
- Change from a stratified sampling to weighting. By weighting, we can use all of our data (instead of holding 10%+ out). The reason we sampled was to allow for organizations that leave the panel to be replaced by similar organizations. This type of event is rare in our panel organizations and was never handled in previous years. Now, by increasing our dataset size (by over 3x), we reduce the impact of any organization leaving the dataset.
- Clearly indicate the biases in our dataset. It is important to us that our users understand where our dataset is strongest. This means (a) ensuring that we’re clear in our messaging that our benchmark is biased towards stable organizations in our report and press releases, and (b) that we contextualize results by both org size and cause.
- Create panels for every year in the dataset, and use those panels to report on past year’s data. This allows us to reduce the minimum length of time organizations appear in the dataset (from 5 years to 3 years), expand the dataset, limiting how much we over-index on long-term stable organizations and ensuring we forever use the same organizations to report on each year’s results.
- Extend the organization size criteria to include more orgs. We went from only including the $100K to $10 million to now include $5K – $25 million. Even by adding organizations at the lower ends of our ability to represent the IRS dataset, we represent over twice as many filers as we did through previous methods.
- Implement outlier detection and elimination throughout the year. We will look out for organizations that are driving topline results disproportionately. If we detect such organizations, we’ll review whether to include or eliminate them in 2021 reporting.
Eager to dive into the technical details? Get more information about these methods and reporting changes.
The Quarterly Report can be accessed via our online dashboard: https://data.givingtuesday.org/fep-report/
