A rule-of-thumb for chi-squared tests in systematic reviews


A chi-squared test with few observations is not a super powerful statistical test (note, apparently termed both chi-square and chi-squared test depending on the discipline and source). Nonetheless, this test useful in systematic reviews to confirm whether observed patterns in the frequency of study of a particular dimension for a topic are statistically different (at least according to about 4/10 referees I have encountered). Not as a vote-counting tool but as a means for the referees and readers of the review to assess whether the counts of approaches, places, species, or some measure used in set of primary studies differed. The mistaken rule-of-thumb is that <5 counts per cell violates the assumptions of chi-squared test. However, this intriguing post reminds that it is not the observed value but the expected value that must be at least 5 (blog post on topic and statistical article describing assumption). I propose that this a reasonable and logical rule-of-thumb for some forms of scientific synthesis such as systematic reviews exploring patterns of research within a set of studies – not the strength of evidence or effect sizes.

An appropriate rule-of-thumb for when you should report a chi-squared test statistic in a systematic review is thus as follows.

When doing a systematic review that includes quantitative summaries of frequencies of various study dimensions, the total sample size of studies summarized (dividend) divided by the potential number of differences in the specific level tested (divisor) should be at least 5 (quotient). You are simply calculating whether the expected values can even reach 5 given your set of studies and the categorical analysis of the frequency of a specific study dimension for the study set applied during your review process.

total number of studies/number of levels contrasted for specific study set dimension >= 5

[In R, I used nrow(main dataframe)/nrow(frequency dataframe for dimension); however, it was a bit clunky. You could use the ‘length’ function or write a new function and use a ‘for loop’ for all factors you are likely to test].

Statistical assumptions aside, it is also reasonable to propose that a practical rule-of-thumb for literature syntheses (systematic reviews and meta-analyses) requires at least 5 studies completed that test each specific level of the factor or attribute summarized.


For example, my colleagues and I were recently doing a systematic review that captured a total of 49 independent primary studies (GitHub repo). We wanted to report frequencies that the specific topic differed in how it was tested by the specific hypothesis (as listed by primary authors), and there were a total of 7 different hypotheses tested within this set of studies.  The division rule-of-thumb for statistical reporting in a review was applied, 49/7 = 7, so we elected to report a chi-squared test in the Results of the manuscript.  Other interesting dimensions of study for the topic had many more levels such as country of study or taxa and violated this rule. In these instances, we simply reported the frequencies in the Results that these aspects were studied without supporting statistics (or we used much simpler classification strategies). A systematic review is a form of formalized synthesis in ecology, and these syntheses typically do not include effect size measure estimates in ecology (other disciplines use the term systematic review interchangeably with meta-analysis, we do not do so in ecology). For these more descriptive review formats, this rule seems appropriate for describing differences in the synthesis of a set studies topologically, i.e. summarizing information about the set of studies, like the meta-data of the data but not the primary data (here is the GitHub repo we used for the specific systematic review that lead to this rule for our team). This fuzzy rule lead to a more interesting general insight. An overly detailed approach to the synthesis of a set of studies likely defeats the purpose of the synthesis.

Rules of thumb for better #openscience and transparent #collaboration

Rules-of-thumb for reuse of data and plots
1. If you use unpublished data from someone else, even if they are done with it, invite them to be a co-author.
2. If you use a published dataset, at the minimum contact authors, and depending on the purpose of the reuse, consider inviting them to become a co-author. Check licensing.
3. If you use plots initiated by another but in a significantly different way/for a novel purpose, invite them to be co-author (within a reasonable timeframe).
4. If you reuse the experimental plots for the exact same purpose, offer the person that set it up ‘right of first refusal’ as first author (within a fair period of time such as 1-2 years, see next rule).
5. If adding the same data to an experiment, first authorship can shift to more recent researchers that do significant work because the purpose shifts from short to long-term ecology.  Prof Turkington (my PhD mentor) used this model for his Kluane plots.  He surveyed for many years and always invited primary researchers to be co-authors but not first.  They often declined after a few years.
6. Set a reasonable authorship embargo to give researchers that have graduated/changed focus of profession a generous chance to be first authors on papers.  This can vary from 8 months to a year or more depending on how critical it is to share the research publicly.  Development pressures, climate change, and extinctions wait for no one sadly.
Rules-of-thumb for collaborative writing
1. Write first draft.
2. Share this draft with all potential first authors so that they can see what they would be joining.
3. Offer co-authorship to everyone that appropriately contributed at this juncture and populate the authorship list as firmly as possible.
4. Potential co-authors are invited to refuse authorship but err on the side of generosity with invitations.
5. Do revisions in serial not parallel.  The story and flow gets unduly challenging for everyone when track changes are layered.

Cuyama Valley Micronet project 2016

With SB county, California, global change will certainly impact coastal zones. However, dryland ecosystems will also be subject to significant change. To examine the importance of shrubs as a buffer to some of these processes, we are deploying a micro-environmental network to measure the amplitude of several key processes. It turned out to be a real gem of a study site.

So much beauty just over the hill away from the roads.










Celebrate #ESA100 & promote #openscience in ecology though synthesis by publishing your synthesis datasets. #ecosynthesis


Summarizing 100 years of ecology and looking forward should incorporate formal synthesis tools. In the spirit of promoting these efforts, for better or worse, I pulled together all the synthesis datasets I have collaborated in building and published any outstanding ones online this week.

I discovered the meta-data we keep for our derived datasets is ‘less than optimal’, that there are some similarities across synthesis datasets (particularly meta-analyses), and that as a rule of thumb figshare or oneshare are great spots for these type of data.  I realize that primary data on ecological systems absolutely needs formal meta-data and should be published in repositories with structured meta-data such as knb, but derived data can still likely have utility in other repositories. Gigascience and Scientific Data are also great homes for more complete data packages.


gif from murally

Published, representative synthesis datasets
Here are all synthesis datasets published to date.  I have only one left to dig up, clean up, and formalize before publication.

A meta-analytic dataset of plant facilitation in coastal dune systems: responses, regions, and research gaps.

Tree invasions dataset: a comparative test of the dominant hypotheses and functional traits

A meta-analysis of the ecological significance of density in tree invasion

The summary data for a review of the relationship with pollen limitation of plant reproduction

Dataset for the diversity of diversity studies: retrospectives and future directions

The relative success of studies of plant facilitation 2009

The dataset for a systematic review of the attractant-decoy and repellent- plant hypotheses: do plants with heterospecific neighbours escape herbivory?

Dataset examining functional assessment of animals with plant facilitation complexes

Dataset for A systematic review and conceptual framework for the mechanistic pathways of nurse plants

Dataset for Land management trumps the effects of climate change and elevated CO2 on grassland functioning

A systematic review of the ecological literature on cushion plants

A systematic review of arthropod community diversity in association with invasive plants

Indirect interactions in terrestrial plant communities: emerging patterns and research gaps Dataset

As a community, I would love to see the other synthesis datasets out there too. I have found quite a few but they are often in the form of online supplements associated with standard publications. There could be some really neat connections across meta-analyses between conservation, ecology, and different taxa.

If you have derived, synthesis datasets published (and done all that work to aggregate independent data), please publish then share them with the tag #ecosynthesis. If you do it leading up the ESA annual meeting, use the tag #ESA100 too and folks can explore them at the meeting!


Changing interactions in a changing world for #ESA2015 & #oe3c

ESA: The future of gradient studies in examining plant-plant interactions for the next 100 years.

That is the crazy ambitious title for my ESA2015 talk.

Instead, I prefer the following running title for the more synthetic version of the talk I will be giving at the oe3c annual meeting.

Changing interactions in a changing world.

Times are a changing. Global changes are real, dramatic, and prominent in ecological research. Even most fundamental research studies on interactions, gradients, or perturbation invoke global change issues as the validation and implication of the respective work reported. However, integrating individual studies is challenging, and ecology must now very rapidly move beyond context specificity to provide useful, reproducible evidence for many of these global issues. Gradients are a changing too in connectance, length, and severity. Herein, a review and conceptual framework of gradient studies that explore ecological interactions are developed. Experimental field manipulations and syntheses are also presented as a means to advance theory and highlight opportunities for future research. Gradients are powerful tools that can be used to shape distributed, collaborative studies of interactions provided interaction estimates are coupled with drivers at multiple scales, network dynamics, and trophic levels. Contrasts of the frequency and/or importance of interactions, positive or negative, are only as useful as their capacity to expand the relevance of the local ecological context.




 1. Generate a framework to identify opportunities & gaps.

2. Find the best case studies from synthesis papers and primary field studies.

3. Connect the dots and ascertain whether gradients are a viable opportunity or hindrance in our capacity to inform global change issues.

4. Link to space for time substitutions (historical/legacy ideas) and new tools such as SEMs.