The Difference Beteween Fraud and Farce, Reflux

... because I must reverse my previously stated positions on Cook et al. (2013)  (hereinafter, C13).  I realize that I can no longer in good conscience defend its design, nor its conclusions as stated.

Summary

Following are some issues that I have previously discounted, but which I now consider serious flaws.  Detailed discussion of each, along with suggestions for improvements/alternatives, are in sections below the break:

1. AGW is inconsistently and therefore ambiguously defined across the eight endorsement categories.  As well, it is vaguely defined in several endorsement categories.
2. The paper reports results in the abstract and body by combining dissimilar AGW definitions into consolidated endorsement buckets, and nowhere reports statistics at the higher detail level of the original eight endorsement categories.

Update 3/27/2016

Brandon Shollenberger has published a reaction to this post here.  The punchline:

So Gates, you know that part where you made a huge fool of yourself by twisting into a pretzel to criticize me on points I was completely correct about? Yeah, suck it

Which hearkens back to a comment in this article:

No consensus? Confused about what "consensus" means? Suck it Shollenberger. At least one oil company grokked it in the early 1980s. Wake up.

Nothing about this ... episode ... doesn't suck for me.  Hence "reflux" not "redux" in the title of this article.  Like slightly bad fish for dinner, it keeps coming back up.  Looks like I picked the wrong week to stop sniffing glue.

AGW is ambiguously and/or vaguely defined

The body of the paper under section 1, Introduction, states:

We examined a large sample of the scientific literature on global CC, published over a 21 year period, in order to determine the level of scientific consensus that human activity is very likely causing most of the current GW (anthropogenic global warming, or AGW).

This is consistent with findings presented by the IPCC in AR5, WGI, Chapter 10, Detection and Attribution of Climate Change: from Global to Regional (p. 869, p. 3 of the .pdf):

Atmospheric Temperatures

More than half of the observed increase in global mean surface temperature (GMST) from 1951 to 2010 is very likely1 due to the observed anthropogenic increase in greenhouse gas (GHG) concentrations.

Even though C13's statement "human activity is very likely causing most of the current GW (anthropogenic global warming, or AGW)" is specific about the amount of warming due to human activities, it is not specific about trend time periods and does not explicitly define "warming" itself.

The IPCC statement is a superior operational definition because warming is defined as global mean surface temperature (GMST) and includes an unambiguous time period (1951-2010).

Both formulations use the term "very likely", which the IPCC defines as "90-100% probability" for AR5.  It seems reasonable to assume C13 uses "very likely" the same way, but it would have been best (and not difficult) to have given a quantified definition.

That C13 more broadly considers any human activity as a putative causal mechanism whereas the AR5 definition limits it to human-caused increased GHG concentrations is not something I consider an inappropriate choice.  My argument here is only that operational definitions should be internally consistent and precise as possible, not that they should conform to some ostensibly authoritative definition given by an organization such as the IPCC.

Moving down in the body of C13, we find:

Table 2. Definitions of each level of endorsement of AGW.

Level of endorsement
Description
Example

(1) Explicit endorsement with quantification
Explicitly states that humans are the primary cause of recent global warming
'The global warming during the 20th century is caused mainly by increasing greenhouse gas concentration especially since the late 1980s'

(2) Explicit endorsement without quantification
Explicitly states humans are causing global warming or refers to anthropogenic global warming/climate change as a known fact
'Emissions of a broad range of greenhouse gases of varying lifetimes contribute to global climate change'

(3) Implicit endorsement
Implies humans are causing global warming. E.g., research assumes greenhouse gas emissions cause warming without explicitly stating humans are the cause
'...carbon sequestration in soil is important for mitigating global climate change'

(4a) No position
Does not address or mention the cause of global warming
   
(4b) Uncertain     Expresses position that human's role on recent global warming is uncertain/undefined
'While the extent of human-induced global warming is inconclusive...'

(5) Implicit rejection
Implies humans have had a minimal impact on global warming without saying so explicitly E.g., proposing a natural mechanism is the main cause of global warming '...anywhere from a major portion to all of the warming of the 20th century could plausibly result from natural causes according to these results'

(6) Explicit rejection without quantification
Explicitly minimizes or rejects that humans are causing global warming
'...the global temperature record provides little support for the catastrophic view of the greenhouse effect'

(7) Explicit rejection with quantification
Explicitly states that humans are causing less than half of global warming
'The human contribution to the CO2 content in the atmosphere and the increase in temperature is negligible in comparison with other sources of carbon dioxide emission'

Here, category 1 uses the phrase, "humans are the primary cause of recent global warming", which is qualitatively different from C13's previous formulation, "human activity is very likely causing most of the current GW".

While it is arguable that the word "primary" often implies "most", in a complex system with multiple (often confounding) causal mechanisms, any single identifiable and quantifiable causal mechanism whose net percentage effect is greater than all others could be considered the primary causal mechanism.

As one point of contention in literature, which C13 attempts to address, is that human activity is not a dominant factor and/or accounts for <50% of observed warming trends over multi-decadal time periods, qualifiers such as "most" or quantified qualifiers, e.g. ">x%" seem more appropriate.


Endorsement categories 2 and 3 use the phrase "humans are causing global warming" which is even more vague than any previous definition of AGW.  Category 2 further muddies the AGW definition with the phrase, "refers to anthropogenic global warming/climate change as a known fact", which is circular.

Categories 5-7 appear intended to be the mirror opposites of 3-1 (reverse order intended).  However, as worded they are not the exact semantic opposites of their counterparts.  Not only do these nuanced differences give more "wiggle room" for subjective interpretation by reviewers and the intended audience, it is inherently confusing due to the additional complexity.

Strictly interpreted, category 3 is not mutually exclusive with its counterpart 5.  Were I to write, as in 3, "humans are causing global warming", it would be entirely valid for me to later argue that what actually intended to convey is that "humans are causing some global warming", or more specifically that "humans are causing <50% of global warming".

Noting that category 5 uses the phrase, "humans have had a minimal impact", and category 6, "minimizes or rejects that humans are causing", compounds the dissimilar definitions of AGW, these categories raise the question, "minimal compared to what?"

Ultimately, the definition of AGW used in C13 resolves to a confusing and self-contradictory circular non-definition, which is not satisfactory.

While such ambiguities and exclusions are normal in everyday communication, a scientific study should attempt to be more rigorous if it is to produce reliable results which inspire confidence in its findings.

I argue that a design such as the following is better for its precision, consistency and simplicity relative to C13:

More than half of the observed increase in global mean surface temperature (GMST) over the past 30 years is very likely (90-100%) due to human activity.

Level of agreement with the above statement:

1. Explicit endorsement with quantification
2. Explicit endorsement without quantification
3. Implicit endorsement without quantification
4. a) no position, b) too uncertain to determine
5. Implicit rejection without quantification
6. Explicit rejection without quantification
7. Explicit rejection with quantification
8. Not applicable (n/a) to this study

Note that the above example does not explicitly define the term AGW itself.  This is intentional because "climate literature" includes multiple human climatic AND other environmental impacts not exclusive to GMST.  As well, some individual studies cover more than one type of impact, often in one or more differing climate/environmental domains.

E.g., numerous papers are limited to regional scope, or discuss other metrics like sea surface temperature, vertically averaged ocean temperature/heat content, ocean pH, global or continental landed ice mass change, bulk upper air temperature trends, etc.  Thus reviewers were often required to make subjective decisions about which impact category the paper MOST represented.

Such conflicts could easily have been avoided by making each impact category self-contained with its own precise definition, thus allowing any given paper to be evaluated by its endorsement of any single or multiple impact categories as appropriate.

While such a scheme might seem to be more complex and unwieldy to administer and analyze, I argue that the additional complexity is at the benefit of more flexibility which allows for more robust results with richer and more complete descriptions of the state of climate literature over time.

One problem with the endorsement categories I propose above is that the list is not exhaustive of all possibilities.  For example, a common argument in the popular debate about human impacts on the environment/climate system is that warming is NOT happening.  Arguments range from lack of statistical observational significance, sparse data, poor analysis, models programmed to produce warming as a function of some or several dubious or physically impossible anthropogenic mechanisms, outright fraudulent data manipulation or some combination.

This could be handled by adding additional categories such as, "effect is not occurring" or "the opposite effect is occurring", etc.  Another mutually inclusive way to approach exhaustion would be to pose additional questions, e.g.:

Warming surface temperature (GMST) trends over the past 30 years are very likely (90-100%) overstated in land-based observational time series.

or:

Nearly all (95-100%) increased atmospheric CO2 concentration since 1850 is virtually certain (99-100%) to be due to human activity.

It is beyond the intended scope of this note to attempt recommending a more exhaustive study design, I only give examples to illustrate how future such studies might improve over C13.

Dissimilar AGW definitions reported as consolidated statistics

The C13 abstract reads in full:

We analyze the evolution of the scientific consensus on anthropogenic global warming (AGW) in the peer-reviewed scientific literature, examining 11 944 climate abstracts from 1991–2011 matching the topics 'global climate change' or 'global warming'. We find that 66.4% of abstracts expressed no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming. Among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming. In a second phase of this study, we invited authors to rate their own papers. Compared to abstract ratings, a smaller percentage of self-rated papers expressed no position on AGW (35.5%). Among self-rated papers expressing a position on AGW, 97.2% endorsed the consensus. For both abstract ratings and authors' self-ratings, the percentage of endorsements among papers expressing a position on AGW marginally increased over time. Our analysis indicates that the number of papers rejecting the consensus on AGW is a vanishingly small proportion of the published research.

The given definition for AGW is vague: "humans are causing global warming".  In the body of the paper, under section 2, Methodology we read:

We classified each abstract according to the type of research (category) and degree of endorsement. Written criteria were provided to raters for category (table 1) and level of endorsement of AGW (table 2). Explicit endorsements were divided into non-quantified (e.g., humans are contributing to global warming without quantifying the contribution) and quantified (e.g., humans are contributing more than 50% of global warming, consistent with the 2007 IPCC statement that most of the global warming since the mid-20th century is very likely due to the observed increase in anthropogenic greenhouse gas concentrations). 

Only the first endorsement category is obviously compatible with "humans are contributing more than 50% of global warming", and yet category 1 does not use the same wording when it very easily could have.  However, this is trivial critique compared to what we read about how the statistics as reported in the abstract were obtained:

To simplify the analysis, ratings were consolidated into three groups: endorsements (including implicit and explicit; categories 1–3 in table 2), no position (category 4) and rejections (including implicit and explicit; categories 5–7).

While I understand that reporting data in consolidated form tends to make for easier comprehension, I consider it actually more "difficult" to do consolidated statistical reporting than not.  In the case of the data gathered for C13, I consider doing either rather trivial.

However, in the case of how C13 inconsistently defined AGW, I consider it inappropriate for the endorsement categories to have been consolidated at all.  And more inappropriate that nowhere does C13 report statistics at the most granular level of the eight total endorsement categories.

I would consider the detailed reporting requirement even if the AGW definitions had been consistent because of the distinct qualitative difference between quantified and unquantified effects and explicit vs. implicit human causality.

Here I allow myself to note an irony -- were I to rate this paper's endorsement level solely on the content of the abstract, it would be:

(2) Explicit endorsement without quantification
Explicitly states humans are causing global warming or refers to anthropogenic global warming/climate change as a known fact

I could however just as easily rate C13 by its own definitions as ...

(4b) Uncertain     Expresses position that human's role on recent global warming is uncertain/undefined

... because the abstract ONLY says ...

Among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming.

... which does not clearly define what AGW means.  Why I would choose 2 over 4b has much to do with my own beliefs, NOT necessarily about what the C13 abstract actually SAYS.  I argue that how C13 was designed and executed may very well say more about what people like me already believed about AGW before reading the paper than it says about the state of what's written in climate literature itself.

As well, the C13 authors are known champions and defenders of "pro-AGW" climate literature who (rightfully so, in my opinion) advocate for policies designed to wean the world away from fossil fuels and toward less carbon-intensive alternatives.  However, I think a compelling argument can be made that it appears the way C13 was designed and executed was to quantify the authors' own opinions about what literature says rather than be a dispassionate review of literature findings.

Or more simply, that C13 was designed -- deliberately or subconsciously -- to conform to some preconceived and too-broadly defined notion of there being a literature consensus that the letter "A" should precede "GW".

Setting aside for a moment that C13 ambiguously and inconsistently defines AGW, let us return to the section of the abstract which discusses the results of the author self-ratings of their own papers:

Compared to abstract ratings, a smaller percentage of self-rated papers expressed no position on AGW (35.5%). Among self-rated papers expressing a position on AGW, 97.2% endorsed the consensus.

In tabular form, those statistics resolve to:

 62.7% support AGW
 35.5% no position
  1.8% reject AGW
------
100.0% total

Yet, when one visits The Consensus Project web page hosted by Skeptical Science, the "among self-rated papers expression a position on AGW" qualifier gets dropped, viz.:

The Cook et al. (2013) 97% consensus result is robust
Why the 97 per cent consensus on climate change still gets challenged
Why we care about the 97% expert consensus on human-caused global warming
Richard Tol accidentally confirms the 97% global warming consensus
How we discovered the 97% scientific consensus on man-made global warming
Skeptical Science Study Finds 97% Consensus on Human-Caused Global Warming in the Peer-Reviewed Literature 

I realize that, "97% of peer-reviewed climate papers taking a position (63% of all papers) endorse what we think most people mean by AGW", is rather wordy and awkward.  However, I think it represents a more honest portrayal of C13's findings.

Conclusions

I think C13 is a flawed and should not have been published in its present form.  While I would personally like to rely upon its results, I find that I cannot.  Nor can I continue to defend its methods or findings as I have done in the recent and more distant past.

Since I am effectively calling C13 a bad paper, this raises the obvious question about whether I think the journal (IOPScience, Environmental Research Letters) should retract it.  The facile answer to that question is that many "bad" papers are published and never retracted -- conventional wisdom holds that they simply don't get cited and fade into obscurity.  However, according to ERL's own statistics, C13 has already been cited 32 times, though at least two of those citations are in later works by one or more of C13's listed authors.

Given the impact on subsequent literature C13 has already achieved, I would be personally satisfied if the authors were to publish a corrigendum which did at least the following:

1. Explicitly defined AGW and consistently used that definition across all endorsement categories.
2. Contained data obtained by all-new ratings of papers using the modified and consistent AGW definition.  Here I include author self-ratings.
3. Published statistics at the most granular endorsement categories in addition to whatever consolidation scheme desired by the authors.
4. Compares the old results with the new results, again at the most granular level of endorsement level.

I would also suggest that the authors themselves do not participate in the rating process, and by extension, that none of the volunteers who previously rated abstracts for the original C13 publication be involved.  Same suggestion for author self-ratings, though perhaps not feasible due to limited response rate.

Re-rating all ~12,000 paper would be a significant amount of work, and may not be a necessary check.  Something on the order of a quarter (3,000) papers, randomly selected, seems reasonable.  It might be nearly as suitable if the re-rating process were limited to only author self-ratings, thus spreading out the workload.  Assuming a similar response rate, this would yield just over 2,000 papers.

One inherent problem with defining AGW as, say, ">x% of since y date at z% confidence level" is that C13 results already suggest that relatively few climate paper are attribution studies -- i.e., they do not explicitly quantify any human influence on any metric of climate change.  Two things may ameliorate this issue;

1. Endorsement level of some >x% proposition could be inferred from cited references.
2. An additional question in the author self-rating survey instrument could be to ask them whether, based on their own personal knowledge of literature, they conclude that some >x% of observed warming is due to anthropogenic causes.

The first one obviously involves some subjectivity, but I argue that it is less subjective than the original protocol which called for abstract-only ratings.  As well, it is obvious to me that the function of primary research literature is to not to "prove" a given proposition from first principles in each paper.  Hence it is to be expected that the vast majority of papers "

[Edit 3/28/2016: Incomplete thought above.  On the basis of how primary literature in any field typically works, it is to be expected that the vast majority climate of papers don't build a case for/against AGW from first principles, and certainly don't attempt to quantify the portion of observed warming due to putative anthropogenic influence.  The TL;DR here is that literature review designed to show a majority of papers endorsing an AGW position of >x% warming due to humankind since y date is therefore rather doomed to fail because it will likely need to prevail on the concept of implicit endorsement ... which is extremely subjective, fuzzy and therefore ripe for abuse by the biases of the authors and ultimate consumers.]

The second item was specifically excluded in the original C13 survey instrument:

Note: we are not asking about your personal opinion but whether each specific paper endorses or rejects (whether explicitly or implicitly) that humans cause global warming:

I understand the import of making that constraint explicit.  On the other hand, I don't consider it an inappropriate question to ask.  Not only do I think it's interesting in its own right, it might also serve as a check for bias on the part of climate researchers themselves by, say, comparing self-ratings to independent ratings by others.  I further note that C13 tabulated results by author and thus have already somewhat "backed into" a statistic of author opinion.  I again refer to this link found on the SkS website to further this argument:

Why we care about the 97% expert consensus on human-caused global warming 

It makes sense to me to make this a formal question on the revised survey instrument for author self-ratings since it's conceivable, if not probable, that not all of an authors' papers fit into any one given endorsement category.

Endnote

I have publicly stated many times that I am not a formally trained scientist of any stripe and have no professional expertise in any of the topics I write about on this blog or elsewhere.  In particular, I have only very general knowledge of how to do scientific surveys, based mostly on undergraduate courses in basic statistics and business (specifically, marketing survey design).  Thus everything I have written above must be considered from the standpoint that they are the lay opinions of an avidly interested, but still an amateur pundit and blog author.  As such, I stand open to corrections and rebuttals by those with superior knowledge of how these things are supposed to work.  Those would be best received if accompanied by literature citations.

I also wish to make it clear that I my own anecdotal experience suggests that C13 is substantively correct to conclude that the majority of climate literature does indeed -- at least implicitly -- consider >50% of GMST increase since 1950 due to human causes (mostly in the form of CO2 emissions) to be essentially factual, if not a cause for concern with an appropriate call for reducing CO2 emissions by any and all reasonable means.

My main argument here rests on the principle that ALL science should ALWAYS be as dispassionately, rigorously and defensibly executed as possible.  Given how important a topic AGW is for me due to its potential to harm present and future generations of humanity -- and due to C13's high profile in the public debate -- exceptional scrutiny is something to be expected.

My hope is that Cook et al. will improve their present and future works on the basis of arguments from their critics and detractors.  I now think some points of their criticism have significant merit.

Addendum (3/23/2016 11:06 PM PDT)

I would be remiss to leave out that Brandon Shollenberger's latest e-book, and my subsequent discussions with him on his blog have been influential.  My review of the book was hasty and and too-dismissive, which I regret.  That said, I still cannot quite bring myself to endorse it either.

I think it also necessary to note how many of his arguments I have deliberately left out of my above critique of C13.  Many of his arguments use materials and communications obtained from SkS servers which the C13 authors clearly would not have wanted published.  I have read much of it, and it informs many of my above opinions even though I don't speak to them directly.

One reason why I left it out was because it's not clear to me that any or all of it was legally obtained, and I don't want the exposure.  Another reason is that some of it may be considered personally and professionally embarrassing to the authors, something which I have no desire to do.

For those reasons, I have chosen to write this note as best I could from the standpoint of someone who had no other information to go on but that which is published in C13 and the supplemental materials themselves.  Such is surely an impossibility, nevertheless it was my intent to try.

Addendum (3/25/2016 4:00 PDT)

ATTP re-posts a blog article by Peter Thorne, who was one the referees who contributed peer-review for Hansen et al. (2016), which has been poorly received by other AGW consensus bloggers.  Thorne writes:

This deliberate publicity surrounding a discussion paper (which to my knowledge is unique) led to unprecedented interest in the paper. By the time the comment period was closed there were over three times as many reviews as to the next most commented discussion paper in the journal’s history. This included many off-topic comments including a long thread on the existence of the greenhouse effect. Ironically, this was one of the better responded to comments by the authors (more later …).

Amongst the greenhouse effect deniers and other off-topic comments were unsolicited reviews from a large number of very well respected scientists expert in many fields pertinent to the paper including several colleagues who were (Coordinating) Lead Authors in the Fifth Assessment Report of IPCC or who have contributed to major works such as the annual state of the climate series. These reviews highlighted very many salient issues that the official reviewers failed to spot, and hence added substantial value.

In my view the responses from the paper author team to very many of the comments they received were inappropriate. Scientific peer review has a set of norms that you respond to the issues raised in a calm and measured manner including point-by-point responses that detail whether changes were made, what these were, and why. Instead, the authors chose to respond in many cases by writing discursive policy pieces that were too often non-responsive and often verged on playing the man and not the ball.

The parallels with C13 are not exact, but they are similar enough so as to be relevant and comparable.  Thorne continues:

It is beholden upon senior members of the community to set an exemplar of expected behaviour. They are role models and they righy or wrongly set or modify expectations of cultural norms, be that in climate science or elsewhere. My view is that the authors treated many of the reviews as a nuisance and did not provide the response that was justified to them that allowed the reviewers to fully understand how each of their review comments was dealt with. This included the public version response to my own invited review. It was not the behviour I would expect from such senior colleagues.

In comments at ATTP's, Willard responds:

I’ve read the word “nuisance” elsewhere, but where? Ah, yes, here:

The publicity was successful in drawing attention to issues that the paper highlights, notably the threat of large sea level rise. Criticism that it got too much attention seems clearly wrong. Would it have been better to keep the process and issues hidden from the public while they were being worked out? The only argument presented for that conclusion is that the publicity resulted in some irrational (bad science) comments from climate change “deniers”. Is there harm in that? On the contrary, it shows a disinterested judge or observer that all opinions are given a hearing. Yes, a few may be of low scientific quality and thus a nuisance, but the public probably wants all to be heard. When an editor cuts off such discussion after it becomes an excessive nuisance, a judge can readily verify that fact and affirm that all parties had a fair opportunity.

http://www.atmos-chem-phys-discuss.net/15/C8226/2015/acpd-15-C8226-2015-supplement.pdf

Emphasis in Willard's original, yellow highlights mine.

Apropos to C13, it's not clear to me that all parties were given a fair opportunity to be heard.  Of course, the SkS public discussion fora are qualitatively different than a public journal discussion forum.

That a judge can factually verify that parties have been given a fair opportunity to be heard,  it cannot be factually determined by anyone whether critiqued parties have been appropriately responsive.

That seems the salient commonality between Thorne's critique of Hansen's public behaviour and my critique of same against the authors of C13.

Update 4/2/2016

RobH, who was one of the more active members of the SkS abstract rating team writes in comments below:

When we applied the exact same rules for SkS ratings (1, 2, & 3's vs 5, 6 & 7's) to the author self-ratings we got almost identical results.

Perhaps I misunderstand his statement.  From the publicly available data files on the SkS website, I summarized the differences in ratings applied by the SkS team vs. the author self-ratings ...

Underrate	1,035	48.5%
Correct	832	39.0%
Overrate	269	12.6%
Total	2,136	100.0%

... where "underrate" means the SkS team's abstract rating understated the AGW endorsement level vs. the author self-rating of the entire paper.  From those calculations I conclude that the results are not almost identical, and that the SkS abstract rating team tended to be conservative in their ratings, i.e., they were biased against AGW endorsement relative to the authors themselves.  This to me speaks well of the SkS team.

By far the largest "error" the SkS team made in the abstract rating phase was ranking papers as category 4 (no position/uncertain) which the authors placed in an endorsement category (1-3).  This happened on 721/2,136 papers, or 33.8% of the time.  12% of the time they rated category 3 (implicit endorsement) compared to author self-ratings of explicit endorsement (categories 1 and 2).

By contrast, the largest "overrate" frequency was for category 3 (implicit endorsement) which the authors self-rated category 4 (no position/uncertain).  That accounts for 144/2,136 papers, or 6.7%.

Following is a detailed breakdown in tabular form.  The "change" column shows the SkS rating as the first numeral, and the author self-rating as the second:

Chg	Count	Pct of All
2>1	54	2.5%
3>1	86	4.0%
3>2	170	8.0%
	256	12.0%
4>1	75	3.5%
4>2	297	13.9%
4>3	349	16.3%
	721	33.8%
5>1	1	0.0%
5>3	2	0.1%
	3	0.1%
6>5	1	0.0%
Chg	Count	Pct of All
1>2	3	0.1%
1>3	3	0.1%
1>4	3	0.1%
1>7	1	0.0%
	10	0.5%
2>3	48	2.2%
2>4	34	1.6%
2>5	1	0.0%
2>7	2	0.1%
	85	4.0%
3>4	144	6.7%
3>5	4	0.2%
3>6	3	0.1%
3>7	1	0.0%
	152	7.1%
4>5	14	0.7%
4>6	3	0.1%
4>7	3	0.1%
	20	0.9%
5>7	2	0.1%