Commentary

The Economist: some problems with the peer-review process

1 Comment

From The Economist, of all places.

Excerpt:

The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers on priming contend). Either way, something is awry.

It is tempting to see the priming fracas as an isolated case in an area of science—psychology—easily marginalised as soft and wayward. But irreproducibility is much more widespread. A few years ago scientists at Amgen, an American drug company, tried to replicate 53 studies that they considered landmarks in the basic science of cancer, often co-operating closely with the original researchers to ensure that their experimental technique matched the one used first time round. According to a piece they wrote last year in Nature, a leading scientific journal, they were able to reproduce the original results in just six. Months earlier Florian Prinz and his colleagues at Bayer HealthCare, a German pharmaceutical giant, reported in Nature Reviews Drug Discovery, a sister journal, that they had successfully reproduced the published results in just a quarter of 67 seminal studies.

Let’s take a look at some of the problems from the article.

Problems with researcher bias:

Other data-heavy disciplines face similar challenges. Models which can be “tuned” in many different ways give researchers more scope to perceive a pattern where none exists. According to some estimates, three-quarters of published scientific papers in the field of machine learning are bunk because of this “overfitting”, says Sandy Pentland, a computer scientist at the Massachusetts Institute of Technology.

Problems with journal referees:

Another experiment at the BMJ showed that reviewers did no better when more clearly instructed on the problems they might encounter. They also seem to get worse with experience. Charles McCulloch and Michael Callaham, of the University of California, San Francisco, looked at how 1,500 referees were rated by editors at leading journals over a 14-year period and found that 92% showed a slow but steady drop in their scores.

As well as not spotting things they ought to spot, there is a lot that peer reviewers do not even try to check. They do not typically re-analyse the data presented from scratch, contenting themselves with a sense that the authors’ analysis is properly conceived. And they cannot be expected to spot deliberate falsifications if they are carried out with a modicum of subtlety.

Problems with fraud:

Fraud is very likely second to incompetence in generating erroneous results, though it is hard to tell for certain. Dr Fanelli has looked at 21 different surveys of academics (mostly in the biomedical sciences but also in civil engineering, chemistry and economics) carried out between 1987 and 2008. Only 2% of respondents admitted falsifying or fabricating data, but 28% of respondents claimed to know of colleagues who engaged in questionable research practices.

Problems releasing data:

Reproducing research done by others often requires access to their original methods and data. A study published last month inPeerJ by Melissa Haendel, of the Oregon Health and Science University, and colleagues found that more than half of 238 biomedical papers published in 84 journals failed to identify all the resources (such as chemical reagents) necessary to reproduce the results. On data, Christine Laine, the editor of the Annals of Internal Medicine, told the peer-review congress in Chicago that five years ago about 60% of researchers said they would share their raw data if asked; now just 45% do. Journals’ growing insistence that at least some raw data be made available seems to count for little: a recent review by Dr Ioannidis which showed that only 143 of 351 randomly selected papers published in the world’s 50 leading journals and covered by some data-sharing policy actually complied.

Critics of global warming have had problems getting at data before, as Nature reported here:

Since 2002, McIntyre has repeatedly asked Phil Jones, director of CRU, for access to the HadCRU data. Although the data are made available in a processed gridded format that shows the global temperature trend, the raw station data are currently restricted to academics. While Jones has made data available to some academics, he has refused to supply McIntyre with the data. Between 24 July and 29 July of this year, CRUreceived 58 freedom of information act requests from McIntyre and people affiliated with Climate Audit. In the past month, the UK Met Office, which receives a cleaned-up version of the raw data from CRU, has received ten requests of its own.

Why would scientists hide their data? Well, recall that the Climategate scandal resulted from unauthorized release of the code used to generate the data used to promote global warming alarmism. The leaked code showed that the scientists had been generating faked data using a “fudge factor”.

Elsewhere, leaked e-mailed from global warmists revealed that they do indeed suppress articles that are critical of global warming alarmism:

As noted previously, the Climategate letters and documents show Jones and the Team using the peer review process to prevent publication of adverse papers, while giving softball reviews to friends and associates in situations fraught with conflict of interest. Today I’ll report on the spectacle of Jones reviewing a submission by Mann et al.

Let’s recall some of the reviews of articles daring to criticize CRU or dendro:

I am really sorry but I have to nag about that review – Confidentially I now need a hard and if required extensive case for rejecting (Briffa to Cook)
…
If published as is, this paper could really do some damage. It is also an ugly paper to review because it is rather mathematical, with a lot of Box-Jenkins stuff in it. It won’t be easy to dismiss out of hand as the math appears to be correct theoretically, (Cook to Briffa)
…
Recently rejected two papers (one for JGR and for GRL) from people saying CRU has it wrong over Siberia. Went to town in both reviews, hopefully successfully. (Jones to Mann)

One last quote from the Economist article. One researcher submitted a completely bogus paper to many journals, and many of them accepted it:

John Bohannon, a biologist at Harvard, recently submitted a pseudonymous paper on the effects of a chemical derived from lichen on cancer cells to 304 journals describing themselves as using peer review. An unusual move; but it was an unusual paper, concocted wholesale and stuffed with clangers in study design, analysis and interpretation of results. Receiving this dog’s dinner from a fictitious researcher at a made up university, 157 of the journals accepted it for publication.

Dr Bohannon’s sting was directed at the lower tier of academic journals. But in a classic 1998 study Fiona Godlee, editor of the prestigious British Medical Journal, sent an article containing eight deliberate mistakes in study design, analysis and interpretation to more than 200 of the BMJ’s regular reviewers. Not one picked out all the mistakes. On average, they reported fewer than two; some did not spot any.

The Economist article did not go into the problem of bias due to worldview presuppositions, though. So let me say something about that.

A while back Casey Luskin posted a list of problems with peer review.

Here was one that stuck out to me:

Point 5: The peer-review system is often biased against non-majority viewpoints.
The peer-review system is largely devoted to maintaining the status quo. As a new scientific theory that challenges much conventional wisdom, intelligent design faces political opposition that has nothing to do with the evidence. In one case, pro-ID biochemist Michael Behe submitted an article for publication in a scientific journal but was told it could not be published because “your unorthodox theory would have to displace something that would be extending the current paradigm.” Denyse O’Leary puts it this way: “The overwhelming flaw in the traditional peer review system is that it listed so heavily toward consensus that it showed little tolerance for genuinely new findings and interpretations.”

Recently, I summarized a podcast on the reviewer bias problem featuring physcist Frank Tipler. His concern in that podcast was that peer-review would suppress new ideas, even if they were correct. He gave examples of this happening. Even a paper by Albert Einstein was rejected by a peer-reviewed journal. Elsewhere, Tipler was explicitly told to remove positive references to intelligent design in order to get his papers published. Tipler’s advice was for people with new ideas to bypass the peer-reviewed journal system entirely.

Speaking about the need to bypass peer-review, you might remember that the Darwinian hierarchy is not afraid to have people sanctioned if they criticize Darwinism in peer-reviewed literature.

Recall the case of Richard Sternberg.

Excerpt:

In 2004, in my capacity as editor of The Proceedings of the Biological Society of Washington, I authorized “The Origin of Biological Information and the Higher Taxonomic Categories” by Dr. Stephen Meyer to be published in the journal after passing peer-review. Because Dr. Meyer’s article presented scientific evidence for intelligent design in biology, I faced retaliation, defamation, harassment, and a hostile work environment at the Smithsonian’s National Museum of Natural History that was designed to force me out as a Research Associate there. These actions were taken by federal government employees acting in concert with an outside advocacy group, the National Center for Science Education. Efforts were also made to get me fired from my job as a staff scientist at the National Center for Biotechnology Information.

So those are some of the issues to consider when thinking about the peer-review process. My view is that peer-reviewed evidence does count for something in a debate situation, but as you can see from the Economist article, it may not count for as much as it used to. I think my view of science in general has been harmed by what I saw from physicist Lawrence Krauss in his third debate with William Lane Craig. If a scientist can misrepresent another scientist and not get fired by his employer, then I think we really need to be careful about the level of honesty in the academy.

News

William Lane Craig writes on “The Feminization of Christianity”

7 Comments

Look, here is a question on the Reasonable Faith web site from a feminist from Canada who objects to the idea of sex differences:

Dear Dr. Craig,

I have usually found your words to be a source of information and reassurance in my Christian faith, and have often sought out your writings and videos in times of doubt or questioning.

So I was really disappointed, almost shocked, when I read your newsletter of April of this year in which you casually stereotypes men and women, and complain that the church is becoming increasingly feminized, and has difficulties in attracting men.

Your compared the audiences at a couple of your speaking engagements to the audience from a clip of a Downton Abbey Q&A at another location – concluding that they were all men at the former and almost all women at the latter “simply because the Downton Abbey program is highly relational, which is more appealing to women, whereas my talks were principally intellectually oriented, which is more appealing to men.”

I believe that you are using stereotypes here, which you justify by making a ridiculous comparison that holds zero statistical significance. Not only is your statement unreasonable, it is potentially damaging – especially when made so carelessly. Stereotypes are shortcuts in classifying people. They can, and often do, limit and distort the way we perceive others and the world. Stereotypes are a lazy way of thinking that can lead to discrimination, and their use should not be encouraged.

I’m also a little disturbed by your claim regarding the feminisation of the church. What do you mean by that, and how do you support that statement?

I’m curious because the church has historically been a largely male-dominated institution (sometimes criminally so), and the bible’s instructions to and about women are often difficult to swallow. If anything, the church has had difficulty in attracting women. And if we are truly seeing more women in leadership roles at the church (I have to assume this is what you meant by feminizing), I believe this is not something to fear and resist. It would be a welcome change, and has every opportunity to challenge how we think about each other – allowing us to love each other better and see each other more clearly.

This newsletter called your expertise in some areas into question for me. Could you help to rebuild some of the faith I’ve lost in your words? I would very much appreciate it.

Sincerely,

Alexandra (Canada)

Canadians are sooooo liberal, especially on social issues like feminism, abortion and marriage. I’m sure this woman has been influenced by feminist ideology so much that she just can’t deal with the fact that men are women are very different.

Anyway, here’s a snip of Dr. Craig’s response:

Third is my claim that the church is becoming increasingly feminized. What I mean by this is that church services and programs are increasingly based on emotional and relational factors that appeal more to women than to men. The problem of the church’s lack of appeal to men has been recognized by men’s movements like Promise Keepers and books like John Eldredge’s Wild at Heart. Nowhere is this feminization more evident than in contemporary worship music. Someone aptly remarked that if you were to replace references to God in many praise songs with “Baby,” they would sound just like romantic songs between a man and a woman! This is not true of classic hymns like “A Mighty Fortress” or “And Can It be?” Talking with young men, I find that many of them are just turned off by these touchy-feely worship services and would rather not go.

We see this same feminization though relational factors in network coverage of sports, traditionally a male bastion. Coverage of Olympic Games has deliberately targeted women in order to increase viewership by the addition of personal stories about athletes’ lives, rather than simply televising the events themselves. In professional sports have you noticed how in recent years television networks have engaged female reporters to go down on the field and interview baseball or football players, usually about how they felt about this or that? Jan and I had to laugh when, following the Broncos’ recent blowout of the Ravens, the female reporter asked Peyton Manning, “Didn’t you feel bad for the other team when you looked up at the scoreboard?” Uh, I don’t think so!

You’re right that the predominance of women in Christianity is a relatively new phenomenon. It is only over the last 200 years that Christianity has become increasingly female in its demographics. I’m very worried that the church is on a course that will end in relatively few men’s being active Christians.

Fourth is my claim that apologetics is a key to making the church and Christian faith relevant to men once more. People think that by having sports programs or men’s barbecues the church will draw in more men. But I’m convinced that the best kept secret to drawing in men is apologetics. Men need to see that Jesus of Nazareth was not only a tough guy but a smart guy. I never suspected that apologetics would have this special effect on men. I had no intention of ministering particularly to men in this ministry. But the appeal of apologetics to men is just undeniable. In my Defenders class we’ve got guys who don’t even attend church but who regularly come for my lectures on Christian doctrine and apologetics. One woman in the class told me, “I don’t understand a lot of what you say. But I’m glad to come because this is the only spiritual activity that my husband will participate in with me.” Wow!

Wow, indeed. Apologetics gets men to engage more in Christianity, and the church should leverage that to bring men in. That’s a fact. So, I’m glad Dr Craig didn’t give an inch to this fact-averse feminist from Canada.

My own post critical of the feminized church can be read here. Keep in mind that this is from two months after I started blogging – very raw stuff.

By the way, Dr. Craig’s ministry Reasonable Faith has a huge matching grant again this year, so if you’re looking for a great ministry to donate to, Reasonable Faith should be on your list!

Related posts

Commentary

Reformation Day celebrates the supremacy of Scripture and reason in theology

3 Comments

The Ligonier Ministries web site has a summary of the event that kicked off the Reformation.

Excerpt:

On October 31, 1517, Martin Luther tacked up 95 theses on the church door at Wittenberg. With this act, he hoped to provoke a discussion among the scholars about the abuses of the indulgences by the Roman Catholic Church. He was not trying to create a public furor by any means, but within a fortnight, these theses had spread through the country like wildfire. The last thing Luther had in mind was to start some kind of major controversy, but nevertheless major controversy did begin.

From the discussions at Wittenberg, the disputations began to accelerate and escalate. Copies of the theses reached Rome and critical meetings were scheduled with the young monk. In these debates, Luther was maneuvered into proclaiming publicly that he had questions about the infallibility of church councils and also that he thought that it was possible that the pope could err. In 1520 a papal encyclical was issued which condemned Martin Luther as a heretic. Luther burned the document in a public bonfire and his defiance before the church was now a matter of record.

In response, Martin Luther picked up his pen to challenge the entire penitential system of the Roman Catholic Church, which undermined in principle the free remission of sins that is ours in the gospel. By doing so, he was unswervingly advocating his commitment to sola fide, the doctrine of justification by faith alone.

In 1521, Luther was summoned to the Imperial Diet, an authoritative meeting that involved the princes of the church, called by the Emperor of the Holy Roman Empire to be held in the city of Worms in Germany. Luther was an outlaw. For him to appear at the Diet was to risk his very life; therefore, he was given safe conduct by the Emperor to attend. With a few friends, Luther traveled from Wittenberg to Worms. The eyewitnesses of that episode tell us that when Luther’s little covered wagon appeared around the corner of the bend, there were lookouts posted in the church tower at Worms. All the people were agog waiting for the arrival of this notorious person. When Luther’s caravan was sighted, people were throwing their hats in the air, blowing trumpets, and creating all the fanfare of the arrival of the hero. It was the 16th century answer to a ticker-tape parade.

Things, however, became very solemn in a hurry because the next day he appeared before the Diet. His books were stacked on a table in the room, and he was asked and ordered to recant of his writings. This surprised Luther because he thought he was going to have an opportunity to defend his writings; but the only question really of any importance that was asked of him was this: “Are these your writings?” And when he said yes, they said, “Are you ready to recant of them?”

Hollywood has their version of Luther standing there boldly with his fist in the air saying, “Here I stand!” and so on. But instead he dropped his chin on his chest and muttered something that nobody could understand, so they asked him to speak up. “What did you say?” He said, “May I have 24 hours to think about it.” And so Luther was granted a reprieve of 24 hours to return to his room to contemplate the seriousness of this occasion.

The prayer that Luther wrote in that ensuing 24-hour period was one of the most moving prayers I have ever read in my life. In that prayer, Luther cried out for God in his sense of total loneliness fearing that God had abandoned him, and proclaimed, “O Lord, I am Thine, and the cause is Thine, give me the courage to stand.”

And on the morrow, Luther was called once again back to the court and was told to reply to the question. He said to the Diet, “Unless I am convinced by sacred Scripture or by evident reason, I cannot recant, for my conscience is held captive by the Word of God, and to act against conscience is neither right nor safe. Here I stand. I can do no other. God help me.” And with that there was an instant uproar.

The Protestant Reformation emphasizes the responsibility of each person to become familiar with what the Bible teaches for themselves and to make sense of it using the laws of logic for themselves. There is an enormous focus on the individual and individual’s responsibility to puzzle theology out for himself, using the tools available to him: reason, science, Scripture, history.  The Reformation put forward the famous five Solas: “Sola Scriptura” (Scripture Alone); “Sola Gratia” (Grace Alone); “Sola Fide” (Faith Alone); “Solus Christus” (Christ Alone); and “Soli Deo Gloria” (To God Alone Be Glory).

My friend J.W. Wartick has a post about the effects of the Reformation on society as a whole.

Here’s one of the effects: (links removed)

Capitalism had already begun before the European Reformations, having its renewal start in Italian city states in the 12th Century (for a detailed and extremely interesting discussion of this, see Rodney Stark, The Victory of Reason, 71ff). The Reformation, however, provided a place for capitalism to flourish. John Calvin’s thought touched upon nearly every contemporary problem, and one of these was usury (money lending at interest). Focusing upon the cultural context of the prohibition of usury in the Hebrew Scriptures, Calvin argued that his contemporary cultural context provided a way for usury to work without being necessarily wrong. Lending money in such a fashion was essential for the later development of capitalism (Alister McGrath, Reformation Thought: An Introduction, 259-260).

The Reformation also paved the way for a “Protestant Work Ethic.” Martin Luther’s notion of the “priesthood of all believers” demolished the hard distinction between the “temporal” and “spiritual” realm which dominated the thought of the church at the time. By breaking down this barrier, hard labor was elevated. It was no longer seen as an inferior life to that of monastic withdrawal; rather, any type of work could be pleasing to God (McGrath, 256-258). In contemporary churches, one can often hear about how the engineer, the retail worker, the auto worker, and the like should all utilize their skills to the glory of God. Such thinking came directly from the Reformation.

The Reformation also kick-started science, because nature held secrets about God’s existence and character.

Excerpt:

Without the Reformation, modern science would probably have developed in any event because of the ethos of rationality and the doctrine of creation conducive to it. The Reformation, however, hastened the development by criticizing scholasticism and by putting emphasis on the direct observation of nature. Luther has been called the Copernicus of theology while, on the other hand, Copernicus has been called the Luther of astronomy. Indeed, Thomas Sprat, an Anglican clergyman and an early member of the Royal Society, emphasized that there was a reformation, some would say revolution, in both philosophy and theology [3]. In natural philosophy or science, questions about nature were no longer answered primarily by quoting Aristotle and the Scholastics, but by turning to observation of and experimentation on nature itself. Similarly, after the Reformation, Protestants no longer answered questions in theology primarily by quoting scholastic philosophers and theologians, but by turning directly to the Bible. Luther interpreted Scripture by asking: what is the clear and straightforward meaning of the text? Scientists interpret nature in the simplest way using the minimum number of hypotheses.

Luther believed that the world was beginning a new age, which would bring not only a reform of religion but a new appreciation of nature. In his informal “Table Talk” he said,

We are at the dawn of a new era, for we are beginning to recover the knowledge of the external world that was lost through the fall of Adam. We now observe creatures properly …. But by the grace of God we already recognize in the most delicate flower the wonders of divine goodness and omnipotence [4].

In the last part of this statement, Luther paraphrased the words of the Apostle Paul in his letter to the Romans (Rom. 1:20).

Luther was open to the authentic scientific advances of his age [5]. He appreciated the mechanical inventions of his day.

He accepted the use of medicine in treating disease and is quoted as having said [6],

It’s our Lord God who created all things and they are good. Wherefore it’s permissible to use medicine, for it is a creature of God.

To someone who said that it is not permissible for a Christian to use medicine, Luther replied rhetorically, “Do you eat when you are hungry?” According to Andrew White [7], this attitude of Luther made the Protestant cities of Germany more ready than others to admit anatomical investigation and dissection.

Luther accepted astronomy as a science, but rejected astrology as a superstition because it cannot be confirmed by demonstration. Astrology, according to Luther, is idolatry and violates the first commandment. He was both amused and distressed by Melanchthon’s interest in astrology, a belief system that was widely accepted at the time [8].

The Reformation means good economics and good science, as well as good theology.

But theology still had some growing to do.

The challenges of the Reformers led to the development of the doctrine of middle knowledge, which reconciles divine sovereignty with human freedom and responsibility, solving a major problem with Reformed theology.

A very good book on middle knowledge is “Salvation and Sovereignty” by Kenneth Heathley. Highly recommended by Paul Gould.