Debate on Misleading Literature Reviews

The Profession

Richard P. Phelps wrote, “To save the research literature, get rid of the literature review,” in the London School of Economics (LSE) Impact Blog:

“The laziest literature reviews are those falsely declaring previous work on a topic to be nonexistent — “dismissive reviews”. When I first began to search for the magnitude of the dismissive review problem, I expected to find hundreds; I found hundreds of thousands…

You can see for yourself. Access a web database that allows searching by phrase (e.g. Google, Yahoo Search, Bing) and try some of these: “this is the first study”; “no previous studies”; “paucity of research”; “there have been no studies”; “few studies”; “little research”; or variations thereof.”

and

“I don’t believe that journals can responsibly review manuscript literature reviews for accuracy. And, because they cannot, literature reviewers are free to trash or ignore competing work. Thus, ironically, literature reviews may degrade our knowledge base more than preserve it.”

Dr. Phelps then gives six reasons for journals to drop the literature review. I encourage you to read his article, linked above.

In response to Dr.  Phelps, Arnaud Vaganay wrote, “To save the research literature, let’s make literature reviews reproducible,” in the same publication outlet.

Dr. Vaganay agreed with the problem of literature reviews, and wrote:

“…editors and peer reviewers do little to ensure that LRs tell the whole story rather than a good story. It is hard to blame them; by some accounts the scientific output doubles every nine years. As it becomes increasingly difficult to stay on top of the literature – even in one’s own field – editors and peer reviewers should pay more attention to the way LRs are conducted.”

He departs from Dr. Phelps in his argument that literature reviews (LR) can be reformed by making some formal requirements:

“Readers who are as frustrated with the current state of LRs as Richard Phelps and me might be interested in Cumulative Literature Reviews (CLRs) (see from slide seven onwards). CLRs start from the premise that LRs and their components (literature dataset, LR protocol) are research assets, just like the protocol and dataset used in the empirical part of a study… The methodology is still at the development stage and is being piloted in two ongoing studies.”

 

 

Advertisement

More Good Advice for Publishing in Journals

The Profession

Hugh McLaughlin, a journal editor, wrote “How to increase your likelihood of publishing in peer reviewed journals.” Some highlights:

Address “SO WHAT?”

“You might want to consider some key questions: does your article contribute new knowledge? Does it offer a unique way to address a social problem or policy dilemma? Will it have implications for practice?”

Revise, Revise,  Revise

“Once you start writing it’s important to remember that each paragraph and page is only a draft.”

Know Your Journal

“Editors like to see that potential authors are aware of the journal and its content.”

Theory is Important

“Infusion refers to the infusion of theory into the manuscript, providing readers with new ways of provoking discussion, challenge and applications to practice. It was noted in our recent publication that “articles that are theoretically strong receive the most citations”. ”

Respond to Each Reviewer

“Respond to each reviewer separately showing how you have addressed the points they have raised.”

Rejection is Good, but Time Is Better

“You may want to consider putting the manuscript aside for a period so that in time you may gain a fresh perspective on it, consider how it can be improved and where you can resubmit it.”

Tips for Grant Finding and Grant Writing

Uncategorized

An interesting primer for early career researchers to obtain grant funding is by Jonathan O’Donnell in a blog post for LSE. With shrinking funding options and more and more PhDs in the world, how to get grants?

Some tips if you are…

A Reviewer of Grants:

Register to review for granting agencies. If you are dreaming of working in another country in the future, register to review their grants, too.

Set clear limits, both in how many reviews you will do, and how long you will spend on each one.

Since you live in a world of measurement, record your effort.

Seek out mentoring, including peer mentoring.

Be kind

A Seeker of Grants:

To gain experience for yourself, work on grants with someone else, perhaps those with successful experience. Be a good collaborator.

There is a world of funding that isn’t peer reviewed. Industry funding, philanthropic funding, crowdfunding, micro-patronage, etc. Get creative with your funding opportunities.

Reject academia altogether, at least for a short time. Go and work in industry, or set up your own entrepreneurial research endeavour.

Set a time limit on how long you will bang your head against your rejection wall.

A Grant Agency:

Make the grant application process less onerous.

Experiment with more inventive methods of review.

Consider changing attitudes to the confidentiality of grant applications. You might choose to build a library of successful applications, or even make all your applications public. That would provide significant benefits to people learning to write good applications.

Reduce grant applications to two pages, concentrating on the core idea. Fixed budgets, double-blind review and no administrivia.

Give better feedback. “After the amount of work that most grant applications entail, it’s always anticlimactic to get a ‘yes/no’ result.”

An Established Scholar and Experienced Grant Writer:

Provide training in peer review, both for grants and publications.

 

 

 

 

Finding and Doing Important Research

Uncategorized

From 80,000 Hours:

Like many new academics, you want to do important research; research that makes the world a better place. The problem is there’s virtually no guidance out there. All you can do is ask experts in specific fields, while there’s no-one providing an overview of which areas might be best.

A high impact research question has three characteristics:

It’s important: If we make progress, then the world will become a better place.
It needs talent: If more people work on it, then more progress will be made. And in particular…
It needs your talent: If you work on it, then more progress will be made
A good research question needs to be both important and in need of more researchers. Building a perpetual motion machine is extremely important – if we could do it, then we’d solve our energy problems – but we have good reason to think it’s impossible, so it’s not worth working on. Similarly, a problem can be important but already hold the attention of many extremely talented researchers, meaning your extra efforts won’t go very far.

Instead, we want to find questions that are important and where one extra person’s efforts can go a long way. What matters is the importance of the problem weighted by the extent to which you can make a difference to the problem.

Finding these questions, however, is difficult. Often, the only way to identify a particularly promising research question is to be an expert in that field! That’s because (when researchers are doing their jobs), they will be taking the most promising opportunities already. The only way for you to have more impact than the average is to have more informed views about what’s important to work on.

Nevertheless, it seems that there are some types of question that tend to get unfairly neglected. So, for someone who prioritises impact, there are special opportunities to uncover. What follows is one way to start narrowing down the huge range of options available.

 

Changing Your Story

Uncategorized

The process of turning an idea into a manageable product is not linear. It may start with a fixed idea, but that idea can change as you collect more information, do more reading, and write-up what you have. New information changes old ideas. And your story changes.

Inspiration comes from anywhere, and everywhere.  Take the story of a guy from Lisbon who is traveling the world and making hand-drawn sketches of everything he sees (the full story is here).

“Luis Simoes has visited 29 countries in the past two years, traversing Europe and Asia and documenting every step. But not in the usual way — there’s no snapping of thousands of photos with his smartphone, or making friends back home jealous with interminable Facebook posts. Instead, Simoes, a 35-year-old former 3D motion designer from Lisbon, Portugal, is sketching his way around the world…”

Why?  Because others travel, so why not him?

“After ten years in the same job he decided that he’d had enough; he wanted to be out there doing what he loved. And that thing, he discovered, was sketching. “I had a friend who was traveling around the world and I was jealous, but then I thought, if he’s doing this then I can also do this.”

How he started, and how he changed. He says:

“Up until now as I’ve been traveling around I do a lot of sightseeing but I try to capture the mood of a traveler when he’s sightseeing. So I’m not just drawing a beautiful building, I’m trying to capture the life around the building, and how people connect with that building. So I try, with my sketches, to bring these moments. I try to build a story behind every sketch.”

His style has changed, he reflects. He no longer draws tourist attractions like the Sagrada Familia in Barcelona or the Eiffel Tower. “Now I’m drawing a story… I don’t have to tell everything, but I give you the subject. I’ve learned how to go sketch faster but keep all the ingredients to keep the viewer interested. Of course the more you sketch the more you grow, so it’s an evolution.”

Your own research is an evolution, not a straight line from a fixed, beginning idea that is ultimately realized.

And it requires a leap into the unknown. Luis is on a five year sketching journey, and he says:

“I needed the ambition — I wanted something big, for myself. Five years is a long time but I want to search for the unknown.”

For more on the evolution of research, see this TED video, “Uri Alon: Why truly innovative science demands a leap into the unknown.”

On the Need for Examining Concepts

Uncategorized

Lewis Coser, American sociologist, wrote in preface to his book, The Functions of Social Conflict (1956: 7):

Before the “facts” can speak, they have to be arranged though some conceptual scheme. The divorce between research, conceived as a quest for “facts,” and theories which too often soar beyond the reach of facts, is responsible for many of the shortcomings of American sociology. And this divorce is responsible as well for the lack of cumulation and continuity in American sociology. Periodic conceptual analysis, in our view, serves to mitigate these two kinds of shortcomings.”

 

On Reviewing Articles for Journals

The Profession

One of the assignments for the “From Idea to…” class was to review an article.  Several classes were devoted to discussions of the structure of empirical articles in the social sciences, and one class was devoted to writing article reviews; students were given a review of an article as an example. Everyone in the class (N = 15) reviewed the same article.

What is striking is how many different views there are about the same article.  Here are some examples of student reviews.

On the style of writing:

— “Article is written very clearly and precisely, all data is analyzed very carefully, research question is answered, all hypotheses are verified.”

— “The structure of the article is clear and simple. The language is academic, hence demanding, but not too hard to understand even for a general public. ”

— “The paper is written clearly and in a manner that makes it easy to follow also for a non-specialist in the field.”

vs.

—  “Values of this in a way pioneer analysis are undermined however by serious shortcomings in the presentation of the study and its results, its chaotic structure, lack of definitional clarity and theoretical profoundness.”

— “My main comment towards the article is that it uses too many intellectual shortcuts, both in theoretical explanation of hypothesis and in the results interpretation. Therefore, I perceived it as more journalistic than academic, despite of being based on the survey data.”

— “The results that follow each other are sometimes describe in such a modest way, that a reader could easily miss some important information or understand it incorrectly.”

Most identified problems such as:

— Article fails to adequately answer, “So what?”

— Article does not cite enough previous literature, and thus the theory and expectations section suffers

— Article does not describe the data used in adequate detail

— Author should not assume that “party image” is a valid indicator of “party system”; a clearer link should be made.

Others thought that the political parties were not adequately described:

—  “The descriptions of the political parties are very journalistic and too simplistic. What does it mean that PO is pragmatic? Or PiS nationalistic? Again it sounds more like repeating the clichés.”

— “[need] to improve the characteristics of the Polish parties, which is not very precise (Samooborona as a “traditionalist Catholic” party, or controversial characteristic of PiS as a nationalistic party).”

Others question whether women and the old are disadvantaged groups:

— “Why women, the old and the poor have been chosen? Are they really disadvantaged in Poland? How do we know that?”

— “The [assumption of the] article is … the idea that women, the old and the poor are disadvantaged social groups and that those groups are less visible during the political process. In terms of a large number of socio-economic indicators those groups definitely have certain disadvantages in their social life, but their little visibility in political process is very questionable.”

A very interesting criticism is this:

— “Another point is that data was collected in 2008, one year after Parliamentary elections: it is known that parties care much more about their image before elections. Perhaps, research during ongoing PR campaign could show different results.”

Most recommended to revise & resubmit or even conditionally accept the article.  Perhaps the students thought that the course instructor is also the author of the article, and decided to be kind; or, the students are too kind anyway, even if their criticisms are direct and rather unfavorable to the work.

On writing article reviews:

Key to understanding how to write an article for review for a journal is that the editor of the journal, the reviewer and the author are all engaged in a debate over the author’s work.  The author presents their research and is supposed to do so clearly and in a detailed fashion.  The editor of the journal asks a “reviewer” to enter this debate; the reviewer’s job is to (a) demonstrate that they understood the article as best they could, (b) present their criticisms in as clear and detailed way as the author had done (or sometimes in an even clearer and more detailed way if the author’s article is insufficiently clear and detailed), and (c) tell the editor whether the article should be rejected outright, rejected but invited to revise the article and resubmit it for a second review (so-called “Revise & Resubmit”), conditionally accept the article (meaning the article has so few flaws that only minor revisions are needed), or accept as is (this almost never happens).

The reviewer has a professional and ethical responsibility to the editor, the author, and the larger scientific community to be clear and detailed in their criticisms.

Remember, too, the audience for reviews: it is for the editor and the author, both of whom are your colleagues in the scientific community.  You have a responsibility to be respectful to them.  Reviews should be direct and honest, but language used by the reviewer can impede the points the reviewer wants to make.  Reviewers should be as careful in their reviews as the author was in writing the article.  Criticisms should be constructive, an attempt to help the author of the article do better.

Other notes:

The beginning of a professional article review necessarily begins with a summary of the article.  This is so that the editor of the journal – the one who makes a decision as to whether the article will be published in their journal – and the author who receives the review are both confident that the reviewer understood the article.

Important to note that in articles which contain tables and figures, editors and reviewers expect that the tables and figures are listed as “–  Table 1 about here –“ in the place where the table would be, and that the table or figure is placed in the back of the paper.

In modern scientific writing in English, it is acceptable to use the first person singular, „I.”  (if the work is co-authored, then “we” is used). This is called, “active writing,” and most modern journals prefer active writing.

As an example, let’s take this sentence from the December 2010 issue of the American Sociological Review, the most prestigious English language journal in sociology (emphasis added):

“Applying this strategy to several different cases, we identify three trajectories scientific propositions assume on their way through contestation to consensus among practicing scientists…”

And these two sentences from a different article in this same issue:

“In this article, I explain variation in token experiences by demonstrating that tokenism is contingent on the local cultural context in which it is embedded. I develop this argument by comparing the experiences of female and African American male investors in the leveraged buyout (LBO) industry.”

You can see these examples here: http://asr.sagepub.com/content/75/6/817.full.pdf+html

and here:  http://asr.sagepub.com/content/75/6/894.full.pdf+html

Unfortunately or not, in modern science writing it is not necessary for authors to state how their research might be “useful for the world.”  World-usefulness is not a criterion for inclusion into social science journals.

The point that the writing style is too “journalistic” can be valid, but you would need to provide a specific example of a sentence where the style impedes on understanding the author’s thought.   At what point is “too journalistic” also “too unscientific?”