Research that's ahead of its time

This past year, I had the opportunity to participate in a NSF panel for the first time.  These panels are convened to evaluate academic research proposals.  Panel members review several proposals each, discuss the merits of each proposal in a group setting, and recommend which ones should be funded to the NSF. Panelists’ recommendations are not binding—NSF makes the final funding decision.

Listed below are some of my thoughts about this experience.

Benefits: I personally gained a lot of insight into how to write a “good proposal” from serving on a panel.  These insights will help me hit the ground running when it comes time for me to write my own proposals as a PI.  If you are a early-career researcher (e.g., a postdoc or new assistant professor), you should volunteer to serve on a panel to gain the same insights.  To volunteer, simply send email to the NSF program director that oversees your research area.  NSF actively encourages early-career researchers to serve on panels, so you will most probably be picked.

Writing a good proposal: A good proposal imagines a future that is 3-5 years out and presents just enough evidence to convince readers that this future is both desirable and attainable.  Writing one is challenging because it requires finding the perfect balance between two competing goals.

First, a good proposal must be “undefined enough” to convince readers that the stated research goals involve overcoming significant and (worthwhile hurdles).  Second, it must be “defined” enough” to convince readers that the research goals can actually be met in a concrete timeframe.   When I discussed this dichotomy with more experienced panelists, they said that the best proposals are ones seem like they will require a couple of PhD dissertations worth of effort.

The best proposals I read described a compelling vision and painted a clear description of how that vision could be attained.  To do so, they broke down the vision into concrete research goals and described how earlier (more straightforward) goals would inform later (less straightforward) ones.  They presented a small amount of preliminary evidence to convince readers that each research goal is attainable.

Reviewing a proposal: I found that reviewing proposals required a very different mindset than reviewing research papers.  The latter generally involves evaluating whether a paper is self-consistent with its stated goals.  The former involves rationalizing whether the potential future that a proposal imagines has a chance of becoming reality.  Stronger proposals made it easier for readers to come to a positive conclusion.

Participating in group discussions about individual proposals: I was impressed by the quality of these discussions.  All of the panelists diligently prepared their reviews.  Also, every every panelist was given equal opportunity to voice their opinions and everyone’s opinions were carefully taken into account in the final recommendations.  There was a concentrated focus on providing constructive feedback to PIs.

My only concern is that I felt it was too easy for panelists to kill a proposal by claiming that it did not compare itself to an important paper (or papers) that other panelists had not yet had the chance to read.  Changing the process so that panelists can see each others’ reviews before the panel meeting would allow panelists more time to read such paper(s) themselves and come to their own conclusions about this important concern.

Final thoughts: In the end, I left the panel with an extremely positive impression of NSF’s review process.  Most of all, I was extremely impressed by NSF’s willingness to include early-career researchers and mentor them in the panel-review process.  You can find more information about how to serve on a NSF panel here.  Other folks have also written about their experiences here and here.

Disclaimer: These are of course my subjective views/opinions as an academic that participated in a NSF peer-review panel. My views/opinions do not bind NSF and may not reflect NSF’s views/stance.