The current system of ethical review for medical research in the United Kingdom is changing from the current system involving large committees of 7–18 members reviewing every individual application to a system involving pre-review by small sub-committees of National Research Ethics Officers (NREAs), who have a remit to approve studies if they believe there are no material ethical issues imposed by the research. The reliability of this new system depends on the reliability of the NREAs and in particular the ability of small groups to effectively identify and appropriately assess the seriousness of all the material ethical issues that may be posed by an application. There is anecdotal evidence to suggest that many individual research ethics committee members have had experience of believing that a study presents no material ethical concerns, then on reaching the committee and discussing the application they realise that the committee feels it does present significant ethical concerns. If this is the case then this casts doubt on the reliability of NREAs or small groups to effectively identify ethically problematic research and appropriately respond to this to protect research participants. In this paper we describe a small questionnaire based piece of research carried out to assess how common this and other relevant experiences are.