Table 2 Quotes from interview participants

Root cause 1. Surgical PAFS are not optimally understood

1

Surgeon

I wouldn’t be a fan of doing a pilot study to see whether the design of the study worked…otherwise people are gonna go ‘oh what’, if you’ve not given them an answer; surgeons are pretty binary

2

Methodologist/funder

My pet hate is something described as a pilot study. When you read it, all of the objectives and everything else reads as if it’s a definitive study but with a massive, thumping great effect size and actually it’s nothing of the sort. It means, we wanted to do the full study but we couldn’t afford it, or couldn’t find enough patients so we’ve done something, we’ve called it pilot, and yes it’s neither fish nor fowl

3

CTU Director

The more uncertainty you’ve got, the more you need a pilot, and if you’re uncertain about surgeon equipoise, uncertain about patient equipoise, uncertain about the actual intervention then, compliance whatever, then that increases the probability you need a pilot

4

Surgeon

Well it’s all about recruiting isn’t it, at the end of the day? And the trial’s got to be attractive to clinicians and the staff who are addressing the patients. So, anything that obstructs recruitment has to be addressed in a pilot study, I would have thought…

Root cause 2. Surgical PAFS are not optimally conducted 

5

Surgeon/funder

I think the truth is most, many surgeons even those involved in trials don’t actually understand what feasibility and pilot work is and confuse it with a… it’s just a smaller trial, well no it’s not, it has a completely different role, maybe it’s stepping stone role but a very different role […] so, it’s not universal, but I think actually if you ask me what the average surgeon understood by that, they wouldn’t have a clue really…

6

Methodologist/funder

I think the key list of things is also a bit of a stumbling block, and that’s in the NIHR feasibility definition, they have this list of things, and I think people think they’ve got to do that

7

Funder

The area that’s probably more neglected, is the feasibility of the intervention and again there’s often a lack of appreciation of the complexity of interventions, lack of awareness of guidance in relation to evaluation of complex interventions, and often a naïve assumption that an intervention that’s been used by an expert in a specialist centre can simply be taken off the shelf and implemented throughout the NHS. And you really need to know about implementation before you start a full-scale pragmatic evaluation

Root cause 3. Surgical PAFS are not optimally reported

8

Surgeon/funder

With my journal editing hat on, I spend a lot of the time trying to persuade investigation groups so, they’ve actually done a pilot study and please report it as such. Even though they try and sell it to you as a definitive project with 20 patients in each group for a complex intervention

Root cause 4. Surgical PAFS are undervalued 

9

CTU Director

I do think there are probably some people out there who spend their career doing pilots and don’t move forward into doing the definitive trial which is harder

10

CTU Director

Sometimes you see pilot feasibility work, is seen as an end in itself […] [Funders have] funded masses of feasibility studies, most of which have gone nowhere and produced nothing of any interest […] I look at these things and say well, why would we go for these things? They’re three years of work, they’re usually very thin on resource, they’re a lot of hard work, and at the end of it we get nothing out of it that is of any use to us […] You can write up your feasibility work, it will go into the Ruritanian Journal of Unreproducible Results, nobody will ever read it again and it ticks a box for our masters that we’re generating income, but it doesn’t generate us any really top-class research output, so most of it goes nowhere

11

CTU director/funder

Regrettably the university won’t see this as being an important paper because, nobody’s going to see it as being three or four star […] it ends up in a low key journal, they look at it, you haven’t collected any data, you haven’t got really hard outcomes other than saying it can’t be done or it can be done, we’re not interested, where’s the main data?

12

CTU director/funder

There’s a lot of confusion about what people should be doing in these studies, and I think a lot of that comes from misunderstandings of journal editors and reviewers and trying to push people in a certain direction

13

CTU director

All the science, all the clever stuff is in the protocol paper, which counts for nothing. And actually, in the feasibility work, because it’s all the positive feasibility work that got you to the point that you could do the main study. It’s where all the clever stuff is. The wonderfully concise paper in ‘The Lancet’ says, we tested it, we found a load of people with whatever it is they’ve got, we gave them whatever these two interventions were and either it worked, or it didn’t. I mean there’s nothing scientific about that

Compounding factor 1: guidance challenges

14

CTU director/funder

I think they [the definitions] push people into particular boxes, whereas different questions are better answered using different approaches. As for almost any research question you care to name there are pros and cons to different approaches but here, the question you’re answering is ‘what’s stopping me doing the main trial'

15

Methodologist/funder

There are some nice papers actually on pilot and feasibility studies, particularly to do with sample size and I think they’re a bit… I mean they are very useful if you’re a methods person, I think they’re a bit tricky if you’re not […] so they’re good, but it’s obvious from the stuff we get from the [name of funder], that that message is not getting across through those articles in a way that is understandable

16

Surgeon/funder

I struggle with that division between feasibility and pilot despite having read and re-read all the crap describing the differences, actually I don’t think the differences are particularly meaningful […] that division is probably not important but only exists, yes in funding scheme applications

17

Methodologist/funder

I think there’s a bit of a failure on our part, meaning methods people like me, to translate for want of a better word, our stuff into a format that people who really have got better things to do, to use. They’ve got better things to do, than read through half a dozen papers. What they really want is to distil the key things that they really, really need, so they can build it into their idea

18

Methodologist

In my opinion most surgeons haven’t got as far as pilot and feasibility studies, there wasn’t very much awareness of them […] I got the impression that not many people were using the IDEAL framework and weren’t aware of it […] They seem to be getting away with doing mini randomised controlled trials and, as I’ve described, they shouldn’t really be doing that […] I felt that they hadn’t got as far as doing pilot and feasibility studies, or embracing them in their work

19

Surgeon/funder

I mean working within a trials unit to run these sorts of trials, you think is absolutely essential really, you can’t imagine working outside of it. I mean I can’t, but I know most, most clinicians do work outside and they have to. There’s a huge barrier between getting access to the trials units, getting CTUs to talk to you. Even well-established people are having trouble engaging with their CTU

20

Surgeon

The major stumbling block is the fact that a lot of bodies require you to have a clinical trials unit, and the clinical trials units are often too expensive

Compounding factor 2: cultural challenges

21

Funder

Deciding whether a trial is worthwhile involves judging value for money and that value for money judgement has to be made from the point of view of the health service and the funder. There are one million-pound questions that are worth one million pounds, and there are five million-pound questions, or even ten million pound questions and we will look at the question, we will judge what it is. But we then need to know how much will a definitive trial cost. Because if a definitive trial is going to cost ten million pounds, and it’s only a one-million-pound question, then there’s no point funding the standalone pilot and, again, I think this is often underappreciated

22

Surgeon

You’ll have one token surgeon with a grant giving body, who I should think, often, is not terribly diplomatic or very experienced in that sort of thing and it’s so competitive. Not just at the grant giving money for trials stage, but at the next level up where we’re going to spend our money, and translational studies and other types of sexy sounding personalised medicines and the humdrum randomised trial is hard to do. If they put their money into knock out mice, teams who know what they’re up to and have got to mould the work, crank that handle, they get the money and out comes a ‘Nature’ paper and we’re… it’s very hard for us to compete with that

23

Surgeon

In terms of funding there are all sorts of funding streams that I’ve used in the past, including industry, and the major stumbling block is the disparity between what the funding bodies actually tell you, how they’re all interested in surgical research and how the minority of surgical research gets funded, and therefore, we’re all mobilising ourselves to make sure that that’s reversed. The fact that that’s not the case, and they’re completely disinterested in surgical research… I can say that safely across the board. I think it’s fair to say that a lot of charities are completely disinterested in anything that’s of clinical value, or that involves surgical research

24

Surgeon/Funder

Speaking candidly, the big companies have got no interest in… in fact it’s almost a disincentive to do head to head comparisons of their technology against whatever other interventions are out there, and because there was no requirement for them to do so and they never actually developed… what you discover is they’ve got no infrastructure to do that

25

Surgeon

The difference is that the pharmaceutical industry, not only has more funding, but has the requirement to carry out the work, whereas the device industry has got less money, but has plenty of money, but has no requirement to carry out the work. There’s a lack of a regulatory requirement

26

Methodologist/Funder

You could get up to £300,000 for a feasibility study and after that you’re looking at NIHR money, so if you wanted £2 million, the only place you have to go is NIHR. HTA really, which means there’s a delay then of at least a year, and probably more than that. So, it really stretches out the development of that trial

27

Surgeon

So if you do it in the linear way, the way you’re supposed to, we’d all be dead before you finished the main trial, which again comes back to my point about the present structure, is just too inefficient

28

CTU Director

Things will move on, and it’s pointless to do a whole bunch of small pilot or feasibility studies, and then actually the question has moved on by the time we’ve worked out whether you can (laugh)…