Episode Transcript
[00:00:03] Speaker A: Welcome to the Ortho Joe Podcast, a joint production of the Journal of Bone and Joint Surgery and Ortho Evidence.
Join hosts Mohit Bhandari and Mark Swankowski as they discuss current topics and publications in the world of orthopedics and beyond.
[00:00:19] Speaker B: Hey, good morning, Mo.
[00:00:20] Speaker C: Morning, Mark. How are you?
[00:00:22] Speaker B: I'm good. I take it at you're at home because I see Ravi Shankar behind you.
[00:00:27] Speaker C: Well, I know. I'm actually at. I'm actually at McMaster University and I'm sitting in the office. In the office.
[00:00:33] Speaker B: So Ravi, he. He. He lives in. At your office, not in your home?
[00:00:38] Speaker C: Yeah, he lives in the office. Yeah, yeah, yeah.
[00:00:40] Speaker B: Look at that. I learned something today.
[00:00:42] Speaker C: Yeah, There you go.
[00:00:43] Speaker B: Right, that's one. That's one of your really best watercolors, I think, Mo.
[00:00:48] Speaker C: Oh, thank you.
I appreciate that.
[00:00:51] Speaker B: Yeah. Well, I'm. I'm in Albuquerque at the AOA meeting.
[00:00:55] Speaker C: Oh, excellent, excellent. Hopefully it's an excellent meeting.
[00:00:58] Speaker B: Yeah. Well, the AOA founded 1887, the oldest orthopedic association in the world. Founded the journal, as you know.
[00:01:06] Speaker C: Yeah.
[00:01:07] Speaker B: 140 some years ago.
But the issue is that I'm away from home and struggle to find a coffee. So here, here's where I came up with.
I went to the Little Bear Coffee shop down.
[00:01:20] Speaker C: Very nice, very nice, very nice.
[00:01:23] Speaker B: And it's interesting how these little independent coffee shops, the flavor is really different than at Tim Hortons or the other chain or what, which is very uniform, you know, but the very interesting kind of smoky, kind of latte flavor. But anyway, I'm here and as I do every time a journal issue comes out, had a look at the most recent one and found a article that was published by the METRC Consortium. And just for the audience to know, perhaps a conflict of interest. I chair the Data Safety Monitoring Board for the METRC Consortium and have done so for, I guess near a decade.
I'd like to just talk about this particular publication.
Secondary analysis of the primary data collected 8, 10 years ago by the Metric Consortium, which I think the audience is pretty well familiar with, the fact that this is a group funded by the Department of Defense that has published over 20 really, really well done randomized control trials surrounding issues with injury and recovery for the benefit of the armed forces as well as the general public.
And this particular study was one of the early ones. It's the Fix it trial, which is a trial which looked at severe open tibia fractures treated with modern ring external fixation versus treatment with nails and or plates.
And the patients were were followed for long periods of time.
This was a pain analysis, a secondary analysis done.
The two groups as stated were modern ring fixation N of 122 versus the internal fixation group 132.
Primary outcomes were pain intensity and interference at 6 and 12 months measured by the Brief Pain Inventory. Secondary outcomes were numerical pain rating scale and incidence of moderate to severe pain.
They compared pain in patients with and without pin site infections and with and without external fixation removed.
The conclusions of this secondary analysis was at six months, the patients treated with modern ring fixators had greater pain interference. In patients treated with internal fixation, partly because of insight infections, there is no difference in pain interference or intensity seen at 12 months.
At 12 months, if the patient still had the fixator in place, the patients had greater pain intensity, which makes good clinical sense then those in which it had been removed. But this was not the case at six months, a third of all the patients had moderate to severe pain at both time points, highlighting that persistent pain is common regardless of treatment type.
They conclude that these findings can guide surgeons in choosing ring external fixation or internal fixation for these fractures.
This is information that was, I think previously pointed out in the LEEP trial, which was a long term study of severe lower extremity open tibia fractures, that patients have pain after they're done with treatment, kind of regardless of how they're treated and that social support, personality, income, education, all those have major impact on the primary outcomes. But the reason why I wanted to chat with you about this, Mo in this episode of Ortho Joe, is the whole issue of secondary analyses. So you have planned and conducted numerous large randomized trials in various aspects of care of the injured patient.
And you start out with a primary analysis, which is the major outcome. But almost always there are these secondary questions or secondary analyses.
And so I want you to educate our audience on what are the best practices when you're planning a trial in regards to these secondary questions.
Sure, take it away, Mo.
[00:06:04] Speaker C: Okay, so I mean, I'll speak and just chime in because you've been involved in many of these trials, but in principle, you know, so there's, you know, there's the pragmatic and then, you know, there's pragmatic approaches too, right? Which is, you know, you're doing a study, you have your primary outcome, you know what you want and you think, well, I don't really know all of the different secondary questions I may get because, you know, we're just evolving, that's fine. I mean, people can, you know, and that's the way a lot of people do it. They'll, they'll, they'll do a trial, they'll have their primary outcome, they'll analyze for outcome and then they'll go back and then, you know, evaluate a series of questions. There are some real challenges with that approach. And I think because of that approach there have been fairly standard guidelines around how we should be thinking about large programs. This is a very large program and similarly we've done some programs like that. The first thing would be, is to the extent that you can think deeply about your protocol and your primary and secondary questions. So usually, right, most studies will have a primary, they might have two or three really important secondary questions, they might even have a couple of tertiary. But think about those. And I think that spending more time on those, Mark, is very helpful thinking about and I think you've been a big proponent of this. If you know the questions in advance, then you collect the data you need and you don't waste data.
So you know, there's the approach of let's collect a bunch of stuff we don't know, we'll probably need it. Probably not a good approach. The better approach would be to say we have hypothesis driven data collection based on, you know, primary and secondary planned analyses.
A way for us to check that in a way that it's reinforced is prior reporting. So either you publish your protocol prior and that gives those who look at your subsequent outputs a way to go back to the original, say, well, did they originally plan all this stuff or did they kind of post hocus make decisions and kind of do a bunch of post hoc analyses which in and of themselves can be done with some rigor. So we can speak to that. But in principle that's one check clinicaltrials.gov for a trial is a good place to start as well. You outline all the things you've done and those are important for a host of reasons because we want to make sure that we haven't just cherry picked things and we haven't decided what makes sense and what messages we want to put out based on, based on what looks most favorable versus what we actually planned. So that would be the first thing. Maybe I'll stop there for a second to see if you have anything else you want to add to that before going to the next level.
[00:08:44] Speaker B: Yeah, so you brought up the concept of trial registration. And we at the journal, along with others that we've collaborated with, BJJ and CORE and Journal of Orthopedic Research, we require trial registration before the first patient is enrolled.
And that is to avoid the issue of authors doing statistical fishing expeditions and altering the outcomes, et cetera. And it's a pretty worldwide accepted standard with human subjects oversight that when you're going to do a randomized trial, you need to register the trial. You need to state what your primary and secondary outcomes are and how those data will be collected and analyzed.
And that's to prevent just looking for statistical findings that you can make a point about which, which is, is not really an ethical, ethical way to proceed.
So that, that's just one real aspect of it. But suppose you have. Let me just pose this question to you. Maybe you can speak to what you did with the float. Well, go ahead.
[00:10:04] Speaker C: Yeah. And I just will add one more point before we jump to the next phase of this, but is another really important reason to think about the, the, the outcomes you're going to be measuring in the, in the interactions or the story, the, the interventions you're going to be comparing up front is you got to make sure you actually have enough statistical study power for that. So, for example, if your primary outcome requires, let's say, 100 patients per arm, but you have a number of secondary outcomes that might be proportional infection, maybe something that might require several hundred per arm, by not thinking through each one separately, you are putting yourself at risk for being grossly underpowered for your analyses, which adds yet another level of, you know, challenge and problem. So the more you think about all the questions that you want to ask up front and document it and put it and plan your research around that, so you're appropriately defending why those measures matter and more importantly, why you'll have enough study power, sample size to manage them all is very important. We did exactly that when we did our first sprint trial. We did exactly that when we did the Health Faith and also the flow trial. But you were going to talk about flow, but that.
[00:11:14] Speaker B: Yeah, no, I was just gonna ask you how you preplanned and managed the secondary analysis for the flow trial. Just what was your experience with that trial?
[00:11:24] Speaker C: Yeah, and so obviously we had put together, you know, a series. You know, we had a prior protocol, we had a trial registration, but there are situations in which, you know, you'll organize a committee, there'll be a writing committee, there'll be an appropriate plan, there'll be appropriate submissions. You know, there's many, many centers in this case. There were many centers, hundreds of investigators around the world who are interested in asking questions after the completion of the primary findings of the trial.
So we Have a secondary, you know, it's almost like a publications committee, Mark, that's organized with the chair and a process. So one thing is develop a clear process. Number two, you know, ideally you want to, before you actually unblind yourself to all the results, is to do, in many ways, start thinking about any sort of quote, post hoc analyses before you've looked at the data. So in other words, start thinking about questions that you may want to ask rather than saying, we've done our analysis now and now we're going to start dredging for more and more things.
It's okay if you didn't think about it 10 years ago when you started the trial, but before you finished the trial and before you've actually done, lock the database and actually analyze, start thinking then, okay, have we learned anything from our database that we may want to additionally ask that we didn't put in and update your trial register? Update things, but be very clear that it's not. After unlocking the database and looking at all the data, now you're starting to say, okay, what works, what doesn't. All this happens before you've even looked at the data and you're getting the committee together again to say, okay, the trial's closing out.
What are some additional questions?
Let's create small protocols for each, make sure they're powered, and then we can make assessments. But there needs to be a clear process and a clear transparency of reporting around how that process is. So when you're talking about publishing that, it should be very clear to the readers that what the processes were to make those decisions about some of these measures, especially if they weren't initially included in your original trial registration.
[00:13:39] Speaker B: Excellent. So let me just summarize once more for the items, because this is why I really wanted to talk about this.
So ideally you would, when you register the trial, you would have all secondary analysis outlined, secondary questions and analysis outlined. And if, if new insights happen that prevent you from having been comprehensive in doing that. Before you unblind and unlock the data set, you should work with your collaborators and decide what are the questions. Make sure you're adequately powered before you unlock the database. Those are the two best practices with dealing with secondary analyses. I think I have it right.
[00:14:22] Speaker C: Yeah, I mean, in principle. Right. It's just a matter of proving to those around and to yourself, right. That we're not just doing a bunch of analyses and seeing what sticks and saying, okay, that one sticks. So therefore, let's just go ahead and write that paper up versus this is important. Here's why we want to do it.
Here are the total number of patients we have, and here's what we anticipate being the difference. And they made a very clear point in this paper. They did an a priori power analysis. They determined they need 119 patients per arm. So you'd say, okay, well, how many patients do we actually have? Oh, we have more than that. Okay, so. So this is a reasonable one for us to think about under the assumption of, you know, when we do the analysis. But it's not like we'll run a bunch of analyses. Oh, pain looks like it's interesting. Let's write a paper about it.
[00:15:06] Speaker B: Yeah.
[00:15:07] Speaker C: Very different.
[00:15:08] Speaker B: Yeah, excellent. And maybe we could conclude with kind of strategies for dissemination. So let's say you've got a trial and there's 10 or 12 different secondary analyses of various questions.
Kind of the two options I would see would be to publish them in a group, and I'll let you speak to how the journal wants to do that or not, versus sending them out one at a time to various different journals.
What have you done with the trials that you've overseen, and what are the options, and what do you recommend?
[00:15:43] Speaker C: Sure. Well, the first thing is that one of the greatest value propositions to large trial collaboratives are that there's an opportunity for those who invest a lot of their time across. You know, whether you're recruiting patients, whether the coordinators are helping, all the work that goes into it. On top of designing and organizing it, there's opportunity that there's going to be lots of opportunity to be involved in the scholarship that's associated. And so there's the primary paper, which is always, you know, a big, exciting period of time. But then there's all these other questions that come up usually, or that have come up in the beginning. So what do you do? Well, I mean, obviously you want to get them disseminated and presented everywhere as widely as you can. But if you want to have great impact, and we see it happen time and time again, it's great to say this is the body of knowledge outside of the primary paper and the learning we've had. And so putting it together as a compendium, as a supplement makes a lot of sense. And negotiating that up front, actually, in your budgeting, because these things often aren't inexpensive. They have cost.
Trial groups will often say, if we're thinking of doing this because we've planned the fact that we might have several secondary papers, let's put some budget in for publication costs, and we've done that in the past as well.
So there's lots of ways that you can do that.
Any host of large trial groups have probably considered this and have done that. The parity group that was a very large trial in tumors and musculoskeletal tumors was a similar thing where there was a supplement that was done. Mark. That this would have been under your tenure, I believe.
And it was good because it gives everybody a chance to look at the overall findings. And the risk here is minimized of coming up with, as you would say, papers that are kind of contradicting each other. When you're putting together a supplement, investigators just naturally think of the storyline and they're very cognizant of making sure the papers all add something new to the overall message rather than if you just put them everywhere out there. You can have situations, and we've seen that in the past. Right. Where there's so many papers that people lose sight of the overall message of the study.
[00:17:59] Speaker B: Yeah, yeah. And this is a bit of a shameless promotion on our part for the journal, but the journal is very interested in. In working with collaboratives to work out how this can be done in a compendium.
To be generally done is open access and just. I'll just say it. We're open for business.
[00:18:24] Speaker C: No, you're absolutely right. I mean, I think. I think. I think, you know, the key thing is with any orthopedic drill and, you know, any large group, you're looking to say, where can I get maximal impact? Right, Mark. And you're saying in doing a large study allows you to have impact in multiple countries all at once or multiple centers all at once. Which is great because you want to disseminate it, but you do want to find a partner. I agree with you. That will help disseminate it beyond your, you know, your own abilities to do so. And obviously shamelessly. I do agree that a lot of. A lot of. Has a lot of, you know, sources, you know, I would say multiple channels in which we can promote and. And do as such. And the question is, you know, deciding and working with authors of large trial groups. And I'm using trials, but could be observational studies, just large groups that have, you know, program of research that is of interest. Working with them early rather than late would be really good for us because then we can really help organize these things in a way that makes sense for them and for us.
[00:19:24] Speaker B: Yeah, great way to end it. Thanks for entertaining me and my desire to. To delve into a little bit deeper on secondary analysis. I appreciate it. And I'm gonna. I'm gonna finish this great coffee from Little Bear and head off to the meeting, so.
[00:19:41] Speaker C: Oh, look at you. Excellent. Enjoy the rest of your day. This is amazing.
[00:19:45] Speaker B: Yeah. Have a great day.
[00:19:47] Speaker C: You too.
[00:19:48] Speaker B: Cheers. Yeah.
[00:19:54] Speaker C: Sam.
