The Clinical Trials Team - Roles & Responsibilities

In a research study, a clinical trial tests a new medical treatment or a new way of using an existing treatment to see whether it will be a better way to avoid and screen for diagnosing or treating a disease. Purpose of clinical trial:

A research study that is performed on individuals for evaluation of a medical, surgical, or behavioral intervention.

Clinical Research Careers

Clinical Research Associate (CRA)

Clinical Research Coordinator (CRC)

Drug Safety Monitor (PV)

Clinical Trial Assistant (CTA)

Clinical Research Nurse (CRN)

Medical Monitor (MM)

Principal Investigator (PI)

All Research Professionals (ICH GCP)

Types of clinical trials:

Prevention trials

Screening trials

Case control studies

Cohort studies

Cross sectional studies

Figure no. 1: clinical trial team flowchart.

Figure no. 1: clinical trial team flowchart

Clinical research trial team:

The success of a quality clinical research program is essential for developing and maintaining an impeccable clinical research trial team. It is the main component of a research program because total time and effort for conducting a clinical trial; nurses and data managers each contribute more than 30%. On the other hand, physician’s contribution to clinical research is only 9%.

Roles and Responsibilities of clinical trial personnel

Clinical research team:

Participants are provided with information about the clinical trial.

The content of the informed consent is explained.

Reporting of adverse events or drug reactions.

report suspected misconduct.

Protect the integrity and confidentiality of records and data during the clinical study

Responsibilities:

Appropriate training

Following of GCP standard

Following required protocols

Investigator: 

• Following ethical principles.

• Provide education programs.

• Design and conduct clinical trials for policies and procedures.

• Refer to GCP course for training.

• Determines the scientific, technical, and administrative aspects of the research project.

Responsibilities: 

• Conduction of trial, statement, protocol, and applicable regulations.

• Protection of rights and welfare of participants.

• Obtaining informed consent.

• Maintenance of proper records.

• Management of all safety reports and financial disclosure reports.

Screen Shot 2021-05-04 at 10.45.38 AM.png

Figure no. 2: Roles of clinical research controller.

Figure no. 3: Responsibilities of clinical research controller.

Figure no. 3: Responsibilities of clinical research controller.

Figure no. 6: Responsibilities of data manager.

Figure no. 6: Responsibilities of data manager.

Sponsor: 

Selection of qualified investigators.

Ensures proper monitoring of the clinical trial.

References: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3092661/ - The Clinical Research Team https://clinicaltrialpodcast.com/clinical-research/ - 15 Clinical Research Job Roles & Responsibilities (2021) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3051859/ - Clinical Investigator Responsibilities https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6042393/ - How to engage stakeholders in research: design principles to support improvement

Advancing Clinical Research Education.

The 3 Secrets To Becoming an Effective Leader In Clinical Research

Guest Column | March 16, 2021

The 3 secrets to becoming an effective leader in clinical research.

By Laurie Halloran and Michelle Pratt, Halloran Consulting Group, Inc.

clinical research study leader

Experience in clinical research provides a solid foundation, which better positions you to launch your career into management- and executive-level roles in the life science industry. However, there’s a caveat regarding career advancement: your responsibilities will become quite different as you progress upward within an organization. Detailed clinical research activities have less of a presence in your day-to-day responsibilities, while driving strategy and influencing the direction of your teams become the focus as you grow into more senior positions within an organization. This is important to consider when mapping your career path. As a future leader, you will be called to develop the skillsets and level of confidence required to lead and guide teams while moving away from the operational execution.

It’s common to aspire to become a senior leader in the clinical research industry, and luckily there are many different directions for growth. If you can take one thing away from this article, let it be the importance of taking time to explore multiple roles before setting your sights on any one position. Enhancing your leadership skills and developing self-awareness and adaptability take time and experience – an essential first step in moving toward a leadership role is understanding the skillsets necessary to be an effective leader. This article will highlight the key skillsets to understand, develop, and refine along your journey to leadership, using stories to illustrate these skillsets in use.

Understand Your Strengths And Weaknesses

The ability to recognize your strengths and weaknesses is important for personal development at any job level.  However, high self-awareness is especially necessary when determining which leadership career path to pursue. For example, if you are interested in following a path from a trial management role in clinical operations toward vice president (VP) of business development at a sponsor company, you will need to build different strengths than those required to become a chief operating officer (COO) at a contract research organization (CRO) or to eventually co-found a clinical technology company. Over the past year, some of our Halloran employees had the opportunity to take a self-evaluation called CliftonStrengths , which includes a set of questions that identify a participant’s top strengths across the spectra of executing, influencing, relationship building, and strategic thinking. Aptitude in each of these categories impacts a person’s ability to be a good leader. Utilizing a tool to help identify your stronger and weaker attributes is a great way to begin determining a career direction and creating a development strategy to help you achieve your advanced career goals.

The expression “If you want to know the road ahead, ask a person who’s been there” describes a key component on the journey to leadership. When leadership becomes the next logical step for you in your career, it’s extremely helpful to reach out to leaders whom you admire and respect. Understanding a person’s journey, the glorious and the not so glorious, is an invaluable way to learn. The perspective of someone “who’s been there” can help shape the decisions you make as you ascend within your career. Initiating these conversations is also an opportunity to establish a mentorship relationship, which can continue to be beneficial even after you have reached a leadership level.

An outside perspective: Shelly has maintained a relationship with Lee, her former instructor from her M.S. in Clinical Research coursework, over the years since her graduation. When her current manager, Alice, asked her to consider a lateral move to an expanded role where she will report to a very difficult leader, Shelly felt anxious about the opportunity. Lee knows Alice as well, so Shelly reached out to him to get his perspective and insights. Lee’s opinion regarding Shelly is that she is a solid but tentative manager, sometimes afraid to advocate for herself and her ideas. After Shelly described her potential new manager, Lee suggested that reporting to a challenging leader in the new role would provide Shelly the opportunity to develop her strengths in managing up and self-advocating. Shelly respected Lee’s experience and perspective and felt she could make a balanced decision rather than one colored by her anxieties.

Be An Adaptable Leader

Regardless of your desired leadership career path, adaptability is key to being successful in your role. If you are looking for guidance on adaptable leadership, the Situational Leadership Model 1,2 provides different considerations to draw upon when assessing leadership opportunities. Situational Leadership Theory states that there is no single "best" style of leadership. Effective leadership is task-relevant, and the most successful leaders are those who adapt their leadership style to an individual or group’s performance readiness (ability and willingness). This means that situational leaders can effectively adjust their management style based on the situation at hand, including understanding and adapting to their team members’ respective development and maturity levels. This type of leadership is often highly successful, but as the theory states, there is no single “best” style of leadership, so it’s best used as a guide rather than a mandate.

To relate this to a clinical research role, someone who has multiple years of experience as a clinical research associate (CRA) at a CRO and has just joined your biotech company as a clinical project manager (CPM) needs a very different level of guidance and coaching on trial management responsibilities, such as selecting sites, than an employee hired just out of college as a clinical trial associate (CTA). A manager who takes a very hands-on approach with all employees could potentially be micromanaging and demotivating to this new CPM. A productive approach is to have a conversation with this CPM about her comfort level with project management and vendor oversight to determine what support she needs. In contrast, the new CTA likely needs an overview of the nuts and bolts of clinical trials and detailed instruction on each task they need to complete.

Paul Hersey and Ken Blanchard described this leadership style in their 1969 book Management of Organizational Behavior . 3 Originally called the “Life Cycle Theory of Leadership,” the authors renamed the theory to “Situational Leadership Theory” several years later. They posited that no singular leadership style is appropriate for every situation and outlined four styles within the overarching theory. They also noted that the leader’s and subordinates’ individual attributes and actions contribute to determining a best-fit leadership style in a given situation.

Table 1: Hersey-Blanchard Leadership Styles

The right leadership style heavily depends on the person or group being led. Below is a set of labels to use for an assessment of an individual’s level of maturity. The maturity levels outlined below are task-specific. This means the person might be generally skilled, confident, and motivated in their job but would still have a low maturity level when asked to perform a task requiring skills they don't possess.

Table 2: The Four Levels of Maturity

To do this effectively adds a layer of assessment for a manager. With the CPM and CTA example above, the manager will need to determine how comfortable, confident, and willing each of the employees are with trying new things, understanding their knowledge and skill gaps, and stretching themselves to support the various trial activities. They may have the same capability level to support their assigned tasks; however, the activities of the manager are going to be very different if, for example, the CTA’s maturity level is lacking confidence to give things a try.

It’s important to understand that leadership styles are not one size fits all. Effective leaders need to be flexible and adaptive to be able to captivate a group and influence a team’s ability to perform. This skillset is challenging to develop.  As a leader, it’s critical to constantly assess the person(s) you are leading and to modify your strategy in order to grow that person’s or group’s skills. This is an exercise in patience and an opportunity for continuous growth.

What it looks like when it works: Jessica grew over the course of her executive tenure as she sought ways to tailor her leadership style to her individual direct reports, her peers, and her management. Each of her Clinical Development VP level direct reports, all very seasoned technical experts, required unique guidance and direction. As she grew more comfortable leading her full team, she pulled back and led each unique group slightly differently based on what she knew of each report’s strategic thinking, leadership qualities, and general capabilities. Mark, one of her VPs, was a very seasoned clinical operations professional, but needed coaching in corporate finance to make better informed project resourcing decisions based on profit, loss, and gross margins. After this coaching, Jessica needed to provide less input on his approach, which allowed her time to participate more fully in executive planning activities and to support the launch of new ideas and programs across her team, promoting strategic skill development and capability expansion. One thing she asked of her VPs was to proactively propose major divisional goals to support and grow the organization over the next three years. Her goal in doing this was to pave the way for her VPs to take initiative so she could step back and provide insight from the executive team only when needed. Mark proposed embarking on a decentralized clinical trial initiative. Given his new strengths in finance, Mark was able to more aptly define the value proposition of this initiative as well as the costs and benefits, such as decreasing some of the sponsor’s reliance on its CROs.

Think Strategically 4,5,6

Self-awareness and adaptability are required to provide leaders and executives time for thinking strategically. Strategic thinking is defined as “a mental process applied by an individual in the context of achieving a goal, or set of goals, in any type of overall venture.” As a cognitive activity, it produces big directional ideas or insights.

When applied in an organizational strategic management process, strategic thinking involves generating and applying distinctive business insights and opportunities intended to create a competitive advantage for a firm or organization. Strategic thinking can be an individual exercise or a collaborative one such as the example described above between Jessica and Mark. Group strategic thinking can create more value than individual strategic thinking, as it enables proactive and creative dialogue through which individuals gain visibility into other people's perspectives on critical and complex issues. This is certainly beneficial in highly competitive and fast-changing business landscapes.

So, are strategic thinkers born or made? In our opinion, if you understand the concepts of strategic thinking, you can refine them.  However, it’s much easier to develop this skillset if you genuinely enjoy big picture thinking. Observing others who do this well during real situations is the best training.

Geoff was a manager of junior level clinical research staff. His manager recognized his strengths in data analysis and had him take on the management of operational metrics across the company. Geoff began leading monthly meetings to enable service area heads to proactively discuss their team’s pipeline, which facilitated discussions about prioritization across the company. Having this opportunity, Geoff was able to digest the data and begin making recommendations for strategic initiatives that could help improve the management team’s visibility to the business coming in and better predict future revenue. He was able to build on his strengths to enhance his strategic thinking skills, and his manager supported this development by giving him the opportunity to take ownership, which grew his role in the company to a valued member of the leadership team.

At Halloran, our executive team has been able to grow their strategic thinking skills as a collaborative group. At strategy sessions, each team member has a well-defined role, and everyone is respectful of each other’s talents, insights, and input, such as prior experience in clinical research. Everyone shares their ideas and no idea is ranked better than any other, which facilitates open communication, problem solving, and consensus. This environment enables our executive team to arrive at decisions that they feel good about, and that will benefit everyone at the company. The team prepares an agenda prior to each session and appoints a facilitator to ensure meetings stay on track. Sometimes the decisions come quickly and easily, and sometimes they require more discussion and time. Ideas requiring further development are discussed. Over time, and particularly through the past year, the team has learned the importance of lifting each other up to better enable building a long-term, multifaceted vision for the business. Having the right combination of perspectives drives better decisions for all.

References:

  • https://situational.com/situational-leadership/
  • https://online.stu.edu/articles/education/what-is-situational-leadership.aspx
  • Management of Organizational Behavior: Utilizing Human Resources, 3rd Ed : Paul Hersey and Kenneth H. Blanchard Englewood Cliffs, N.J.: Prentice-Hall, 1977.
  • http://www.inc.com/paul-schoemaker/6-habits-of-strategic-thinkers.html
  • https://www.forbes.com/sites/terinaallen/2018/11/20/3-unmistakable-signs-youre-a-strategic-thinker/?sh=118ddbfa6921
  • https://hbr.org/2019/09/how-to-demonstrate-your-strategic-thinking-skills

About The Authors:

Author

Like what you are reading?

Sign up for our free newsletter, newsletter signup.

Newsletter Signup

Get the latest articles from Clinical Leader delivered to your inbox.

Trends in Medicine

8 elements of a successful clinical research career.

Ebrahim Barkoudah teaching a class

As medical professionals continue to make great strides in understanding the epidemiology of diseases and developing innovative ways to prevent and treat them, it’s important that they take the time to fully develop their clinical research skills in order to make the most of their efforts.

Physicians, scientists, investigators, pharmacists and pharmaceutical leaders who have a strong clinical research foundation to build on will play an increasingly significant role in moving the medical field forward in exciting new directions over the next decade, according to Ebrahim Barkoudah, MD, MPH, FACP,  Medical Director at Brigham and Women’s Hospital in Boston and Assistant Professor of Medicine at Harvard Medical School.

Building on Healthcare Experience to Advance Clinical Research

Dr. Barkoudah points out that while patient care continues to be the main focus for many physicians, a growing number of them are also building on their experiences in the exam room by broadening out into the investigative side as well to deepen their understanding of disease states and determine how best to treat them.

“Many of the skills needed to excel at clinical research extend beyond what is taught in medical school,” he says. That’s why healthcare professionals can benefit by being proactive in seeking out opportunities to sharpen their research knowledge and skills so they can maximize their contributions to the field, both now and moving forward, he stresses. Further, when clinicians focus on building up their strengths in this area earlier in their career path, it can help them establish themselves for success in the field.

Key Competencies for Clinical Researchers

Following are eight key areas that Dr. Barkoudah recommends healthcare providers and others focus on in order to set themselves up for success in the clinical research arena.

  • Develop your understanding of best research practices. Anyone with aspirations in the clinical research field needs to be up-to-speed on epidemiology and research design. For instance, Dr. Barkoudah says that as a clinical researcher, you’ll need to know how to develop relevant questions, formulate hypotheses you can test, and design clinical studies to gather important insights.
  • Recognize how to effectively track and interpret your data. Sharpening your ability to perform statistical analytics on your findings is also essential, he says. This can encompass a wide range, from establishing baseline criteria and determining what you are measuring and why, to creating a database to gather your findings, and determining how best to report the results in a meaningful way.
  • Consider the patient safety and ethical concerns involved in your research efforts. These are important details that need to be taken into account throughout every phase of your explorations—from protecting the safety of clinical trial participants, to ensuring informed consent, to identifying and reporting adverse events, to complying with the Health Insurance Portability and Accountability Act (HIPAA). These are just some of the many sensitive elements that must be in the forefront of your mind every step of the way, Dr. Barkoudah says.
  • Familiarize yourself with regulatory guidelines. Whether you are exploring the biology of a disease or the effectiveness of a pharmacological intervention or device to treat it, it’s crucial that you understand the framework and regulations guiding your efforts so you can comply with them appropriately. Just keep in mind that the regulations can vary from one country to the next, so you need to stay abreast of the specifics for your situation, Dr. Barkoudah stresses.
  • Strengthen your leadership skills. Clinical researchers need to be able to build  strong multi-disciplinary teams and lead them to success. This requires strong negotiation skills, as well as a deep understanding of how to designate duties and manage staff. If you are working with colleagues from different backgrounds, you also need to be sensitive to their cultural norms and make sure you are finding common ground. You also need to be able to create strong networks that cut across different disciplines for best results.
  • Communicate in a language that will resonate with different audiences. This is essential to get everyone on the same page. For instance, ensuring effective communication among different sites, sponsors, clinical research organizations and regulators can be the key to a successful research effort. Dr. Barkoudah points out that you also need to be able to organize the results of your research for publication in timely medical journals. If you’ll be sharing your findings with a lay audience, you also will need to know how to translate scientific techniques into user-friendly language.
  • Be prepared to navigate the funding world. Whether you are seeking support for novel research or exploring a tried-and-true concept, you’ll need to find the best funding option for your specific situation and constraints. Make sure you understand any criteria, restrictions or expectations that go along with support for your efforts. You will also need to be familiar with how the funders want your findings reported back to them.
  • Translate your findings into meaningful steps for patients. The most successful clinical researchers can critically evaluate medical literature. This requires looking beyond the research setting to determine how best to apply a variety of discoveries in real-world scenarios. Such critical thinking skills are essential to bring real benefit, such as through the creation of new treatments that lead to improved outcomes for patients.

Apply Clinical Research Skills Broadly

The good news is that the most effective clinical researchers bring to the table a wide range of strengths—including epidemiology, biostatistics, study design, ethics in research, evidence-based medicine, and communicating scientific research—that can be essential for success in a wide range of roles and settings.

When you make a conscious effort to strengthen your knowledge in all of these areas, Dr. Barkoudah says that in the process, you will elevate your career to new heights. At the same time, you will also be positioned to help unravel a variety of medical mysteries and lead to better outcomes for patients.

Written by Lisa D. Ellis

Related Articles

Sanya Thomas.

A Physician’s Journey into the World of Clinical Investigation and Bench Science

Two female health care workers talk in a lab.

Challenges and Opportunities for Women in Clinical Research

Women working in clinical research can bring real value when they serve in leadership positions. Yet…

Headshot of Miriam S. Teixeira.

Opening Doors to a New Career and Clinical Protocol

© 2024 by the President and Fellows of Harvard College

clinical research study leader

Leadership and Line Management in Clinical Research

clinical research study leader

Anatoly Gorkun

Senior manager, global clinical management, ppd.

clinical research study leader

Hugh Devine

Senior director, global clinical management, ppd.

Abstract : Line management is generic, utilizing the same approaches throughout all industries; line management brings together company needs and its workforce to deliver company objectives. This article provides an overview of line management styles and line-management-through-leadership approaches in the clinical research environment. The line management cycle – team building, support, motivation, and development—is described. Real-life examples illustrate how line management can support clinical research deliverables.

Introduction to Line Management

Line management brings together both the company needs and its workforce to deliver company objectives. Also, line management is not an exact science. The management style used needs to be adapted to reflect the needs of the direct reports. While the goal of line management is to deliver company objectives, it is important to support direct reports so that they can develop and grow in their roles.

Both line managers and direct reports have expectations. Based on an internal survey, line managers expect the following from their direct reports:

  • Reliability
  • Dependability
  • Accountability
  • Business mindset
  • Flexibility
  • Smart working
  • Delivery of metrics

Direct reports responded to the survey saying that they expected the following from their line managers:

  • Work-life balance
  • Development
  • Transparency
  • Knowledge and guidance

Management Styles

Application of appropriate management styles result in a happy and productive work environment.

The authors use a variety of management styles, adjusting the style that they use to reflect the needs of the individuals being managed. There are six main management styles described in the literature 1 :

  • Authoritative
  • Affiliative
  • Participative
  • Pacesetting

Each management style has its own strengths and weaknesses and works best in certain situations.

A directive (coercive) management style (“do it the way I tell you”) may be the best during emergencies or when the person doing the job is less experienced.

An authoritative management style (visionary) shares the manager’s vision and provides long-term direction.

An affiliative management style puts “people first, task second.” As an example, a CRA started the first 1:1 meeting with his line manager saying, “I know all managers are interested in metrics, so I am ready to start with that.” Instead, the manager suggested that the CRA start talking about himself first: how happy he was doing his job, any challenges and successes he experienced, if he was achieving his own goals for career growth. Turning the conversation towards himself helped to develop a better rapport between manager and CRA. An affiliative management style works because it is people who deliver the metrics. If the line manager takes care of the individual, the individual will try harder to deliver on their metrics.

In a participative or democratic management style, “everyone has input.” The participative management style especially works with the direct reports who are experienced enough and may not appreciate the directive (coercive) approach. Brainstorming with the team or asking team members for their input on how to complete a task are ways to use the participative management technique, engaging team members and giving them a sense of ownership.

Pacesetting (“do it myself”) and coaching (“developmental manager”) management styles are also necessary and very important.

Rosalind Cardinal, author of The Resilient Employee, said , “The key to being an effective leader is to have a broad repertoire of styles and to use them appropriately.” 3       

Managing through Leadership

A manager is the person who is responsible for controlling or administering an organization or a group of staff. Some managers achieve this by technical administering, telling their direct reports what to do and following up with reminders to have the work done. The opposite of technical administering and controlling is managing through leadership; a leader is a person who:

  • Has and shares their vision
  • Provides support
  • Has empathy
  • Is creative
  • Achieves objectives through motivation
  • Builds the team
  • Listens and hears
  • Provides continuous improvement
  • Takes risks

The cycle of managing through leadership has four main components: Team building, support, motivation , and development.

Team Building and Continuous Improvement

A group of people is not necessarily a team. Table 1 provides an overview of team building and continuous improvement. Continuous improvement is necessary because the team is a living organism that changes over time. Building and maintaining the team is important because a happy and professional team is a productive team.

Recruiting the right people is a key to forming a productive team. A good job description is necessary. The individual being considered must have the right qualifications and experience as well as enthusiasm and the right attitude. The line manager must also determine whether the individual will be a good fit for the team and must assess his/her decision-making and critical thinking abilities.

Expectations can be set beginning with the job interview and/or the introductory meeting. Together with asking questions during an interview, it might be a good idea to share the company’s expectations. It’s also important to remind team members about company expectations when appropriate, the benefits of exceeding expectations, and the consequences of not meeting expectations. The authors find it important to praise team members and the team as a whole for their achievements as well as to make high achievement the norm.

Creating a transparent and friendly environment is another part of the team building technique. Ways to do this include hiring the right people, sharing values and expectations, and explaining the purpose of tasks and goals. Line managers provide or arrange for coaching as necessary for the team members.

Team building also requires being able to manage both conflicts and difficult team members. Even when the line manager hires the right people, there is no guarantee that people will not change over time. The line manager needs to try to understand whether any team members are unhappy and manage complaints proactively, have open dialogue with team members, take actions to resolve conflict, and handle difficult team members quickly. Balancing strengths and weaknesses is another approach to team building. The line manager should get to know team members in order to understand their likes and dislikes, their technical knowledge, their strengths, and the opportunities that need to be developed. Tasks need to be allocated to the appropriate people, and the management style should reflect the way in which each individual is going to be managed in the best way.

If a team member under performs, the line manager should step in to manage poor performance as soon as possible. Most people do not come to work to do a bad job; however, if someone on the team isn’t performing, this affects the other team members. Once the line manager identifies under performance, the manager should perform a root cause analysis to determine the problem. There are many reasons for poor performance, which include personal or professional issues. After identifying the root cause, the line manager should take immediate action to resolve the problem.

Team Support

Supporting the team (Table 2) gives the team members confidence. It works better when line managers are approachable, listen to their team members, and act upon information received.

The line manager should be patient, open, and fair, while remembering that patience is rewarded. As an example, a manager once asked one of his direct reports if she would like to deliver a presentation at an investigators’ meeting. She said “no” because she had never done it before. The manager decided to wait. She came back in a few days asking if the presentation was still available. They thoroughly worked together to prepare. The presentation appeared to be a great success, being a very strong further motivator for this individual.

Supporting the team also includes:

  • A fair distribution of the workload
  • Sharing expectations including those aligned with the end of the year performance review meeting
  • Fair promotion, salary increases, and bonuses
  • Not showing favoritism to any team members

Line managers should show their commitment to the team and be positive, serving as an example.

Protecting team members from unreasonable demands and serving as a buffer is also part of motivating the team. Nowadays at most international companies, people work in virtual teams, so they need to be culturally savvy to work effectively and efficiently while understanding different cultures, otherwise communication issues may occur. There may be situations regarding cultural misinterpretations where the line manager may need to step in to help.   

For example, a newly hired in-house CRA had great credentials and was a very nice individual to work with; however, there were issues with two sites where the standards of communication were different. In this case, the CRA’s manager had an open and fair conversation with the CRA discussing the issue. Together, they developed a plan to overcome the problem. The CRA’s communication style was adjusted, and the issue was successfully resolved.

Team Motivation

Motivating the team is another important part ofmanaging through leadership (Table 3). Leading, not ordering, and setting up clear achievable goals are parts of motivating the team while eliminating distractions. In the case of changes at an organizational level, such as a restructuring or an acquisition, the line manager may not be able to do much. However, the manager can help to explain and support team members. Distractions at the team level should be identified and managed quickly.

Line managers empower and encourage team members. Ways to do this include:

  • Demonstrating trust
  • Communicating vision and expectations
  • Encouraging development and improvement
  • Delegating tasks
  • Encouraging team members to go above and beyond
  • Accepting flexibility (there are many ways of doing things)
  • Inspiring creative and critical thinking
  • Showing appreciation

Several years ago, one of the authors hired a research administrator. The administrator had a scientific degree and was brilliant in that role; however, this person had low self-confidence. It was thought that the administrator was a great candidate to become a clinical research associate (CRA); however, when approached about this, this person was afraid to make the move. The author continued encouraging and motivating the administrator, and at some point that person gained the confidence to become a CRA, followed by a clinical trial manager. Now that person is a very successful global project manager. Without such manager’s support, that successful project manager might still be working at an administrative level.

Line managers should not refrain from praising their direct reports, as this motivates and encourages them, thus often boosting their confidence. In turn, it helps to set higher expectations. As an example, one of the authors maintains an Achievement Board for all of his direct reports using OneNote. All of the achievements are posted on the board – nothing is missed and no one is forgotten. The achievements are reviewed on a monthly basis at planned 1:1 meetings, and at the end of the year the Achievement Board serves as a very useful tool for the end-of-year performance review and possible rationale for promotions.

Recognizing achievements is a way to raise the bar and help team members to develop. Other ways to stretch individuals is to continue to challenge and support them while sharing successes with the broader team.

Team development

Team development (Table 4) can start with determining the aspirations, likes, and dislikes of the team members. Line managers should try to take these preferences into account when possible. However, development can be achieved by challenging them as well. There was an interesting real-life example where a very experienced Principal CRA joined the company. The CRA’s experience was primarily in oncology, and there was the agreement that this person would be allocated to work on oncology studies. However, it happened that there was no need for a CRA for an oncology study but there was a need for a CRA on a HIV trial. The CRA kindly agreed to take on the challenge, and we were going to review the allocation in six months. When the time came, the CRA requested to remain on the HIV study until the trial ended. The CRA has now been working on the HIV study for nearly three years, saying that some very valuable experience was gained and personal horizons were extended. So, this example suggests that it might be a great development exercise to take team members out of their comfort zone in order to promote their development.

Managing through leadership includes identifying development needs. We do this by reviewing career-building plans at different management levels by reviewing the plans with direct reports from time to time to implement changes, if needed. Providing reasonable flexibility can be very useful in facilitating staff retention. There are several ways to provide flexibility including allowing employees to work from home occasionally, perhaps start work later sometimes, or take on additional responsibilities or a specific study. A flexible approach should be used when it benefits both the team member and the company. At the same time, it is important to ensure that team members remember standard expectations so that flexibility is not abused.

Training is also an important element in managing through leadership. The global trend is to deliver training online, thereby giving up classroom-style training. However, line managers can still support their reports by:

  • Providing suitable opportunities to learn by doing
  • Conducting cross-functional team calls
  • Delegating new tasks or allocating studies with new indications to team members
  • Learning from others, including systems and processes
  • Sharing best practices
  • Adopting Q&A sessions and discussions

Coaching by the line manager and mentoring by more experienced colleagues are also ways to support team development.

Managing through leadership also includes boosting the confidence of team members. This is done through training and supporting a “you can do it” approach. Line managers identify the subject matter experts who can provide help when needed and be available to provide advice themselves. Other ways to boost confidence include appointing a mentor, providing the opportunity to suggest possible solutions, praising achievements, and encouraging colleagues and managers to acknowledge a job well done.

Learning Line Management

In line management, it is important to realize that you never stop learning. We can learn from both positive and negative real-life examples, learning by doing, shadowing, self-learning, training courses, and experimenting.

We, the line managers, should remember that our goal is to bring together the company needs and its workforce to deliver company objectives. We achieve it by building, supporting, motivating and developing happy and productive teams. There are mutual expectations between line managers and their direct reports. Line managers should be creative in identifying the management style that works in each particular situation. The successful manager develops a culture of leading by example, moving forward through a non-stop and ongoing cycle.

Table 5 details resources related to line management in clinical research.

  • Recruit the right people
  • Set up expectations
  • Create a transparent and friendly environment
  • Manage conflict and difficult team members
  • Balance strengths and weaknesses
  • Manage underperformance
  • Be approachable and listen
  • Be patient, open, and fair
  • Show commitment and positivity
  • Direct and delegate
  • Protect team members
  • Leading, not ordering
  • Setting up clear goals to achieve
  • Eliminating distractions
  • Empowering, encouraging, and praising
  • Raising the bar

Team Development

  • Knowing likes and dislikes
  • Identifying development needs
  • Providing reasonable flexibility
  • Training, coaching, and mentoring
  • Boosting confidence

Resource on Line Management in Clinical Research

What is Leadership? 10 Ways To Define It. https://www.game-learn.com/what-is-leadership-ways-to-define . Accessed 12/18/19.

6 Management Styles and When Best to Use Them – The Leaders Tool Kit.

https://leadersinheels.com/career/6-management-styles-and-when-best-to-use-them-the-leaders-tool-kit/ . Accessed 12/18/19.

Sean McFeat. A 20 Point Checklist For Effective Leadership

https://www.linkedin.com/pulse/20-point-checklist-effective-leadership-sean-mcpheat . Accessed 12/18/19.

Training Line Managers, CIPD Podcast. June 2016. https://www.cipd.co.uk/podcasts/training-line-managers . Accessed 12/18/19.

Line managers play crucial role in supporting employee well-being and engagement. CIPD. February 2017. http://www.cipd.co.uk/news-views/news-articles/line-managers-support-wellbeing-engagement. Accessed 12/18/19.

Trust the Process: 10 Tips to Empower and Encourage Your Staff

https://www.business.com/articles/trust-the-process-10-tips-to-empower-and-encourage-your-staff/

Accessed 12/18/19.

Conger JA. The necessary art of persuasion. Harvard Business Review , May–June 1998. https://hbr.org/1998/05/the-necessary-art-of-persuasion. Accessed 12/18/19.

[1] Ros Cardinal. 6 Management styles and when to use them – the leaders toolkit, April 2013

2 Sean McFeat. A 20 Point Checklist For Effective Leadership.

https://www.linkedin.com/pulse/20-point-checklist-effective-leadership-sean-mcpheat. Accessed 12/18/19 .

3 Rosalind Cardinal. The Resilient Employee: The essential guide to coping with change and thriving in today’s workplace

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

This site uses Akismet to reduce spam. Learn how your comment data is processed .

Clinical research leadership-"A blueprint"

Affiliations.

  • 1 Periodontology, Clinical Trials Unit, Bristol Dental School, University of Bristol, Bristol, UK. Electronic address: [email protected].
  • 2 Periodontology, Clinical Trials Unit, Bristol Dental School, University of Bristol, Bristol, UK.
  • PMID: 31075371
  • DOI: 10.1016/j.jdent.2019.05.004

The principles of leadership in academic research, reflect those of life in general and differ only by circumstance. A great leader is one who inspires and energizes, motivating and empowering the whole team to achieve. They articulate a vision, establish direction, clarify the big picture and set clear strategies in a positive culture. A great leader needs to align and connect people by fostering excellent communication channels, gaining commitment and building teams and coalitions.

Keywords: Clinical research; Leadership; Models of academia; Quality.

Copyright © 2019. Published by Elsevier Ltd.

Publication types

  • Leadership*

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Int J Environ Res Public Health

Logo of ijerph

Leadership Effectiveness in Healthcare Settings: A Systematic Review and Meta-Analysis of Cross-Sectional and Before–After Studies

Vincenzo restivo.

1 Department of Health Promotion, Maternal and Infant Care, Internal Medicine and Medical Specialties (PROMISE) “G. D’Alessandro”, University of Palermo, Via del Vespro 133, 90127 Palermo, Italy

Giuseppa Minutolo

Alberto battaglini.

2 Vaccines and Clinical Trials Unit, Department of Health Sciences, University of Genova, Via Antonio Pastore 1, 16132 Genova, Italy

Alberto Carli

3 Santa Chiara Hospital, Largo Medaglie d’oro 9, 38122 Trento, Italy

Michele Capraro

4 School of Public Health, Vita-Salute San Raffaele University, Via Olgettina 58, 20132 Milan, Italy

Maddalena Gaeta

5 Department of Public Health, Experimental and Forensic Medicine, University of Pavia, Via Forlanini 2, 27100 Pavia, Italy

Cecilia Trucchi

6 Planning, Epidemiology and Prevention Unit, Liguria Health Authority (A.Li.Sa.), IRCCS San Martino Hospital, Largo R. Benzi 10, 16132 Genoa, Italy

Carlo Favaretti

7 Centre on Leadership in Medicine, Catholic University of the Sacred Heart, Largo F. Vito 1, 00168 Rome, Italy

Francesco Vitale

Alessandra casuccio, associated data.

Data will be available after writing correspondence to the author.

To work efficiently in healthcare organizations and optimize resources, team members should agree with their leader’s decisions critically. However, nowadays, little evidence is available in the literature. This systematic review and meta-analysis has assessed the effectiveness of leadership interventions in improving healthcare outcomes such as performance and guidelines adherence. Overall, the search strategies retrieved 3,155 records, and 21 of them were included in the meta-analysis. Two databases were used for manuscript research: PubMed and Scopus. On 16th December 2019 the researchers searched for articles published in the English language from 2015 to 2019. Considering the study designs, the pooled leadership effectiveness was 14.0% (95%CI 10.0–18.0%) in before–after studies, whereas the correlation coefficient between leadership interventions and healthcare outcomes was 0.22 (95%CI 0.15–0.28) in the cross-sectional studies. The multi-regression analysis in the cross-sectional studies showed a higher leadership effectiveness in South America (β = 0.56; 95%CI 0.13, 0.99), in private hospitals (β = 0.60; 95%CI 0.14, 1.06), and in medical specialty (β = 0.28; 95%CI 0.02, 0.54). These results encourage the improvement of leadership culture to increase performance and guideline adherence in healthcare settings. To reach this purpose, it would be useful to introduce a leadership curriculum following undergraduate medical courses.

1. Introduction

Over the last years, patients’ outcomes, population wellness and organizational standards have become the main purposes of any healthcare structure [ 1 ]. These standards can be achieved following evidence-based practice (EBP) for diseases prevention and care [ 2 , 3 ] and optimizing available economical and human resources [ 3 , 4 ], especially in low-industrialized geographical areas [ 5 ]. This objective could be reached with effective healthcare leadership [ 3 , 4 ], which could be considered a network whose team members followed leadership critically and motivated a leader’s decisions based on the organization’s requests and targets [ 6 ]. Healthcare workers raised their compliance towards daily activities in an effective leadership context, where the leader succeeded in improving membership and performance awareness among team members [ 7 ]. Furthermore, patients could improve their health conditions in a high-level leadership framework. [ 8 ] Despite the leadership benefits for healthcare systems’ performance and patients’ outcomes [ 1 , 7 ], professionals’ confidence would decline in a damaging leadership context for workers’ health conditions and performance [ 4 , 9 , 10 ]. On the other hand, the prevention of any detrimental factor which might worsen both team performance and healthcare systems’ outcomes could demand effective leadership [ 4 , 7 , 10 ]. However, shifting from the old and assumptive leadership into a more effective and dynamic one is still a challenge [ 4 ]. Nowadays, the available evidence on the impact and effectiveness of leadership interventions is sparse and not systematically reported in the literature [ 11 , 12 ].

Recently, the spreading of the Informal Opinion Leadership style into hospital environments is changing the traditional concept of leadership. This leadership style provides a leader without any official assignment, known as an “opinion leader”, whose educational and behavioral background is suitable for the working context. Its target is to apply the best practices in healthcare creating a more familiar and collaborative team [ 2 ]. However, Flodgren et al. reported that informal leadership interventions increased healthcare outcomes [ 2 ].

Nowadays, various leadership styles are recognized with different classifications but none of them are considered the gold standard for healthcare systems because of heterogenous leadership meanings in the literature [ 4 , 5 , 6 , 12 , 13 ]. Leadership style classification by Goleman considered leaders’ behavior [ 5 , 13 ], while Chen DS-S proposed a traditional leadership style classification (charismatic, servant, transactional and transformational) [ 6 ].

Even if leadership style improvement depends on the characteristics and mission of a workplace [ 6 , 13 , 14 ], a leader should have both a high education in healthcare leadership and the behavioral qualities necessary for establishing strong human relationships and achieving a healthcare system’s goals [ 7 , 15 ]. Theoretically, any practitioner could adapt their emotive capacities and educational/working experiences to healthcare contexts, political lines, economical and human resources [ 7 ]. Nowadays, no organization adopts a policy for leader selection in a specific healthcare setting [ 15 ]. Despite the availability of a self-assessment leadership skills questionnaire for aspirant leaders and a pattern for the selection of leaders by Dubinsky et al. [ 15 ], a standardized and universally accepted method to choose leaders for healthcare organizations is still argued over [ 5 , 15 ].

Leadership failure might be caused by the arduous application of leadership skills and adaptive characteristics among team members [ 5 , 6 ]. One of the reasons for this negative event could be the lack of a standardized leadership program for medical students [ 16 , 17 ]. Consequently, working experience in healthcare settings is the only way to apply a leadership style for many medical professionals [ 12 , 16 , 17 ].

Furthermore, the literature data on leadership effectiveness in healthcare organizations were slightly significant or discordant in results. Nevertheless, the knowledge of pooled leadership effectiveness should motivate healthcare workers to apply leadership strategies in healthcare systems [ 12 ]. This systematic review and meta-analysis assesses the pooled effectiveness of leadership interventions in improving healthcare workers’ and patients’ outcomes.

2. Materials and Methods

A systematic review and meta-analysis was conducted according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement guidelines [ 18 ]. The protocol was registered on the PROSPERO database with code CRD42020198679 on 15 August 2020. Following these methodological standards, leadership interventions were evaluated as the pooled effectiveness and influential characteristic of healthcare settings, such as leadership style, workplace, settings and the study period.

2.1. Data Sources and Search Strategy

PubMed and Scopus were the two databases used for the research into the literature. On 16th December 2019, manuscripts in the English language published between 2015 and 2019 were searched by specific MeSH terms for each dataset. Those for PubMed were “leadership” OR “leadership” AND “clinical” AND “outcome” AND “public health” OR “public” AND “health” OR “public health” AND “humans”. Those for Scopus were “leadership” AND “clinical” AND “outcome” AND “public” AND “health”.

2.2. Study Selection and Data Extraction

In accordance with the PRISMA Statement, the following PICOS method was used for including articles [ 18 ]: the target population was all healthcare workers in any hospital or clinical setting (Population); the interventions were any leader’s recommendation to fulfil quality standards or performance indexes of a healthcare system (Intervention) [ 19 ]; to be included, the study should have a control group or reference at baseline as comparison (Control); and any effectiveness measure in terms of change in adherence to healthcare guidelines or performances (Outcome). In detail, any outcome implicated into healthcare workers’ capacity and characteristics in reaching a healthcare systems purposes following the highest standards was considered as performance [ 19 ]. Moreover, whatever clinical practices resulted after having respected the recommendations, procedures or statements settled previously was considered as guideline adherence [ 20 ]. The selected study design was an observational or experimental/quasi-experimental study design (trial, case control, cohort, cross-sectional, before-after study), excluding any systematic reviews, metanalyses, study protocol and guidelines (Studies).

The leaders’ interventions followed Chen’s leadership styles classification [ 6 ]. According to this, the charismatic leadership style can be defined also as an emotive leadership because of members’ strong feelings which guide the relationship with their leader. Its purpose is the improvement of workers’ motivation to reach predetermined organizational targets following a leader’s planning strategies and foresights. Servant leadership style is a sharing leadership style in whose members can increase their skills and competences through steady leader support, and they have a role in an organization’s goals. The transformational leadership style focuses on practical aspects such as new approaches for problem solving, new interventions to reach purposes, future planning and viewpoints sharing. Originality in a transformational leadership style has a key role of improving previous workers’ and healthcare system conditions in the achievement of objectives. The transactional leadership style requires a working context where technical skills are fundamental, and whose leader realizes a double-sense sharing process of knowledge and tasks with members. Furthermore, workers’ performances are improved through a rewarding system [ 6 ].

In this study, the supervisor trained the research team for practical manuscript selection and data extraction. The aim was to ensure data homogeneity and to check the authors’ procedures for selection and data collection. The screening phase was performed by four researchers reading each manuscript’s title and abstract independently and choosing to exclude any article that did not fulfill the inclusion criteria. Afterwards, the included manuscripts were searched for in the full text. They were retrieved freely, by institutional access or requesting them from the authors.

The assessment phase consisted of full-text reading to select articles following the inclusion criteria. The supervisor solved any contrasting view about article selection and variable selection.

The final database was built up by collecting the information from all included full-text articles: author, title, study year, year of publication, country/geographic location, study design, viability and type of evaluation scales for leadership competence, study period, type of intervention to improve leadership awareness, setting of leader intervention, selection modality of leaders, leadership style adopted, outcomes assessed such as guideline adherence or healthcare workers’ performance, benefits for patients’ health or patients’ outcomes improvement, public or private hospitals or healthcare units, ward specialty, intervention in single specialty or multi-professional settings, number of beds, number of healthcare workers involved in leadership interventions and sample size.

Each included article in this systematic review and meta-analysis received a standardized quality score for the specific study design, according to Newcastle–Ottawa, for the assessment of the quality of the cross-sectional study, and the Study Quality Assessment Tools by the National Heart, Lung, and Blood Institute were used for all other study designs [ 21 , 22 ].

2.3. Statistical Data Analysis

The manuscripts metadata were extracted in a Microsoft Excel spreadsheet to remove duplicate articles and collect data. The included article variables for the quantitative meta-analysis were: first author, publication year, continent of study, outcome, public or private organization, hospital or local healthcare unit, surgical or non-surgical ward, multi- or single-professionals, ward specialty, sample size, quality score of each manuscript, leadership style, year of study and study design.

The measurement of the outcomes of interest (either performance or guidelines adherence) depended on the study design of the included manuscripts in the meta-analysis:

  • for cross-sectional studies, the outcome of interest was the correlation between leadership improvement and guideline adherence or healthcare performance;
  • the outcome derived from before–after studies or the trial was the percentage of leadership improvement intervention in guideline adherence or healthcare performance;
  • the incidence occurrence of improved results among exposed and not exposed healthcare workers of leadership interventions and the relative risks (RR) were the outcomes in cohort studies;
  • the odds ratio (OR) between the case of healthcare workers who had received a leadership intervention and the control group for case-control studies.

Pooled estimates were calculated using both the fixed effects and DerSimonian and Laird random effects models, weighting individual study results by the inverse of their variances [ 23 ]. Forest plots assessed the pooled estimates and the corresponding 95%CI across the studies. The heterogeneity test was performed by a chi-square test at a significance level of p < 0.05, reporting the I 2 statistic together with a 25%, 50% or 75% cut-off, indicating low, moderate, and high heterogeneity, respectively [ 24 , 25 ].

Subgroup analysis and meta-regression analyses explored the sources of significant heterogeneity. Subgroup analysis considered the leadership style (charismatic, servant, transactional and transformational), continent of study (North America, Europe, Oceania), median cut-off year of study conduction (studies conducted between 2005 and 2011 and studies conducted between 2012 and 2019), type of hospital organization (public or private hospital), type of specialty (surgical or medical specialty) and type of team (multi-professional or single-professional team).

Meta-regression analysis considered the following variables: year of starting study, continent of study conduction, public or private hospital, surgical or non-surgical specialty ward, type of healthcare service (hospital or local health unit), type of healthcare workers involved (multi- or single-professional), leadership style, and study quality score. All variables included in the model were relevant in the coefficient analysis.

To assess a potential publication bias, a graphical funnel plot reported the logarithm effect estimate and related the standard error from each study, and the Egger test was performed [ 26 , 27 ].

All data were analyzed using the statistical package STATA/SE 16.1 (StataCorp LP, College 482 Station, TX, USA), with the “metan” command used for meta-analysis, and “metafunnel”, “metabias” and “confunnel” for publication bias assessment [ 28 ].

3.1. Studies Characteristics

Overall, the search strategies retrieved 3,155 relevant records. After removing 570 (18.1%) duplicates, 2,585 (81.9%) articles were suitable for the screening phase, of which only 284 (11.0%) articles were selected for the assessment phase. During the assessment phase, 263 (92.6%) articles were excluded. The most frequent reasons of exclusion were the absence of relevant outcomes ( n = 134, 51.0%) and other study designs ( n = 61, 23.2%). Very few articles were rejected due to them being written in another language ( n = 1, 0.4%), due to the publication year being out of 2015–2019 ( n = 1, 0.4%) or having an unavailable full text ( n = 3, 1.1%).

A total of 21 (7.4%) articles were included in the qualitative and quantitative analysis, of which nine (42.9%) were cross-sectional studies and twelve (57.1%) were before and after studies ( Figure 1 ).

An external file that holds a picture, illustration, etc.
Object name is ijerph-19-10995-g001.jpg

Flow-chart of selection manuscript phases for systematic review and meta-analysis on leadership effectiveness in healthcare workers.

The number of healthcare workers enrolled was 25,099 (median = 308, IQR = 89–1190), including at least 2,275 nurses (9.1%, median = 324, IQR = 199–458). Most of the studies involved a public hospital ( n = 16, 76.2%). Among the studies from private healthcare settings, three (60.0%) were conducted in North America. Articles which analyzed servant and charismatic leadership styles were nine (42.9%) and eight (38.1%), respectively. Interventions with a transactional leadership style were examined in six (28.6%) studies, while those with a transformational leadership style were examined in five studies (23.8%). Overall, 82 healthcare outcomes were assessed and 71 (86.6%) of them were classified as performance. Adherence-to-guidelines outcomes were 11 (13.4%), which were related mainly to hospital stay ( n = 7, 64.0%) and drug administration ( n = 3, 27.0%). Clements et al. and Lornudd et al. showed the highest number of outcomes, which were 19 (23.2%) and 12 (14.6%), respectively [ 29 , 30 ].

3.2. Leadership Effectiveness in before–after Studies

Before–after studies ( Supplementary Table S1 ) involved 22,241 (88.6%, median = 735, IQR = 68–1273) healthcare workers for a total of twelve articles, of which six (50.0%) consisted of performance and five (41.7%) of guidelines adherence and one (8.3%) of both outcomes. Among healthcare workers, there were 1,294 nurses (5.8%, median = 647, IQR = 40–1,254). Only the article by Savage et al. reported no number of involved healthcare workers [ 31 ].

The number of studies conducted after 2011 or between 2012–2019 was seven (58.3%), while only one (8.3%) article reported a study beginning both before and after 2011. Most of studies were conducted in Northern America ( n = 5, 41.7%). The servant leadership style and charismatic leadership style were the most frequently implemented, as reported in five (41.7%) and four (33.3%) articles, respectively. Only one (8.3%) study adopted a transformational leadership style.

The pooled effectiveness of leadership was 14.0% (95%CI 10.0–18.0%), with a high level of heterogeneity (I 2 = 99.9%, p < 0.0001) among the before–after studies ( Figure 2 ).

An external file that holds a picture, illustration, etc.
Object name is ijerph-19-10995-g002.jpg

Effectiveness of leadership in before after studies. Dashed line represents the pooled effectiveness value [ 29 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 ].

The highest level of effectiveness was reported by Weech-Maldonado R et al. with an effectiveness of 199% (95%CI 183–215%) based on the Cultural Competency Assessment Tool for Hospitals (CCATH) [ 39 ]. The effectiveness of leadership changed in accordance with the leadership style ( Supplementary Figure S1 ) and publication bias ( Supplementary Figure S2 ).

Multi-regression analysis indicated a negative association between leadership effectiveness and studies from Oceania, but this result was not statistically significant (β = −0.33; 95% IC −1.25, 0.59). On the other hand, a charismatic leadership style affected healthcare outcomes positively even if it was not statistically relevant (β = 0.24; 95% IC −0.69, 1.17) ( Table 1 ).

Correlation coefficients and multi-regression analysis of leadership effectiveness in before–after studies.

3.3. Leadership Effectiveness in Cross Sectional Studies

A total of 2858 (median = 199, IQR = 110–322) healthcare workers were involved in the cross-sectional studies ( Supplementary Table S2 ), of which 981 (34.3%) were nurses. Most of the studies were conducted in Asia ( n = 4, 44.4%) and North America ( n = 3, 33.3%). All of the cross-sectional studies regarded only the healthcare professionals’ performance. Multi-professional teams were involved in seven (77.8%) studies, and they were more frequently conducted in both medical and surgical wards ( n = 6, 66.7%). The leadership styles were equally distributed in the articles and two (22.2%) of them examined more than two leadership styles at the same time.

The pooled effectiveness of the leadership interventions in the cross-sectional studies had a correlation coefficient of 0.22 (95%CI 0.15–0.28), whose heterogeneity was remarkably high (I 2 = 96.7%, p < 0.0001) ( Figure 3 ).

An external file that holds a picture, illustration, etc.
Object name is ijerph-19-10995-g003.jpg

Effectiveness of leadership in cross-sectional studies. Dashed line represents the pooled effectiveness value [ 30 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ].

The effectiveness of leadership in the cross-sectional studies changed in accordance with the leadership style ( Supplementary Figure S3 ) and publication bias ( Supplementary Figure S4 ).

Multi-regression analysis showed a higher leadership effectiveness in studies conducted in South America (β = 0.56 95%CI 0.13–0.99) in private hospitals (β = 0.60; 95%CI 0.14–1.06) and in the medical vs. surgical specialty (β = −0.22; 95%CI −0.54, −0.02) ( Table 2 ).

Multi-regression analysis of leadership effectiveness in cross-sectional studies.

* 0.05 ≤ p < 0.01.

4. Discussion

Leadership effectiveness in healthcare settings is a topic that is already treated in a quantitative matter, but only this systematic review and meta-analysis showed the pooled effectiveness of leadership intervention improving some healthcare outcomes such as performance and adherence to guidelines. However, the assessment of leadership effectiveness could be complicated because it depends on the study methodology and selected outcomes [ 12 ]. Health outcomes might benefit from leadership interventions, as Flodgren et al. was concerned about opinion leadership [ 2 ], whose adhesion to guidelines increased by 10.8% (95% CI: 3.5–14.6%). On the other hand, other outcomes did not improve after opinion leadership interventions [ 2 ]. Another review by Ford et al. about emergency wards reported a summary from the literature data which acknowledged an improvement in trauma care management through healthcare workers’ performance and adhesion to guidelines after effective leadership interventions [ 14 ]. Nevertheless, some variables such as collaboration among different healthcare professionals and patients’ healthcare needs might affect leadership intervention effectiveness [ 14 ]. Therefore, a defined leadership style might fail in a healthcare setting rather than in other settings [ 5 , 13 , 14 ].

The leadership effectiveness assessed through cross-sectional studies was higher in South America than in other continents. A possible explanation of this result could be the more frequent use of a transactional leadership style in this area, where the transactional leadership interventions were effective at optimizing economic resources and improving healthcare workers’ performance through cash rewards [ 48 ]. Financing methods for healthcare organizations might be different from one country to another, so the effectiveness of a leadership style can change. Reaching both economic targets and patients’ wellness could be considered a challenge for any leadership intervention [ 48 ], especially in poorer countries [ 5 ].

This meta-analysis showed a negative association between leadership effectiveness and studies by surgical wards. Other research has supported these results, which reported surgical ward performance worsened in any leadership context (charismatic, servant, transactional, transformational) [ 47 ]. In those workplaces, adopting a leadership style to improve surgical performance might be challenging because of nervous tension and little available time during surgical procedures [ 47 ]. On the other hand, a cross-sectional study declared that a surgical team’s performance in private surgical settings benefitted from charismatic leadership-style interventions [ 42 ]. This style of leadership intervention might be successful among a few healthcare workers [ 42 ], where creating relationships is easier [ 6 ]. Even a nursing team’s performance in trauma care increased after charismatic leadership-style interventions because of better communicative and supportive abilities than certain other professional categories [ 29 , 47 ]. However, nowadays there is no standardized leadership in healthcare basic courses [ 5 , 6 , 12 ]. Consequently, promoting leadership culture after undergraduate medical courses could achieve a proper increase in both leadership agreement and working wellness as well as a higher quality of care. [ 17 ]. Furthermore, for healthcare workers who have already worked in a healthcare setting, leadership improvement could consist of implementing basic knowledge on that topic. Consequently, they could reach a higher quality of care practice through working wellness [ 17 ] and overcoming the lack of previous leadership training [ 17 ].

Although very few studies have included in a meta-analysis examined in private healthcare settings [ 35 , 38 , 40 , 41 , 42 ], leadership interventions had more effectiveness in private hospitals than in public hospitals. This result could be related to the continent of origin, and indeed 60.0% of these studies were derived from North America [ 38 , 41 , 42 ], where patients’ outcomes and healthcare workers’ performance could influence available hospital budgets [ 38 , 40 , 41 , 42 ], especially in peripheral healthcare units [ 38 , 41 ]. Private hospitals paid more attention to the cost-effectiveness of any healthcare action and a positive balance of capital for healthcare settings might depend on the effectiveness of leadership interventions [ 40 , 41 , 42 ]. Furthermore, private healthcare assistance focused on nursing performance because of its impact on both a patients’ and an organizations’ outcomes. Therefore, healthcare systems’ quality could improve with effective leadership actions for a nursing team [ 40 ].

Other factors reported in the literature could affect leadership effectiveness, although they were not examined in this meta-analysis. For instance, professionals’ specialty and gender could have an effect on these results and shape leadership style choice and effectiveness [ 1 ]. Moreover, racial differences among members might influence healthcare system performance. Weech-Maldonado et al. found a higher compliance and self-improvement by black-race professionals than white ones after transactional leadership interventions [ 39 ].

Healthcare workers’ and patients’ outcomes depended on style of leadership interventions [ 1 ]. According to the results of this meta-analysis, interventions conducted by a transactional leadership style increased healthcare outcomes, though nevertheless their effectiveness was higher in the cross-sectional studies than in the before–after studies. Conversely, the improvement by a transformational leadership style was higher in before–after studies than in the cross-sectional studies. Both a charismatic and servant leadership style increased effectiveness more in the cross-sectional studies than in the before–after studies. This data shows that any setting required a specific leadership style for improving performance and guideline adherence by each team member who could understand the importance of their role and their tasks [ 1 ]. Some outcomes had a better improvement than others. Focusing on Savage et al.’s outcomes, a transformational leadership style improved checklist adherence [ 31 ]. The time of patients’ transport by Murphy et al. was reduced after conducting interventions based on a charismatic leadership style [ 37 ]. Jodar et al. showed that performances were elevated in units whose healthcare workers were subjected to transactional and transformational leadership-style interventions [ 1 ].

These meta-analysis results were slightly relevant because of the high heterogeneity among the studies, as confirmed by both funnel plots. This publication bias might be caused by unpublished articles due to either lacking data on leadership effectiveness, failing appropriate leadership strategies in the wrong settings or non-cooperating teams [ 12 ]. The association between leadership interventions and healthcare outcomes was slightly explored or gave no statistically significant results [ 12 ], although professionals’ performance and patients’ outcomes were closely related to the adopted leadership style, as reported by the latest literature sources [ 7 ]. Other aspects than effectiveness should be investigated for leadership. For example, the evaluation of the psychological effect of leadership should be explored using other databases.

The study design choice could affect the results about leadership effectiveness, making their detection and their statistical relevance tough [ 12 ]. Despite the strongest evidence of this study design [ 50 ], nowadays, trials about leadership effectiveness on healthcare outcomes are lacking and have to be improved [ 12 ]. Notwithstanding, this analysis gave the first results of leadership effectiveness from the available study designs.

Performance and adherence to guidelines were the main two outcomes examined in this meta-analysis because of their highest impact on patients, healthcare workers and hospital organizations. They included several other types of outcomes which were independent each other and gave different effectiveness results [ 12 ]. The lack of neither an official classification nor standardized guidelines explained the heterogeneity of these outcomes. To reach consistent results, they were classified into performance and guideline adherence by the description of each outcome in the related manuscripts [ 5 , 6 , 12 ].

Another important aspect is outcome assessment after leadership interventions, which might be fulfilled by several standardized indexes and other evaluation methods [ 40 , 41 ]. Therefore, leadership interventions should be investigated in further studies [ 5 ], converging on a univocal and official leadership definition and classification to obtain comparable results among countries [ 5 , 6 , 12 ].

5. Conclusions

This meta-analysis gave the first pooled data estimating leadership effectiveness in healthcare settings. However, some of them, e.g., surgery, required a dedicated approach to select the most worthwhile leadership style for refining healthcare worker performances and guideline adhesion. This can be implemented using a standardized leadership program for surgical settings.

Only cross-sectional studies gave significant results in leadership effectiveness. For this reason, leadership effectiveness needs to be supported and strengthened by other study designs, especially those with the highest evidence levels, such as trials. Finally, further research should be carried out to define guidelines on leadership style choice and establish shared healthcare policies worldwide.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph191710995/s1 , Figure S1. Leadership effectiveness by leadership style in before after studies; Figure S2. Funnel plot of before after studies; Figure S3. Leadership effectiveness in cross sectional studies by four leadership style; Figure S4. Funnel plot of cross-sectional studies; Table S1. Before after studies included in this systematic review and meta-analysis; Table S2. Cross-sectional studies included in this systematic review and meta-analysis. All outcomes were performance.

Funding Statement

This research received no external funding.

Author Contributions

Conceptualization, V.R., A.C. (Alessandra Casuccio), F.V. and C.F.; methodology, V.R., M.G., A.O. and C.T.; software, V.R.; validation, G.M., A.B., A.C. (Alberto Carli) and M.C.; formal analysis, V.R.; investigation, G.M., A.B., A.C. (Alberto Carli) and M.C.; resources, A.C. (Alessandra Casuccio); data curation, G.M. and V.R.; writing—original draft preparation, G.M.; writing—review and editing, A.C. (Alessandra Casuccio), F.V., C.F., M.G., A.O., C.T., A.B., A.C. (Alberto Carli) and M.C.; visualization, G.M.; supervision, V.R.; project administration, C.F.; funding acquisition, A.C. (Alessandra Casuccio), F.V. and C.F. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Ethical review and approval were waived for this study due to secondary data analysis for the systematic review and meta-anlysis.

Informed Consent Statement

Not applicable.

Data Availability Statement

Conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

  • Submitting Data
  • Accessing Data
  • About the Genomic Data Sharing (GDS) Policy
  • Key Documents
  • Preparing Genomic Data
  • Extramural Grantees
  • Non-NCI Funded Investigators
  • Intramural Investigators
  • Accessing Genomic Data
  • Genomic Data Sharing Policy Contact Information
  • Data Standards Services
  • Cloud Resources
  • Data Commons Framework
  • Repositories
  • ARPA-H BDF Toolbox
  • Accelerating Therapeutics for Opportunities in Medicine (ATOM)
  • NCI-DOE Collaboration AI/ML Resources
  • NCI-DOE Publications
  • U.S.-EU Artificial Intelligence Administrative Arrangement
  • NCI Data Catalog
  • CDISC Terminology
  • FDA Terminology
  • NCPDP Terminology
  • Pediatric Terminology
  • Metadata for Cancer Research
  • Informatics Technology for Cancer Research (ITCR) Tools
  • Generating and Collecting Data
  • Cleaning Data
  • Exploring and Analyzing Data
  • Predictive Modeling
  • Visualizing Data
  • Sharing Data
  • Cancer Data Science Course
  • Training Guide Library
  • Cancer Data Science Pulse Blog
  • Data Science Seminar Series
  • Contact CBIIT
  • Organization
  • CBIIT Director
  • Staff Directory
  • Application Support
  • Genomic Data Sharing
  • NCI-Department of Energy Collaborations
  • Cancer Vocabulary
  • Learn About Cancer Data Science
  • Improve My Data Science Skills

CBIIT’s Hiring—Apply Today to be a Clinical Research Informatics Leader

Are you skilled in conceptualizing, developing, and managing projects in clinical research informatics? Come work with us at NCI’s CBIIT as a clinical research informatics specialist! Here, you can help translate data into real-world solutions for cancer care.

To qualify you should be able to:

  • demonstrate expertise in using biomedical informatics tools and approaches for clinical research;
  • show experience in managing projects using informatics for precision medicine ;
  • communicate and collaborate with others to develop informatics approaches;
  • supervise and direct other staff; and
  • confirm that you have a bachelor’s (or graduate/higher level) degree in a major study in an academic field related to the medical, health science, or allied scientific fields.

A sampling of your duties include:

  • managing projects to support clinical trials and precision medicine research programs;
  • managing biomedical informatics projects to support clinical trials;
  • working with others (individuals and teams) to develop successful solutions in clinical and translational informatics;
  • assuring that security plans protect patient confidentiality and corporate proprietary information; checking that disaster recovery plans are cost effective and cover a minimal loss of information and system down time; and
  • using your skills to advance studies in biomedical research, cancer clinical trials, cancer biology, and medical informatics research.

Image CAPTCHA

Enter the characters shown in the image.

SUBSCRIBE TO UPDATES

What do you think.

Let us know if this new training section is meeting your needs and what we can do to make it even better!

Scientific Leadership

George Clinical supports sponsors in the setup of key trial committees by selecting, and contracting world renowned and experienced committee members that we access through our global scientific networks.

George Clinical

Site performance is transformed by scientific leadership.

Scientific leadership in clinical trials facilitates study operations through peer-to-peer engagement with investigators.

The George Clinical Scientific Leadership model offers:

  • A scientifically sound, pragmatic approach that informs strong operational delivery
  • Protocol alignment with clinical practice across geographies
  • Motivated and engaged investigators
  • Robust communication conduits between the investigators, study teams, steering committees and the sponsor

Professor Stephen MacMahon

Professor Stephen MacMahon, Principal Director, The George Institute for Global Health

Delivery of our scientific leadership model leverages the reputation and extensive global clinical investigator networks of our partner in research, The George Institute of Global Health.

In the traditional study operations model, site engagement is typically performed by clinical research associates (CRAs). With George Clinical’s scientific leadership model, we provide a framework that enables active peer-to-peer communication and action pathways between global scientific committee members, national scientific leaders and principal investigators.

Examples of activities within our scientific leadership service model include:

  • Trial design and strategy
  • Protocol development
  • Publication oversight and writing
  • National leader selection and appointment
  • Content development for investigator meetings
  • Liaison with the various trial committees, national leaders and operations teams
  • Support site identification and recruitment activities
  • Support patient recruitment and retention activities
  • Advise operational teams on dealing with challenges arising in the clinical trial environment
  • Scientific committee management, such as selection, appointment, contracting and charter preparations
  • Administrative and communication support between various study committees

By engaging its scientific leadership, George Clinical is able to recruit international experts to Steering, Adjudication and Data Safety Committees.

Meeting

George Clinical supports sponsors in the setup of key trial committees by selecting, and contracting experienced committee members whom we access through our global scientific networks.

George Clinical’s scientific teams are experts in the establishment and operational management of committees, including charter development, meeting organisation and chairing roles.

Investigator Initiated Trials (IIT) are an essential part of clinical research offering unique insight from a real-world clinical perspective. George Clinical’s scientific leaders are well placed to identify and address requirements for innovative and pragmatic clinical trials.

George Clinical’s Investigator Initiated Trials are fully supported by the qualified staff of George Clinical, who manage all aspects of the project from study design through data-lock, analysis, and manuscript. Our team possesses a deep understanding of the clinical landscape and emerging trends in clinical practice.

Meet our Team

Ckd summit 2024.

March 19 - 21st, 2024

George Clinical

[email protected]

clinical research study leader

  • Corporate News
  • Case Studies

Add George Clinical to your network

© George Clinical 2024 All Rights Reserved | Privacy Policy | Modern Slavery & Human Trafficking Statement |  GDPR  Compliance Statement

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • Topic Collection
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Volume 7, Issue 3
  • Attributes, skills and actions of clinical leadership in nursing as reported by hospital nurses: a cross-sectional study
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • http://orcid.org/0000-0001-8401-4976 Majd T Mrayyan 1 ,
  • http://orcid.org/0000-0002-6393-3022 Abdullah Algunmeeyn 2 ,
  • http://orcid.org/0000-0002-2639-9991 Hamzeh Y Abunab 3 ,
  • Ola A Kutah 2 ,
  • Imad Alfayoumi 3 ,
  • Abdallah Abu Khait 1
  • 1 Department of Community and Mental Health Nursing, Faculty of Nursing , The Hashemite University , Zarqa , Jordan
  • 2 Advanced Nursing Department, Faculty of Nursing , Isra University , Amman , Jordan
  • 3 Basic Nursing Department, Faculty of Nursing , Isra University , Amman , Jordan
  • Correspondence to Dr Majd T Mrayyan, Department of Community and Mental Health Nursing, Faculty of Nursing, The Hashemite University, Zarqa 13133, Jordan; mmrayyan{at}hu.edu.jo

Background Research shows a significant growth in clinical leadership from a nursing perspective; however, clinical leadership is still misunderstood in all clinical environments. Until now, clinical leaders were rarely seen in hospitals’ top management and leadership roles.

Purpose This study surveyed the attributes and skills of clinical nursing leadership and the actions that effective clinical nursing leaders can do.

Methods In 2020, a cross-sectional design was used in the current study using an online survey, with a non-random purposive sample of 296 registered nurses from teaching, public and private hospitals and areas of work in Jordan, yielding a 66% response rate. Data were analysed using descriptive analysis of frequency and central tendency measures, and comparisons were performed using independent t-tests.

Results The sample consists mostly of junior nurses. The ‘most common’ attributes associated with clinical nursing leadership were effective communication, clinical competence, approachability, role model and support. The ‘least common’ attribute associated with clinical nursing leadership was ‘controlling’. The top-rated skills of clinical leaders were having a strong moral character, knowing right and wrong and acting appropriately. Leading change and service improvement were clinical leaders’ top-rated actions. An independent t-test on key variables revealed substantial differences between male and female nurses regarding the actions and skills of effective clinical nursing leadership.

Conclusions The current study looked at clinical leadership in Jordan’s healthcare system, focusing on the role of gender in clinical nursing leadership. The findings advocate for clinical leadership by nurses as an essential element of value-based practice, and they influence innovation and change. As clinical leaders in various hospitals and healthcare settings, more empirical work is needed to build on clinical nursing in general and the attributes, skills and actions of clinical nursing leadership of nursing leaders and nurses.

  • clinical leadership
  • health system
  • leadership assessment

Data availability statement

Data are available on request due to privacy/ethical restrictions. https://authorservices.taylorandfrancis.com/data-sharing/share-your-data/data-availability-statements/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/leader-2022-000672

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Clinical leadership was limited to service managers; however, currently, all clinicians are invited to participate in leadership practices. Clinical leaders are needed in various healthcare settings to produce positive outcomes.

WHAT THIS STUDY ADDS

This study outlined clinical leadership attributes, skills and actions to understand clinical nursing leadership better. The current study highlighted the role of gender in clinical nursing leadership, and it asserts that effective clinical nursing leadership is warranted to improve the efficiency and effectiveness of care. The results call for nurses’ clinical leadership as essential in today’s turbulent work environment.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

Nurses and clinical leaders need additional attributes, skills and actions. Clinical nursing leaders should use innovative interventions and have skills or actions to manage current work environments. Further work is needed to build on clinical nursing in general and the attributes, skills and actions of clinical nursing leadership. Clinical leadership programmes must be integrated into the nursing curricula.

Introduction

Clinical leadership is a matter of global importance. Currently, all clinicians are invited to participate in leadership practices. 1 This invitation is based on the fact that people deliver healthcare within complex systems. Effective clinicians must understand systems of care to function effectively. 1 2 Engaging in clinical leadership is an obligation, not a choice, for all clinicians at all levels. This obligation is more critical in nursing with many e merging global health issues , 2 such as the COVID-19 pandemic.

The systematic literature review of Cummings et al 3 shows the differences in leadership literature. In early 2000, clinical leadership emerged in scientific literature. 4 It is about having the knowledge, skills and competencies needed to effectively balance the needs of patients and team members within resource constraints. 4 Clinical leadership is vital in nursing as nurses face complex challenges in clinical settings, especially in acute care settings. 4 Although developed from the management domain, leadership and management are two concepts used interchangeably, 5–9 leading to further misunderstanding of the relationship between clinical leadership and management. While different types of leadership have been evident in nursing and health industry literature, clinical leadership is still misunderstood in clinical environments. 8 Clinical leadership is not fully understood among health professionals trained to care for patients, as clinical leadership is a management concept, leaving the concept open to different interpretations. 10 For example, Gauld 10 reported that clinical leaders might be professionals (such as doctors and nurses) who are no longer clinically active, mandating that clinical leaders should also be involved in delivering care. 10

There is no clear definition of ‘clinical leadership’. However, effective clinical leadership involves individuals with the appropriate clinical leadership skills and attributes at different levels of an organisation, focusing on multidisciplinary and interdisciplinary work. 10 The main skills associated with clinical leadership were having values and beliefs consistent with their actions and interventions, being supportive of colleagues, communicating effectively, serving as a role model and engaging in reflective practice. 4–9 The main attributes associated with clinical leadership were using effective communication, clinical competence, being a role model, supportive and approachability. 4–9 Stanley and colleagues reported that clinical leaders are found across health organisations and are presented in all clinical environments. Clinical leaders are often found at the highest level for clinical interaction but not commonly found at the highest management level in wards or units. 4–9

With the increasing urgency to improve the efficiency and effectiveness of care, effective nursing leadership is warranted. 4 11–17 Clinical leaders can be found in various healthcare settings, 4 most often at the highest clinical level, but they are uncommon at the top executive level. 6–9 18–24 In the UK, the National Health Service (NHS) 25 empowers clinicians and front-line staff to build their decision-making capabilities, which is required for clinical leadership. This empowerment encourages a broader practice of clinical leadership without being limited to top executives alone. 25 26

Purpose and significance

This study assesses clinical nursing leadership in Jordan. More specifically, it answers the following research questions: (1) What attributes are associated with clinical nursing leadership in Jordanian hospitals? (2) What skills are important for effective clinical nursing leadership? (3) What actions are important for effective clinical nursing leadership? (4) What are the differences in skills critical to effective clinical nursing leadership based on the sample’s characteristics? (5) What are the differences in effective clinical nursing leaders’ actions based on the sample’s characteristics?

Nursing leadership studies are abundant; however, clinical leadership research is not well established. 8 27 Until fairly recently, clinical leadership in nursing has tended to focus on nursing leaders in senior leadership positions, ignoring nurse managers in clinical positions. 8 There has been significant growth in research exploring clinical leadership from a nursing perspective. 4 8 9 14–17 24 26–32 A new leadership theory, ‘congruent leadership’, has emerged, claiming that clinical leaders acted on their values and beliefs about care and thus were followed. 6–9 20 This study is the first in Jordan’s nursing and health-related research about clinical leadership. Clarifying this concept from nurses’ perspectives will support greater healthcare delivery efficiencies.

Search methods

The initial search was done using ‘clinical nursing leadership’ at the Clarivate database and Google Scholar database from 2017 to 2021, yielded 35 studies, of which, after abstracting, 14 studies were selected. However, Stanley’s work (12 studies), including those before 2017, was included because we followed the researcher’s passion and methodology of studying clinical leadership; also, some classical models of clinical leadership because they were essential for the conceptualisation of the study as well as the discussion, such as the NHS Leadership Academy (three studies; ref 25 33 34 ).

Another search was run using the words ‘attributes’, ‘skills’ or ‘actions’ using the same time frame; most of the yielded studies were not relevant, this search year was expanded to 2013–2021 because the years 2013–2015 were the glorious time of studying these concepts. Using ‘clinical leadership’ rather than ‘leadership studies’, 15 studies were yielded; however, Stanley’s above work was excluded to avoid repetition, resulting in using three studies (ref 29 30 35 ). A relevant reference of 2022 similar to our study (ref 36 ) was added at the stages of revisions. The remaining 16 of 49 references were related to the methodology and explanation of some results, such as those related to gender differences in leadership. The following limits were set: the language was English; and the year of publication was basically the last 5 years to ensure that the search was current.

Clinical leadership

Clinical leadership ensures quality patient care by providing safe and efficient care and creating a healthy clinical work environment. 4 10–17 27 31 32 It also decreases the high costs of clinical litigation settlements and improves the safety of service delivery to consumers. 4 11–17 32 For these reasons, healthcare organisations should initiate interventions to develop clinical leadership among front-line clinicians, including nurses. 8 9

Literature was scarce on clinical leadership in nursing. 4 8–10 14–17 27 28 31 Stanley and Stanley 8 defined clinical leadership as developing a culture and leading a set of tasks to improve the quality and safety of service delivery to consumers.

Clinical leadership is about focusing on direct patient care, delivering high-quality direct patient care, motivating members of the team to provide effective, safe and satisfying care, promoting staff retention, providing organisational support and improving patient outcomes. 31 Clinical leadership roles include providing the vision, setting the direction, promoting professionalism, teamwork, interprofessional collaborations, good practice and continued medical education, contributing to patient care and performing tasks effectively. 31 Moreover, the researchers added that clinical leadership is having the approachability and the ability to communicate effectively, the ability to gain support and influence others, role modelling, visibility and availability to support, the ability to promote change, advise and guide. 31 Clinical leadership competencies include demonstrating clinical expertise, remaining clinically focused and engaged and comprehending clinical leadership roles and decision-making. In addition, clinical leadership was not associated with a position within the management and organisational structure, unlike health service management. 31 33

Clinical leadership is hindered by many barriers that include the lack of time and the high clinical/client demand on their time. 8 9 Clinical leadership is limited because of the deficit in intrapersonal and interpersonal capabilities among team members and interdisciplinary and organisational factors, such as a lack of influence in interdisciplinary care planning and policy. 37 Other barriers include limited organisational leadership opportunities, the perceived need for leadership development before serving in leadership roles and a lack of funding for advancement. 38

This paper aligns with the theory of congruent leadership proposed by Stanley. 19 This theory is best suited for understanding clinical leadership because it defines leadership as a congruence between the activities and actions of the leader and the leader’s values, beliefs and principles, and those of the organisation and team.

Attributes of clinical leadership

The clinical leadership attributes needed for nurses 8 28 to perform their roles effectively are: (1) personal attributes: nurses are confident in their abilities to provide best practice, communicate effectively and have emotional intelligence; (2) team attributes: encouraging trust and commitment to others, team focus and valuing others’ skills and expertise; and (3) capabilities: encouraging contribution from others, building and maintaining relationships, creating clear direction and being a role model. 8 28 Clinical leadership attributes are linked to communicating effectively, role modelling, promoting change, providing advice and guidance, gaining support and influencing others. 28–30 Other attributes to include are clinical leaders’ engagement in reflective practice, 29 provision of the vision; setting direction, having the resources to perform tasks effectively and promoting professionalism, teamwork, interprofessional collaborations, effective practice and continued education. 27 28 31

Skills of clinical leadership

Clinical leadership skills include (1) a ‘clinical focus’: being expert knowledge, providing evidence-based rationale and systematic thinking, understanding clinical leadership, understanding clinical decision-making, being clinically focused, remaining clinically engaged and demonstrating clinical expertise; (2) a ‘follower/team focus’: being supportive of colleagues, effectively communicating communication skills, serving as a role model and empowering the team; and (3) a ‘personal qualities focus’: engaging in reflective practice, initiating change and challenging the status quo. 17 30 32 Clinical leaders have advocacy skills, facilitate and maintain healthier workplaces by driving changes in cultural issues among all health professionals. 17 29 Moreover, the overlap between the attributes and skills of clinical leaders includes being credible to colleagues because of clinical competence and the skills and capacity to support multidisciplinary teams effectively. 17 29 32

Actions of clinical leadership

A clinical leader is anyone in a clinical position exercising leadership. 26 The clinical leader’s role is to continuously instil in clinicians the capability to improve healthcare on small and large scales. 26 Furthermore, Stanley et al 9 demonstrated that clinical leaders are not always managers or higher-ups in organisations. Clinical leaders act following their values and beliefs, are approachable and provide superior service to their clients. 9 Clinical leaders define and delegate safety and quality responsibilities and roles. 14 32 39 They also ensure safety and quality of care, manage the operation of the clinical governance system, implement strategic plans and implement the organisation’s safety culture. 14 32 39 The Australian Commission on Safety and Quality in Health Care 39 also reported that clinical leaders might support other clinicians by reviewing safety and quality performance data, supervising the clinical workforce, conducting performance appraisals and ensuring that the team understands the clinical governance system.

In summary, clinical leadership attributes, skills and actions were outlined to understand clinical nursing leadership. The literature shows limited nursing research on clinical leadership, calling for clinical leadership that paves the road for nurses in the current turbulent work environment.

Study design

A descriptive quantitative analysis was developed to collect data about the attributes and skills of clinical nursing leadership and the actions that effective nursing clinical leaders can take. A cross-sectional design was employed to measure clinical leadership using an online survey in 2020. This design was appropriate for such a study as it allows the researchers to measure the outcome and the exposures of the study participants at the same time. 40

Sample and settings

The general population was registered nurses in medical centres in Jordan. The target population was registered nurses in teaching, public and private hospitals. Most nurses in Jordan are females working at different shifts on a full-time basis in different types of healthcare services. The baccalaureate degree is the minimum entry into the clinical practice of registered nurses. As previous nurses, we would like to attest that nurses in Jordanian hospitals commonly use team nursing care delivery models with different decision-making styles. The size of the sample was calculated by using Thorndike’s rule as follows: N≥10(k)+50 (where N was the sample size, k is the number of independent variables) (attributes, skills, actions), the minimum sample size should be 80 participants. 40 From experience, the researcher considers the sample’s demographics and subscales as independent variables (k=17); the overall sample should not be less than 220.

Research participants were recruited through a ‘direct recruitment strategy’ from the hospitals where the nursing students were trained. A survey was used to collect data using non-random purposive sampling; of possible 450 Jordanian nurses, 296 were recruited from different types of hospitals: teaching (51 of possible 120 nurses), public (180 of possible 210 nurses) and private (65 of possible 120 nurses), with a response rate of 66%, which is adequate for an online survey. The inclusion criteria were that nurses should work in hospital settings, and any nurses who work in non-hospital settings were excluded. No incentives were applied.

Using a direct measurement method, Stanley’s Clinical Leadership Scale ( online supplemental file 1 ) was used to collect the data using the English version of the scale because English is the official education language of nursing in Jordan. 8 9 The original questionnaire consists of 24 questions: 12 quantitative and qualitative questions relevant to clinical leadership, and 12 related to the sample’s demographics. Several studies about clinical leadership among nurses and paramedics in the UK and Australia used modified versions of a survey tool 5 8 9 18–24 ; construct validity was ensured using exploratory factor analysis or triangulation of validation. Cronbach’s alpha measures the homogeneity in the survey, and it was reported to be 0.87 8 9 and 0.88 in the current study.

Supplemental material

Several questions were measured on a 5-point Likert scale in the original scale, and others were qualitative. The survey for the current study consists of 12 quantitative and qualitative questions related to clinical leadership and 14 questions related to the sample’s demographics. However, the qualitative data obtained were scattered and incomplete; thus, only the quantitative questions were analysed and reported, and another qualitative study about clinical leadership was planned. For the current study, three quantitative questions only focused on clinical leadership, leadership skills and the actions of clinical leaders, and 14 questions focused on the sample’s characteristics relevant to the Jordanian healthcare system developed by the first author. The sample characteristics were gender, marital status, shift worked, time commitment, level of education, age, years of experience in nursing, years of experience in leadership and the number of employees directly supervised. Other characteristics include the type of unit/ward, model of nursing care, ward/unit’s decision-making style, formal leadership-related education (yes/no) and formal management-related education (yes/no). Before data collection, permission to use the tool was granted.

Ethical considerations

Nurses were invited to answer the survey while assuring the voluntary nature of their participation. The participants were told that their participation in the survey was their consent form. Participants’ anonymity and confidentiality of information were assured; all questionnaires were numerically coded, and the overall results were shared with nursing and hospital administrators. 40

Patient and public involvement

There was no patient or public involvement in this research’s design, conduct, reporting or dissemination.

Data collection procedures

After a pilot study on 12 December 2020, which checked for the suitability of the questionnaire for the Jordanian healthcare settings, data were collected over a month on 23 December 2020. Data were collected through Google Forms; the survey was posted on various WhatsApp groups and Facebook pages. Using purposive snowball sampling, nurses were asked to invite their contacts and to submit the survey once. To assure one submission, the Google Forms was designed to allow for one submission only.

No problem was encountered during data collection. The two attrition prevention techniques used were effective communication and asserting to the participants that the study was relevant to them.

The researchers controlled for all possible extraneous and confounding variables by including them in the study. A possible non-accounted extraneous variable is the organisational structure; a centralised organisational structure may hinder the use of clinical nursing leadership.

Data analyses

After data cleaning and checking wild codes and outliers, all coded variables were entered into the Statistical Package for Social Sciences (SPSS) (V.25), 35 which was used to generate statistics according to the level of measurement. A descriptive analysis focused on frequency and central tendency measures. 40 Part 1 of the scale comprises 54 qualities or characteristics to answer the first research question. Responses related to skills were measured on a 1–5 Likert scale; thus, means and SDs were reported to answer the second research question. Eight actions were rated on a 1–5 Likert scale; thus, means and SDs were reported to answer the third research question. Independent t-tests using all sample characteristics were performed to answer the fourth and fifth research questions.

The preanalysis phase of data analysis was performed; data were eligible and complete as few missing data were found; thus, they were left without intervention. The assumption of normality was met; both samples are approximately normally distributed, and there were no extreme differences in the sample’s SDs.

Characteristics of the sample

There were 296 nurses in the current study from different types of hospitals: teaching (51 nurses), public (180 nurses) and private (65 nurses), with a response rate of 66%. Most nurses were females (209, 70.6%), single (87, 29.4%), working a day shift (143, 48.3%) or rotating shifts (92, 31.1%), on a full-time basis (218, 73.6%), with a baccalaureate degree (236, 79.7%), aged less than 25 years (229, 77.4%) and 25–34 years (45, 15.2%), respectively. Also, 65.1% (166) of nurses reported having less than 1 year of experience in nursing; thus, they have few nurses under them to supervise (145, 49% supervise one to two nurses), and 23.3% (69) of nurses reported having 1–9 years of experience in leadership. Nurses reported that their unit or ward has a primary (81, 27.4%) or team nursing care delivery model (162, 54.7%), with a mixed (94, 31.8%) or participatory decision-making style (113, 38.2%), and had formal leadership-related education (191, 64.5%), and had no formal management-related education (210, 70.9%) ( table 1 ).

  • View inline

Sample’s characteristics (N=296*)

Attributes of clinical nursing leadership

Nurses were asked to think about the attributes and features of clinical leadership. Based on Stanley’s Clinical Leadership Scale, 8 9 nurses were given a list of 54 qualities and characteristics and asked to select the most strongly associated with clinical leadership, followed by those least strongly associated with clinical leadership. Table 2 shows the respondents’ ‘top ten’ selected qualities in ranked order.

'Most’ and ‘Least’ important attributes associated with clinical nursing leadership (N=296)

Skills of effective clinical nursing leaders

On a Likert scale of 1–5, respondents were asked to rank the skills of effective clinical leaders from ‘not relevant’ or ‘not important’ to 5=‘very relevant’ or ‘very important’. The top skills were having a strong moral character, knowing right and wrong and acting appropriately which received a high rating, with a mean of 4.17 out of 5 (0.92). Being in a management position to be effective was ranked as the least skill of an effective leader, with a mean value of 3.78 out of 5 (1.00). As indicated by respondents, other skills of effective clinical leaders are shown in table 3 .

Skills of effective clinical nursing leaders (N=296)

Actions of effective clinical nursing leaders

On a Likert scale of 1–5, respondents were asked to rank the actions of effective clinical leaders. Leading change and service management achieved a high rating of 4.07 out of 5 points (0.90). Influencing organisational policy was rated last, with a mean score of 3.95 out of 5 (1.01), which may reflect the very junior nature of the majority of the sample. As described by respondents, some of the other actions of effective leaders are shown in table 4 .

Actions effective clinical nursing leaders can do (N=296)

Significant differences in skills of effective clinical nursing leaders based on gender

Independent t-tests using all sample’s characteristics were performed to answer the fourth research question. Gender was the only characteristic variable that differentiated clinical leadership skills. An independent t-test demonstrates that males and females have distinct perspectives on 3 out of 10 items measuring clinical leadership skills. Female participants outperform male participants in terms of ‘working within the team (p value=0.021)’, ‘being visible in the clinical environment (p value=0.004)’ and ‘recognizing optimal performance and expressing appreciation promptly (p value=0.042) ( table 5 )’.

Significant differences in skills and actions of effective clinical nursing leaders based on gender (n=296)

Significant differences in actions of effective clinical nursing leaders based on gender

Independent t-tests using all sample’s characteristics were performed to answer the fifth research question. Gender was the only characteristic variable that differentiated clinical leadership actions, and it was discovered that five of the eight propositions varied in their actions: the way clinical care is administered (p=0.010); participating in staff development education (p=0.006); providing valuable staff support (p=0.033); leading change and service improvement (p=0.014); and encouraging and leading service management (p=0.019). The independent t-test results revealed that female participants scored higher in those acts, corresponding to effective leaders’ competencies. The mean values of participants’ responses to the actions of effective clinical leaders are shown in table 5 .

The characteristics of the current sample are similar to those of the structure of the task force in Jordan. The remaining question is how men in Jordan be supported in nursing to develop clinical leadership skills on par with females. Al-Motlaq et al 41 proposed using a part-time nurses policy to address nurses’ gender imbalances. Although this is necessary for both genders, we propose to develop a clinical leadership training package to promote working male nurses’ clinical leadership. In Jordan, we apply the modern trend of using leadership in nursing rather than management. About 65% of the nurses reported having formal leadership-related education, while around 71% reported no formal management-related education.

The findings clearly showed what nurses seek in a clinical leader. They appear to refer to a good communicator who values relationships and encouragement, is flexible, approachable and compassionate, can set goals and plans, resource allocation, is clinically competent and visible and has integrity. They necessitate clinical nursing leaders who can be role models for others in practice and deal with change. They should be supportive decision-makers, mentors and motivators. They should be emphatic; otherwise, they should not be in a position of control. These findings align with other research on clinical leadership. 7–9 21 Clinical leaders should be visible and participate in team activities. They should be highly skilled clinicians who instil trust and set an example, and their values should guide them in providing excellent patient care. 8 9

Participants chose other terms or functions associated with leadership roles less frequently or perceived as unrelated to clinical leadership functions. Management, creativity and vision were among the terms and functions mentioned. The absence of the word ‘visionary’ from the list of the most important characteristics suggests that traditional leadership theories, as transformational leadership and situational leadership, do not provide a solid foundation for understanding clinical leadership approaches in the clinical setting. This result can also be influenced by the junior level of the majority of the sample.

Skills of clinical nursing leadership

Numerous studies have documented the characteristics and skills of clinical leaders. 27 29 31 Clinical leaders’ skills include advocacy, facilitation and healthier workplaces. 27 29 31 Our participants were rated as having high morals (similar to other studies) 27 29 31 and worked within teams. 29 In turn, they were flexible and expressed appreciation promptly. 7–9 21 They were clinically competent; thus, they improvised and responded to various situations with appropriate skills and interventions. They recognised optimal performance, initiated interventions, led actions and procedures and had the skills and resources necessary to perform their tasks.

The lowest mean was ‘ being in a management position to be effective ’. This lowest meaning ‘ somehow ’ makes sense; all nurses can be effective leaders rather than managers, assuming effective clinical leadership roles without having management positions. 28 42

Actions of clinical nursing leadership

Influential nursing leaders are clinically competent and can initiate interventions and lead actions; these skills translate to actions. Clinical leaders are qualified to lead and manage the service improvement change (similar to Major). 42 This role will not suddenly happen; it requires clinical nursing leaders who encourage and participate in staff development education (consistent with Major). 42 This is an essential milestone and an example of providing valuable staff support. As these were the lowest reported actions, clinical nursing leaders should initiate and lead improvement initiatives in their clinical settings, 42 resulting in service improvement. They also have to influence evidence-based policies to improve work–life integration 43 and enhance patients, nurses and organisational outcomes. These outcomes include quality of care, nurses’ empowerment, job satisfaction, quality of life and work engagement. 4 11–17 32

Female nurses had more clinical leadership skills. Because the findings of this study have never been reported in the previous clinical leadership research literature, they are considered novel. This finding indicates that one possible explanation is that the overwhelming majority of respondents were females, with the proportion of females in favour (70.6%) exceeding that of males (29.4%). Furthermore, the current findings could be explained because the study was conducted in Jordan, a traditionally female-dominated gender nursing career.

This study discovered that there are gender differences in the characteristics of nurses and their clinical leadership skills, with female clinical nursing leaders scoring higher on the t-test than male clinical nursing leaders in the following areas: this is contrary to Masanotti et al , 43 who reported that male nurses have a greater sense of coherence and, in turn, more teamwork than female nurses, who commonly have job dissatisfaction and less teamwork. These could apply to female clinical nursing leaders. These female nurses had more ‘visibility in the clinical environment’, as expected in female-dominated gender nursing careers. As they were commonly dissatisfied as nurses, 43 clinical nursing leaders would be competent in caring for their nurses’ psychological status. These leaders know that even ‘thank you’ is the simplest way to show appreciation and recognition; however, this should be given promptly.

In Arab and developing countries, the perception that females have more skills with effective clinical leadership characteristics than males is consistent with Alghamdi et al 44 and Yaseen. 45 They found that females outperform males on leadership scales, which may also apply to clinical leadership. This study shows consistency between female and male clinical nursing leaders’ general perceptions of clinical leadership skills in female-dominated gender nursing careers but not in male-dominated, gender-segregated countries, including Jordan.

Female nurses had more clinical leadership actions, which differed in five out of eight actions. Female clinical nursing leaders were better at impacting clinical care delivery, participating in staff development education, providing valuable staff support, leading change and improving service.

It is aware that the nursing profession has a difficult context in some Arab and developing countries. For example, a study conducted in Saudi Arabia could explain the current findings that male nurses face various challenges, including a lack of respect and discrimination, resulting in fewer opportunities for professional growth and development. 46 The researchers reported that female clinical nursing leaders are preferred over male nurses because nursing is a nurturing and caring profession; it has been dubbed a ‘female profession’. 46 Additionally, this study corroborates a study that found many males avoid the nursing profession entirely due to its negative connotations 47 ; the profession is geared towards females. These and other stereotypes have influenced male nurses to pursue masculine nursing roles.

The study’s findings are unique because they have never been published in the previous clinical leadership research literature. However, these results can be explained indirectly based on non-clinical leadership literature. Consistent with Khammar et al , 48 as it is a female-dominated profession, it is apparent that female clinical nursing leaders are better at delivering clinical care. This result could also be related to female clinical nursing leaders having a better attitude towards clinical conditions and managing different conditions. 48 Female clinical nursing leaders, in turn, are better at influencing patient care and improving patient safety 36 and overall care and services. This improvement will not happen suddenly; it should be accompanied by paying more attention to providing continuous support, especially during induced change.

The current study reported that female clinical nursing leaders supported staff development and education because it is a female-oriented sample. Yet, Khammar et al 48 reported that men had more opportunities to educate themselves in nursing; this is true in a male-dominated country like Jordan. They also noted that males could communicate better during nursing duties. Regardless of gender, all of us should pay attention to our staff’s working environment and related issues, including promoting open communication, providing support, encouraging continuing education, managing change and improving the overall outcomes.

Limitations

Even though the study’s findings are intriguing, further investigation is needed to comprehend them. Because of the cross-sectional design used in the current study, we cannot establish causality. For this reason, the results should be interpreted with caution. Also, the purposive sample limits the generalisability; thus, this research should be carried out again with a broader selection of nursing candidates and clinical settings. Moreover, the sample consists mostly of nurses with minimal experience compared with nurses in other international countries such as Canada, the UK and the USA. 5 The current study also included nurses in their 40s and above, with male nurses less represented, and this causes misunderstanding of the true clinical leadership in nursing.

Implications

For practice, our sample consists of nurses with minimal experience compared with nurses in other developed counties. Our sample reported ‘influencing organizational policy’ as the last clinical leadership skill, which reflects the very junior nature of the sample. Unlike our study, in their systematic review, Guibert-Lacasa and Vázquez-Calatayud 36 reported that the profiles of the care clinical nurses’ experience usually varied, ranging from recent graduates to senior nurses. If our nurses were more experienced, it might lead to different results. More nurses’ clinical experience would increase nurses’ abilities at the bedside, especially in areas related to reasoning and problem solving. 36 More experienced nurses tend to work collaboratively within the team with greater competency and autonomy. 36 More experienced nurses would provide high-quality care, 36 resulting in patient satisfaction. To generate positive outcomes of clinical nursing leadership, such early-career nurses should be qualified. Guibert-Lacasa and Vázquez-Calatayud 36 suggested using the nursing clinical leadership programme based on the American Organization for Nursing Leadership 34 competency model, pending the presence of organisational support for such an initiative. 36

‘Most’ important clinical nursing leadership attributes should be promoted at all organisational and clinical levels. Clinical nursing leadership’s ‘least’ important attributes should be defeated to achieve better outcomes. Clinical nursing leaders should use innovative interventions and have skills or actions conducive to a healthy work environment. These interventions include being approachable to enable their staff to cope with change, 28 using open and consistent communication, 28–30 being visible and consistently available as role models and mentors and taking risks. 28 Hospital administrators must help their clinical leaders, including nursing leaders, to effectively use their authority, responsibility and accountability; clinical leadership is not only about complying with the job description. A good intervention to start with to promote the culture of clinical leadership is setting an award for the ‘ideal nursing leaders’. This award will bring innovative attributes, skills and actions.

Moreover, as they are in the front line of communication, nurses and clinical nursing leaders should be involved in policy-related matters and committees. 49 An interventional programme that gives nurses more autonomy in making decisions is warranted. In turn, various patient, nurse and organisational outcomes will be improved. 13–17 32

The study’s findings revealed statistically significant differences in the skills and actions of effective clinical leaders, with female nurses scoring higher in many skills and actions. Hence, healthcare organisations must re-evaluate current leadership and staff development policies and prioritise professional development for nurses while also introducing new modes of evaluation and assessment that are explicitly geared at improving clinical leadership among nurses, particularly males.

For education, this study outlined clinical leadership attributes, skills and actions to understand clinical nursing leadership in Jordan better. Nevertheless, nurses and clinical leaders need additional attributes, skills and actions. Consequently, undergraduate nursing students might benefit from clinical leadership programmes integrated into the academic curriculum to teach them the fundamentals of clinical leadership. A master’s degree programme in ‘Clinical Nursing Leadership’ would prepare nurses for this pioneering role and today and tomorrow’s clinical nursing leaders. However, all nurses are clinical leaders regardless of their degrees and experience. Conducting presentations, convening meetings, overseeing organisational transformation and settling disagreements are common ways to hone these abilities.

For research purposes, it is worth exploring the concept of clinical leadership from a practice nurse’s perspective to provide insight into practice nurses’ feelings and perceptions. Thus, a longitudinal quantitative design or a phenomenological qualitative design might be adopted to assess the subjective experience of the nurses involved. It is better in future research to focus on both young and veteran clinical leaders; some of our nurses were aged 45 years and above, and those nurses may not be clinically focused.

Summary and conclusion

The current study put clinical leadership into the context of the healthcare system in Jordan. This study highlighted the role of gender in clinical nursing leadership. Nurses’ clinical leadership is a milestone for influencing innovation and change. The current study identified the ‘most’ and ‘least’ important attributes, skills and actions associated with clinical leadership. However, the male and female nurses found substantial differences in effective clinical nursing leadership skills and actions. This study is unique; little is known about the collective concepts of attributes, skills and actions necessary for clinical nursing leadership.

Nurses need leadership attributes, skills and actions to influence policy development and change in their work environments. Leadership attributes can help develop programmes that give nurses more autonomy in making decisions. As a result, nurses will be more active as clinical leaders.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

This study involves human participants and was approved by The Hashemite University, Jordan (IRB number: 1/1/2020/2021) on 18 October 2020. Participants gave informed consent to participate in the study before taking part.

Acknowledgments

The researchers thank the subjects who participated in the study, and Mrs Othman and Mr Sayaheen who collected the data.

  • Folaron I ,
  • Colburn JA , et al
  • Edmonson C ,
  • McCarthy C ,
  • Trent-Adams S , et al
  • Cummings GG ,
  • Tate K , et al
  • Connolly M ,
  • Stanley D ,
  • Blanchard D ,
  • Hohol A , et al
  • Cao T , et al
  • Cordeiro R ,
  • Rengel-Diaz C ,
  • Duignan M ,
  • Drennan J ,
  • McCarthy VJC
  • Tao Z , et al
  • Cuthbertson J ,
  • Metcalfe H , et al
  • Latimer K ,
  • NHS Leadership Academy
  • Loewenthal L , et al
  • Claus S , et al
  • Jackson D ,
  • Mannix J , et al
  • Siegel EO ,
  • American Organization for Nursing Leadership
  • Guibert-Lacasa C ,
  • Vázquez-Calatayud M
  • McNamara MS ,
  • Casey M , et al
  • Peltzer JN ,
  • Shen Q , et al
  • Australian Commission on Safety and Quality in Health Care
  • Al-Motlaq M ,
  • Masanotti GM ,
  • Paolucci S ,
  • Abbafati E , et al
  • Alghamdi MG ,
  • Abdelkader MA ,
  • Abdelrahman S
  • Abushaikha L ,
  • Mahadeen A ,
  • AbdelKader R , et al
  • Khammar A ,
  • Poursadeghiyan M ,
  • Marioryad H , et al
  • Kitson AL ,
  • Gifford W , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Data supplement 1

Contributors MTM developed the study conception, abstract, introduction, literature review and methods; collected the data and wrote the first draft of this research paper and the final proofreading. HAN analysed the data and wrote the results. AA wrote the discussion and updated the literature review. OK wrote the limitations, implications, and summary and conclusion. IAF and AAK did the critical revisions and the final proofreading. All authors contributed to the current work.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

  • Department of Health and Human Services
  • National Institutes of Health

Clinical Center Logo

Nursing at the NIH Clinical Center

Clinical research nurse roles.

Health Care Technician     The Health Care Technician is a certified nursing assistant in the state of Maryland or has completed the Fundamentals of Nursing course work in a current accredited nursing program.  This role supports the activities of the professional nurse by providing patient care functions to assigned patients while maintaining a safe environment.

Patient Care Technician     The Patient Care Technician is a certified nursing assistant in the state of Maryland who supports the activities of the professional nurse by independently providing patient care functions to assigned patients while maintaining a safe environment.

Clinical Research Nurse I      The Clinical Research Nurse (CRN) I has a nursing degree or diploma from a professional nursing program approved by the legally designated state accrediting agency.  The CRN I is a newly graduated registered nurse with 6 months or less of clinical nursing experience.  The incumbent functions under the direction of an experienced nurse to provide patient care, while using professional judgment and sound decision making.

Clinical Research Nurse II      The CRN II has a nursing degree or diploma from a professional nursing program approved by the legally designated state accrediting agency and has practiced nursing at the NIH Clinical Center for at least 6 months.  This nurse independently provides nursing care; identifies and communicates the impact of the research process on patient care; adjusts interventions based on findings; and reports issues/variances promptly to the research team. The CRN II administers research interventions; collects patient data according to protocol specifications; evaluates the patient response to therapy; and integrates evidence-based practice into nursing practice. The CRN II contributes to teams, workgroups and the nursing shared governance process.  New skills and knowledge are acquired that are based on self-assessment, feedback from peers and supervisors, and changing clinical practice requirements.

Clinical Research Nurse III     The CRN III has a nursing degree or diploma from a professional nursing program approved by the legally designated state accrediting agency at the time the program was completed by the applicant.

The CRN III has practiced nursing at the NIH Clinical Center for at least 1 year.  The role spans the professional nursing development from “fully competent” to “expert” nursing practice.  The CRN III provides care to acute and complex patient populations, and utilizes appropriate professional judgment and critical decision making in planning and providing care. S/he masters all nursing skills and associated technology for a particular Program of Care and assists in assessing the competency of less experienced nurses. The CRN III participates in the planning of new protocol implementation on the patient care unit; administers research interventions; collects patient data according to protocol specifications; evaluates the patient’ response to therapy; responds to variances in protocol implementation; reports variances to the research team; integrates evidence-based practice into nursing practice; and evaluates patient outcomes. The CRN III assumes the charge nurse and preceptor roles as assigned. Formal and informal feedback is provided by the CRN III to peers and colleagues in support of individual growth and improvement of the work environment.

Senior Clinical Research Nurse      The Senior CRN has a nursing degree or diploma from a professional nursing program approved by the legally designated state accrediting agency at the time the program was completed by the applicant.

The Senior CRN serves as a leader in all aspects of nursing practice. S/he demonstrates expertise in the nursing process; professional judgment and decision making; planning and providing nursing care; and knowledge of the biomedical research process.  The Senior CRN utilizes basic leadership principles and has an ongoing process of questioning and evaluating nursing practice. 

The Senior CRN may have one of three foci:

Specialty Practice- as a clinical expert for a designated patient population or program of care Management- Clinical Manager     The Senior CRN-Clinical Manager (CM) is an experienced staff nurse who collaborates with the Nurse Manager and other departmental leadership to assist with operations and management of a patient care area. The CM supervises the delivery of high quality patient care and assures the appropriate use of resources (i.e. staffing, rooms, etc.). The CM interacts with the research teams and support services to promote positive patient care outcomes and maintain protocol integrity. The CM models effective leadership, outstanding communication skills, while promoting a safe, supportive and professional environment. Education-  Clinical Educator     The Senior CRN-Clinical Educator (CE) is an experienced staff nurse who collaborates with the Nurse Manager and other departmental leadership to oversee educational needs of unit staff. The CE develops/coordinates/evaluates orientation for new unit staff, trains/mentors unit preceptors, serves as a liaison/resource for departmental/Clinical Center/professional educational opportunities, identifies educational needs, coordinates unit in-services, and plans unit educational days. The CE designs, implements and evaluates learning experiences for all staff levels to acquire, maintain, or increase their knowledge and competence. The Clinical Educator teaches at the unit and departmental level.

Nurse Manager     The Nurse Manager has 3 to 5 years of recent management experience;  advanced preparation (Masters Degree) is preferred. The Nurse Manager has experience in change management, creative leadership and program development; an demonstrates strong communication and collaboration skills to foster an effective partnership with institute personnel.. The Nurse Manager demonstrates a high level of knowledge in a particular specialty practice area and utilizes advanced leadership skills to meet organization goals.

Clinical Nurse Specialist     The Clinical Nurse Specialist (CNS) has a Masters or Doctorate Degree in Nursing from a state-approved school of nursing accredited by either the National League for Nursing Accrediting Commission (NLNAC) or the Commission on Collegiate Nursing Education (CCNE) with a major in the clinical nursing specialty to which the nurse is assigned.  The CNS has a minimum of 5 years experience, is certified in a specialty area, and is accountable for a specific patient population within a specialized program of care.

Nurse Consultant     The Nurse Consultant serves as an expert advisor and program manager for a specific area of clinical administration or clinical practice management. The incumbent serves as liaison to Clinical Center departments and the ICs for issues related area of expertise and assigned responsibility and to provide communication and consultative services to all credentialed nurses at the Clinical Center.

Nurse Scientist     The Nurse Scientist is a nurse with advanced preparation (PhD in nursing or related field) in research principles and methodology, who also has expert content knowledge in a specific clinical area. The primary focus of the role is to (1) provide leadership in the development, coordination and management of clinical research studies; (2) provide mentorship for nurses in research; (3) lead evaluation activities that improve outcomes for patients participating in research studies at the Clinical Center; and (4) contribute to the overall health sciences literature. The incumbent is expected to develop a portfolio of independent research that provides the vehicle for achieving these primary objectives.

NOTE: PDF documents require the free Adobe Reader .

This page last updated on 04/08/2022

You are now leaving the NIH Clinical Center website.

This external link is provided for your convenience to offer additional information. The NIH Clinical Center is not responsible for the availability, content or accuracy of this external site.

The NIH Clinical Center does not endorse, authorize or guarantee the sponsors, information, products or services described or offered at this external site. You will be subject to the destination site’s privacy policy if you follow this link.

More information about the NIH Clinical Center Privacy and Disclaimer policy is available at https://www.cc.nih.gov/disclaimers.html

  • Systematic review
  • Open access
  • Published: 19 February 2024

‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice

  • Annette Boaz   ORCID: orcid.org/0000-0003-0557-1294 1 ,
  • Juan Baeza 2 ,
  • Alec Fraser   ORCID: orcid.org/0000-0003-1121-1551 2 &
  • Erik Persson 3  

Implementation Science volume  19 , Article number:  15 ( 2024 ) Cite this article

1758 Accesses

68 Altmetric

Metrics details

The gap between research findings and clinical practice is well documented and a range of strategies have been developed to support the implementation of research into clinical practice. The objective of this study was to update and extend two previous reviews of systematic reviews of strategies designed to implement research evidence into clinical practice.

We developed a comprehensive systematic literature search strategy based on the terms used in the previous reviews to identify studies that looked explicitly at interventions designed to turn research evidence into practice. The search was performed in June 2022 in four electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched from January 2010 up to June 2022 and applied no language restrictions. Two independent reviewers appraised the quality of included studies using a quality assessment checklist. To reduce the risk of bias, papers were excluded following discussion between all members of the team. Data were synthesised using descriptive and narrative techniques to identify themes and patterns linked to intervention strategies, targeted behaviours, study settings and study outcomes.

We identified 32 reviews conducted between 2010 and 2022. The reviews are mainly of multi-faceted interventions ( n  = 20) although there are reviews focusing on single strategies (ICT, educational, reminders, local opinion leaders, audit and feedback, social media and toolkits). The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Furthermore, a lot of nuance lies behind these headline findings, and this is increasingly commented upon in the reviews themselves.

Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been identified. We need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of research perspectives (including social science) in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed.

Peer Review reports

Contribution to the literature

Considerable time and money is invested in implementing and evaluating strategies to increase the implementation of research into clinical practice.

The growing body of evidence is not providing the anticipated clear lessons to support improved implementation.

Instead what is needed is better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice.

This would involve a more central role in implementation science for a wider range of perspectives, especially from the social, economic, political and behavioural sciences and for greater use of different types of synthesis, such as realist synthesis.

Introduction

The gap between research findings and clinical practice is well documented and a range of interventions has been developed to increase the implementation of research into clinical practice [ 1 , 2 ]. In recent years researchers have worked to improve the consistency in the ways in which these interventions (often called strategies) are described to support their evaluation. One notable development has been the emergence of Implementation Science as a field focusing explicitly on “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice” ([ 3 ] p. 1). The work of implementation science focuses on closing, or at least narrowing, the gap between research and practice. One contribution has been to map existing interventions, identifying 73 discreet strategies to support research implementation [ 4 ] which have been grouped into 9 clusters [ 5 ]. The authors note that they have not considered the evidence of effectiveness of the individual strategies and that a next step is to understand better which strategies perform best in which combinations and for what purposes [ 4 ]. Other authors have noted that there is also scope to learn more from other related fields of study such as policy implementation [ 6 ] and to draw on methods designed to support the evaluation of complex interventions [ 7 ].

The increase in activity designed to support the implementation of research into practice and improvements in reporting provided the impetus for an update of a review of systematic reviews of the effectiveness of interventions designed to support the use of research in clinical practice [ 8 ] which was itself an update of the review conducted by Grimshaw and colleagues in 2001. The 2001 review [ 9 ] identified 41 reviews considering a range of strategies including educational interventions, audit and feedback, computerised decision support to financial incentives and combined interventions. The authors concluded that all the interventions had the potential to promote the uptake of evidence in practice, although no one intervention seemed to be more effective than the others in all settings. They concluded that combined interventions were more likely to be effective than single interventions. The 2011 review identified a further 13 systematic reviews containing 313 discrete primary studies. Consistent with the previous review, four main strategy types were identified: audit and feedback; computerised decision support; opinion leaders; and multi-faceted interventions (MFIs). Nine of the reviews reported on MFIs. The review highlighted the small effects of single interventions such as audit and feedback, computerised decision support and opinion leaders. MFIs claimed an improvement in effectiveness over single interventions, although effect sizes remained small to moderate and this improvement in effectiveness relating to MFIs has been questioned in a subsequent review [ 10 ]. In updating the review, we anticipated a larger pool of reviews and an opportunity to consolidate learning from more recent systematic reviews of interventions.

This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and Boaz, Baeza and Fraser [ 8 ] overview articles. To ensure optimal retrieval, our search strategy was refined with support from an expert university librarian, considering the ongoing improvements in the development of search filters for systematic reviews since our first review [ 11 ]. We also wanted to include technology-related terms (e.g. apps, algorithms, machine learning, artificial intelligence) to find studies that explored interventions based on the use of technological innovations as mechanistic tools for increasing the use of evidence into practice (see Additional file 1 : Appendix A for full search strategy).

The search was performed in June 2022 in the following electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched for articles published since the 2011 review. We searched from January 2010 up to June 2022 and applied no language restrictions. Reference lists of relevant papers were also examined.

We uploaded the results using EPPI-Reviewer, a web-based tool that facilitated semi-automation of the screening process and removal of duplicate studies. We made particular use of a priority screening function to reduce screening workload and avoid ‘data deluge’ [ 12 ]. Through machine learning, one reviewer screened a smaller number of records ( n  = 1200) to train the software to predict whether a given record was more likely to be relevant or irrelevant, thus pulling the relevant studies towards the beginning of the screening process. This automation did not replace manual work but helped the reviewer to identify eligible studies more quickly. During the selection process, we included studies that looked explicitly at interventions designed to turn research evidence into practice. Studies were included if they met the following pre-determined inclusion criteria:

The study was a systematic review

Search terms were included

Focused on the implementation of research evidence into practice

The methodological quality of the included studies was assessed as part of the review

Study populations included healthcare providers and patients. The EPOC taxonomy [ 13 ] was used to categorise the strategies. The EPOC taxonomy has four domains: delivery arrangements, financial arrangements, governance arrangements and implementation strategies. The implementation strategies domain includes 20 strategies targeted at healthcare workers. Numerous EPOC strategies were assessed in the review including educational strategies, local opinion leaders, reminders, ICT-focused approaches and audit and feedback. Some strategies that did not fit easily within the EPOC categories were also included. These were social media strategies and toolkits, and multi-faceted interventions (MFIs) (see Table  2 ). Some systematic reviews included comparisons of different interventions while other reviews compared one type of intervention against a control group. Outcomes related to improvements in health care processes or patient well-being. Numerous individual study types (RCT, CCT, BA, ITS) were included within the systematic reviews.

We excluded papers that:

Focused on changing patient rather than provider behaviour

Had no demonstrable outcomes

Made unclear or no reference to research evidence

The last of these criteria was sometimes difficult to judge, and there was considerable discussion amongst the research team as to whether the link between research evidence and practice was sufficiently explicit in the interventions analysed. As we discussed in the previous review [ 8 ] in the field of healthcare, the principle of evidence-based practice is widely acknowledged and tools to change behaviour such as guidelines are often seen to be an implicit codification of evidence, despite the fact that this is not always the case.

Reviewers employed a two-stage process to select papers for inclusion. First, all titles and abstracts were screened by one reviewer to determine whether the study met the inclusion criteria. Two papers [ 14 , 15 ] were identified that fell just before the 2010 cut-off. As they were not identified in the searches for the first review [ 8 ] they were included and progressed to assessment. Each paper was rated as include, exclude or maybe. The full texts of 111 relevant papers were assessed independently by at least two authors. To reduce the risk of bias, papers were excluded following discussion between all members of the team. 32 papers met the inclusion criteria and proceeded to data extraction. The study selection procedure is documented in a PRISMA literature flow diagram (see Fig.  1 ). We were able to include French, Spanish and Portuguese papers in the selection reflecting the language skills in the study team, but none of the papers identified met the inclusion criteria. Other non- English language papers were excluded.

figure 1

PRISMA flow diagram. Source: authors

One reviewer extracted data on strategy type, number of included studies, local, target population, effectiveness and scope of impact from the included studies. Two reviewers then independently read each paper and noted key findings and broad themes of interest which were then discussed amongst the wider authorial team. Two independent reviewers appraised the quality of included studies using a Quality Assessment Checklist based on Oxman and Guyatt [ 16 ] and Francke et al. [ 17 ]. Each study was rated a quality score ranging from 1 (extensive flaws) to 7 (minimal flaws) (see Additional file 2 : Appendix B). All disagreements were resolved through discussion. Studies were not excluded in this updated overview based on methodological quality as we aimed to reflect the full extent of current research into this topic.

The extracted data were synthesised using descriptive and narrative techniques to identify themes and patterns in the data linked to intervention strategies, targeted behaviours, study settings and study outcomes.

Thirty-two studies were included in the systematic review. Table 1. provides a detailed overview of the included systematic reviews comprising reference, strategy type, quality score, number of included studies, local, target population, effectiveness and scope of impact (see Table  1. at the end of the manuscript). Overall, the quality of the studies was high. Twenty-three studies scored 7, six studies scored 6, one study scored 5, one study scored 4 and one study scored 3. The primary focus of the review was on reviews of effectiveness studies, but a small number of reviews did include data from a wider range of methods including qualitative studies which added to the analysis in the papers [ 18 , 19 , 20 , 21 ]. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. In this section, we discuss the different EPOC-defined implementation strategies in turn. Interestingly, we found only two ‘new’ approaches in this review that did not fit into the existing EPOC approaches. These are a review focused on the use of social media and a review considering toolkits. In addition to single interventions, we also discuss multi-faceted interventions. These were the most common intervention approach overall. A summary is provided in Table  2 .

Educational strategies

The overview identified three systematic reviews focusing on educational strategies. Grudniewicz et al. [ 22 ] explored the effectiveness of printed educational materials on primary care physician knowledge, behaviour and patient outcomes and concluded they were not effective in any of these aspects. Koota, Kääriäinen and Melender [ 23 ] focused on educational interventions promoting evidence-based practice among emergency room/accident and emergency nurses and found that interventions involving face-to-face contact led to significant or highly significant effects on patient benefits and emergency nurses’ knowledge, skills and behaviour. Interventions using written self-directed learning materials also led to significant improvements in nurses’ knowledge of evidence-based practice. Although the quality of the studies was high, the review primarily included small studies with low response rates, and many of them relied on self-assessed outcomes; consequently, the strength of the evidence for these outcomes is modest. Wu et al. [ 20 ] questioned if educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes. Although based on evaluation projects and qualitative data, their results also suggest that positive changes on patient outcomes can be made following the implementation of specific evidence-based approaches (or projects). The differing positive outcomes for educational strategies aimed at nurses might indicate that the target audience is important.

Local opinion leaders

Flodgren et al. [ 24 ] was the only systemic review focusing solely on opinion leaders. The review found that local opinion leaders alone, or in combination with other interventions, can be effective in promoting evidence‐based practice, but this varies both within and between studies and the effect on patient outcomes is uncertain. The review found that, overall, any intervention involving opinion leaders probably improves healthcare professionals’ compliance with evidence-based practice but varies within and across studies. However, how opinion leaders had an impact could not be determined because of insufficient details were provided, illustrating that reporting specific details in published studies is important if diffusion of effective methods of increasing evidence-based practice is to be spread across a system. The usefulness of this review is questionable because it cannot provide evidence of what is an effective opinion leader, whether teams of opinion leaders or a single opinion leader are most effective, or the most effective methods used by opinion leaders.

Pantoja et al. [ 26 ] was the only systemic review focusing solely on manually generated reminders delivered on paper included in the overview. The review explored how these affected professional practice and patient outcomes. The review concluded that manually generated reminders delivered on paper as a single intervention probably led to small to moderate increases in adherence to clinical recommendations, and they could be used as a single quality improvement intervention. However, the authors indicated that this intervention would make little or no difference to patient outcomes. The authors state that such a low-tech intervention may be useful in low- and middle-income countries where paper records are more likely to be the norm.

ICT-focused approaches

The three ICT-focused reviews [ 14 , 27 , 28 ] showed mixed results. Jamal, McKenzie and Clark [ 14 ] explored the impact of health information technology on the quality of medical and health care. They examined the impact of electronic health record, computerised provider order-entry, or decision support system. This showed a positive improvement in adherence to evidence-based guidelines but not to patient outcomes. The number of studies included in the review was low and so a conclusive recommendation could not be reached based on this review. Similarly, Brown et al. [ 28 ] found that technology-enabled knowledge translation interventions may improve knowledge of health professionals, but all eight studies raised concerns of bias. The De Angelis et al. [ 27 ] review was more promising, reporting that ICT can be a good way of disseminating clinical practice guidelines but conclude that it is unclear which type of ICT method is the most effective.

Audit and feedback

Sykes, McAnuff and Kolehmainen [ 29 ] examined whether audit and feedback were effective in dementia care and concluded that it remains unclear which ingredients of audit and feedback are successful as the reviewed papers illustrated large variations in the effectiveness of interventions using audit and feedback.

Non-EPOC listed strategies: social media, toolkits

There were two new (non-EPOC listed) intervention types identified in this review compared to the 2011 review — fewer than anticipated. We categorised a third — ‘care bundles’ [ 36 ] as a multi-faceted intervention due to its description in practice and a fourth — ‘Technology Enhanced Knowledge Transfer’ [ 28 ] was classified as an ICT-focused approach. The first new strategy was identified in Bhatt et al.’s [ 30 ] systematic review of the use of social media for the dissemination of clinical practice guidelines. They reported that the use of social media resulted in a significant improvement in knowledge and compliance with evidence-based guidelines compared with more traditional methods. They noted that a wide selection of different healthcare professionals and patients engaged with this type of social media and its global reach may be significant for low- and middle-income countries. This review was also noteworthy for developing a simple stepwise method for using social media for the dissemination of clinical practice guidelines. However, it is debatable whether social media can be classified as an intervention or just a different way of delivering an intervention. For example, the review discussed involving opinion leaders and patient advocates through social media. However, this was a small review that included only five studies, so further research in this new area is needed. Yamada et al. [ 31 ] draw on 39 studies to explore the application of toolkits, 18 of which had toolkits embedded within larger KT interventions, and 21 of which evaluated toolkits as standalone interventions. The individual component strategies of the toolkits were highly variable though the authors suggest that they align most closely with educational strategies. The authors conclude that toolkits as either standalone strategies or as part of MFIs hold some promise for facilitating evidence use in practice but caution that the quality of many of the primary studies included is considered weak limiting these findings.

Multi-faceted interventions

The majority of the systematic reviews ( n  = 20) reported on more than one intervention type. Some of these systematic reviews focus exclusively on multi-faceted interventions, whilst others compare different single or combined interventions aimed at achieving similar outcomes in particular settings. While these two approaches are often described in a similar way, they are actually quite distinct from each other as the former report how multiple strategies may be strategically combined in pursuance of an agreed goal, whilst the latter report how different strategies may be incidentally used in sometimes contrasting settings in the pursuance of similar goals. Ariyo et al. [ 35 ] helpfully summarise five key elements often found in effective MFI strategies in LMICs — but which may also be transferrable to HICs. First, effective MFIs encourage a multi-disciplinary approach acknowledging the roles played by different professional groups to collectively incorporate evidence-informed practice. Second, they utilise leadership drawing on a wide set of clinical and non-clinical actors including managers and even government officials. Third, multiple types of educational practices are utilised — including input from patients as stakeholders in some cases. Fourth, protocols, checklists and bundles are used — most effectively when local ownership is encouraged. Finally, most MFIs included an emphasis on monitoring and evaluation [ 35 ]. In contrast, other studies offer little information about the nature of the different MFI components of included studies which makes it difficult to extrapolate much learning from them in relation to why or how MFIs might affect practice (e.g. [ 28 , 38 ]). Ultimately, context matters, which some review authors argue makes it difficult to say with real certainty whether single or MFI strategies are superior (e.g. [ 21 , 27 ]). Taking all the systematic reviews together we may conclude that MFIs appear to be more likely to generate positive results than single interventions (e.g. [ 34 , 45 ]) though other reviews should make us cautious (e.g. [ 32 , 43 ]).

While multi-faceted interventions still seem to be more effective than single-strategy interventions, there were important distinctions between how the results of reviews of MFIs are interpreted in this review as compared to the previous reviews [ 8 , 9 ], reflecting greater nuance and debate in the literature. This was particularly noticeable where the effectiveness of MFIs was compared to single strategies, reflecting developments widely discussed in previous studies [ 10 ]. We found that most systematic reviews are bounded by their clinical, professional, spatial, system, or setting criteria and often seek to draw out implications for the implementation of evidence in their areas of specific interest (such as nursing or acute care). Frequently this means combining all relevant studies to explore the respective foci of each systematic review. Therefore, most reviews we categorised as MFIs actually include highly variable numbers and combinations of intervention strategies and highly heterogeneous original study designs. This makes statistical analyses of the type used by Squires et al. [ 10 ] on the three reviews in their paper not possible. Further, it also makes extrapolating findings and commenting on broad themes complex and difficult. This may suggest that future research should shift its focus from merely examining ‘what works’ to ‘what works where and what works for whom’ — perhaps pointing to the value of realist approaches to these complex review topics [ 48 , 49 ] and other more theory-informed approaches [ 50 ].

Some reviews have a relatively small number of studies (i.e. fewer than 10) and the authors are often understandably reluctant to engage with wider debates about the implications of their findings. Other larger studies do engage in deeper discussions about internal comparisons of findings across included studies and also contextualise these in wider debates. Some of the most informative studies (e.g. [ 35 , 40 ]) move beyond EPOC categories and contextualise MFIs within wider systems thinking and implementation theory. This distinction between MFIs and single interventions can actually be very useful as it offers lessons about the contexts in which individual interventions might have bounded effectiveness (i.e. educational interventions for individual change). Taken as a whole, this may also then help in terms of how and when to conjoin single interventions into effective MFIs.

In the two previous reviews, a consistent finding was that MFIs were more effective than single interventions [ 8 , 9 ]. However, like Squires et al. [ 10 ] this overview is more equivocal on this important issue. There are four points which may help account for the differences in findings in this regard. Firstly, the diversity of the systematic reviews in terms of clinical topic or setting is an important factor. Secondly, there is heterogeneity of the studies within the included systematic reviews themselves. Thirdly, there is a lack of consistency with regards to the definition and strategies included within of MFIs. Finally, there are epistemological differences across the papers and the reviews. This means that the results that are presented depend on the methods used to measure, report, and synthesise them. For instance, some reviews highlight that education strategies can be useful to improve provider understanding — but without wider organisational or system-level change, they may struggle to deliver sustained transformation [ 19 , 44 ].

It is also worth highlighting the importance of the theory of change underlying the different interventions. Where authors of the systematic reviews draw on theory, there is space to discuss/explain findings. We note a distinction between theoretical and atheoretical systematic review discussion sections. Atheoretical reviews tend to present acontextual findings (for instance, one study found very positive results for one intervention, and this gets highlighted in the abstract) whilst theoretically informed reviews attempt to contextualise and explain patterns within the included studies. Theory-informed systematic reviews seem more likely to offer more profound and useful insights (see [ 19 , 35 , 40 , 43 , 45 ]). We find that the most insightful systematic reviews of MFIs engage in theoretical generalisation — they attempt to go beyond the data of individual studies and discuss the wider implications of the findings of the studies within their reviews drawing on implementation theory. At the same time, they highlight the active role of context and the wider relational and system-wide issues linked to implementation. It is these types of investigations that can help providers further develop evidence-based practice.

This overview has identified a small, but insightful set of papers that interrogate and help theorise why, how, for whom, and in which circumstances it might be the case that MFIs are superior (see [ 19 , 35 , 40 ] once more). At the level of this overview — and in most of the systematic reviews included — it appears to be the case that MFIs struggle with the question of attribution. In addition, there are other important elements that are often unmeasured, or unreported (e.g. costs of the intervention — see [ 40 ]). Finally, the stronger systematic reviews [ 19 , 35 , 40 , 43 , 45 ] engage with systems issues, human agency and context [ 18 ] in a way that was not evident in the systematic reviews identified in the previous reviews [ 8 , 9 ]. The earlier reviews lacked any theory of change that might explain why MFIs might be more effective than single ones — whereas now some systematic reviews do this, which enables them to conclude that sometimes single interventions can still be more effective.

As Nilsen et al. ([ 6 ] p. 7) note ‘Study findings concerning the effectiveness of various approaches are continuously synthesized and assembled in systematic reviews’. We may have gone as far as we can in understanding the implementation of evidence through systematic reviews of single and multi-faceted interventions and the next step would be to conduct more research exploring the complex and situated nature of evidence used in clinical practice and by particular professional groups. This would further build on the nuanced discussion and conclusion sections in a subset of the papers we reviewed. This might also support the field to move away from isolating individual implementation strategies [ 6 ] to explore the complex processes involving a range of actors with differing capacities [ 51 ] working in diverse organisational cultures. Taxonomies of implementation strategies do not fully account for the complex process of implementation, which involves a range of different actors with different capacities and skills across multiple system levels. There is plenty of work to build on, particularly in the social sciences, which currently sits at the margins of debates about evidence implementation (see for example, Normalisation Process Theory [ 52 ]).

There are several changes that we have identified in this overview of systematic reviews in comparison to the review we published in 2011 [ 8 ]. A consistent and welcome finding is that the overall quality of the systematic reviews themselves appears to have improved between the two reviews, although this is not reflected upon in the papers. This is exhibited through better, clearer reporting mechanisms in relation to the mechanics of the reviews, alongside a greater attention to, and deeper description of, how potential biases in included papers are discussed. Additionally, there is an increased, but still limited, inclusion of original studies conducted in low- and middle-income countries as opposed to just high-income countries. Importantly, we found that many of these systematic reviews are attuned to, and comment upon the contextual distinctions of pursuing evidence-informed interventions in health care settings in different economic settings. Furthermore, systematic reviews included in this updated article cover a wider set of clinical specialities (both within and beyond hospital settings) and have a focus on a wider set of healthcare professions — discussing both similarities, differences and inter-professional challenges faced therein, compared to the earlier reviews. These wider ranges of studies highlight that a particular intervention or group of interventions may work well for one professional group but be ineffective for another. This diversity of study settings allows us to consider the important role context (in its many forms) plays on implementing evidence into practice. Examining the complex and varied context of health care will help us address what Nilsen et al. ([ 6 ] p. 1) described as, ‘society’s health problems [that] require research-based knowledge acted on by healthcare practitioners together with implementation of political measures from governmental agencies’. This will help us shift implementation science to move, ‘beyond a success or failure perspective towards improved analysis of variables that could explain the impact of the implementation process’ ([ 6 ] p. 2).

This review brings together 32 papers considering individual and multi-faceted interventions designed to support the use of evidence in clinical practice. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been conducted. As a whole, this substantial body of knowledge struggles to tell us more about the use of individual and MFIs than: ‘it depends’. To really move forwards in addressing the gap between research evidence and practice, we may need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of perspectives, especially from the social, economic, political and behavioural sciences in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed. Harvey et al. [ 53 ] suggest that when context is likely to be critical to implementation success there are a range of primary research approaches (participatory research, realist evaluation, developmental evaluation, ethnography, quality/ rapid cycle improvement) that are likely to be appropriate and insightful. While these approaches often form part of implementation studies in the form of process evaluations, they are usually relatively small scale in relation to implementation research as a whole. As a result, the findings often do not make it into the subsequent systematic reviews. This review provides further evidence that we need to bring qualitative approaches in from the periphery to play a central role in many implementation studies and subsequent evidence syntheses. It would be helpful for systematic reviews, at the very least, to include more detail about the interventions and their implementation in terms of how and why they worked.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Before and after study

Controlled clinical trial

Effective Practice and Organisation of Care

High-income countries

Information and Communications Technology

Interrupted time series

Knowledge translation

Low- and middle-income countries

Randomised controlled trial

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30. https://doi.org/10.1016/S0140-6736(03)14546-1 .

Article   PubMed   Google Scholar  

Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it.” J Am Board Fam Pract. 2005;18:541–5. https://doi.org/10.3122/jabfm.18.6.541 .

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1–3. https://doi.org/10.1186/1748-5908-1-1 .

Article   PubMed Central   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:2–14. https://doi.org/10.1186/s13012-015-0209-1 .

Article   Google Scholar  

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1–8. https://doi.org/10.1186/s13012-015-0295-0 .

Nilsen P, Ståhl C, Roback K, et al. Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Sci. 2013;8:2–12. https://doi.org/10.1186/1748-5908-8-63 .

Rycroft-Malone J, Seers K, Eldh AC, et al. A realist process evaluation within the Facilitating Implementation of Research Evidence (FIRE) cluster randomised controlled international trial: an exemplar. Implementation Sci. 2018;13:1–15. https://doi.org/10.1186/s13012-018-0811-0 .

Boaz A, Baeza J, Fraser A, European Implementation Score Collaborative Group (EIS). Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4:212. https://doi.org/10.1186/1756-0500-4-212 .

Article   PubMed   PubMed Central   Google Scholar  

Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior – an overview of systematic reviews of interventions. Med Care. 2001;39 8Suppl 2:II2–45.

Google Scholar  

Squires JE, Sullivan K, Eccles MP, et al. Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implement Sci. 2014;9:1–22. https://doi.org/10.1186/s13012-014-0152-6 .

Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Development of an efficient search filter to retrieve systematic reviews from PubMed. J Med Libr Assoc. 2021;109:561–74. https://doi.org/10.5195/jmla.2021.1223 .

Thomas JM. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evid Based Med. 2013;1:1–6.

Effective Practice and Organisation of Care (EPOC). The EPOC taxonomy of health systems interventions. EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016. epoc.cochrane.org/epoc-taxonomy . Accessed 9 Oct 2023.

Jamal A, McKenzie K, Clark M. The impact of health information technology on the quality of medical and health care: a systematic review. Health Inf Manag. 2009;38:26–37. https://doi.org/10.1177/183335830903800305 .

Menon A, Korner-Bitensky N, Kastner M, et al. Strategies for rehabilitation professionals to move evidence-based knowledge into practice: a systematic review. J Rehabil Med. 2009;41:1024–32. https://doi.org/10.2340/16501977-0451 .

Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–8. https://doi.org/10.1016/0895-4356(91)90160-b .

Article   CAS   PubMed   Google Scholar  

Francke AL, Smit MC, de Veer AJ, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:1–11. https://doi.org/10.1186/1472-6947-8-38 .

Jones CA, Roop SC, Pohar SL, et al. Translating knowledge in rehabilitation: systematic review. Phys Ther. 2015;95:663–77. https://doi.org/10.2522/ptj.20130512 .

Scott D, Albrecht L, O’Leary K, Ball GDC, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:1–17. https://doi.org/10.1186/1748-5908-7-70 .

Wu Y, Brettle A, Zhou C, Ou J, et al. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14. https://doi.org/10.1016/j.nedt.2018.08.026 .

Yost J, Ganann R, Thompson D, Aloweni F, et al. The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis. Implement Sci. 2015;10:1–15. https://doi.org/10.1186/s13012-015-0286-1 .

Grudniewicz A, Kealy R, Rodseth RN, Hamid J, et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:2–12. https://doi.org/10.1186/s13012-015-0347-5 .

Koota E, Kääriäinen M, Melender HL. Educational interventions promoting evidence-based practice among emergency nurses: a systematic review. Int Emerg Nurs. 2018;41:51–8. https://doi.org/10.1016/j.ienj.2018.06.004 .

Flodgren G, O’Brien MA, Parmelli E, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD000125.pub5 .

Arditi C, Rège-Walther M, Durieux P, et al. Computer-generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2017. https://doi.org/10.1002/14651858.CD001175.pub4 .

Pantoja T, Grimshaw JM, Colomer N, et al. Manually-generated reminders delivered on paper: effects on professional practice and patient outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD001174.pub4 .

De Angelis G, Davies B, King J, McEwan J, et al. Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review. JMIR Med Educ. 2016;2:e16. https://doi.org/10.2196/mededu.6288 .

Brown A, Barnes C, Byaruhanga J, McLaughlin M, et al. Effectiveness of technology-enabled knowledge translation strategies in improving the use of research in public health: systematic review. J Med Internet Res. 2020;22:e17274. https://doi.org/10.2196/17274 .

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35. https://doi.org/10.1016/j.ijnurstu.2017.10.013 .

Bhatt NR, Czarniecki SW, Borgmann H, et al. A systematic review of the use of social media for dissemination of clinical practice guidelines. Eur Urol Focus. 2021;7:1195–204. https://doi.org/10.1016/j.euf.2020.10.008 .

Yamada J, Shorkey A, Barwick M, Widger K, et al. The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review. BMJ Open. 2015;5:e006808. https://doi.org/10.1136/bmjopen-2014-006808 .

Afari-Asiedu S, Abdulai MA, Tostmann A, et al. Interventions to improve dispensing of antibiotics at the community level in low and middle income countries: a systematic review. J Glob Antimicrob Resist. 2022;29:259–74. https://doi.org/10.1016/j.jgar.2022.03.009 .

Boonacker CW, Hoes AW, Dikhoff MJ, Schilder AG, et al. Interventions in health care professionals to improve treatment in children with upper respiratory tract infections. Int J Pediatr Otorhinolaryngol. 2010;74:1113–21. https://doi.org/10.1016/j.ijporl.2010.07.008 .

Al Zoubi FM, Menon A, Mayo NE, et al. The effectiveness of interventions designed to increase the uptake of clinical practice guidelines and best practices among musculoskeletal professionals: a systematic review. BMC Health Serv Res. 2018;18:2–11. https://doi.org/10.1186/s12913-018-3253-0 .

Ariyo P, Zayed B, Riese V, Anton B, et al. Implementation strategies to reduce surgical site infections: a systematic review. Infect Control Hosp Epidemiol. 2019;3:287–300. https://doi.org/10.1017/ice.2018.355 .

Borgert MJ, Goossens A, Dongelmans DA. What are effective strategies for the implementation of care bundles on ICUs: a systematic review. Implement Sci. 2015;10:1–11. https://doi.org/10.1186/s13012-015-0306-1 .

Cahill LS, Carey LM, Lannin NA, et al. Implementation interventions to promote the uptake of evidence-based practices in stroke rehabilitation. Cochrane Database Syst Rev. 2020. https://doi.org/10.1002/14651858.CD012575.pub2 .

Pedersen ER, Rubenstein L, Kandrack R, Danz M, et al. Elusive search for effective provider interventions: a systematic review of provider interventions to increase adherence to evidence-based treatment for depression. Implement Sci. 2018;13:1–30. https://doi.org/10.1186/s13012-018-0788-8 .

Jenkins HJ, Hancock MJ, French SD, Maher CG, et al. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ. 2015;187:401–8. https://doi.org/10.1503/cmaj.141183 .

Bennett S, Laver K, MacAndrew M, Beattie E, et al. Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies. Int Psychogeriatr. 2021;33:947–75. https://doi.org/10.1017/S1041610220001702 .

Noonan VK, Wolfe DL, Thorogood NP, et al. Knowledge translation and implementation in spinal cord injury: a systematic review. Spinal Cord. 2014;52:578–87. https://doi.org/10.1038/sc.2014.62 .

Albrecht L, Archibald M, Snelgrove-Clarke E, et al. Systematic review of knowledge translation strategies to promote research uptake in child health settings. J Pediatr Nurs. 2016;31:235–54. https://doi.org/10.1016/j.pedn.2015.12.002 .

Campbell A, Louie-Poon S, Slater L, et al. Knowledge translation strategies used by healthcare professionals in child health settings: an updated systematic review. J Pediatr Nurs. 2019;47:114–20. https://doi.org/10.1016/j.pedn.2019.04.026 .

Bird ML, Miller T, Connell LA, et al. Moving stroke rehabilitation evidence into practice: a systematic review of randomized controlled trials. Clin Rehabil. 2019;33:1586–95. https://doi.org/10.1177/0269215519847253 .

Goorts K, Dizon J, Milanese S. The effectiveness of implementation strategies for promoting evidence informed interventions in allied healthcare: a systematic review. BMC Health Serv Res. 2021;21:1–11. https://doi.org/10.1186/s12913-021-06190-0 .

Zadro JR, O’Keeffe M, Allison JL, Lembke KA, et al. Effectiveness of implementation strategies to improve adherence of physical therapist treatment choices to clinical practice guidelines for musculoskeletal conditions: systematic review. Phys Ther. 2020;100:1516–41. https://doi.org/10.1093/ptj/pzaa101 .

Van der Veer SN, Jager KJ, Nache AM, et al. Translating knowledge on best practice into improving quality of RRT care: a systematic review of implementation strategies. Kidney Int. 2011;80:1021–34. https://doi.org/10.1038/ki.2011.222 .

Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10Suppl 1:21–34. https://doi.org/10.1258/1355819054308530 .

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci. 2012;7:1–10. https://doi.org/10.1186/1748-5908-7-33 .

Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5:e008592. https://doi.org/10.1136/bmjopen-2015-008592 .

Metz A, Jensen T, Farley A, Boaz A, et al. Is implementation research out of step with implementation practice? Pathways to effective implementation support over the last decade. Implement Res Pract. 2022;3:1–11. https://doi.org/10.1177/26334895221105585 .

May CR, Finch TL, Cornford J, Exley C, et al. Integrating telecare for chronic disease management in the community: What needs to be done? BMC Health Serv Res. 2011;11:1–11. https://doi.org/10.1186/1472-6963-11-131 .

Harvey G, Rycroft-Malone J, Seers K, Wilson P, et al. Connecting the science and practice of implementation – applying the lens of context to inform study design in implementation research. Front Health Serv. 2023;3:1–15. https://doi.org/10.3389/frhs.2023.1162762 .

Download references

Acknowledgements

The authors would like to thank Professor Kathryn Oliver for her support in the planning the review, Professor Steve Hanney for reading and commenting on the final manuscript and the staff at LSHTM library for their support in planning and conducting the literature search.

This study was supported by LSHTM’s Research England QR strategic priorities funding allocation and the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. Grant number NIHR200152. The views expressed are those of the author(s) and not necessarily those of the NIHR, the Department of Health and Social Care or Research England.

Author information

Authors and affiliations.

Health and Social Care Workforce Research Unit, The Policy Institute, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE, UK

Annette Boaz

King’s Business School, King’s College London, 30 Aldwych, London, WC2B 4BG, UK

Juan Baeza & Alec Fraser

Federal University of Santa Catarina (UFSC), Campus Universitário Reitor João Davi Ferreira Lima, Florianópolis, SC, 88.040-900, Brazil

Erik Persson

You can also search for this author in PubMed   Google Scholar

Contributions

AB led the conceptual development and structure of the manuscript. EP conducted the searches and data extraction. All authors contributed to screening and quality appraisal. EP and AF wrote the first draft of the methods section. AB, JB and AF performed result synthesis and contributed to the analyses. AB wrote the first draft of the manuscript and incorporated feedback and revisions from all other authors. All authors revised and approved the final manuscript.

Corresponding author

Correspondence to Annette Boaz .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix a., additional file 2: appendix b., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Boaz, A., Baeza, J., Fraser, A. et al. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implementation Sci 19 , 15 (2024). https://doi.org/10.1186/s13012-024-01337-z

Download citation

Received : 01 November 2023

Accepted : 05 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01337-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation
  • Interventions
  • Clinical practice
  • Research evidence
  • Multi-faceted

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

clinical research study leader

Weill Cornell Medicine

Less Invasive Early Lung Cancer Study Receives Top 10 Clinical Research Achievement Award

  • Share to Twitter
  • Share to Facebook
  • Share to LinkedIn

A physician in a white coat behind a red backdrop

Dr. Nasser Altorki. Credit: Tiffany Walling/Getty Images for WCM. 

A Weill Cornell Medicine-led research team has been awarded a 2024 Top 10 Clinical Research Achievement Award from the Clinical Research Forum in recognition of an influential 2023 New England Journal of Medicine study on early-stage lung cancer resection.

The award is one of 10 given annually by the Clinical Research Forum for highly innovative and clinically translatable research with the potential to provide major benefits to patients. The Washington, D.C.-based organization is an influential advocate for government funding of clinical research and the interests of American clinical research institutions generally. The winners will present their award-winning research April 4 at the Clinical Research Forum’s annual meeting in Las Vegas.

The clinical trial results were published  Feb. 9, 2023 by a team led by Dr. Nasser Altorki , chief of the Division of Thoracic Surgery in the Department of Cardiothoracic Surgery at Weill Cornell Medicine and NewYork-Presbyterian/Weill Cornell Medical Center, and co-investigators from Duke University as well as investigators from 83 hospitals across the United States, Canada and Australia. The trial found that a surgery that removes only a portion of one of the five lobes that comprise a lung is as effective as removing an entire lobe for certain early-stage lung cancer patients.

“This award means a lot to me, as it recognizes an important advance in the surgical treatment of patients with early-stage lung cancer,” said Dr. Altorki, who is also the David B. Skinner, M.D. Professor of Thoracic Surgery and a professor of cardiothoracic surgery at Weill Cornell Medicine, and a thoracic surgeon at NewYork-Presbyterian/Weill Cornell Medical Center. “I think the award also recognizes the contribution of Weill Cornell Medicine and NewYork-Presbyterian to cooperative group trials supported by the National Cancer Institute.”

In the trial, investigators compared outcomes for nearly 700 patients with early-stage lung cancer, about half of whom were randomly assigned to “lobectomy” surgery, which removes the whole lobe, while the other half had “sublobar resection” surgery, which removes part of the affected lobe. Over a median follow-up period of seven years after surgery, the two groups did not differ significantly in terms of disease-free or overall survival, and the sublobar group had modestly better lung function.

Lobectomy has been the standard approach for early-stage lung cancer surgery for almost 50 years, but the study’s results indicate that a subset of early-stage lung cancer patients would be better off, or at least no worse, with the more tissue-conserving sublobar surgery.

“We started the trial in 2007 and it took about 10 years to complete,”  said  Dr. Altorki, who is also a member of the Sandra and Edward Meyer Cancer Center at Weill Cornell Medicine. “We then we had to wait until we got the results, which unexpectedly came in May of 2022. They were amazing results, and it was worth the wait, and it changed practice.” 

Related News

  • Genetic Signature May Predict Response to Immunotherapy for Non-Small Cell Lung Cancer
  • New Insight into How an Old IBD Drug Works
  • Helping Patients with Low Income Overcome Eating Disorders

Back to News

Weill Cornell Medicine Office of External Affairs New York, NY --> Phone: (646) 962-9476

  • See us on facebook
  • See us on twitter
  • See us on youtube
  • See us on linkedin
  • See us on instagram

Stanford Medicine study identifies distinct brain organization patterns in women and men

Stanford Medicine researchers have developed a powerful new artificial intelligence model that can distinguish between male and female brains.

February 20, 2024

sex differences in brain

'A key motivation for this study is that sex plays a crucial role in human brain development, in aging, and in the manifestation of psychiatric and neurological disorders,' said Vinod Menon. clelia-clelia

A new study by Stanford Medicine investigators unveils a new artificial intelligence model that was more than 90% successful at determining whether scans of brain activity came from a woman or a man.

The findings, published Feb. 20 in the Proceedings of the National Academy of Sciences, help resolve a long-term controversy about whether reliable sex differences exist in the human brain and suggest that understanding these differences may be critical to addressing neuropsychiatric conditions that affect women and men differently.

“A key motivation for this study is that sex plays a crucial role in human brain development, in aging, and in the manifestation of psychiatric and neurological disorders,” said Vinod Menon , PhD, professor of psychiatry and behavioral sciences and director of the Stanford Cognitive and Systems Neuroscience Laboratory . “Identifying consistent and replicable sex differences in the healthy adult brain is a critical step toward a deeper understanding of sex-specific vulnerabilities in psychiatric and neurological disorders.”

Menon is the study’s senior author. The lead authors are senior research scientist Srikanth Ryali , PhD, and academic staff researcher Yuan Zhang , PhD.

“Hotspots” that most helped the model distinguish male brains from female ones include the default mode network, a brain system that helps us process self-referential information, and the striatum and limbic network, which are involved in learning and how we respond to rewards.

The investigators noted that this work does not weigh in on whether sex-related differences arise early in life or may be driven by hormonal differences or the different societal circumstances that men and women may be more likely to encounter.

Uncovering brain differences

The extent to which a person’s sex affects how their brain is organized and operates has long been a point of dispute among scientists. While we know the sex chromosomes we are born with help determine the cocktail of hormones our brains are exposed to — particularly during early development, puberty and aging — researchers have long struggled to connect sex to concrete differences in the human brain. Brain structures tend to look much the same in men and women, and previous research examining how brain regions work together has also largely failed to turn up consistent brain indicators of sex.

test

Vinod Menon

In their current study, Menon and his team took advantage of recent advances in artificial intelligence, as well as access to multiple large datasets, to pursue a more powerful analysis than has previously been employed. First, they created a deep neural network model, which learns to classify brain imaging data: As the researchers showed brain scans to the model and told it that it was looking at a male or female brain, the model started to “notice” what subtle patterns could help it tell the difference.

This model demonstrated superior performance compared with those in previous studies, in part because it used a deep neural network that analyzes dynamic MRI scans. This approach captures the intricate interplay among different brain regions. When the researchers tested the model on around 1,500 brain scans, it could almost always tell if the scan came from a woman or a man.

The model’s success suggests that detectable sex differences do exist in the brain but just haven’t been picked up reliably before. The fact that it worked so well in different datasets, including brain scans from multiple sites in the U.S. and Europe, make the findings especially convincing as it controls for many confounds that can plague studies of this kind.

“This is a very strong piece of evidence that sex is a robust determinant of human brain organization,” Menon said.

Making predictions

Until recently, a model like the one Menon’s team employed would help researchers sort brains into different groups but wouldn’t provide information about how the sorting happened. Today, however, researchers have access to a tool called “explainable AI,” which can sift through vast amounts of data to explain how a model’s decisions are made.

Using explainable AI, Menon and his team identified the brain networks that were most important to the model’s judgment of whether a brain scan came from a man or a woman. They found the model was most often looking to the default mode network, striatum, and the limbic network to make the call.

The team then wondered if they could create another model that could predict how well participants would do on certain cognitive tasks based on functional brain features that differ between women and men. They developed sex-specific models of cognitive abilities: One model effectively predicted cognitive performance in men but not women, and another in women but not men. The findings indicate that functional brain characteristics varying between sexes have significant behavioral implications.

“These models worked really well because we successfully separated brain patterns between sexes,” Menon said. “That tells me that overlooking sex differences in brain organization could lead us to miss key factors underlying neuropsychiatric disorders.”

While the team applied their deep neural network model to questions about sex differences, Menon says the model can be applied to answer questions regarding how just about any aspect of brain connectivity might relate to any kind of cognitive ability or behavior. He and his team plan to make their model publicly available for any researcher to use.

“Our AI models have very broad applicability,” Menon said. “A researcher could use our models to look for brain differences linked to learning impairments or social functioning differences, for instance — aspects we are keen to understand better to aid individuals in adapting to and surmounting these challenges.”

The research was sponsored by the National Institutes of Health (grants MH084164, EB022907, MH121069, K25HD074652 and AG072114), the Transdisciplinary Initiative, the Uytengsu-Hamilton 22q11 Programs, the Stanford Maternal and Child Health Research Institute, and the NARSAD Young Investigator Award.

About Stanford Medicine

Stanford Medicine is an integrated academic health system comprising the Stanford School of Medicine and adult and pediatric health care delivery systems. Together, they harness the full potential of biomedicine through collaborative research, education and clinical care for patients. For more information, please visit med.stanford.edu .

Artificial intelligence

Exploring ways AI is applied to health care

Stanford Medicine Magazine: AI

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • My Account Login
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Open access
  • Published: 19 February 2024

Selection, optimization and validation of ten chronic disease polygenic risk scores for clinical implementation in diverse US populations

  • Niall J. Lennon   ORCID: orcid.org/0000-0002-2874-7371 1   na1 ,
  • Leah C. Kottyan 2   na1 ,
  • Christopher Kachulis   ORCID: orcid.org/0000-0003-2095-7419 1 ,
  • Noura S. Abul-Husn   ORCID: orcid.org/0000-0002-5179-1944 3 ,
  • Josh Arias   ORCID: orcid.org/0000-0001-6545-6656 4 ,
  • Gillian Belbin 3 ,
  • Jennifer E. Below   ORCID: orcid.org/0000-0002-1346-1872 5 ,
  • Sonja I. Berndt 4 ,
  • Wendy K. Chung   ORCID: orcid.org/0000-0003-3438-5685 6 ,
  • James J. Cimino   ORCID: orcid.org/0000-0003-4101-1622 7 ,
  • Ellen Wright Clayton   ORCID: orcid.org/0000-0002-0308-4110 5 ,
  • John J. Connolly 8 ,
  • David R. Crosslin 9 , 10 ,
  • Ozan Dikilitas   ORCID: orcid.org/0000-0002-9906-8608 11 ,
  • Digna R. Velez Edwards 5 ,
  • QiPing Feng   ORCID: orcid.org/0000-0002-6213-793X 5 ,
  • Marissa Fisher 1 ,
  • Robert R. Freimuth 11 ,
  • Tian Ge 12 ,
  • The GIANT Consortium ,
  • The All of Us Research Program ,
  • Joseph T. Glessner   ORCID: orcid.org/0000-0001-5131-2811 8 ,
  • Adam S. Gordon   ORCID: orcid.org/0000-0002-2058-7289 13 ,
  • Candace Patterson 1 ,
  • Hakon Hakonarson   ORCID: orcid.org/0000-0003-2814-7461 8 ,
  • Maegan Harden   ORCID: orcid.org/0000-0002-3607-6416 1 ,
  • Margaret Harr 8 ,
  • Joel N. Hirschhorn 1 , 14 ,
  • Clive Hoggart 3 ,
  • Li Hsu   ORCID: orcid.org/0000-0001-8168-4712 15 ,
  • Marguerite R. Irvin 7 ,
  • Gail P. Jarvik 10 ,
  • Elizabeth W. Karlson 12 ,
  • Atlas Khan   ORCID: orcid.org/0000-0002-6651-2725 6 ,
  • Amit Khera 1 ,
  • Krzysztof Kiryluk   ORCID: orcid.org/0000-0002-5047-6715 6 ,
  • Iftikhar Kullo   ORCID: orcid.org/0000-0002-6524-3471 11 ,
  • Katie Larkin 1 ,
  • Nita Limdi 7 ,
  • Jodell E. Linder   ORCID: orcid.org/0000-0002-0081-4712 5 ,
  • Ruth J. F. Loos 16 , 17 ,
  • Yuan Luo   ORCID: orcid.org/0000-0003-0195-7456 13 ,
  • Edyta Malolepsza 1 ,
  • Teri A. Manolio   ORCID: orcid.org/0000-0001-5844-4382 4 ,
  • Lisa J. Martin   ORCID: orcid.org/0000-0001-8702-9946 2 ,
  • Li McCarthy 1 ,
  • Elizabeth M. McNally 13 ,
  • James B. Meigs 12 ,
  • Tesfaye B. Mersha   ORCID: orcid.org/0000-0002-9189-8447 2 ,
  • Jonathan D. Mosley   ORCID: orcid.org/0000-0001-6421-2887 5 ,
  • Anjene Musick   ORCID: orcid.org/0000-0001-7770-299X 18 ,
  • Bahram Namjou   ORCID: orcid.org/0000-0003-4452-7878 2 ,
  • Nihal Pai 1 ,
  • Lorenzo L. Pesce 13 ,
  • Ulrike Peters 15 ,
  • Josh F. Peterson 5 ,
  • Cynthia A. Prows 2 ,
  • Megan J. Puckelwartz 13 ,
  • Heidi L. Rehm   ORCID: orcid.org/0000-0002-6025-0015 1 ,
  • Dan M. Roden   ORCID: orcid.org/0000-0002-6302-0389 5 ,
  • Elisabeth A. Rosenthal   ORCID: orcid.org/0000-0001-6042-4487 10 ,
  • Robb Rowley 4 ,
  • Konrad Teodor Sawicki 13 ,
  • Daniel J. Schaid 11 ,
  • Roelof A. J. Smit 3 ,
  • Johanna L. Smith   ORCID: orcid.org/0000-0002-5861-0413 11 ,
  • Jordan W. Smoller   ORCID: orcid.org/0000-0002-0381-6334 12 ,
  • Minta Thomas 15 ,
  • Hemant Tiwari 7 ,
  • Diana M. Toledo 1 ,
  • Nataraja Sarma Vaitinadin 5 ,
  • David Veenstra 10 ,
  • Theresa L. Walunas   ORCID: orcid.org/0000-0002-7653-3650 13 ,
  • Zhe Wang   ORCID: orcid.org/0000-0002-8046-4969 3 ,
  • Wei-Qi Wei   ORCID: orcid.org/0000-0003-4985-056X 5 ,
  • Chunhua Weng 6 ,
  • Georgia L. Wiesner 5 ,
  • Xianyong Yin   ORCID: orcid.org/0000-0001-6454-2384 19 &
  • Eimear E. Kenny 3  

Nature Medicine volume  30 ,  pages 480–487 ( 2024 ) Cite this article

11k Accesses

373 Altmetric

Metrics details

  • Clinical genetics
  • Risk factors

Polygenic risk scores (PRSs) have improved in predictive performance, but several challenges remain to be addressed before PRSs can be implemented in the clinic, including reduced predictive performance of PRSs in diverse populations, and the interpretation and communication of genetic results to both providers and patients. To address these challenges, the National Human Genome Research Institute-funded Electronic Medical Records and Genomics (eMERGE) Network has developed a framework and pipeline for return of a PRS-based genome-informed risk assessment to 25,000 diverse adults and children as part of a clinical study. From an initial list of 23 conditions, ten were selected for implementation based on PRS performance, medical actionability and potential clinical utility, including cardiometabolic diseases and cancer. Standardized metrics were considered in the selection process, with additional consideration given to strength of evidence in African and Hispanic populations. We then developed a pipeline for clinical PRS implementation (score transfer to a clinical laboratory, validation and verification of score performance), and used genetic ancestry to calibrate PRS mean and variance, utilizing genetically diverse data from 13,475 participants of the All of Us Research Program cohort to train and test model parameters. Finally, we created a framework for regulatory compliance and developed a PRS clinical report for return to providers and for inclusion in an additional genome-informed risk assessment. The initial experience from eMERGE can inform the approach needed to implement PRS-based testing in diverse clinical settings.

Polygenic risk scores (PRSs) aggregate the effects of many genetic risk variants and can be used to predict an individual’s genetic predisposition to a disease or phenotype 1 . PRSs are being calculated and disseminated at a prodigious rate 1 , 2 , but their development and application to clinical care, particularly among ancestrally diverse individuals, present substantial challenges 3 , 4 , 5 . Incorporation of genomic risk information has the potential to improve risk estimation and management 4 , 6 , particularly at younger ages 7 . Clinical use of PRSs may ultimately prevent disease or enable its detection at earlier, more treatable stages 7 , 8 , 9 , 10 . Improved estimation of risk may also enable targeting of preventive or therapeutic interventions to those most likely to benefit from them while avoiding unnecessary testing or overtreatment 10 , 11 . However, clinical use of Eurocentric PRSs in diverse patient samples risks exacerbating existing health disparities 12 , 13 , 14 .

PRSs for individual conditions are typically generated from summary statistics derived from genome-wide association studies (GWASs), which are themselves derived from populations that are heavily overrepresented by individuals of European ancestry 12 . Such scores have been shown to have limited prediction accuracy with increasing genetic distance from European populations 12 , 15 . PRSs can be improved if developed and validated using multiancestry cohorts 16 . Clinical and environmental data combined with monogenic and polygenic risk measurements can improve risk prediction, as demonstrated in ref. 17 and other studies 18 . Approaches for combining genomic and nongenomic information, optimizing models for populations of diverse genetic ancestry and across age groups, and conveying this information to clinicians and patients have yet to be developed and applied in clinical care. Various forms of PRSs are available to consumers through commercial platforms such as 23andMe, Myriad Genetics (riskScore), Allelica, Ambry Genetics, and others, and several noncommercial studies have explored the clinical use of PRSs in direct-to-participant models 19 , 20 , 21 ; however, there is limited information on the clinical implementation considerations of returning PRSs across multiple phenotypes in a primary care setting 20 . Even before assessing the ability of PRSs to improve health outcomes, reduce risk and enhance clinical care, large multicenter prospective pragmatic studies are needed to assess how patients and care providers interact with and respond to PRSs in a primary care setting 22 .

The Electronic Medical Records and Genomics (eMERGE) Network is a multicenter consortium established in 2007 to conduct genomic research in biobanks with electronic medical records 23 , 24 . In 2020, eMERGE embarked on a study of genomic risk assessment and management in 5,000 children and 20,000 adults of diverse ancestry, beginning with efforts to identify and validate published PRSs across multiple race-ethnic groups (and inferred genetic ancestries) in ten common diseases with complex genetic etiologies. The study plans for 25,000 individuals (aged 3–75 years) to be recruited from general healthcare system populations. Six of the ten recruitment sites are committed to recruiting an ‘enhanced diversity cohort’, meaning that their enrollment will target 75% of enrolled individuals belonging to a racial or ethnic minority or medically underserved population, whereas the remainder of clinical sites will target 35% minority participants 22 . Enrollment is not targeted to individuals with specific conditions, although individuals with prevalent conditions can be included. For this prospective, pragmatic study, the primary outcome being measured is the number of new healthcare actions after return of the genome-informed risk assessment. This paper describes (1) identification, selection and optimization of the PRSs that are included in the study; (2) calibration of ancestry for PRS estimation using a modified method developed for eMERGE; (3) development and launch of clinical reporting tools; and (4) an overview of the first 2,500 samples processed as part of the study.

PRS auditing and evaluation

To select the PRSs for clinical implementation, the Network conducted a multistage process to evaluate proposed scores (Fig. 1 ). An initial set of 23 conditions was selected based on considerations including relevance to population health (condition prevalence and heritability), strength of evidence for PRS performance, clinical expertise in the eMERGE Network, and data availability that would facilitate validation of the PRS in diverse populations. These conditions were abdominal aortic aneurysm, age-related macular degeneration, asthma, atopic dermatitis, atrial fibrillation, bone mineral density, breast cancer, Crohn’s disease, chronic kidney disease, colorectal cancer, coronary heart disease, depression, hypercholesterolemia, hypertension, ischemic stroke, lupus, nonalcoholic fatty liver disease, obesity, primary open angle glaucoma, prostate cancer, rheumatoid arthritis, type 1 diabetes and type 2 diabetes.

figure 1

a ,Timeline and process for selection, evaluation, optimization, transfer, validation and implementation of the clinical PRS test pipeline. Dashed lines represent pivotal moments in the progression of the project with duration between these events indicated in months (mo) above the blue arrow. Numbers in white represent the number of conditions being examined at each stage and their fates. List of ten conditions on the right-hand side indicates the conditions that were implemented in the clinical pipeline for this study. b , Overview of the eMERGE PRS process. Participant DNA is genotyped using the Illumina Global Diversity Array, which assesses 1.8 million sites. Genotyping data are phased and imputed with a reference panel derived from the 1,000 Genomes Project. For each participant, raw PRSs are calculated for each condition ( PRS raw ). Each participant’s genetic ancestry is algorithmically determined in the projection step. For each condition, an ancestry calibration model is applied to each participant’s z- scores based on model parameters derived from the All of Us Research Program (Calibration) and an adjusted z -score is calculated ( PRS adjusted ). Participants whose adjusted scores cross the predefined threshold for high PRS are identified and a pdf report is generated. The report is electronically signed after data review by a clinical laboratory director and delivered to the study portal for return to the clinical sites.

Network sites completed a comprehensive literature review on 23 proposed conditions and the corresponding PRSs. A summary of the features of the PRS for each of the final conditions chosen is shown in Supplementary Table 1 . The collated information included analytic viability—a description of covariates, the age, and ancestry effects of the original PRS model; feasibility—access to sufficiently diverse validation datasets (genetic ancestry and age) as well as condition prevalence and relevance to preventative care; potential clinical actionability—existing screening or treatment strategies, and magnitude (odds ratio) of risk in the high-risk group; and translatability—expected public health impact across diverse populations. Candidate PRSs were restricted to those that were either previously validated and published (journal or preprint) or for which there was sufficient access to information to develop and/or optimize new PRSs, which could then be validated.

In auditing and evaluating evidence of PRS performance, the eMERGE steering committee considered PRSs for conditions that could be implemented in pediatric and/or adult populations, and for diseases with a range of age of onset (0 to >65 years of age). We considered published single nucleotide polymorphism (SNP)-based heritability estimates available for ten of the 23 conditions, ranging from 3% to 58%. The majority of PRSs under consideration aimed to identify individuals at high risk for disease; however, PRSs to predict disease severity and drug response were also considered. Two of the conditions, breast cancer and prostate cancer, were only considered for implementation in individuals whose biological sex was female or male, respectively. As the eMERGE Network plans to enroll >50% participants from underrepresented groups (including racial and ethnic minority groups; people with lower socioeconomic status; underserved rural communities; sexual and gender minority groups) 25 , emphasis was placed on PRSs that were already available for, or could be developed and validated in, diverse population groups.

To define population groups, study-level population descriptors were first extracted from published literature, preprints or information shared directly by collaborators on data used to develop and/or optimize and/or validate PRSs. Methods for defining population groups across studies ranged from self-reporting, extraction from health system data and/or analysis of genetic ancestry. We designated four population groups: European ancestry (that is, study population descriptors included European, European-American or other European descent diaspora groups), African (African, African American or other African descent diaspora groups), Hispanic (that is, Hispanic, Latina/o/x or those who have origins in countries in the Caribbean and Latin America) and Asian (that is, South Asian, East Asian, South-East Asian, Asian-American or other diaspora Asian groups).

Thirteen conditions were considered and not selected for clinical implementation (Fig. 1 ). Of the six conditions dropped from consideration in August 2020, low disease prevalence across ancestral groups (age-related macular degeneration), availability of diverse genetic datasets for validation (primary open angle glaucoma, rheumatoid arthritis and Crohn’s disease) and the lack of a validated algorithm to identify patients and controls based upon electronic health record (EHR) (bone mineral density) were the driving factors. In March 2021, five additional conditions were dropped from consideration for clinical implementation based upon the progress of the development and validation of a multiancestral PRSs (depression, ischemic stroke), the low predictive value of candidate PRSs (hypertension, nonalcoholic fatty liver disease) and ethical considerations around returning results to a condition with low population prevalence (lupus).

Conditions not prioritized for implementation continued on a ‘developmental’ pathway for further refinement. Each of the 12 conditions that were selected to move forward from the March 2021 review was assigned a ‘lead’ and ‘co-lead’ site, which worked together to develop, validate and transfer the score to the clinical laboratory for instantiation and Clinical Laboratory Improvement Amendments (CLIA) validation. Assignment of leads was based on site preference, expertise and distribution of workload.

Selection, optimization and validation

A systematic framework was developed to evaluate the performance for the remaining 12 PRSs, in accordance with best practices outlined in ref. 26 . An in-depth evaluation matrix of the 12 chosen conditions can be found in Supplementary Table 2 . The Network carefully considered a variety of strategies to optimize PRS generalizability and portability. The Network prioritized validation across four ancestries with an emphasis on African and Hispanic ancestry due to their underrepresentation in genetic research and projected representation within the study cohort. We determined that a PRS was validated if the odds ratios were statistically significant in a minimum of two and up to four ancestral populations: African/African American, Asian, European ancestry, and Hispanic/Latino. The PRS Working Group members conducted an extensive scoping exercise to identify suitable datasets of multiple ancestries for disease-specific PRS validation. These included datasets from early phases of eMERGE (2007–2019) as well as external datasets such as the UK Biobank and Million Veteran Program. These larger population-level databases had the advantage of large sample sizes and less case–control ascertainment bias (though other sources of bias can still be an issue; ‘Discussion’). A standardized set of questions was addressed by the disease leads that included the source of discovery and validation datasets, the availability of multiancestry validation datasets, the availability of cross-ancestry PRSs (that is, PRS models that were developed and validated in more than one genetic ancestry), proposed percentile thresholds for identifying high-risk status, model discrimination (AUC) and effect sizes (odds ratios) associated with high-risk versus not high-risk status (Supplementary Table 2 ). For seven out of the 12 candidate scores, no further optimization of the original model was performed. For five scores, an additional optimization effort was undertaken to further refine the score performance in multiple ancestries. Details of the optimization can be found in Supplementary Table 3 . A specific score optimization was applied for chronic kidney disease. This optimization consisted of adding the effect of APOL1 risk genotypes to a polygenic component, which has been found to improve risk predictions in African ancestry cohorts 27 .

For the final selection of PRSs to be included in the prospective clinical study, the steering committee considered the score performance summaries (presented by condition leads) in addition to the actionable and measurable recommendations relevant for return, for each condition, in the prospective cohort. Abdominal aortic aneurysm was removed from the clinical pathway in June 2021 based on inability to pull a critical risk factor from the EHR (smoking) and a relatively low disease prevalence in Asian and Hispanic populations. Colorectal cancer was removed in June 2021 because the development and validation of the PRS was not complete for all the ancestral groups (Fig. 1 ). For the ten remaining phenotypes, the prospective pragmatic study required a small number of measurable primary clinical recommendations per phenotype so that the utility of the PRS to change physician and patient behavior can be measured. These recommendations can be found in Supplementary Tables 2 and 4 of ref. 22 .

Population-based z -score calibration

In this study, the focus is on integration and implementation of validated PRSs in clinical practice rather than novel PRS development. Ultimately, the Network opted to balance generalizability and feasibility by validating and returning cross-ancestry PRSs. However, even with cross-ancestry scores, differences remain in the distribution of z -scores for the PRSs across genetic ancestries that can result in inconsistent categorization of individuals into ‘high’ or ‘not high’ polygenic risk categories for a given condition 28 . To that end, the Network chose to develop methods to genetically infer each participant’s ancestry and calibrate the distribution of resulting z -scores through a population-based calibration model 28 , 29 (see below). An alternative would have been to apply existing PRSs in available samples of different ancestries and derive ancestry-specific effect estimates. However, returning ancestry-specific risk estimates is challenging in real-world implementations as it would require self-reporting of ancestry by patients (who may not be able to provide this with accuracy) and developing multiple ancestry-specific reports for each health condition. In addition, such PRSs would be problematic to return to patients of mixed ancestry.

PRSs often have different means and standard deviations for individuals from different genetic ancestries. While some of these differences could be due to true biological differences in risk, they also result from allele frequency and linkage disequilibrium structure differences between populations 30 . This problem is more acute when a PRS is calculated for an individual whose ancestry does not match the ancestries used to develop the PRS. A clinically implemented PRS test to return disease risk estimates, therefore, must be adjusted to account for these differences due to ancestral background. A calibration method based on principal component analysis (PCA), which was initially described in ref. 28 , was modified to model both the variance and means of scores as ancestry dependent, as compared to the previous method ( Methods ), which modeled only the means as dependent on ancestry. This modification was found to be necessary because some conditions were found to exhibit highly ancestry-dependent variance, which would have led to many more or fewer participants of certain ancestries receiving a ‘high PRS’ determination than was intended. One option considered to create and train the calibration model was to enroll and process a representative number of participants then pause on the return of results while the model was trained and the older data reprocessed. This stop–start approach was deemed suboptimal. Instead, the model was fit, with permission, to a portion of the All of Us (AoU) Research Program ( https://www.researchallofus.org/ ) cohort genotyping data, which allowed for continuous return of results to eMERGE participants once the study began. Of note, the All of Us Research Program cohorts selected for both training and testing the calibration model exhibited high degrees of genetic admixture, which would be expected to accurately reflect the study enrollment population. Importantly, because no ancestry group is homogenous, when individuals are compared directly to other individuals in their assigned population group, a dependence between admixture fraction and PRS can result. This dependence is removed by the described PCA calibration method, and the resulting calibrated PRSs are independent of admixture fraction. More details about the ancestry calibration can be found in Methods .

Transfer and implementation

Once the final ten conditions had been selected, condition leads worked with computational scientists at the clinical laboratory (Clinical Research Sequencing Platform, LLC at the Broad Institute) to transfer the PRS models and create the sample and data-processing workflow (Fig. 2 ). Condition-specific models were run with outputs from the lab’s genotyping (Illumina Global Diversity Array (GDA)), phasing (Eagle2 (ref. 31 ) https://github.com/poruloh/Eagle ) and imputation (Minimac4 (ref. 32 ) https://genome.sph.umich.edu/wiki/Minimac4 ) pipelines to assess genomic site representation (see Methods for more information on the architecture and components of the pipeline). Several rounds of iteration between the clinical laboratory and condition leads followed in which any issues with the pipeline were resolved and the effect of genomic site missingness was assessed (Table 1 ). The final version of the implemented models was returned to the condition leads to recalculate effect sizes in the validation cohorts.

figure 2

‘High-PRS threshold’ represents the percentile that is deemed to be the cutoff for a specific condition above which a high-PRS result is reported for that condition. Odds ratios are reported as the mean odds ratios (square dot) associated with having a score above the specified threshold, compared to having a score below the specified threshold, along with 95% confidence intervals (CIs), shown in the whiskers. The number of case and control samples used to derive these odds ratios and CIs for each condition can be found in Supplementary Table 2 . Note that the odds ratio for obesity is not reported here, as it will be published by the Genetic Investigation of ANthropometric Traits consortium (Smit et al., manuscript in preparation). ‘Number of SNPs’ represents the range of numbers or sites included in each score. ‘Age ranges for return’ indicates the participant ages at which a PRS is calculated for a given condition. AFIB, atrial fibrillation; BC, breast cancer; CKD, chronic kidney disease; CHD, coronary heart disease; HC, hypercholesterolemia; PC, prostate cancer; T1D, type 1 diabetes; T2D, type 2 diabetes.

Finally, as part of the implementation of the PRS pipelines as a clinical test in a CLIA laboratory, a validation study was performed (see Methods for a detailed description; Table 1 summarizes some of the results). Briefly, this study leveraged 70 reference cell lines from diverse ancestry groups (Coriell) where 30X whole genome sequencing data were generated to form a variant truth set from which the technical accuracy and reproducibility of imputation and PRS calling was assessed. A second sample set of 20 matched donor blood and saliva specimens was procured to assess the performance of the pipeline with different input materials. A set of three samples, each with six replicates, was run end-to-end through the wet lab and analytical pipelines as an assessment of reproducibility. As a verification of the clinical validity of the scores, cohorts of cases for eight of the ten conditions were created using the eMERGE phase III imputed dataset (available on https://anvil.terra.bio/#workspaces/anvil-datastorage/AnVIL_eMERGE_GWAS/data (registration required)). PRS performance measures were calculated to confirm associations between scores and conditions. Due to limitations in the eMERGE phase III imputation (no chromosome X, different imputation pipeline), the odds ratios from this analysis were not included in the final reports; rather, the odds ratios calculated in the condition-specific validation cohorts (using the final clinical lab pipeline) were used (Fig. 2 and Table 1 ). A validation report was created for each condition. This report was reviewed and approved by the Laboratory Director in compliance with CLIA regulations for the development of a laboratory-developed test. Personnel were trained on laboratory and analytical procedures, and standard operating procedures were implemented. Data review metrics were established, sample pass/fail criteria were defined, and order and report data-transfer pipelines were built as described in ref. 22 .

Creation of pipeline for report creation, review, sign-out and release

A software pipeline was built to facilitate the data review and clinical report generation. Reports were created both as documents (in pdf format) and structured data (in JSON format; a sample report is included in the Supplementary Information ). Automated rules for case triage were built into the PRS calculation and reporting pipeline to account for differences in return based on age and sex at birth for certain conditions. For instance, the PRS for breast cancer is only calculated for participants who report sex at birth as female; similarly, prostate cancer scores are only generated for participants who report sex at birth as male. Age-related restrictions were similarly coded into the pipeline to account for study policies on return. Data review by an appropriately qualified, trained individual is required for high complexity clinical testing. In the PRS clinical pipeline, this review takes the form of a set of metrics that are exposed by the pipeline to the reviewer. These include a z -score range for each condition (passing samples will have a score −5 <  z  < +5), a PCA plot per batch against a reference sample set (visual representation of outlier samples), monitoring the z -score range for each control per condition (one control on each plate; NA12878) and flagging any samples with multiple ‘high risk’ results for further review.

Each participant’s sample is also run on an orthogonal fingerprinting assay (Fluidigm biomark) that creates a genotype-based fingerprint for that DNA aliquot. Infinium genotyping data are compared to this fingerprint as a primary check of sample chain-of-custody fidelity and to preclude sample or plate swaps during lab processing. Reviewed and approved data for a participant are processed into a clinical report. The text and format of this report were created during an iterative review process by consortium work groups. For this pragmatic clinical implementation study, two results are returned to participants: ‘high risk’ or ‘not high risk’ based on the PRS 22 . In the clinical report, a qualitative framework has been developed to indicate for which condition(s) a participant has been determined to have a high PRS (if any). Quantitative values ( z- scores) are not included for any condition in the main results panel. For breast cancer and CHD, the z -score is presented in another section of the report for inclusion in integrated score models for those conditions. For breast cancer specifically, the provided z -score is used with the BOADICEA 33 model to generate an integrated risk that is included in the genome-informed risk assessment (GIRA), as described in ref. 22 .

Overview of the first 2,500 clinical samples processed

Between the launch in July 2022 and May 2023, 2,500 participants were processed through the clinical PRS pipeline (representing ∼ 10% of the proposed cohort). Of the first 2,500 participants processed, 64.5% (1,612) indicated sex at birth as female, while 35.5% (886) indicated male. Median age at sample collection was 51 years (range: 3 years to 75 years). Participants self-reported race/ancestry, with 32.8% (820) identifying as ‘White (for example, English, European, French, German, Irish, Italian, Polish, etc.)’; 32.8% (820) identified as ‘Black, African American or African (for example, African American, Ethiopian, Haitian, Jamaican, Nigerian, Somali, etc.)’; 25.4% (636) identified as ‘Hispanic, Latino or Spanish (for example, Colombian, Cuban, Dominican, Mexican or Mexican American, Puerto Rican, Salvadoran, etc.)’; 5% (124) identified as ‘Asian (for example, Asian, Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, etc.)’; 1.5% (38) identified as American Indian or Alaska Native (for example, Aztec, Blackfeet Tribe, Mayan, Navajo Nation, Native Village of Barrow (Utqiagvik) Inupiat Traditional Government, Nome Eskimo Community, etc.); 0.9% (22) identified as Middle Eastern or North African (for example, Algerian, Egyptian, Iranian, Lebanese, Moroccan, Syrian, etc.); 0.8% (21) selected ‘None of these fully describe [me_or_my_child]’; 0.7% (17) selected ‘Prefer not to answer’; 0.1% (2) participants had incomplete data. A summary of the performance of the first 2,500 samples and resulting high-PRS metrics are shown in Fig. 3 . In the first 2,500 participants, we identified 515 participants (20.6%) with a high PRS for one of the ten conditions, 64 participants (2.6%) had a high PRS for two conditions, and two participants (0.08%) had a high PRS for three conditions. The remaining 1,919 participants had no high PRS found. High-PRS participants spanned the spectrum of genetic ancestry when projected onto principal component space (Fig. 3 ). Observed numbers of high-PRS assessments were largely consistent with the corresponding expected numbers. The P values in Fig. 3c are two-sided P values, which are calculated taking into account both the finite size of the eMERGE cohort and the finite size of the training data used to estimate the ancestry adjustment parameters. The P values are further adjusted for multiple hypothesis testing using the Holm–Šidák procedure 34 .

figure 3

a , PCA of ancestry indicating participants with a result of ‘high PRS’ for any condition (red dots) compared to participants who did not have a high PRS identified (gray dots). b , Summary of number of high-risk conditions found per participant. c , Observed numbers of high PRS called per condition compared to the expected numbers of high PRS per condition. P values are two-sided P values calculated by simulation to account for the uncertainty in the All of Us (AoU) derived ancestry calibration parameters due to the finite size of the AoU training cohort, and further adjusted for multiple hypothesis testing using the Holm–Šidák procedure. Note not all participants get scored for every condition based on age and sex at birth filters.

While the predictive performance of PRSs has improved substantially in recent years, challenges remain in ensuring that PRSs are applicable and effective in diverse populations. In particular, the vast majority of GWASs have focused on individuals of European ancestry, and the predictive accuracy of PRSs declines with increasing genetic distance from the discovery population 5 , 30 , 35 . This risks exacerbating existing health disparities, as clinical use of Eurocentric PRSs in diverse patient samples may not accurately reflect disease risk in non-European populations. To address these challenges, the eMERGE Network has conducted a multistage process to evaluate and optimize PRS selection, development and validation. The Network has prioritized conditions with high prevalence and heritability, existing literature, clinical actionability and the potential for health disparities, and has developed strategies to optimize PRS generalizability and portability across diverse populations. In particular, the Network has emphasized performance across four major ancestry groups (African, Asian, European, Hispanic, as reflected by self-identified race/ethnicity) and has developed a pipeline for clinical PRS implementation, a framework for regulatory compliance and a PRS clinical report.

The potential impact of PRS-based risk assessment in clinical practice is substantial. By enabling targeted interventions and preventative measures, PRS-based risk assessment has the potential to reduce the burden of a range of conditions 22 . Moreover, the development of PRS-based risk assessment in diverse populations has the potential to reduce health disparities by ensuring that clinical use of PRSs accurately reflects disease risk in diverse populations.

However, challenges remain in the successful implementation of PRS-based risk assessment in clinical practice. Participation bias in training or validation datasets that do not accurately represent the broader populations, for example the United Kingdom BioBank, can lead to skewed results and reduced generalizability in PRS test development 36 . Other challenges include concerns about genetic determinism, the potential for stigmatization and the need for robust regulatory frameworks to ensure that PRS-based risk assessment is deployed safely and effectively. Furthermore, to have more clinical utility, an individual’s PRS-based risk would be calculated as age-based absolute risk. Challenges also remain in healthcare provider and patient understanding and interpretation of PRS results and how to effectively communicate these results. Additionally, one of the biggest challenges is the implementation of effective disease prevention strategies after the return of the results. Return of the results will not result in a benefit without effective disease prevention or early detection strategies. The eMERGE Network’s work provides a blueprint for addressing these challenges, but ongoing research and evaluation will be necessary to ensure that PRS-based risk assessment is implemented in a responsible and effective manner. While this study will not answer all of the unanswered challenges and questions, the results from the 25,000 subjects from the eMERGE study will provide additional data to existing risk stratification to model harms and benefits over patient lifetimes.

Future groups developing, transferring and implementing PRSs into a clinical setting could build upon the eMERGE experience. Slightly less than half of the phenotypes originally considered for PRS development were able to be continued through clinical implementation based on varying considerations, suggesting that a moderately high number of phenotypes with measurable genetic contributions will be appropriate for PRS-based clinical tools. Thresholds for returning ‘high risk’ PRS were identified by each phenotype working group based in part upon the statistical significance between the ‘high-risk’ and ‘not high-risk’ groups. Future studies might consider standardizing the analyses and methods used to define these thresholds. Additionally, to have more clinical utility, an individual’s PRS-based risk would be calculated as an age-based absolute risk. While data for these risk assessments are available for some phenotypes (for example, cardiovascular and cancer), age of onset data are lacking for many clinically important phenotypes. Finally, the standards, guidance and the development of best practices for the integration of PRSs into clinical processes are yet to be developed. Future studies can learn from eMERGE and other groups' experiences will be a foundation for ongoing opportunities for the integration of polygenic risk predictions in clinical care settings.

In conclusion, the eMERGE Network’s work in PRS development represents an important step forward in the implementation of PRS-based risk assessment (in combination with other risk estimates from monogenic testing and family history) in clinical practice.

Consent and ethical approval

The study was conducted in accordance with the Declaration of Helsinki, and the central institutional regulatory board protocol was approved by the Ethics Committee of Vanderbilt University. All participants for eMERGE are consented, using a global primary consent and a site-specific consent. Minors acknowledge study participation by signing an assent (if local policy dictates) and the child’s parent/guardian signs a parental permission form. The Vanderbilt University Medical Center Co-ordinating Center is the institutional review board of record (no. 211043) for the Network’s single institutional review board, approved in July 2021.

For the All of Us Research Program, informed consent for all participants is conducted in person or through an eConsent platform that includes primary consent, Health Insurance Portability and Accountability Act authorization for research EHRs and consent for return of genomic results. The protocol was reviewed by the Institutional Review Board (IRB) of the All of Us Research Program. The All of Us Institutional Review Board follows the regulations and guidance of the National Institutes of Health Office for Human Research Protections for all studies, ensuring that the rights and welfare of research participants are overseen and protected uniformly.

Clinical trials registration

The eMERGE genomic risk assessment study is a registered, prospective, interventional clinical trial registered with clinicaltrials.gov (Identifier: NCT05277116 ). The purpose of the study is to determine if providing a GIRA will impact clinical actions taken by providers and patients to manage disease risk and the propensity of participants to develop a disease reported in the GIRA. For this prospective, pragmatic study, the primary outcome being measured is the number of new healthcare actions after return of the genome-informed risk assessment. Number of new healthcare actions will be measured by electronic health record data and participant-reported outcomes through a REDCap survey. Prespecified actions will include a condition-specific composite of new encounters, clinical orders or specialty referrals for clinical evaluation associated with the condition(s), placed by a provider within six months of result disclosure.

Secondary outcomes are the number of newly diagnosed conditions after return of the genome-informed risk assessment and the number of risk-reducing interventions after return of the genome-informed risk assessment (time frame: six months and 12 months post return of results to participant).

Population group definition

In the score auditing and evaluation phase, condition leads cataloged population groups used in the development or validation of given scores from available publications, preprints or information shared directly from collaborators. Across the initial list of evaluated scores, methods for defining population groups included self-reporting, extraction from health system data and/or analysis of genetic ancestry. In the optimization phase, populations were defined using either computational analysis alone or both computational analysis and self-reported ancestry, as indicated in Supplementary Table 3 . For creation of the training model for PRS ancestry calibration, populations were computationally determined as described in ‘PRS ancestry calibration overview’ below.

Populations with that are underserved and more frequently experience health disparities include racial and ethnic minority groups; people with lower socioeconomic status; underserved rural communities; sexual and gender minority groups; and people with disabilities 25 .

Analytical and technical validation studies

Broad imputation pipeline overview.

An imputation pipeline that takes as an input a variant call format (VCF) file generated from a genotyping microarray and imputes the genotypes at additional sites across the genome was developed. The pipeline architecture and composition was based on the widely used University of Michigan Imputation Server, which uses a software called Eagle ( https://github.com/poruloh/Eagle ) for phasing and Minimac4 ( https://genome.sph.umich.edu/wiki/Minimac4 ) for the imputation. The pipeline uses a curated version of the 1,000 Genomes Project (1KG, www.internationalgenome.org ) as the reference panel. Additional details on the imputation pipeline can be found at https://broadinstitute.github.io/warp/docs/Pipelines/Imputation_Pipeline/README .

Broad curated 1KG reference panel

During the validation process, we determined that some sites in the 1KG reference panel were incorrectly genotyped compared to the sites in matching whole genome sequencing data. To increase accuracy of the imputation and PRS scoring, we curated the original panel by removing sites that were likely incorrectly genotyped based on comparing allele frequencies to those reported in gnomAD v.2 ( https://gnomad.broadinstitute.org/ ). Documentation of this curation can be found at https://broadinstitute.github.io/warp/docs/Pipelines/Imputation_Pipeline/references_overview and a publicly available version of the panel at the following Google Cloud location (accessible via the gsutil utility): gs://broad-gotc-test-storage/imputation/1000G_reference_panel/.

Selection of a reference panel for imputation as an input to a PRS is an important consideration. Some reference panels (for example, Trans-Omics for Precision Medicine (TOPMed)) have more samples than the default used in our pipeline (that is, 1KG). This leads to more variants being imputed. The question is whether this would materially change the PRSs calculated from samples imputed with the TOPMed panel. Access to this panel computationally is restricted (and local download prohibited) so it was deemed infeasible to implement in our clinical production environment. The performance of a non-eMERGE PRS (for CHD; ref. 28 ) using the two different reference panels was determined for 20 GDA saliva specimens and for 42 AoU array v.1 specimens. The cohort was imputed both by the Broad imputation pipeline with curated 1KG as the reference panel as well as on the TOPMed imputation server with TOPMed as the reference panel. Imputed arrays were scored by the PRS pipeline.

The PRS percentiles computed with each method are highly concordant for both cohorts. The Pearson correlation coefficient is 0.996 for both cohorts, the P value of the Welch two-sample t- test is equal to 0.93 and 0.85 (indicating no statistical difference between the methods) for GDA and AoU v.1 cohorts, respectively.

Performance verification of the imputation pipeline

Imputation accuracy was determined for 42 specimens that were processed through a genotyping microarray (AoU v.1 array—the precursor to the commercial Global Diversity Array) and imputed with curated 1KG as the reference panel where corresponding deep-coverage (>30X) PCR-free whole genome sequencing data were used as a truth call set to calculate sensitivity and specificity. The arrays were also imputed on the Michigan Imputation Server with 1KG as the reference panel.

Within the cohort, four different ancestries were represented: non-Finnish Europeans, East Asians, South Asian (SAS), African (AFR). Broad imputation pipeline sensitivity for SNPs is >97% and insertions/deletions (INDELs) >95% for all ancestries. Similarly, specificity for SNPs from the Broad imputation pipeline is above 99% and the specificity for INDELs is >98%. See Extended Data Table 1 for a table of results. Results were highly concordant with those returned by the remote server at Michigan.

Performance evaluation of different input material types

To assess the performance of specimens derived from both saliva and whole blood, a set of 20 matched blood and saliva pairs were run through the GDA genotyping process and the resulting VCFs were imputed using the Broad pipeline to be compared against results for matched blood-derived whole genome data. The Pearson correlation between sensitivity and specificity of blood- and saliva-derived samples are equal to 100% and 100%, respectively. For the same pairs, the Welch two-sample t- test statistic is 0.997 and 0.987, respectively. There is no significant difference between the different input sample types.

Imputation repeatability and reproducibility

Imputation pipeline repeatability was assessed by repeating imputation of a cohort of 1,000 Global Screening Array arrays ten times over the course of two weeks and was found to be 100% concordant. Imputation pipeline precision (reproducibility) was also tested on technical replicates. Three individual samples derived from saliva were each genotyped six times, followed by an imputation in a cohort of all saliva-derived samples. In each set of technical replicates, all pairs and variants in each pair were compared (making a total of 45 pairs for which genotypes were compared). Reproducibility is measured using Jaccard scores. ‘Reproducibility over variants’ was calculated only over sites where at least one of the two replicates in a pair calls a non hom-ref genotype and was found to be 99.91% (95% CI 99.89–99.93) for SNPs and 99.87% (95% CI 99.85–99.90) for INDELs. ‘Reproducibility over all sites’ was calculated over all genotyped sites, including sites genotyped as hom-ref in both replicates and was found to be 100% (95% CI 100–100) for both SNPs and INDELs.

Imputation performance as a function of variant frequency

Because we expect accuracy to be impacted by the frequency of a variant in the population (rare variants are less likely to be in the reference panel and therefore less accurately imputed), we further subdivided the performance assessment by allele frequencies on two cohorts: 42 AoU v.1 arrays and 20 blood–saliva pairs of GDA arrays. Accuracy of imputation of variants as a function of population allele frequency performed as expected, with rare variants in the population not being as accurately represented. Imputation is more accurate for variants that are more frequently observed in the population (≥0.1 allele frequency (AF)).

Impact of genotyping array call rate on imputation performance

The impact of call rate on the imputation was assessed by generating a downsampled series of 42 arrays, each with call rates of 90%, 95%, 97% and 98%. Pearson correlation values for SNPs and INDELs were calculated across bins of allele frequencies, assessed against gnomAD common variants (AF > 0.1), for the cohorts with downsampled call rates. Call rates below 95% were found to produce suboptimal results. At this rate the mean R 2 dosage score for sites with AF ≥ 0.1 was found to be 0.98% (95% CI 0.98–0.98) for both SNPs and INDELs compared to 0.99% for call rates of 97% and 98%.

Impact of imputation batch size on performance

Batch size effect of the imputation pipeline was assessed by imputing and analyzing arrays in a cohort of size 1,000 (randomly chosen), ten cohorts of size 100 (nonoverlapping subsets of the 1,000 cohort) and ten cohorts of size ten (nonoverlapping subsets of one of the 100 cohorts). Pearson correlations of dosage scores were calculated across bins for allele frequencies (assessed against gnomAD) for smaller cohorts versus larger cohorts. The data show that imputation is highly correlated across batch sizes with batches down to as few as ten samples, producing acceptable performance. The mean R 2 correlation of dosage scores for sites with allele frequency greater or equal to 0.1 is above 0.97 in all cases both for SNPs and INDELs and increases to 0.98 for the larger studied cohorts. Increasing batch sizes produces very slight improvements in imputation but these are not significant and the choice of imputation batch size (above or equal to ten samples) can be made on practical and operational grounds.

Broad PRS pipeline overview

The PRS pipeline begins by calculating a raw score using plink2 ( https://www.cog-genomics.org/plink/2.0/ ). For each condition, effect alleles and weights are defined for a set of genomic sites stored in a weights file. At each site, the effect allele dosage observed in the imputed VCF is multiplied by the effect weight in the weights file. The raw score is the sum of these products over all the specified sites.

Validation of technical and analytical performance of the PRS pipeline

For each of the ten conditions chosen by the consortium for clinical return, a validation study was performed to assess the technical and analytical performance as well as to verify the association between score and disease risk. See Extended Data Table 2 for a summary of the validation measures.

PRS pipeline accuracy

Accuracy of the pipeline was determined by calculating the Pearson correlation between PRSs calculated from 70 specimens imputed from GDA array data and PRSs of corresponding deep-coverage PCR-free whole genome sequencing data (used as a truth call set).

Input material performance

Accuracy of PRS scoring when different sample types (blood or saliva) are used as inputs was determined by comparing the PRSs from matched blood and saliva pairs collected from 20 individuals.

PRS pipeline repeatability

PRS pipeline repeatability was assessed by running the pipeline on the same dataset of 70 imputed GDA arrays ten times over the course of two weeks (without call caching). Scores generated from the different processing runs were compared to determine if there are any differences observed for a given PRS when the pipeline is run at different times.

PRS pipeline reproducibility

PRS pipeline precision (reproducibility) was assessed using three samples each run six times end-to-end and then compared in a pairwise manner. The z -score standard deviation is used as a measure of variability.

PRS site representation

The SNP weight sites that are not called during genotyping or imputation were determined. These are sites not present in the intersection of an imputed GDA array and the reference panel. Ideally, all sites required for PRS calculation are present either as genotyped or imputed sites; however, in practice, a small number of sites are not present due to differences in the data used to create the score and the specific array and imputation reference panel used in this study.

Performance verification using eMERGE I–III cohort

A cohort of samples with known phenotypic information was used to verify the relationship between PRS as determined by our pipeline and disease risk. For conditions where cases and controls could be identified in the eMERGE I–III cohort, we determined performance using metrics outlined in the ClinGen working group recommendations 26 . Specifically, we determined the PRS distributions for cases and controls, we examined the impact of ancestry adjustment on the distributions and we examined the relationship between observed and predicted risk. An example of this analysis (for T2D is shown below).

The T2D weight file used for PRSs in this validation report comes from a GWAS by Ge et al. 29 where they reported that individuals in the top 2% of the PRSs in the population have an increased risk of developing T2D.

The T2D cohort in the eMERGE I–III dataset consisted of 19,145 cases and 68,823 control samples. The mean adjusted PRS for case samples was 0.435, while the mean for control samples was −0.042. Individuals with higher adjusted PRS scores tend to be more likely to develop disease (see Extended Data Fig. 1 for a histogram of T2D PRSs in cases and controls).

There are some limitations to this analysis: (1) the eMERGE I–III dataset being used for this analysis was generated from different array platforms and was imputed with a different pipeline including a different version of the 1KG reference panel than the one currently implemented; (2) the eMERGE I–III imputed dataset does not include variants from chromosomes X or Y. For these reasons, the PRS disease association analysis represents a verification of the clinical validation performed by eMERGE condition leads rather than the quantitative measure of the impact of the score on risk. The clinical associations (odds ratios) that are reported on the clinical report for each condition were independently determined by eMERGE disease-specific expert teams.

Validation of pipeline and ancestry adjustment in original case–control cohorts

The final pipeline was made available to computational scientists at each of the eMERGE disease-specific expert teams who had access to appropriate case–control cohorts. These groups confirmed the performance of the final pipeline on their cohorts. The odds ratios for each condition that are reported on the clinical reports come from these cohorts rather than the eMERGE cohort for the reasons described above.

PRS ancestry calibration overview

Pca method description.

For a PRS, which is a sum of SNP effects (linear weights), the central limit theorem states that the distribution of scores in a homogenous population will tend towards a normal distribution as the number of SNPs becomes large. When two different homogenous populations are randomly mixed, the additive property of the PRS leads the resulting distribution to be similarly normally distributed, with mean and variance depending on the mean and variance of the original homogenous populations 37 , 38 . We can therefore model the distribution of the PRS as being normally distributed, with mean and variance being functions of genetic ancestry. Practically, we implement this as

with genetic ancestry being represented by projection into principal component (PC) space 39 . The α and β parameters are found by jointly fitting them to a cohort of training data. This fit is performed by minimizing the negative log likelihood:

where i runs over the individuals in the training cohort, prs i is the i th individual’s raw PRS, and μ i and σ i are calculated using equations ( 2 ) and ( 3 ) above by projecting the i th individual into PC space. Note that, due to the simplicity of the model, overfitting is unlikely to be a problem, and so no regularization or other overfitting avoidance technique is implemented. An individual’s PRS z -score can then be calculated as

where μ and σ have again been calculated based on the specific individual’s projection into PC space. In this way, once the model has been trained, the z -score calculation is fully defined by the fitted model parameters, and z -scores can be calculated without needing additional access to the original training cohort.

Generating trained models from All of Us data

Generating the trained models consisted of three steps: (1) selecting the training cohort; (2) imputation of the training cohort; and (3) training the models on the training cohort. A test cohort was also generated to test the performance of the training.

Ancestry-balanced training and test cohorts were generated by subsampling from an initial cohort of around 100,000 All of Us samples. For the purposes of balancing the cohort, each sample was assigned to one of the five 1KG super populations. Principal component analysis was first performed on a random subset of 20,000 samples. The 1KG samples were projected onto these principal components, and a support vector machine was trained on 1KG to predict ancestry. The support vector machine was then used to assign 108,000 AoU samples to one of the five 1KG super populations. A balanced training cohort was selected based on these predicted ancestries, and principal components were recalculated using this balanced training cohort. A similarly balanced test cohort was selected based on ancestries estimated from projection on the training set PCs. The resulting breakdown of the cohorts by estimated ancestry is shown in Extended Data Table 3 .

Both the training and testing cohorts include a number of individuals with highly admixed ancestry. Admixture was quantified using the tool Admixture 40 with five ancestral populations. The resulting admixtures of each cohort are shown in Extended Data Fig. 2 , and the most common admixed ancestries in each cohort are summarized in Extended Data Table 4 .

Each cohort was imputed using the imputation pipeline described above, with 1KG as the reference panel. By keeping the imputation pipeline identical to the pipeline used for the eMERGE dataset, and because the AoU dataset uses the same GDA array as the eMERGE dataset, any potential biases resulting from differing data production and processing methods were removed. The training cohort was scored for each of the ten conditions, and model parameters were fit by minimizing the negative log likelihood as described. The test cohort was then used to evaluate the generalizability of these model parameters.

Performance on test cohort

Extended Data Fig. 3 illustrates the distribution of calibrated z -scores in the test cohort using the parameters fit in the training cohort. As can be seen, all ancestries show the intended standard normal distribution of calibrated scores.

One of the main improvements of this method over previous methods is the inclusion of an ancestry-dependent variance in addition to the ancestry-dependent mean. The importance of this is shown for the hypercholesterolemia PRS in Extended Data Fig. 4 . The variance of this score differs significantly across ancestries, so that a method that only fits the mean of the distribution as ancestry dependent can result in z -score distributions that have been attenuated towards zero or expanded away from zero for some ancestries. By also treating variance as ancestry dependent, this method results in z -score distributions that are more standardized across ancestries.

In addition to improving calibration across ancestries, this method can improve calibration within ancestries, particularly for highly admixed individuals. An example of this can be seen in Extended Data Fig. 5 . Because no ancestry group is homogenous, when individuals are compared directly to other individuals in their assigned population group, a dependence between admixture fraction and PRS can result. This dependence is removed by the described PCA calibration method, and the resulting calibrated PRSs are independent of admixture fraction.

Reporting summary

Further information on the research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

Underlying data used to verify the performance of the PRS pipeline are available in dbGaP https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?study_id=phs001584.v1.p1 . De-identified data relating to trial participants will be available through dbGaP ( https://www.ncbi.nlm.nih.gov/gap/ ) access and the AnVIL platform ( https://anvil.terra.bio/ ) as an interim analysis in 2024 and final dataset at the end of the study, expected in 2026. Information (sites and weights) on the implemented scores can be found at https://github.com/broadinstitute/eMERGE-implemented-PRS-models-Lennon-et-al and also on the UCSC browser https://genome.ucsc.edu/s/Max/emerge . Additionally, PGS Catalog IDs for most of the implemented scores are indicated in Supplementary Table 3 .

Code availability

Codes used in this work to create and operate the imputation and PRS pipelines are hosted at https://github.com/broadinstitute/palantir-workflows/tree/main/ImputationPipeline . Code for the PRS ancestry calibration can also be found in the AoU demonstration workspace https://workbench.researchallofus.org/workspaces/aou-rw-bef5bf62/demopolygenicriskscoregeneticancestrycalibration/data (open access but researcher registration required).

Lambert, S. A. et al. The Polygenic Score Catalog as an open database for reproducibility and systematic evaluation. Nat. Genet. 53 , 420–425 (2021).

Article   CAS   PubMed   Google Scholar  

Lewis, C. M. & Vassos, E. Polygenic risk scores: from research tools to clinical instruments. Genome Med. 12 , 44 (2020).

Article   PubMed   PubMed Central   Google Scholar  

Polygenic Risk Score Task Force of the International Common Disease Alliance. Responsible use of polygenic risk scores in the clinic: potential benefits, risks and gaps. Nat. Med. 27 , 1876–1884 (2021).

Article   CAS   Google Scholar  

Torkamani, A., Wineinger, N. E. & Topol, E. J. The personal and clinical utility of polygenic risk scores. Nat. Rev. Genet. 19 , 581–590 (2018).

Duncan, L. et al. Analysis of polygenic risk score usage and performance in diverse human populations. Nat. Commun. 10 , 3328 (2019).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Hurson, A. N. et al. Prospective evaluation of a breast-cancer risk model integrating classical risk factors and polygenic risk in 15 cohorts from six countries. Int. J. Epidemiol. 50 , 1897–1911 (2022).

Article   PubMed   Google Scholar  

Inouye, M. et al. Genomic risk prediction of coronary artery disease in 480,000 adults: implications for primary prevention. J. Am. Coll. Cardiol. 72 , 1883–1893 (2018).

Guo, F. et al. Polygenic risk score for defining personalized surveillance intervals after adenoma detection and removal at colonoscopy. Clin. Gastroenterol. Hepatol. 21 , 210–219.e11 (2023).

Fantus, R. J. & Helfand, B. T. Germline genetics of prostate cancer: time to incorporate genetics into early detection tools. Clin. Chem. 65 , 74–79 (2019).

Pharoah, P. D. P., Antoniou, A. C., Easton, D. F. & Ponder, B. A. J. Polygenes, risk prediction, and targeted prevention of breast cancer. N. Engl. J. Med. 358 , 2796–2803 (2008).

Willoughby, A., Andreassen, P. R. & Toland, A. E. Genetic testing to guide risk-stratified screens for breast cancer. J. Pers. Med 9 , 15 (2019).

Martin, A. R. et al. Clinical use of current polygenic risk scores may exacerbate health disparities. Nat. Genet. 51 , 584–591 (2019).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mars, N. et al. Systematic comparison of family history and polygenic risk across 24 common diseases. Am. J. Hum. Genet. 109 , 2152–2162 (2022).

Wang, Z. et al. The value of rare genetic variation in the prediction of common obesity in European ancestry populations. Front. Endocrinol. 13 , 863893 (2022).

Ruan, Y. et al. Improving polygenic prediction in ancestrally diverse populations. Nat. Genet. 54 , 573–580 (2022).

Márquez-Luna, C., Loh, P.-R., South Asian Type 2 Diabetes (SAT2D) Consortium & Price, A. L. Multiethnic polygenic risk scores improve risk prediction in diverse populations. Genet. Epidemiol. 41 , 811–823 (2017).

Hujoel, M. L. A., Loh, P.-R., Neale, B. M. & Price, A. L. Incorporating family history of disease improves polygenic risk scores in diverse populations. Cell Genom. 2 , 100152 (2022).

Elliott, J. et al. Predictive accuracy of a polygenic risk score-enhanced prediction model vs a clinical risk score for coronary artery disease. JAMA 323 , 636–645 (2020).

Folkersen, L. et al. Impute.me: an open-source, non-profit tool for using data from direct-to-consumer genetic testing to calculate and interpret polygenic risk scores. Front. Genet. 11 , 515901 (2020).

Article   Google Scholar  

Hao, L. et al. Development of a clinical polygenic risk score assay and reporting workflow. Nat. Med. 28 , 1006–1013 (2022).

Vassy, J. L. et al. Cardiovascular disease risk assessment using traditional risk factors and polygenic risk scores in the million veteran program. JAMA Cardiol. 8 , 564–574 (2023).

Linder, J. E. et al. Returning integrated genomic risk and clinical recommendations: the eMERGE study. Genet. Med. 25 , 100006 (2023).

McCarty, C. A. et al. The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Med. Genomics 4 , 13 (2011).

Gottesman, O. et al. The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future. Genet. Med. 15 , 761–771 (2013).

NIMHD. Overview. https://www.nimhd.nih.gov/about/overview/ . Accessed 11 Dec 2023.

Wand, H. et al. Improving reporting standards for polygenic scores in risk prediction studies. Nature 591 , 211–219 (2021).

Khan, A. et al. Genome-wide polygenic score to predict chronic kidney disease across ancestries. Nat. Med. 28 , 1412–1420 (2022).

Khera, A. V. et al. Genome-wide polygenic scores for common diseases identify individuals with risk equivalent to monogenic mutations. Nat. Genet. 50 , 1219–1224 (2018).

Ge, T. et al. Development and validation of a trans-ancestry polygenic risk score for type 2 diabetes in diverse populations. Genome Med. 14 , 70 (2022).

Martin, A. R. et al. Human demographic history impacts genetic risk prediction across diverse populations. Am. J. Hum. Genet. 100 , 635–649 (2017).

Loh, P.-R. et al. Reference-based phasing using the Haplotype Reference Consortium panel. Nat. Genet. 48 , 1443–1448 (2016).

Howie, B., Fuchsberger, C., Stephens, M., Marchini, J. & Abecasis, G. R. Fast and accurate genotype imputation in genome-wide association studies through pre-phasing. Nat. Genet. 44 , 955–959 (2012).

Lee, A. et al. BOADICEA: a comprehensive breast cancer risk prediction model incorporating genetic and nongenetic risk factors. Genet. Med. 21 , 1708–1718 (2019).

Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 , 65–70 (1979).

Ding, Y. et al. Large uncertainty in individual polygenic risk score estimation impacts PRS-based risk stratification. Nat. Genet. 54 , 30–39 (2022).

Schoeler, T. et al. Participation bias in the UK Biobank distorts genetic associations and downstream analyses. Nat. Hum. Behav. 7 , 1216–1227 (2023).

Privé, F. et al. Portability of 245 polygenic scores when derived from the UK Biobank and applied to 9 ancestry groups from the same cohort. Am. J. Hum. Genet. 109 , 12–23 (2022).

Ding, Y. et al. Polygenic scoring accuracy varies across the genetic ancestry continuum. Nature 618 , 774–781 (2023).

Scutari, M., Mackay, I. & Balding, D. Using genetic distance to infer the accuracy of genomic prediction. PLoS Genet. 12 , e1006288 (2016).

Alexander, D. H., Novembre, J. & Lange, K. Fast model-based estimation of ancestry in unrelated individuals. Genome Res. 19 , 1655–1664 (2009).

Download references

Acknowledgements

We thank the past and future participants of the eMERGE Network projects. We thank M. O’Reilly for help with figure creation. We thank our All of Us Research Program colleagues, A. Ramirez, S. Lim, B. Mapes, A. Green and A. Musick, for providing their support and input throughout the ancestry calibration demonstration project lifecycle. We also thank the All of Us Science Committee and All of Us Steering Committee for their efforts evaluating and finalizing the approved demonstration projects. The All of Us Research Program would not be possible without the partnership of contributions made by its participants. To learn more about the All of Us Research Program’s research data repository, please visit https://www.researchallofus.org/ . This phase of the eMERGE Network was initiated and funded by the National Human Genome Research Institute through the following grants: U01HG011172 (Cincinnati Children’s Hospital Medical Center); U01HG011175 (Children’s Hospital of Philadelphia); U01HG008680 (Columbia University); U01HG011176 (Icahn School of Medicine at Mount Sinai); U01HG008685 (Mass General Brigham); U01HG006379 (Mayo Clinic); U01HG011169 (Northwestern University); U01HG011167 (University of Alabama at Birmingham); U01HG008657 (University of Washington); U01HG011181 (Vanderbilt University Medical Center); U01HG011166 (Vanderbilt University Medical Center serving as the Coordinating Center). The All of Us Research Program is supported by the National Institutes of Health, Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA#: AOD 16037; Federally Qualified Health Centers: 75N98019F01202; Data and Research Center: 1 OT2 OD35404; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 OT2 OD030043; Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276.

Author information

These authors contributed equally: Niall J. Lennon, Leah C. Kottyan.

Full lists of members and their affiliations appear in the Supplementary Information.

Authors and Affiliations

Broad Institute of MIT and Harvard, Cambridge, MA, USA

Niall J. Lennon, Christopher Kachulis, Marissa Fisher, Joel Hirschhorn, Candace Patterson, Maegan Harden, Joel N. Hirschhorn, Amit Khera, Katie Larkin, Edyta Malolepsza, Li McCarthy, Nihal Pai, Heidi L. Rehm & Diana M. Toledo

Cincinnati Children’s Hospital Medical Center, University of Cincinnati, Cincinnati, OH, USA

Leah C. Kottyan, Lisa J. Martin, Tesfaye B. Mersha, Bahram Namjou & Cynthia A. Prows

Icahn School of Medicine at Mount Sinai, New York, NY, USA

Noura S. Abul-Husn, Gillian Belbin, Clive Hoggart, Roelof A. J. Smit, Zhe Wang & Eimear E. Kenny

National Human Genome Research Institute, National Institutes of Health, Bethesda, MD, USA

Josh Arias, Sonja I. Berndt, Sonja Berndt, Teri A. Manolio & Robb Rowley

Vanderbilt University Medical Center, Nashville, TN, USA

Jennifer E. Below, Ellen Wright Clayton, Digna R. Velez Edwards, QiPing Feng, Jodell E. Linder, Jonathan D. Mosley, Josh F. Peterson, Dan M. Roden, Nataraja Sarma Vaitinadin, Wei-Qi Wei & Georgia L. Wiesner

Columbia University, New York, NY, USA

Wendy K. Chung, Atlas Khan, Krzysztof Kiryluk & Chunhua Weng

University of Alabama at Birmingham, Birmingham, AL, USA

James J. Cimino, Marguerite R. Irvin, Nita Limdi & Hemant Tiwari

Children’s Hospital of Philadelphia, Philadelphia, PA, USA

John J. Connolly, Joseph T. Glessner, Hakon Hakonarson & Margaret Harr

Tulane University, New Orleans, LA, USA

David R. Crosslin

University of Washington, Seattle, WA, USA

David R. Crosslin, Gail P. Jarvik, Elisabeth A. Rosenthal & David Veenstra

Mayo Clinic, Rochester, MI, USA

Ozan Dikilitas, Robert R. Freimuth, Iftikhar Kullo, Daniel J. Schaid & Johanna L. Smith

Mass General Brigham, Boston, MA, USA

Tian Ge, Elizabeth W. Karlson, James B. Meigs & Jordan W. Smoller

Northwestern University, Evanston, IL, USA

Adam S. Gordon, Yuan Luo, Elizabeth M. McNally, Lorenzo L. Pesce, Megan J. Puckelwartz, Konrad Teodor Sawicki & Theresa L. Walunas

Boston Children’s Hospital, Boston, MA, USA

Joel Hirschhorn & Joel N. Hirschhorn

Fred Hutchinson Cancer Center, Seattle, WA, USA

Li Hsu, Ulrike Peters & Minta Thomas

Novo Nordisk Foundation Center for Basic Metabolic Research, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark

Ruth Loos & Ruth J. F. Loos

The Charles Bronfman Institute for Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA

National Institutes of Health, Bethesda, MD, USA

  • Anjene Musick

Nanjing Medical University, Nanjing, China

Xianyong Yin

You can also search for this author in PubMed   Google Scholar

The GIANT Consortium

  • Sonja Berndt
  • , Joel Hirschhorn
  • , Ruth Loos
  • , Roelof A. J. Smit
  •  & Xianyong Yin

The All of Us Research Program

Contributions.

N.J.L. and L.C.K. contributed equally. N.J.L., L.C.K., D.R.C., O.D., T.G., J.T.G., H.H., L.H., E.W.K., R.L., E.M.M., J.B.M., B.N., R.A.J.S. and E.E.K. were responsible for PRS development. N.J.L., L.C.K., D.R.C., O.D., T.G., J.T.G., A.S.G., H.H., L.H., E.W.K., J.E.L., R.L., Y.L., E.M., L.M., J.B.M., B.N., L.L.P., J.F.P., M.J.P., R.R., K.T.S., R.A.J.S., J.L.S., C.W., W.-Q.W. and E.E.K. conducted PRS evaluation. PRS selection, optimization and validation was done by N.J.L., L.C.K., D.R.C., Q.F., O.D., T.G., J.T.G., A.S.G., H.H., L.H., E.W.K., J.E.L., R.L., Y.L., E.M., L.M., B.N., L.L.P., M.J.P., R.R., K.T.S., R.A.J.S., J.L.S., G.L.W., C.W., W.-Q.W. and E.E.K. Population-based z- score calibration was done by N.J.L., C.K., T.G., B.N. and E.E.K. N.J.L., L.C.K., C.K., J. J. Connolly., D.R.C., T.G., E.M., B.N., M.J.P., R.A.J.S. and E.E.K. were responsible for PRS transfer and implementation. Assessment of the first 2,500 participants was done by N.J.L., C.K., Q.F., B.N. and E.E.K. N.J.L., L.C.K., C.K., J.E.L., T.A.M., J.W.S., J.F.P. and E.E.K. wrote the first draft of the paper. All authors reviewed the paper.

Corresponding author

Correspondence to Niall J. Lennon .

Ethics declarations

Competing interests.

N.S.A.-H. is an employee and equity holder of 23andMe; serves as a scientific advisory board member for Allelica, Inc; received personal fees from Genentech Inc, Allelica Inc, and 23andMe; received research funding from Akcea Therapeutics; and was previously employed by Regeneron Pharmaceuticals. E.E.K. received personal fees from Illumina Inc, 23andMe and Regeneron Pharmaceuticals and serves as a scientific advisory board member for Encompass Bioscience, Foresite Labs and Galateo Bio. J.N.H. has equity in Camp4 Therapeutics and has been a consultant to Amgen, AstraZeneca, Cytokinetics, PepGen, Pfizer and Tenaya Therapeutics and is the founder of Ikaika Therapeutics. J.F.P. is a paid consultant for Natera Inc. A. Khera. is an employee of Verve Therapeutics. N.L. received personal fees from Illumina Inc and is a scientific advisory board member for FYR Diagnostics. J.F.P. is a consultant for Myome. D.V. is a consultant for Illumina and has grant support from GeneDx. T.L.W. has grant funding from Gilead Sciences, Inc. The other authors declare no competing interests.

Peer review

Peer review information.

Nature Medicine thanks Cathryn Lewis, Lili Milani, Bjarni Vilhjálmsson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Anna Maria Ranzoni, in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1 case-control prs histograms..

Histograms of T2D PRS scores for case and control samples in the eMERGE I-III dataset.

Extended Data Fig. 2 Representation of the genetic ancestry admixture composition of both the Test and Training cohorts.

The x-axis represents individuals within the cohorts and the color-coding highlights the proportion of genetic admixture observed.

Extended Data Fig. 3 Calibrated z-scores.

The distributions of calibrated z-scores in the test cohort when the training cohort parameters are applied.

Extended Data Fig. 4 Hypercholesterolemia PRS calibrated z-scores of training cohort.

Note the improvement when an ancestry dependent variance is used over a constant variance method.

Extended Data Fig. 5 PRS z-score as a function of African Admixture Fraction, for individuals of African ancestry.

In the ‘Bucketing’ method, a z-score is calculated by comparing to the mean and variance of all individuals of African ancestry in the cohort. The ‘PCA Calibrated’ method is the method described above. Note the dependence on admixture fraction in the ‘Bucketing’ method, which has been removed in the ‘PCA Calibrated’ method.

Supplementary information

Supplementary information.

Sample clinical report and list of consortia members.

Reporting Summary

Supplementary table 1.

Supplementary Tables 1–3 (tabs in a single worksheet).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lennon, N.J., Kottyan, L.C., Kachulis, C. et al. Selection, optimization and validation of ten chronic disease polygenic risk scores for clinical implementation in diverse US populations. Nat Med 30 , 480–487 (2024). https://doi.org/10.1038/s41591-024-02796-z

Download citation

Received : 25 May 2023

Accepted : 02 January 2024

Published : 19 February 2024

Issue Date : February 2024

DOI : https://doi.org/10.1038/s41591-024-02796-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

clinical research study leader

IMAGES

  1. 1 Roles of the Clinical Laboratory Leader

    clinical research study leader

  2. What is a Clinical Research Director

    clinical research study leader

  3. Clinical Research Associate Career Overview and Outlook

    clinical research study leader

  4. Insight into the Different Clinical Trial Study Designs

    clinical research study leader

  5. The Role and Duties of a Clinical Research Coordinator

    clinical research study leader

  6. What does a Clinical Research Associate do? // Talentmark

    clinical research study leader

COMMENTS

  1. Clinical Research Study Leader jobs

    11,575 Clinical Research Study Leader jobs available on Indeed.com. Apply to Clinical Research Associate, Clinical Research Coordinator, Clinic Manager and more!

  2. 4,193 Clinical research study leader Jobs in US

    Search Clinical research study leader jobs. Get the right Clinical research study leader job with company ratings & salaries. 4,193 open jobs for Clinical research study leader.

  3. The Clinical Trials Team

    Screening trials Case control studies Cohort studies Cross sectional studies Figure no. 1: clinical trial team flowchart Clinical research trial team: The success of a quality clinical research program is essential for developing and maintaining an impeccable clinical research trial team.

  4. Clinical Leader

    Clinical Leader - Helping clinical research professionals connect and collaborate Partnering Clinical Technologies Regulatory Market Research Ramblings From A First-Time SCOPE Attendee

  5. Study Leader

    Responsible with per needed-basis oversight from the Study Director community Lead (SD-CL) for the execution and delivery of the GCO supported clinical studies of standard to medium complexity and priority per the Operational Execution Plan (OEP) and clinical study protocol.The Senior Study Leader is the leader of the cross-functional clinical trial team (CTT), guides planning and management ...

  6. The 3 Secrets To Becoming an Effective Leader In Clinical Research

    The 3 Secrets To Becoming an Effective Leader In Clinical Research By Laurie Halloran and Michelle Pratt, Halloran Consulting Group, Inc. This article is the third in a series providing insight and guidance to help professional clinical researchers effectively navigate their careers. Check out Part 1 and Part 2.

  7. Clinical Research Team

    Clinical Research: Team Members and Their Roles. Conducting clinical research requires a team of people with a variety of skills and experience. If you decide to participate in clinical research, you may interact with scientists, medical experts, and others who work together to coordinate all aspects of a study.

  8. Clinical Research Team Leader Jobs, Employment

    Clinical Research Team Leader jobs. Sort by: relevance - date. 2,031 jobs. Research Associate - CLINICAL FORMULATION - CELL THERAPY. Novo Nordisk. Fremont, CA. $65,850 - $115,240 a year. ... Provides assistance on clinical research studies as per study guidelines and protocols.

  9. Salary: Clinical Research Study Leader in United States 2023

    The estimated total pay for a Clinical Research Study Leader is $69,229 per year in the United States area, with an average salary of $65,782 per year. These numbers represent the median, which is the midpoint of the ranges from our proprietary Total Pay Estimate model and based on salaries collected from our users.

  10. 8 Elements of a Successful Clinical Research Career

    For instance, Dr. Barkoudah says that as a clinical researcher, you'll need to know how to develop relevant questions, formulate hypotheses you can test, and design clinical studies to gather important insights. Recognize how to effectively track and interpret your data. Sharpening your ability to perform statistical analytics on your ...

  11. Senior Study Leader

    Location - Hybrid working roleAssociate Director level role The Senior Study Leader is responsible for the execution and delivery of the Global Clinical Operations (GCO), clinical studies in accordance with the Operational Execution Plan (OEP) and clinical study protocol. You will lead the cross-functional Clinical Trial Team (CTT) and guide the planning and management of assigned clinical ...

  12. Clinical Research Manager Jobs, Employment

    1,292 Clinical Research Manager jobs available on Indeed.com. Apply to Clinic Manager, Clinical Research Associate, Site Manager and more! ... Lead investigator meetings prior to initiation of clinical trials to ensure compliance with protocol and overall study logistics. Assist key opinion leaders in the conception, development and conduct of ...

  13. Clinical research leadership—"A blueprint"

    1. Introduction Within academia, programmes of research traditionally take place in higher education institutions such as universities, developing strong leaders in the field.

  14. Leadership and Line Management in Clinical Research

    Abstract: Line management is generic, utilizing the same approaches throughout all industries; line management brings together company needs and its workforce to deliver company objectives. This article provides an overview of line management styles and line-management-through-leadership approaches in the clinical research environment.

  15. The Career Path of a Clinical Research Coordinator

    A clinical research coordinator is an integral part of the research team for medical studies. They conduct and manage clinical trials, providing outcomes that shape medical advances in preventative care, curing diseases, and immunizations, among other areas. With employment options available in hospitals, pharmaceutical companies, and private ...

  16. Clinical research leadership-"A blueprint"

    Clinical research leadership-"A blueprint" 2019 Aug:87:20-23. doi: 10.1016/j.jdent.2019.05.004. Epub 2019 May 7. N X West , N C Claydon , Affiliations 31075371 The principles of leadership in academic research, reflect those of life in general and differ only by circumstance.

  17. Clinical Study Leadership < Yale Cardiovascular Research Group

    Clinical data from YCRG-designed trials have provided the basis for market approval of numerous products in the US and OUS, and advanced the methodology of cardiovascular clinical trials. Biostatistics. YCRG provides expert statistical leadership for academic and industry-sponsored clinical trials and related projects.

  18. Leaders, leadership and future primary care clinical research

    Leaders, leadership and future primary care clinical research John Furler, 1 Jennifer Cleland, 2 Chris Del Mar, 3 Barbara Hanratty, 4 Umesh Kadam, 5 Daniel Lasserson, 6 Colin McCowan, 7 Parker Magin, 8 Caroline Mitchell, 9 Nadeem Qureshi, 10 Greta Rait, 11 Nick Steel, 12 Mieke van Driel, 13 and Alison Ward 14

  19. Leadership Effectiveness in Healthcare Settings: A Systematic Review

    Abstract To work efficiently in healthcare organizations and optimize resources, team members should agree with their leader's decisions critically. However, nowadays, little evidence is available in the literature.

  20. CBIIT's Hiring—Apply Today to be a Clinical Research Informatics Leader

    To qualify you should be able to: demonstrate expertise in using biomedical informatics tools and approaches for clinical research; show experience in managing projects using informatics for precision medicine;; communicate and collaborate with others to develop informatics approaches;

  21. Scientific Leadership

    The George Clinical Scientific Leadership model offers: A scientifically sound, pragmatic approach that informs strong operational delivery. Protocol alignment with clinical practice across geographies. Motivated and engaged investigators. Robust communication conduits between the investigators, study teams, steering committees and the sponsor.

  22. Attributes, skills and actions of clinical leadership in nursing as

    As clinical leaders in various hospitals and healthcare settings, more empirical work is needed to build on clinical nursing in general and the attributes, skills and actions of clinical nursing leadership of nursing leaders and nurses. Data availability statement

  23. Clinical Research Nurse Roles

    The Nurse Scientist is a nurse with advanced preparation (PhD in nursing or related field) in research principles and methodology, who also has expert content knowledge in a specific clinical area. The primary focus of the role is to (1) provide leadership in the development, coordination and management of clinical research studies; (2) provide ...

  24. Fred Hutch to lead new federal Cancer Screening Research Network

    Fred Hutchinson Cancer Center is at the helm of a new clinical trials consortium, the Cancer Screening Research Network, or CSRN, created by the National Cancer Institute, part of the National Institutes of Health, to advance President Joe Biden's Cancer Moonshot by improving early detection of cancers.. Biden announced the formation of the Cancer Moonshot, intended to jumpstart progress in ...

  25. 'It depends': what 86 systematic reviews tell us about what strategies

    The gap between research findings and clinical practice is well documented and a range of strategies have been developed to support the implementation of research into clinical practice. The objective of this study was to update and extend two previous reviews of systematic reviews of strategies designed to implement research evidence into clinical practice.

  26. Less Invasive Early Lung Cancer Study Receives Top 10 Clinical Research

    A Weill Cornell Medicine-led research team has been awarded a 2024 Top 10 Clinical Research Achievement Award from the Clinical Research Forum in recognition of an influential 2023 New England Journal of Medicine study on early-stage lung cancer resection.. The award is one of 10 given annually by the Clinical Research Forum for highly innovative and clinically translatable research with the ...

  27. Stanford Medicine study identifies distinct brain organization patterns

    A new study by Stanford Medicine investigators unveils a new artificial intelligence model that was more than 90% successful at determining whether scans of brain activity came from a woman or a man.. The findings, published Feb. 20 in the Proceedings of the National Academy of Sciences, help resolve a long-term controversy about whether reliable sex differences exist in the human brain and ...

  28. Selection, optimization and validation of ten chronic disease ...

    A new study from the eMERGE consortium provides insights on the development of a pipeline for the generation and reporting of polygenic risk scores for ten diseases to diverse populations in a ...