Competency Assessment vs Proficiency Testing. Do IVF Labs Need Both? Tl;DR – YES! We explain why.
“One thing worse than training employees and losing them… is not training them and keeping them.” -Ed Metcalf
Competency Assessment vs Proficiency Testing. Does my IVF lab need to perform both? The short answer is yes!
Learning Objectives
— What is the difference between proficiency testing and competency assessment?
— When do you need to perform PT?
— When do you need to perform CA?
— What are the 6 elements needed to perform a CAP level CA?
— When do you need to perform PT?
— Define the criteria for satisfactory proficiency testing (PT) performance.
— Describe the consequences of unsatisfactory performance of a PT event.
— Take the appropriate steps to investigate a PT failure.
— Discuss the advantages of standardization and digitization for documentation of training, performance, and corrective actions.
Until each staff member has documented training, followed by successful assessment of competency of that task, they should not be permitted to perform or result patient testing independently of their trainer. Embryologist training, competency acquisition and documentation of these activities for inspections hasn’t advanced in 40 years, until now. ART Compass is outsmarting the embryologist shortage, reducing non-conformance events, and helping you pass quality inspections, defeating CAPs number one most cited deficiency, staff-related checklist items- download our complete full length, peer reviewed study here, “Assuring quality in assisted reproduction laboratories: assessing the performance of ART Compass – a digital ART staff management platform.”
Whether you’re subject to CLIA, CAP or your local state DOH requirements, you will be required to provide proof (documentation) of both training AND competency for each employee, for each task that they perform. Documentation of all of these tasks can be organized through the use of advanced digital IVF lab management tools, like ART Compass. ART Compass helps savvy IVF lab directors keep track of what each staff members’ specific job junctions include, which they need to be trained in, and which tasks have been completed by each trainer.
Proficiency Testing
Before passage of the Clinical Laboratory Improvement Amendments of 1988 (CLIA), participation in proficiency testing (PT) was voluntary for many labs and the standards for these tests were not defined. With passage of CLIA ‘88, proficiency testing evolved from an educational self-assessment and improvement tool to the primary measure of continued reimbursement and licensure.
Proficiency testing determines the performance of individuals for specific tests or measurements. Performance on PT has been found to be an indicator of the quality of patient testing. By CLIA standards, clinical laboratories are required to enroll in a CMS-approved PT program for all regulated nonwaived testing, and their performance is compared to a target value using statutory criteria for acceptable performance. For nonregulated, nonwaived testing or for tests that have no formal PT available, CLIA requires alternative assessment, twice per year.
Failure of a single event (<80%, or <100% for immunohematology) requires an investigation to ensure that processes were followed correctly throughout all phases of testing: preanalytic, analytic, and postanalytic. This type of unsuccessful performance does not require that a response be returned to the PT program or to the facility’s accreditation agency. However, if two consecutive events or two out of three consecutive events for the same procedure receive an unsuccessful grading, consequences are more severe. The laboratory must take steps to discontinue the procedure. It is best to notify the accrediting agency of the voluntary discontinuance before the accrediting agency notifies the laboratory.
Next, the problem should be investigated, corrective action implemented, and two consecutive successful PT events must be performed. The events can be independently obtained through the ART Compass program. Finally, the accrediting agency must be notified of the corrective action and the intent to reinstate testing.
Embryology procedures fall under nonregulated, nonwaived testing. This requirement can be met by subscribing to a formal program, such as ART Compass, which offers testing for all embryology procedures. The target values in ARTC are determined two ways: by the mean of all peer participant responses after removal of outliers, and by IVF Lab Director consensus. Acceptable performance is considered to be 80%.
You may access a Proficiency Testing Policy and Procedure in ART Compass, by logging in online via a Chrome Browser and navigating to the green tab “Documents and Forms,” then click on the “Quality” folder. Use the “Show to” feature to select any individual who needs to read and review the policy.
Competency Assessment
Embryologist competency assessment is an ongoing, continuous process of monitoring individuals’ abilities to perform their specific job functions.
Once training has been completed and documented in ART Compass, you will assess each staff member’s clinical decision-making ability to successfully perform these tasks using our robust suite of subjective “surveys.” This is their competency, where you assess if the training was successful and staff are able to perform each assigned task correctly.
You may access a Competency Assessment Policy and Procedure in ART Compass, by logging in online via a Chrome Browser and navigating to the green tab “Documents and Forms,” then click on the “Quality” folder. Use the “Show to” feature to select any individual who needs to read and review the policy.
CAP Requirements for Competency Assessment:
ART Compass contains a robust suite of tools for staff management, including a Staff-Quality Audit and subjective competency assessments for each embryology procedure, as well as objective surveys for topics of interest such as Media and pH, FDA regulations, HIPAA Compliance, PPE and Bloodborne Pathogens and much more! Two ART Compass digital tools, a conventional checklist and our Competency Assessment 2.0 document capture module (use your phone camera to record evidence) will guide your competency evaluations and document evidence of completion for all required procedural steps.
To fully demonstrate successful competency all 6 of the following elements must be documented for each employee, for each task:
Direct observation of routine patient testing — Watch (and snap a picture!) as employee performs routine work processes and procedures, including; patient preparation, specimen handling, processing, and testing.
Monitoring the recording and reporting of test results — Review worksheets, computer printouts, or manual test logs for results or entries that are incorrect or incomplete. The worksheets, logs, and printouts should be compared to final test reports, evaluated turnaround times, and for compliance with policies and procedures.
Review of intermediate test results, records, and proficiency testing results — Review testing documentation, to include test results or worksheets, documentation forms, QC logs, proficiency testing documentation, and other applicable documentation for completion of proper policies and procedure. Documentation of review will include follow up of corrective action related to problems in the laboratory.
Direct observation of instrument maintenance and function checks — Watch employee perform instrument maintenance and function checks as compared to documentation for completeness and correctness of process and expected outcome.
Assessment of test performance through testing previously analyzed specimens, internal blind testing samples or external proficiency testing samples — Assessment of previously tested samples; internal blind samples or external proficiency testing samples.
Assessment of problem-solving skills — Include review of problem logs, incident reports, and QC failures; Review corrective actions employed to resolve the problem. Personnel interviews (staff narrative) may also be included for documentation.
PT Failure: Now What?
Unsatisfactory PT performance makes up approximately 3.2% of all CMS survey deficiencies. Inappropriate PT enrollment makes up 1.8%, and failure to perform alternative assessment of nonregulated testing makes up 6% of deficiencies (Yost, 2011). The suggested steps for investigating PT failure are:
- Survey Materials: On receipt of the survey materials, verify that they were received in acceptable condition. Should any unacceptable conditions or criteria be noted, notify the PT program as soon as possible for shipment of replacement samples. Ensure that the samples are stored according to the manufacturers’ instructions until analysis.
- Clerical Error: Errors of this type are the most commonly reported and may be of various types. They include typographical errors as well as errors related to interpretation, transcription, transposition, misidentification, answer-form coding, miscalculation, or acceptance of a nonlinear result. Manual entry offers more chance for transcription or transposition errors. Electronic entry still allows for incorrect keystrokes and/or options to be submitted.
- Technical Problems: The most common assumption when assessing technical issues is that the reagent and/or instrumentation reported an inaccurate result. This may be exhibited by the quality control value falling out of the acceptable range for the analyte and the run being accepted. Quality control results that demonstrate excessive repeats, shifts, or trends may also affect PT results and should be carefully scrutinized in the investigation.
- Method Problems: These types of errors include instrument problems, maintenance issues, or faulty reagents and/or standards. Verify that the instrument was within performance guidelines the day of PT. Were there repairs or component replacements recently done that could have affected the PT results?
- Could the error have affected patient testing? If so, document the actions taken to ensure that the patient testing reported at the time of the unsatisfactory PT was valid or whether amendment is needed. It is necessary to notify providers if any inaccurate results were reported.
- The final steps of the investigation are to determine the problem(s) and summarize the findings in your documentation. Include any supporting documents for the findings. These may include proficiency report records, instrument data, quality control records, calibration records, maintenance logs, and any other records found to be of assistance. Document the corrective action taken to remedy the problem and that steps that will be taken to prevent the problem from recurring.
Documenting PT performance in ARTC
A logical approach and use of a standardized investigation forms may be a reassurance that possible critical steps in the investigative process were not overlooked, and variables were examined for the root cause of unsatisfactory proficiency testing performance. You can find free resources in ART Compass to troubleshoot failed or unsatisfactory PT events. 1. In the surveys module of ARTC you will find The IVF Lab Digital Playbook -> Investigation of Proficiency Testing Results. 2. Access ARTC online, navigate to the green tab “Documents and Forms” and assign the “Investigation of Proficiency Testing Results”to a staff member.
Interested in Embryologist Training and KPI management? Book a one on one meeting with ART Compass today! Missed our Digital Staff Management and Novel KPI Webinar? Catch the replay, anytime, on demand!!
“One thing worse than training employees and losing them… is not training them and keeping them.” -Ed Metcalf
IVF labs have been affected by unprecedented workforce shortages, and are at the forefront of a great wave of pending retirements. Less skilled and experienced staff means there is more room for grave errors to occur, and less people than ever to lead the training of new staff. Until each staff member has documented training, followed by successful assessment of competency of that task, they should not be permitted to perform or result patient testing independently of their trainer. Embryologist training, competency acquisition and documentation of these activities for inspections hasn’t advanced in 40 years, until now. ART Compass is outsmarting the workforce shortage, reducing non-conformance events, and helping you pass quality inspections, defeating CAPs number one most cited deficiency, staff related checklist items.
Whether you’re subject to CLIA, CAP or your local state DOH requirements, you will be required to provide proof (documentation) of both training AND competency for each employee, for each task that they perform. Documentation of all of these tasks can be organized through the use of advanced digital IVF lab management tools, like ART Compass. ART Compass helps savvy IVF lab directors keep track of what each staff members’ specific job functions include, which ones they will need to be trained on, and which tasks have been completed by each trainer. Competency assessment is an ongoing, continuous process of monitoring individuals’ abilities to perform their specific job functions.
When Embryologist Training is Complete- Maintain Competencies and KPIs
Once training has been completed and documented in ART Compass, you will assess each staff member’s ability to successfully perform these tasks using our robust suite of subjective “surveys”. This is competency, where you will assess if the training was successful and if staff can perform each assigned task correctly. ART Compass contains competency assessments for every embryology procedure. To fully demonstrate successful competency all 6 of the CAP Checklist items must be documented for each employee, for each task.
Our webinar will detail the ART Compass platform to standardize competency assessment, ART lab training, and reduce new staff on-boarding time and much more, including discussion of the KPIs or the future – in a fully digital embryology lab!
How can ART Compass Help?
ART Compass is a mobile application for the assisted reproductive technology (ART) industry that puts physicians, administrators, and lab staff on the “same page” as their patients. It is a LIMS , a patient portal, and a laboratory quality management system. It documents IVF staff clinical decision-making, streamlines reporting, and automates data analysis.
Apple App Store:
https://itunes.apple.com/us/app/art-compass/id1442048939?mt=8
Google Play Store:
https://play.google.com/store/apps/details?id=com.artproficiencyapp
We achieve this unprecedented level of information management, starting in the heart of any clinical practice: the IVF (in vitro fertilization) laboratory. The IVF laboratory is responsible for creating the patient’s embryos, reporting their quality, status, and genetic results to both patients and physicians.
ART Compass takes the pain out of managing dozens of parameters for automated regulatory reporting, simplifying the quality control workflow for labs, saving time and money, all while simultaneously improving patient care by providing direct and timely access to hundreds of data points pertaining to each patient’s IVF cycle.
Physicians go on to use this information to make clinical care decisions, such as which embryo to transfer. Administrators use this information to track pregnancy and success rates for their clinic. Patients have a secure and HIPAA compliant portal to view their frozen embryo inventory, and IVF cycle information, and direct access to physicians and laboratory staff.
www.artcompass.io
While you are here… check out the entire ARTC digital Playbook…
| September | FDA Audit |
| October | Staff Audit |
| November | Patient Satisfaction and Security |
| December | Space and Supplies |
| January | PMs, Air Quality, and Deep Clean |
| February | Biennial P&P Updates, Review, and Sign Off |
| March | |
| April | |
| May | |
| June | |
| July | |
| August |
ART Compass — The world’s first mobile IVF laboratory information management system (LIMS)
For immediate release: ART Compass Partners with WHO— The world’s first mobile IVF laboratory information management system (LIMS) powered by artificial intelligence, announced today that it is partnering with the World Health Organization to deliver educational content for Andrology competency assessments using high definition images from the planned 6th Edition Laboratory manual for the examination and processing of human semen.
“We are thrilled to be bringing our user’s high definition, high-quality andrology images from the World Health Organization’s 6th Edition. Semen analysis is of paramount importance to study potential male fertility and a couple’s infertility. It is was very important for us to deliver educational and competency assessment content in Andrology from the world’s foremost authority.” ART Compass Founder, Dr. Carol Lynn Curchoe says.
“2/3RDS OF IVF CLINICS ARE STILL USING A PAPER REPORTING SYSTEM.
THERE IS NO INDUSTRY-STANDARD QUALITY CONTROL. HOW CAN PATIENTS AND CLINICS BE CONFIDENT IN THEIR RESULTS?“
About ART Compass:
The IVF industry has a CAGR of 10.2%, resulting in a technology explosion, but a major lag in Quality Control Infrastructure; despite having the highest levels of documentation and reporting required for local, state, and national compliance, of any medical field. These failures cost the industry millions annually, but most important, destroy the dreams of couples desperate to make a family. ART Compass is the world’s first industry QA/QC platform powered by artificial intelligence that puts physicians, administrators, and lab staff on the “same page” as their patients. ART Compass documents IVF staff’s clinical decision making streamlines reporting, automates data analysis, and makes critical information available to all stakeholders.
About the WHO manual:
ART Compass Partners with WHO The sixth edition of the WHO manual will update the information provided in the fifth edition on sperm preparation for clinical use or specialized assays and on cryopreservation, quality control in the semen analysis laboratory, and evidence-based reference ranges and reference limits for various semen characteristics. The methods described are intended to improve the quality of semen analysis and the comparability of results from different laboratories.
Semen analysis may be useful in both clinical and research settings, for investigating male fertility status as well as monitoring spermatogenesis during and following male fertility regulation and other interventions. This manual provides updated, standardized, evidence-based procedures and recommendations for laboratory managers, scientists and technicians to follow in examining human semen in a clinical or research setting. Detailed protocols for routine, optional and research tests are elaborated.
Our IVF Clinic Management Software Approach
Validation Studies
The ART Compass Approach To Competency Assessment Over 80 competency assessment modules were created for sperm and embryo morphology, quality, viability, and common clinical decision time points in our IVF Clinic Management Software. Validation studies are underway, using images of PAP stained morphology slides for andrology modules, and 112 cleavage stage images captured on EmbryoScope 66 hours post insemination, and 168 blastocyst stage images captured on EmbryoScope 115 and 139 hours post insemination for embryology modules. Each slide was rotated and repeated 3 times throughout the modules to allow for intra-observer variability measurements. This work was presented as an abstract and poster at the Foundation for Reproductive Medicine Conference in New York City, November 17-19, 2018.
Test Protocol Standardization and Check for Understanding
Rigorously standardized test protocols are essential for meaningful comparisons across multiple sites. Before each assessment can begin, a standardized and detailed protocol is displayed, followed by a quick “Check for Understanding” module to ensure that test takers understand the instructions and how to take the test. ART Compass is flexible; it can serve standardized specimens to each technologist at each study site simultaneously, allowing even very small IVF clinics to compare individual values to the mean of all technologists, or to technologists in a central laboratory.
Clinical Decision Making
Embryologists rely on morphological assessments when assessing sperm morphology, sperm quality, and when performing semen analyses and when selecting sperm for Intracytoplasmic Sperm Injection (ICSI). Once a sperm has been identified for injection, embryologists immobilize the cell by breaking the tail with an ICSI needle. Breaking the cell membrane invokes subsequent physiological and biochemical reactions, which promote decondensation of the sperm head and activation of the oocyte. Staff is trained to avoid contact with the midpiece region of the sperm as this contains the centriole, which plays a major role in the cleavage patterns of the developing embryo. The location of the immobilization step can be visually determined in The ART Compass Approach To Competency Assessment app.
Additionally, cleavage stage embryo morphological assessments are performed to evaluate embryo quality. These scores are used to make clinical decisions such as determining the day of transfer (Day 3 vs Day 5), determining the number of embryos to transfer, selecting the top-quality embryo(s) for transfer, as well as determining the final disposition of embryos. ART Compass provides assessments for each of these parameters.
Blastocyst stage embryo morphological assessments are performed to evaluate embryo quality. These scores are used to make clinical decisions regarding the number of embryos to transfer, selecting the top-quality embryo(s) for transfer, as well as determining the final disposition of embryos. ART Compass provides assessments for each of these parameters and more.
American Society for Reproductive Medicine – ASRM Celebrating 75 Years of History and Innovation in Philadelphia, October 12-16
Exclusive continuing education content will be made available for American Society for Reproductive Medicine – ASRM 2019 attendees. The hands-on training opportunities offered by EmbryoDirector are complemented by the ART Compass competency assessment and continuing education mobile app platform.
How does your clinical decision making or subject test knowledge compare to senior IVF industry professionals? Grab a friend or come make new ones, as you sharpen your content knowledge at our interactive exhibit.