ÃÞ»¨ÌÇÖ±²¥

Method verification studies are standard practices in any clinical laboratory. These studies are required by the Clinical Laboratory Improvement Amendments (CLIA) () for before reporting patient results. Non-waived systems, which include tests of moderate or high complexity, refers to the skills, reagents and steps needed to perform the assay and require documentation that they can be reliably used in the operator’s environment. This should be done for any new assay or equipment and when there are major changes in procedures or instrument re-location. However, the process can be confusing and verifications for microbiological methods don’t always fit the parameters for analytical assays. So how do you begin to confirm that your new test panel is ready to use?

Determine the Purpose of the Study

Is it a Verification or a Validation?

The terms validation and verification are sometimes used interchangeably. However, they are different. A validation is a process meant to establish that an assay works as intended. This applies to non-FDA cleared tests (e.g., laboratory developed methods) and modified FDA-approved tests. Modifications are changes to the assay not specified as acceptable by the manufacturer and can include using different , or test parameters such as changing incubation times. These changes could affect the performance of the assay and would need to be validated before implementing. A verification is for unmodified FDA-approved or cleared tests. It is a one-time study meant to demonstrate that a test performs in line with previously established performance characteristics when used as intended by the manufacturer.

A Qualitative or Quantitative Assay?

It is important to know the type of assay being implemented since it can influence how the CLIA standards are met. Testing methods can be divided into 3 main categories based on the results that will be reported:

• Qualitative: provides a binary result such as “detected” or “not detected.”
• Quantitative: provides a numerical value.
• Semi-quantitative: uses numerical values to determine an acceptable cutoff but reports a qualitative result (e.g., cycle threshold (Ct) cutoff for the detection of mecA by real-time polymerase chain reaction (PCR)).

Qualitative and semi-quantitative are the more common assay types for a microbiology lab.

Establish the Study Design

Once you know the purpose of the study, you can determine what criteria needs to be tested. For an , laboratories are required to verify the characteristics listed below:

• Accuracy: to confirm the acceptable agreement of results between the new method and a comparative method.
• Precision: to confirm acceptable within-run, between-run and operator variance.
• Reportable Range: to confirm the acceptable upper and lower limit of the test system.
• Reference Range: to confirm the normal result for the tested patient population. 

Below are suggestions to meet the verification criteria for qualitative/semi-quantitative assays:

Accuracy

For the number of samples, use a minimum of 20 clinically relevant isolates. For qualitative assays, use a combination of positive and negatives samples and for semi-quantitative assays, use a range of samples with high to low values. The acceptable specimens can come from standards or controls, reference materials, proficiency tests, de-identified clinical samples, if tested previously or in parallel with a validated method or consider including different sample matrices, if applicable. For the calculations, use the number of results in agreement over total number of results multiplied by 100. The acceptable percentage of accuracy should meet the stated claims of the manufacturer or what the CLIA director determines.

Precision

For the number of samples, use a minimum of 2 positive and 2 negatives tested in triplicate for 5 days by 2 operators. If system is fully automated, user variance is not needed. For qualitative assays, use a combination of positive and negatives samples and for semi-quantitative assays, use a combination of samples with high to low values. The acceptable specimens can come from controls or de-identified clinical samples. For the calculations, use the number of results in agreement over total number of results multiplied by 100. The acceptable percentage of precision should meet the stated claims of the manufacturer or what the CLIA director determines.

Reportable Range

For the number of samples, verify using a minimum of 3 samples. For qualitative assays, use known samples positive for the detected analyte and for semi-quantitative assays, use a range of positive samples near the upper and lower ends of the manufacturer determined cutoff values. To evaluate, the reportable range for a qualitative or semi-quantitative assay will be defined as what the laboratory establishes as a reportable result (e.g., Detected, Not detected, Ct value cutoff), verified by testing samples that fall within the reportable range.

Reference Range

For the number of samples, verify using a minimum of 20 isolates. For qualitative and semi-quantitative assays use de-identified clinical samples or reference samples with a result known to be standard for the laboratory’s patient population. This can be provided by the manufacturer, such as, samples negative for methicillin resistant-Staphylococcus aureus (MRSA) for an assay detecting MRSA. The reference range for a qualitative or semi-quantitative assay will be defined as what the laboratory establishes as an expected result for a typical sample, verified by testing samples representative to the laboratory’s patient population. If the manufacturer’s reference range does not represent the laboratory’s typical patient population, additional samples from the laboratory’s population should be screened and the reference range re-defined.

Once you have the above areas outlined, write a verification plan for what needs to be done before starting the study.

Create a Verification Plan

If a written verification plan is required by your lab, it will need to be reviewed and signed off by the lab director. The verification plan should include:

• Type of verification and purpose of study.
• Purpose of test and method description.
• Details of study design, including:
  • Number and type(s) of samples.
  • Type of quality assurance (QA) and quality controls (QC) that will be used.
  • Number of replicates, including how many days and how many analysts.
  • Performance characteristics that will be evaluated and the acceptance criteria.
• Materials, equipment and any other resources needed.
• Safety considerations.
• Expected timeline for completion.

CLIA regulations will specify under what circumstances verifications must be conducted and provides some guidelines but not specific details on how to do it. This can lead to some confusion, making implementation of new assays challenging. This is especially evident with antimicrobial susceptibility test methods. Knowing what organisms to use, how to interpret results and what to consider when using non-FDA breakpoints with an FDA-cleared AST panel is not clear-cut. Therefore, when planning a method verification study, it is important to get input. Laboratory leaders such as are valuable resources, specifically trained to oversee this process.

Ultimately, verifications and validations are only one part when it comes to a laboratory test system. Clinical laboratories must create an on-going process to monitor and re-assess the assay and determine if the test continues to meet the desired purpose. Understanding the patient population, why the test is being performed as well as the costs, quality monitoring and training associated with the testing is just as important to ensure the reliability of the laboratory.

Helpful Resources

• : User Protocol for Evaluation of Qualitative Test Performance.
• : A Framework for Using CLSI Documents to Evaluate Clinical Laboratory Measurement Procedures.
• : Verification of Commercial Microbial Identification and AST Systems.
• : Molecular Diagnostic Methods for Infectious Diseases.
• Cumitech 31A: Verification and Validation of Procedures in the Clinical ÃÞ»¨ÌÇÖ±²¥ Laboratory.

---

ÃÞ»¨ÌÇÖ±²¥, in collaboration with CLSI and CAP, has developed an Individualized Quality Control Plan (IQCP) template for Quality Control (QC) of a commercial cartridge-based molecular test system for detection of a single or multiple targets.