Costs of inaccurate or inadequate data can be steep. Problems with data quality can result in tangible and intangible damage ranging from loss of customer/user confidence to loss of life and mission. – Department of Defense (DoD) Guidelines on Data Quality Management
…..more than a decade’s experience has demonstrated that integrity is not a safe assumption. United States Environmental Protection Agency (EPA), 2002 Guidance on Environmental Data Verification and Data Validation
Data validation is absolutely essential at key decision points, such as determining the boundaries of groundwater contamination. – EPA, Region 9
We must ensure project objectives are met through adequate, accurate data. We must also take steps to ensure project decisions are based upon legally defensible data.
In the terminology of the United States Environmental Protection Agency (EPA) and the Department of Defense (DoD), measurement performance criteria (MPC*) or data quality indicators (DQIs) are the criteria for evaluation of project data. Your MPC support your data quality objectives (DQOs), which are objectives your data must satisfy to be “good enough” to support your project decisions. This means MPC support your DQOs, which in turn support the project objectives. However, your MPC do not have to be perfectly met for your DQOs to be achieved.
How do you determine your MPC and DQOs? Your MPC and DQOs are set during project planning. The optimized Uniform Federal Policy – Quality Assurance Project Plan (UFP-QAPP) lays out steps and some guidelines to assist you in determining what they should be. The prompts in the optimized UFP-QAPP format can be useful for project planning, even if this challenging document is not required for your project, and you don’t go on to use the template. Referencing the prompts can help you meet EPA criteria for project plans and determine when to engage your engineers, geologists, and chemists during planning. Though this may seem time-consuming, this approach to planning will save you valuable project execution time once your project is in progress.
Your chemists can help develop and/or identify your MPC. It may be appropriate to default to the current DoD Quality Systems Manual (QSM) criteria, method criteria, or historical laboratory criteria. Because your planning time is not unlimited and because you cannot predict every field and laboratory condition that may affect your data, using established criteria as your MPC is usually a good approach. Your MPC are intended to be a guide for evaluating how well you met your DQOs and not meeting MPC is an indication that a closer review of the data may be needed, but MPC are not a prescriptive measure of your DQOs.**
So, how do you know if your MPC have been met well enough for you to make sound project decisions? How do you know your project decisions are based upon legally defensible data? Data validation is how you do that.
During project execution, your validators evaluate your data to determine if any of your data are not usable for project purposes. Ideally, this evaluation is not simply an “in is in, out is out” approach to evaluating quality control (QC) outliers, but rather takes your overall project goal into account. With use of Automated Data Review (ADR) and similar tools, 60-80% of what a validator does can be automated. However, not all of the required assessments can be automated, and experience and the ability to look at the big picture have value in minimizing risks.
Although laboratories perform several levels of data review, it is illegal for them to “validate” their own data in most cases; this is considered a conflict of interest. Under deadline and holding-time pressure, mistakes can occur.
Validation is required by your client for your project which involves a simple dig and haul remediation of lead in soil with no migration to groundwater. Easy, right? You get your preliminary results which are all nondetects, place your clean fill, and move on. Your final data arrives a few weeks later and your validators determine the calibration for lead was improperly performed or calculated by the laboratory and your nondetect values for lead reported at a 2 milligram per kilogram (mg/kg) quantitation limit (QL) should have been reported at 20 mg/kg.
If an “in is in, out is out, get it done as quickly as possible” approach is used, this could lead the validators to reject these nondetect values, and you may have to remobilize, re-excavate, and collect and analyze additional samples from your site. Your project has just cost twice as much as anticipated.
However, if your validators have access to your project goals and are taking a “whole project” approach, they will have access to your project action limits (PALs) or decision criteria, which is likely to be between 100 mg/kg and 400 mg/kg for lead, in which case, nondetect values with QLs raised to 20 mg/kg are clearly usable to determining you have met your criteria.
If any of your project documents, inclusive of your Request for Proposal (RFP), Statement of Objectives (SOO), Statement of Work (SOW), or Performance Work Statement (PWS) require data review per the DoD QSM, the UFP-QAPP, or reference ADR, your client is requiring data validation. Clients expect you to understand their requirements and to know the content of their guidance documents.
Data validation is ideally performed and often required for confirming remedial action is complete, for monitoring and operations assessments, and for determining that materials are suitable to be put into or back into the ground (ex: using clean site soils as backfill). Data validation may also be needed for site characterization, depending upon purpose of characterization. Data validation provides assurance that data are adequate for the intended use. Data that are adequate for the intended use lead to sound project decisions. Data validation may also save you a day in court.
The new owner of the property in Scenario 1 plans to sue the previous owner, your client. You are called upon to defend your assertion that you completed remediation to the satisfaction of the regulatory requirements of the time; however, ten years have passed since you completed this project and you don’t recall the details of this small project all that well. You are concerned when your data are called into question. However, because your validators clearly and thoroughly documented the calibration issue, raised the reporting limits, and showed your soil samples were clean to 20 mg/kg, the case is dismissed.
Data validation is, however, not always needed. It is rarely needed for waste characterization, and data indicating additional actions are needed (such as excavating wider or deeper) do not require validation. In these cases, responsibility is being assumed by another party (i.e. the waste acceptance facility or landfill) or your team will be taking additional actions before you make the final project decisions. In some cases, the level of validation (such as those commonly referred to as Levels 2, 3, and 4) may be minimized unless issues arise, depending on project objectives. Data validation performed in a manner that is tailored to your project helps ensure project objectives are met and risk is minimized.
If you have questions or comments, please leave them in the comments section below. If you’d like to find out if Oak Services is the right company to write or review your project plan, perform your environmental data validation or assess your validation needs, please contact us and let us know what you’re looking for.
- Environmental Data Quality at DoD Superfund Sites in Region 9
- DoD Guidelines on Data Quality Management
- DoD QSM for Environmental Laboratories
- EPA QA/G-8
- Intergovernmental Data Quality Task Force, Uniform Federal Policy for Quality Assurance Project Plans, Optimized UFP-QAPP Worksheets
*The parameters evaluated with MPC were historically referred to as “PARCC”, which is an acronym for precision, accuracy, representativeness, comparability, and completeness. PARCC is generally considered to be an obsolete term now, and MPC includes an evaluation of data precision, accuracy, bias, sensitivity, and completeness.
** Although it is possible to tailor your MPC very specifically to your project, it is not possible to anticipate every possible field and laboratory condition that may be encountered during project execution. It is reasonable to use default MPC with room for professional judgement. This approach can be formalized in your QAPP, along with a requirement that professional judgment calls will be briefly explained in your data validation summaries or reports.
About the author: Dianne McNeill is a Proposal Manager and Senior Scientist with Oak Services. She has 23 years of experience in the environmental sciences, with 15 years environmental data validation experience inclusive of 10 years experience training new validators. She also has 7 years of environmental laboratory experience, with a focus on GC/MS analysis of VOCs and SVOCs and LC/MS and LC/MS/MS analysis of explosives, dyes, and specialty parameters.