Chapter 06
Data integrity
The least glamorous chapter, and the one that decides whether anything in the previous six means anything. There is a specific mechanism by which laboratory quality problems turn into false geology inside your model. Understand that mechanism and you will never again treat QA/QC as somebody else's job.
Start with the mechanism
Here is the failure, stated as compactly as possible.
Samples are assayed in batches. A lab's calibration drifts, or a reagent lot changes, and for three weeks every result reads 8% high. Meanwhile, drilling happens in campaigns: those three weeks of samples come from holes that are physically adjacent, because you drill out one part of a deposit at a time.
So a purely temporal instrument artifact maps onto a contiguous region of space. It is not noise — it is a smooth, spatially coherent 8% grade anomaly with sharp boundaries at campaign edges. Your variogram will model it. Your prospectivity model will find it. Your resource estimate will bank it. Nothing in a standard validation pipeline catches it, because it looks exactly like a geological domain.
In ML vocabulary: structured, non-random label noise correlated with your primary covariate. Every intuition you have about noise averaging out is invalid, because this noise does not average out — it integrates.
The provenance chain
A grade in a database is the end of a long physical and clerical process. Each arrow is a place where the number can change.
rock in the ground
→ drilled core / RC chips # sample recovery can be <100%
→ split (half core to lab) # which half? biased if mineralisation is asymmetric
→ crushed # comminution
→ riffle split # subsampling error — Gy's theory lives here
→ pulverised to pulp
→ split again → charge (30–50 g)
→ digested / fused # partial vs total digestion: different quantities
→ instrument (AAS / ICP-OES / ICP-MS / fire assay)
→ LIMS certificate # a PDF or CSV
→ import into the drillhole DB # transcription, unit conversion, merges
→ your dataframeNote how much of the variance is created before the instrument. The fundamental sampling error from taking a 30 gram charge to represent a metre of rock is often larger than the analytical error — dramatically so for coarse gold, where a single visible grain in or out of the split moves the answer by an order of magnitude. This is Pierre Gy's sampling theory, and it is why "the assay" is not a measurement of the interval but a noisy estimate of it.
The control suite
The industry manages this with control samples inserted blind into the sample stream — the lab must not be able to tell them from real material. Typical insertion rates run around one control in twenty samples, split across four types. Each type isolates a different error mode.
| Control | What it is | What it detects |
|---|---|---|
| CRM / standard | Certified reference material with a known certified value and standard deviation | Accuracy / bias. Is the lab reading high or low? |
| Blank | Material certified to contain none of the element | Contamination and carryover — especially a blank inserted right after a high-grade sample |
| Field duplicate | Second split taken at the rig | Total variance: sampling + subsampling + analytical |
| Coarse duplicate | Second split of the crushed reject | Subsampling + analytical only |
| Pulp duplicate | Second split of the pulverised pulp | Analytical precision alone |
| Umpire / check assay | Pulps re-assayed at a second, independent laboratory | Inter-lab bias — the one your primary lab's own controls cannot see |
The three duplicate types are not redundant — they are a designed experiment. Because each captures a nested subset of the error sources:
σ²field = σ²sampling + σ²subsampling + σ²analytical σ²coarse = σ²subsampling + σ²analytical σ²pulp = σ²analytical
differencing recovers each component. It is a hierarchical ANOVA that tells you where to spend money: if subsampling dominates, buying a better instrument is wasted — you need a bigger charge or finer pulverisation.
Reading the controls
CRMs are plotted on a control chart against the certified value, with warning and failure limits conventionally at ±2 and ±3 standard deviations. Beyond single-point failures, the informative patterns are runs — several consecutive results on the same side of the certified value indicate drift, even if every one is individually within limits. If you have implemented statistical process control, you already know these rules.
Duplicate pairs are summarised by relative difference. Two conventions, and people conflate them constantly:
RPD = |A − B| / ((A + B) / 2) × 100 %
HARD = |A − B| / (A + B) × 100 % ( = RPD / 2 )
Acceptance is usually stated as a fraction of pairs below a threshold — for example, most pulp duplicate pairs within 10% HARD. Thresholds are commodity- and deposit-specific and should come from the project's QP configuration, never from a hard-coded default.
Relative difference blows up near the detection limit — two values of 0.011 and 0.009 differ by 20% RPD and mean nothing. Always exclude pairs where both values sit near the detection limit before computing pass rates, and state the exclusion. Thompson-Howarth plots and rank-HARD plots exist precisely to show precision as a function of concentration rather than as a single number.
Failure modes worth knowing by name
- Over-limit truncation. Standard multi-element packages saturate — copper above ~1% may return as
>10000 ppm. If the ore-grade re-assay is never requested or never merged back, your highest-grade material is systematically censored at exactly the values that matter most. - Sample swaps. Two samples exchanged in the rack. Undetectable by control charts; visible only through twin holes, re-assay programmes, or a grade that contradicts the logged lithology.
- Silent unit changes. A database maintained across decades quietly switches ppm to %, or feet to metres, at a company acquisition. The join succeeds; the numbers are off by 10,000×.
- Method heterogeneity. Aqua regia (partial) and four-acid (near-total) digestions measure different quantities. Merging them without normalisation creates a step change at the boundary — and the boundary is usually a date, which is usually a place.
- Missing recovery. If core recovery was 60% in a broken, mineralised shear zone, that interval's grade is unreliable and probably biased. Recovery is a required covariate, and it is routinely absent.
- Copy-paste in spreadsheets. Genuinely one of the most common real-world causes of bad numbers. A duplicated block of rows is trivially detectable by hashing and almost never checked.
What good looks like
The design principles below are what Aethermine Audit is built on, and they generalise to any data-integrity system you might build.
- Read-only, connector-first. Attach to the client's existing dataroom or database with viewer/SELECT-only credentials. Never ask a geologist to migrate their data to use your tool; enforce the read-only property at the platform level rather than by convention.
- Continuous, not annual. QA/QC is traditionally reviewed in a batch before a resource statement — sometimes months after the samples were assayed, when the reject material may be gone. Checking every batch as it lands turns a post-mortem into a correction.
- Provenance as a first-class column set. Every value carries
source_id,source_locator(file / sheet / row / page),ingested_at, and acontent_hash. A finding you cannot trace to a cell in a file is not actionable. - Idempotent, hash-gated ingestion. Re-running on unchanged sources is a no-op. This is what makes continuous operation affordable and reproducible.
- Every rule cites its threshold and its source. A finding says which rule fired, what threshold applied, whether that threshold was a documented default or a QP override, and what evidence triggered it.
- Known-answer tests with planted errors. Build fixtures with deliberately seeded failures and assert both detection and non-detection — false-positive guards matter as much as recall, because an alerting system nobody trusts is worse than none.
- Flag, never clear. The system flags issues and marks material as requiring review. It does not verify, clear or certify. A Qualified Person decides. This is not legal hedging — it is an accurate description of what a statistical check can and cannot establish.
Deterministic rules do the flagging, because a QP needs to know exactly why something fired and to reproduce it. The language model's job is explanation and triage — reading the finding, the surrounding context and the source documents, then summarising what happened and what to check next, with every claim citing evidence it actually retrieved. The moment a model is allowed to assert a number it did not look up, the entire trust argument collapses.
What to do about it as a modeller
- Get the QA/QC data. Ask for the control sample results, not just the assay table. If nobody can produce them, that is itself the most important finding of your project.
- Add batch as a covariate and test whether it explains variance. If lab batch is significant after conditioning on location, you have found a problem, not a feature.
- Plot grade against assay date. Thirty seconds of work, and it catches drift, method changes and unit switches that no summary statistic will show you.
- Validate spatially and temporally. Hold out whole campaigns, not random rows. If performance collapses across campaign boundaries, suspect batch effects before you suspect geology.
- Propagate assay uncertainty. You measured it with the duplicate programme. Feed it in as the nugget in Chapter 8 and as observation noise in Chapter 10, rather than treating assays as exact.
Sources for this chapter
- Abzalov, Applied Mining Geology (Springer, 2016). The most practical treatment of QA/QC design in print — control sample types, insertion rates, control charts, duplicate analysis. The closest thing to a manual for this chapter.
- Pitard, Pierre Gy's Sampling Theory and Sampling Practice (CRC, 3rd ed. 2019). The theory of sampling error, and why the fundamental sampling error from a 30 g charge often dominates everything the laboratory does.
- Thompson & Howarth, on estimating analytical precision from duplicate pairs — The Analyst 101 (1976) and Journal of Geochemical Exploration 9 (1978). The origin of the precision-versus-concentration plots still used today.
- Stanley & Lawie, "Average relative error in geochemical determinations", Geochemistry: Exploration, Environment, Analysis 7 (2007). Careful treatment of the relative-difference measures (RPD / HARD) and their behaviour near the detection limit.
- CIM Mineral Exploration Best Practice Guidelines — the industry's own statement of expected QA/QC practice, and what a QP will be measured against.
- ISO/IEC 17025 — the accreditation standard commercial assay laboratories operate under. Worth knowing it exists, and knowing that accreditation constrains process, not accuracy on your particular sample.