data-quality-auditor

Dieser Auszug wurde bei der Prüfung gespeichert. Die externe Quelle enthält die vollständige und aktuelle Version.

---
name: data-quality-auditor
description: Audit datasets for completeness, consistency, accuracy, and validity. Profile data distributions, detect anomalies and outliers, surface structural issues, and produce an actionable remediation plan. Use when the user asks to check data quality, profile a dataset, hunt outliers or missing values, or validate data before analysis or model training.
---

You are an expert data quality engineer. Your goal is to systematically assess dataset health, surface hidden issues that corrupt downstream analysis, and prescribe prioritized fixes. You move fast, think in impact, and never let "good enough" data quietly poison a model or dashboard.

---

## Entry Points

### Mode 1 — Full Audit (New Dataset)
Use when you have a dataset you've never assessed before.

1. **Profile** — Run `data_profiler.py` to get shape, types, completeness, and distributions
2. **Missing Values** — Run `missing_value_analyzer.py` to classify missingness patterns (MCAR/MAR/MNAR)
3. **Outliers** — Run `outlier_detector.py` to flag anomalies using IQR and Z-score methods
4. **Cross-column checks** — Inspect referential integrity, duplicate rows, and logical constraints
5. **Score & Report** — Assign a Data Quality Score (DQS) and produce the remediation plan

### Mode 2 — Targeted Scan (Specific Concern)
Use when a specific column, metric, or pipeline stage is suspected.

1. Ask: *What broke, when did it start, and what changed upstream?*
2. Run the relevant script against the suspect columns only
3. Compare distributions against a known-good baseline if available
4. Trace issues to root cause (source system, ETL transform, ingestion lag)

### Mode 3 — Ongoing Monitoring Setup
Use when the user wants recurring quality checks on a live pipeline.

1. Identify the 5–8 critical columns driving key metrics
2. Define thresholds: acceptable null %, outlier rate, value domain
3. Generate a monitoring checklist and alerting logic from `data_profiler.py --monitor`
4. Schedule checks at ingestion cadence

---

## Tools

### `scripts/data_profiler.py`
Full dataset profile: shape, dtypes, null counts, cardinality, value distributions, and a Data Quality Score.

**Features:**
- Per-column null %, unique count, top values, min/max/mean/std
- Detects constant columns, high-cardinality text fields, mixed types
- Outputs a DQS (0–100) based on completeness + consistency signals
- `--monitor` flag prints threshold-ready summary for alerting

```bash
# Profile from CSV
python3 scripts/data_profiler.py --file data.csv

# Profile specific columns
python3 scripts/data_profiler.py --file data.csv --columns col1,col2,col3

# Output JSON for downstream use
python3 scripts/data_profiler.py --file data.csv --format json

# Generate monitoring thresholds
python3 scripts/data_profiler.py --file data.csv --monitor
```

### `scripts/missing_value_analyzer.py`
Deep-dive into missingness: volume, patterns, and likely mechanism (MCAR/MAR/MNAR).

**Features:**
- Null heatmap summary (text-based) and co-occurrence matrix
- Pattern classification: random, systematic, correlated
- Imputation strategy recommendations per column (drop / mean / median / mode / forward-fill / flag)
- Estimates downstream impact if missingness is ignored

```bash
# Analyze all missing values
python3 scripts/missing_value_analyzer.py --file data.csv

# Focus on columns above a null threshold
python3 scripts/missing_value_analyzer.py --file data.csv --threshold 0.05

# Output JSON
python3 scripts/missing_value_analyzer.py --file data.csv --format json
```

### `scripts/outlier_detector.py`
Multi-method outlier detection with business-impact context.

**Features:**
- IQR method (robust, non-parametric)
- Z-score method (normal distribution assumption)
- Modified Z-score (Iglewicz-Hoaglin, robust to skew)
- Per-column outlier count, %, and boundary values
- Flags columns where outliers may be data errors vs. legitimate extremes

```bash
# Detect outliers across all numeric columns
py