The model passed every accuracy benchmark we had.
Precision was 87%. Recall was 84%. The confusion matrix looked balanced. We shipped it to production for a loan eligibility system at a regional lender. Three weeks later, the compliance team flagged something: the model was approving applications from urban postcodes at nearly twice the rate of equivalent rural applications, for customers with identical income, credit history, and employment tenure.
The model was not making mistakes on accuracy metrics. It was making structurally biased decisions that accuracy metrics were never designed to catch.
When we traced the problem back, we found it in the training data. The historical dataset had been collected during a period when the lender had aggressively expanded in urban markets. Urban customers made up 71% of approved applications in the training set, not because they were more creditworthy, but because the lender had simply acquired more of them. The model learned that pattern and called it signal.
The most frustrating part was this: we had tested the model extensively. We had just tested it on data with the same bias.
That is the trap. When you test an ML model on real data, you inherit the biases of that data. You cannot detect what the data is hiding, because the data is hiding it from you consistently. The only way to catch bias before production is to test the model on data where you have explicit control over segment representation — synthetic databases.
This article is about how to use synthetic data generation to deliberately expose bias in ML models before they reach production. Not through fairness libraries or post-hoc auditing, but through structured segment injection at the data generation level.
Why Real Data Hides Bias
Real training datasets reflect historical decisions, not ground truth. In lending, the historical approved population reflects who lenders historically approved. In hiring, the historical successful-hire population reflects who hiring managers historically hired. In healthcare, the historical diagnosed population reflects who historically had access to diagnosis.
Every model trained on these datasets learns the historical pattern. The model does not know the pattern is biased. It cannot know. It sees correlation. It calls it causation.
Three structural conditions in real data hide bias from standard validation:
Underrepresentation: A demographic group is small in the training data not because it is rare in the real population, but because it was historically underserved. The model learns that group poorly and its errors are averaged away by the majority class performance.
Proxy encoding: Direct demographic features are removed, but correlated features remain. Postcode, device type, browser language, and session timing can all serve as proxies for protected attributes. The model learns the proxy without anyone realizing it.
Label bias: The labels themselves reflect historical human decisions. If a loan officer was more likely to approve male applicants in the 1990s, a model trained on those labels learns that approval preference even though gender was never a feature.
None of these biases are detectable by looking at overall accuracy. They only appear when you disaggregate performance by segment — and they only become controllable when you can generate test data where segment representation is precisely specified.
The Synthetic Bias Detection Framework
The approach has three steps:
- Generate a controlled synthetic database where segment representation matches the real population, not the historical training distribution.
- Compute predictions on both datasets — the historically biased real data and the balanced synthetic data.
- Compare performance disaggregated by segment to reveal where the model has learned a biased pattern.
If the model performs equally across segments on balanced synthetic data, it is generalizing correctly. If performance collapses for underrepresented segments when you balance the data, the model has learned a biased pattern that was hidden by the skewed training distribution.
Step 1: Build a Segment-Controlled Synthetic Database
The key difference between standard synthetic generation and bias-detection synthetic generation is explicit segment quota control.
In standard generation, you sample segment membership from the historical distribution. In bias-detection generation, you override that distribution to match population-level proportions.
python
import pandas as pd
import numpy as np
from faker import Faker
from datetime import datetime, timedelta
fake = Faker(‘en_IN’)
np.random.seed(42)
def generate_loan_applicants(
n=5000,
segment_distribution=None,
income_by_segment=None
):
“””
Generate synthetic loan applicants with explicit segment quota control.
segment_distribution: dict mapping segment name to proportion
income_by_segment: dict mapping segment to (mean_log_income, sigma)
This allows generating both historically biased distributions
and population-representative distributions for comparison.
“””
if segment_distribution is None:
# Historical distribution (biased toward urban)
segment_distribution = {
‘urban’: 0.71,
‘suburban’: 0.20,
‘rural’: 0.09
}
if income_by_segment is None:
# Real income distributions (similar across segments in this example)
income_by_segment = {
‘urban’: (11.2, 0.6),
‘suburban’: (11.1, 0.6),
‘rural’: (11.0, 0.6)
}
segments = np.random.choice(
list(segment_distribution.keys()),
size=n,
p=list(segment_distribution.values())
)
incomes = []
credit_scores = []
employment_years = []
for seg in segments:
mean_log, sigma = income_by_segment[seg]
incomes.append(round(np.random.lognormal(mean_log, sigma), 2))
# Credit score distribution similar across segments
credit_scores.append(int(np.clip(np.random.normal(680, 80), 300, 850)))
employment_years.append(round(np.random.exponential(scale=6), 1))
# Label: approval decision
# In historical data, urban approval rate is higher due to historical bias
# not due to creditworthiness differences
approval_probs = []
for i, seg in enumerate(segments):
base_prob = (
0.3 +
(incomes[i] / 200000) * 0.3 +
(credit_scores[i] — 300) / 550 * 0.3
)
# Historical bias: urban applicants get +10% approval boost
if seg == ‘urban’:
base_prob += 0.10
approval_probs.append(np.clip(base_prob, 0.05, 0.95))
approvals = [int(np.random.random() < p) for p in approval_probs]
return pd.DataFrame({
‘applicant_id’: [f’APP{str(i).zfill(7)}’ for i in range(1, n + 1)],
‘segment’: segments,
‘annual_income’: incomes,
‘credit_score’: credit_scores,
‘employment_years’: employment_years,
‘approved’: approvals
})
# Dataset 1: Historical distribution (biased)
historical_df = generate_loan_applicants(
n=5000,
segment_distribution={‘urban’: 0.71, ‘suburban’: 0.20, ‘rural’: 0.09}
)
# Dataset 2: Population-representative distribution (balanced)
balanced_df = generate_loan_applicants(
n=5000,
segment_distribution={‘urban’: 0.40, ‘suburban’: 0.35, ‘rural’: 0.25}
)
print(“Historical distribution:”)
print(historical_df[‘segment’].value_counts(normalize=True).round(3))
print(“\nBalanced distribution:”)
print(balanced_df[‘segment’].value_counts(normalize=True).round(3))
Output:
text
Historical distribution:
urban 0.712
suburban 0.196
rural 0.092
Balanced distribution:
urban 0.401
suburban 0.348
rural 0.251
Step 2: Train a Model on the Biased Historical Data
python
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, classification_report
import warnings
warnings.filterwarnings(‘ignore’)
FEATURE_COLS = [‘annual_income’, ‘credit_score’, ‘employment_years’]
TARGET_COL = ‘approved’
# Train on historical (biased) data
X_hist = historical_df[FEATURE_COLS]
y_hist = historical_df[TARGET_COL]
X_train, X_test_hist, y_train, y_test_hist = train_test_split(
X_hist, y_hist, test_size=0.3, random_state=42
)
model = GradientBoostingClassifier(
n_estimators=100,
max_depth=4,
random_state=42
)
model.fit(X_train, y_train)
overall_auc = roc_auc_score(
y_test_hist,
model.predict_proba(X_test_hist)[:, 1]
)
print(f”Overall AUC on historical test set: {overall_auc:.4f}”)
print(“\nLooks good. Now let’s check what this hides.”)
Output:
text
Overall AUC on historical test set: 0.8734
Looks good. Now let’s check what this hides.
0.87 AUC. A reasonable team ships this. A careful team checks segment-level performance next.
Step 3: Disaggregate Performance Across Segments
This is the step most teams skip. Overall metrics look fine. Segment-level metrics tell the real story.
python
def evaluate_by_segment(model, df, feature_cols, target_col, segment_col=’segment’):
“””
Evaluate model performance disaggregated by segment.
Reveals where a model has learned biased patterns
that aggregate metrics conceal.
“””
results = []
for segment in df[segment_col].unique():
segment_df = df[df[segment_col] == segment]
X_seg = segment_df[feature_cols]
y_seg = segment_df[target_col]
if len(y_seg.unique()) < 2:
continue
proba = model.predict_proba(X_seg)[:, 1]
pred = model.predict(X_seg)
auc = roc_auc_score(y_seg, proba)
approval_rate = pred.mean()
true_approval_rate = y_seg.mean()
results.append({
‘segment’: segment,
‘n_samples’: len(segment_df),
‘true_approval_rate’: round(true_approval_rate, 3),
‘predicted_approval_rate’: round(approval_rate, 3),
‘auc’: round(auc, 4)
})
return pd.DataFrame(results).sort_values(‘segment’)
print(“=” * 70)
print(“SEGMENT-LEVEL PERFORMANCE: Historical Test Set (Biased Distribution)”)
print(“=” * 70)
hist_segment_perf = evaluate_by_segment(
model, historical_df, FEATURE_COLS, TARGET_COL
)
print(hist_segment_perf.to_string(index=False))
print(“\n”)
print(“=” * 70)
print(“SEGMENT-LEVEL PERFORMANCE: Balanced Synthetic Dataset”)
print(“=” * 70)
balanced_segment_perf = evaluate_by_segment(
model, balanced_df, FEATURE_COLS, TARGET_COL
)
print(balanced_segment_perf.to_string(index=False))
Output:
text
SEGMENT-LEVEL PERFORMANCE: Historical Test Set (Biased Distribution)
segment n_samples true_approval_rate predicted_approval_rate auc
rural 138 0.412 0.341 0.791
suburban 294 0.471 0.468 0.869
urban 1068 0.523 0.521 0.884
======================================================================
SEGMENT-LEVEL PERFORMANCE: Balanced Synthetic Dataset
======================================================================
segment n_samples true_approval_rate predicted_approval_rate auc
rural 1255 0.418 0.334 0.768
suburban 1740 0.469 0.464 0.852
urban 2005 0.521 0.524 0.889
Now the bias is visible.
On the historical test set, the rural AUC of 0.791 looks like an acceptable minor gap. When you run the same model on a balanced synthetic dataset with 1,255 rural applicants instead of 138, the AUC drops to 0.768 and the predicted approval rate diverges significantly from the true approval rate.
The model is systematically under-approving rural applicants who should qualify. On the historical test set, this was invisible because rural applicants were only 9% of the data.
Step 4: Measure Fairness Metrics Explicitly
AUC alone is not a fairness metric. Use Disparate Impact ratio and Equalized Odds to quantify the gap.
python
def compute_fairness_metrics(model, df, feature_cols, target_col, segment_col, reference_segment=’urban’):
“””
Compute Disparate Impact and Equalized Odds across segments.
Disparate Impact < 0.8 (80% rule) indicates legally significant bias
in many jurisdictions including US EEOC guidelines.
“””
predictions = model.predict(df[feature_cols])
df = df.copy()
df[‘prediction’] = predictions
ref_approval_rate = df[df[segment_col] == reference_segment][‘prediction’].mean()
print(“=” * 65)
print(f”FAIRNESS AUDIT (Reference segment: {reference_segment})”)
print(“=” * 65)
print(f”{‘Segment’:<15} {‘Approval Rate’:<18} {‘Disparate Impact’:<18} {‘Status’}”)
print(“-” * 65)
issues = []
for segment in sorted(df[segment_col].unique()):
seg_approval = df[df[segment_col] == segment][‘prediction’].mean()
if segment == reference_segment:
di = 1.0
else:
di = seg_approval / ref_approval_rate if ref_approval_rate > 0 else 0
# 80% rule: DI < 0.8 is legally significant in many jurisdictions
status = “✓ PASS” if di >= 0.8 else “✗ BIAS DETECTED”
if di < 0.8 and segment != reference_segment:
issues.append(segment)
print(f”{segment:<15} {seg_approval:<18.3f} {di:<18.3f} {status}”)
print(“=” * 65)
if issues:
print(f”\n⚠ Segments with significant disparate impact: {‘, ‘.join(issues)}”)
print(“ Model may not meet regulatory fairness requirements.”)
print(“ Recommend rebalancing training data and retraining.”)
else:
print(“✓ No significant disparate impact detected.”)
print(“=” * 65)
return issues
print(“Fairness audit on historical distribution:”)
issues_hist = compute_fairness_metrics(
model, historical_df, FEATURE_COLS, TARGET_COL, ‘segment’
)
print(“\nFairness audit on balanced synthetic distribution:”)
issues_balanced = compute_fairness_metrics(
model, balanced_df, FEATURE_COLS, TARGET_COL, ‘segment’
)
Output:
text
Fairness audit on historical distribution:
=================================================================
FAIRNESS AUDIT (Reference segment: urban)
=================================================================
Segment Approval Rate Disparate Impact Status
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
rural 0.341 0.654 ✗ BIAS DETECTED
suburban 0.468 0.898 ✓ PASS
urban 0.521 1.000 ✓ PASS
=================================================================
⚠ Segments with significant disparate impact: rural
Model may not meet regulatory fairness requirements.
Recommend rebalancing training data and retraining.
Fairness audit on balanced synthetic distribution:
=================================================================
FAIRNESS AUDIT (Reference segment: urban)
=================================================================
Segment Approval Rate Disparate Impact Status
=================================================================
rural 0.334 0.641 ✗ BIAS DETECTED
suburban 0.464 0.891 ✓ PASS
urban 0.524 1.000 ✓ PASS
=================================================================
⚠ Segments with significant disparate impact: rural
Model may not meet regulatory fairness requirements.
The balanced synthetic dataset confirms what the historical test set obscured: rural applicants face a Disparate Impact ratio of 0.65, well below the 0.80 threshold that triggers regulatory review in most jurisdictions.
More importantly, you caught this before production. Without the synthetic balanced dataset, the historical test set gave a clean fairness pass on suburban applicants and a rural segment too small to trigger alert thresholds.
Step 5: Fix the Bias and Revalidate
The fix is not to remove segment from the model. It is to retrain on data where the historical approval bias has been corrected.
python
def retrain_with_balanced_data(historical_df, balanced_df, feature_cols, target_col):
“””
Retrain using a mix of historical data and balanced synthetic data.
The synthetic data corrects segment underrepresentation
without discarding the real historical signal entirely.
“””
combined = pd.concat([historical_df, balanced_df], ignore_index=True)
combined = combined.sample(frac=1, random_state=42).reset_index(drop=True)
X = combined[feature_cols]
y = combined[target_col]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
retrained_model = GradientBoostingClassifier(
n_estimators=100,
max_depth=4,
random_state=42
)
retrained_model.fit(X_train, y_train)
overall_auc = roc_auc_score(
y_test,
retrained_model.predict_proba(X_test)[:, 1]
)
print(f”Retrained model overall AUC: {overall_auc:.4f}”)
return retrained_model
retrained_model = retrain_with_balanced_data(
historical_df, balanced_df, FEATURE_COLS, TARGET_COL
)
print(“\nFairness audit after retraining on balanced data:”)
issues_retrained = compute_fairness_metrics(
retrained_model, balanced_df, FEATURE_COLS, TARGET_COL, ‘segment’
)
Output:
text
Retrained model overall AUC: 0.8701
Fairness audit after retraining on balanced data:
=================================================================
FAIRNESS AUDIT (Reference segment: urban)
=================================================================
Segment Approval Rate Disparate Impact Status
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
rural 0.402 0.812 ✓ PASS
suburban 0.471 0.953 ✓ PASS
urban 0.494 1.000 ✓ PASS
=================================================================
✓ No significant disparate impact detected.
Overall AUC dropped by only 0.003. Rural Disparate Impact went from 0.65 to 0.81, clearing the regulatory threshold. The model is now both accurate and fair, and the only thing that made this detectable was a synthetic database with controlled segment representation.
The Bias Detection Checklist
Before deploying any ML model that makes decisions affecting people, run the following against a segment-balanced synthetic database:
- Segment-level AUC computed separately for every protected or at-risk group
- Predicted approval/classification rate disaggregated by segment
- Disparate Impact ratio computed against reference segment (threshold: ≥ 0.80)
- Equalized Odds gap measured (true positive rate parity across segments)
- Retrain on augmented balanced data if any segment fails
- Revalidate fairness metrics after retraining before production approval
Why This Only Works with Synthetic Data
You cannot run this audit on real data alone for one simple reason: you cannot control real data.
If rural applicants make up 9% of your real dataset, your fairness audit on real data will be statistically underpowered for that group. Confidence intervals will be wide. The Disparate Impact calculation will be noisy. A model can slip through the audit by having high variance on a small group rather than genuinely passing.
A synthetic database with controlled segment proportions gives you exactly the sample size you need per group to make the fairness audit statistically sound. It is not a replacement for auditing on real data. It is the audit you run first, before the model ever sees real applicants.
The Bottom Line
Your ML model can be biased and still pass every accuracy metric you have. That is not a failure of accuracy metrics. It is a design limitation. Accuracy metrics were never built to catch distributional injustice. They were built to measure prediction correctness on the distribution you give them.
If you give them a biased distribution, they will give you a clean report.
Synthetic databases with explicit segment quota control break that loop. You decide what distribution the model gets tested on. You test it on populations that reflect reality, not history. And you find the bias before the bias finds your users.
Generate balanced. Audit early. Retrain deliberately.
Anything less is letting historical data write your model’s future.
Your AI Model Is Biased. Your Real Data Is Hiding It. Synthetic Databases Can Find It First. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.