Cross-Validation & Data Triangulation

Cross-Validation & Data Triangulation

The Core Principle

Never trust a single system. Never make decisions based on one chart. Always cross-compare.

This is not marketing advice - this is fundamental data science.

---

Why Cross-Validation Matters

Single System View = Blind Spots

Every system has limitations:

• Facebook Ads: Can’t track users who decline consent, iOS users with ATT disabled, ad blocker users
• Google Ads: Similar consent issues, conversion delay reporting
• PostHog/Analytics: Might miss server-side transactions, can have client-side tracking failures
• Billing System: Only sees completed transactions, not attempts or abandoned checkouts

No single system sees the full picture.

The Solution: Data Triangulation

Compare multiple systems to find truth:

Facebook Ads says: 80 conversions
PostHog says: 100 conversions
Billing system says: 95 conversions

Reality: ~95 conversions (billing is most accurate)
Facebook undercounting: ~16% (due to tracking consent)
PostHog overcounting: ~5% (possibly duplicate events or test transactions)

Now you know:

• Real conversion count: 95
• Facebook tracking gap: 16%
• PostHog has minor tracking issues to investigate

---

The Three-System Cross-Check

System 1: Ad Platforms (Facebook Ads, Google Ads)

What they track:

• Ad impressions
• Clicks
• Conversions (limited by consent)
• Ad spend
• ROAS (as they see it)

Limitations:

• Consent tracking (GDPR, iOS ATT, Android Privacy)
• Ad blockers
• Cross-device attribution issues
• 7-day/28-day attribution windows (miss longer conversion paths)
• Platform wants to show good results (bias toward optimistic attribution)

Use for:

• Campaign optimization (within platform)
• Creative/audience testing
• Understanding tracking limitations
• Rough directional guidance

Don’t use as sole source of truth - they undercount significantly

---

System 2: Attribution/Analytics (PostHog, Moon, Google Analytics)

What they track:

• All user sessions (first-party tracking)
• Full funnel events (landing, lead, conversion)
• User journeys across devices/sessions
• Original attribution (UTM parameters)
• Multi-touch attribution

Limitations:

• Client-side tracking can be blocked by privacy tools
• Requires proper implementation (if broken, all data is wrong)
• Can have duplicate events (if not deduped properly)
• Test transactions might pollute data
• Relies on cookies/local storage (can be cleared)

Use for:

• Campaign performance analysis
• Funnel optimization
• Attribution analysis
• User behavior insights
• Primary decision-making (with validation against billing)

Most accurate for attribution - first-party tracking with full context

---

System 3: Billing/Payment System (Stripe, Payment Processor, Internal Billing)

What they track:

• Actual transactions
• Money received
• Transaction IDs
• Card information
• Billing addresses
• Invoice records

Limitations:

• Only sees completed transactions (not attempts)
• Doesn’t see upper funnel (landing, leads)
• No attribution data (unless you pass it through)
• Can’t answer “which ad drove this sale” without external attribution

Use for:

• Revenue validation
• Transaction counting
• Reconciliation (comparing against marketing systems)
• Ultimate source of truth for “did money change hands?”

Most accurate for revenue - this is definitive, money doesn’t lie

---

The Two-Tier Validation System

Tier 1: Daily Check - Ad Platforms vs Marketing Attribution

Check every morning:

• Ad system dashboards (Facebook Ads, Google Ads)
• Your marketing attribution (PostHog, Moon, etc.)

Purpose: Real-time campaign decisions

Example:

Facebook Ads Dashboard: 80 conversions
PostHog Attribution:    100 conversions

Analysis: FB undercounting by 20% (consent gap)
Decision: Campaigns are working, keep running

Tier 2: Weekly/Bi-Weekly Check - Marketing Attribution vs Billing

Check weekly or bi-weekly:

• Marketing attribution system (PostHog, Moon)
• Billing/Product system (Stripe, payment processor, accounting)

Purpose: Validate tracking accuracy, find unattributed sales

Example:

Marketing Attribution: 100 sales
Billing System:        120 sales

Gap: 20 unattributed sales (16.7%)
Analysis: Some sales not tracked by marketing system

This is normal. Unattributed sales happen due to:

• Direct sales (customer called/emailed directly)
• Offline conversions
• Tracking failures
• Users who cleared cookies/changed devices
• Manual orders entered by team

Action:

• Document the unattributed percentage (e.g., 16.7%)
• Adjust marketing attribution expectations by this percentage
• Investigate if percentage is too high (>25%)

---

Daily Cross-Validation Workflow

Morning Routine: The Three-System Check

Step 1: Check Ad Platform (Facebook Ads)

Yesterday's Facebook Ads Dashboard:
- Ad Spend: $5,000
- Conversions: 80
- Reported ROAS: 4.0x
- Revenue (per FB): $20,000

Step 2: Check Attribution System (PostHog/Moon)

Yesterday's Attribution Data:
- Conversions attributed to FB: 100
- Revenue attributed to FB: $25,000
- ROAS: 5.0x

Step 3: Check Billing System

Yesterday's Transactions:
- Total transactions: 95
- Total revenue: $23,750
- FB-attributed transactions (via email match): 95

Step 4: Reconcile and Understand

Facebook:     80 conversions, $20,000 revenue, 4.0x ROAS
PostHog:     100 conversions, $25,000 revenue, 5.0x ROAS
Billing:      95 conversions, $23,750 revenue, 4.75x ROAS

Analysis:
- Reality: 95 conversions, $23,750 revenue, 4.75x ROAS
- Facebook undercounting: 16% fewer conversions (consent tracking)
- PostHog overcounting: 5% more conversions (need to investigate)
- Billing matches FB-attributed emails: All 95 traced to FB campaign

Decision:
- Actual ROAS is 4.75x (excellent, keep scaling)
- Facebook showing 4.0x (misleading, but expected due to tracking)
- Don't panic about FB's lower numbers - they're normal

---

Common Scenarios & How to Handle

Scenario 1: Facebook Shows Low Conversions, Your System Shows High

What you see:

• Facebook Ads: 50 conversions
• PostHog: 100 conversions
• Billing: 95 conversions

Analysis:

• Facebook is undercounting by ~47%
• PostHog is slightly overcounting by ~5%
• Billing shows reality: 95 conversions

Why this happens:

• iOS users declined tracking (Facebook can’t see them)
• GDPR consent rejections (Facebook can’t track)
• Ad blockers
• Users converted days later (outside FB attribution window)

Decision: ✅ Keep campaigns running - real conversions are happening ✅ Scale based on billing data ($4,750 revenue on $5,000 spend) ✅ Adjust FB expectations - know that FB will show ~50% of actual ❌ Don’t panic and pause - that would kill profitable campaigns

---

Scenario 2: Your System Shows More Sales Than Billing

What you see:

• PostHog: 100 conversions
• Billing: 80 conversions

Analysis:

• PostHog is overcounting by 25%
• Something is wrong with PostHog tracking

Investigation:

1. Check for duplicate events (same transaction firing twice)
2. Check for test transactions (internal testing pollution)
3. Check for abandoned checkouts counted as conversions
4. Check event timing (are you counting “checkout started” instead of “payment completed”?)

Fix:

• Dedupe events by transaction_id
• Filter out test emails/IDs
• Use correct conversion event (payment_completed, not checkout_initiated)
• Reconcile PostHog data with billing weekly

---

Scenario 3: Billing Shows More Sales Than Your System (Unattributed Sales)

What you see:

• PostHog: 100 conversions
• Billing: 120 conversions

Analysis:

• 20 unattributed sales (16.7%)
• This is normal and expected

Why unattributed sales happen:

1. Direct/Offline conversions

   • Customer called sales team directly
   • Customer emailed support and purchased
   • Manual order entry by team

2. Tracking limitations

   • User cleared cookies between landing and purchase
   • User switched devices (landed on mobile, purchased on desktop)
   • JavaScript tracking failed to fire
   • Adblocker prevented tracking pixel

3. Attribution window expired

   • User landed 60 days ago (outside typical attribution window)
   • Cookie expired before purchase

4. Edge cases

   • API/server-side purchases not tracked client-side
   • Third-party checkout flows without tracking
   • Gift purchases (different person buys)

What’s normal:

• 5-20% unattributed: Normal, acceptable
• 20-30% unattributed: High but manageable, investigate

• 30% unattributed: Too high, tracking is significantly broken

How to handle:

Step 1: Document the baseline

Week 1: Marketing 100, Billing 120 → 16.7% unattributed
Week 2: Marketing 95, Billing 115 → 17.4% unattributed
Week 3: Marketing 105, Billing 125 → 16.0% unattributed

Average: ~17% unattributed (consistent)

Step 2: Adjust marketing expectations

Marketing attribution shows: 100 sales
Actual (adjusted for unattributed): 100 / 0.83 = 120 sales

When calculating ROAS:
Reported: 100 sales, $50,000 revenue, $10,000 spend = 5.0x ROAS
Actual: 120 sales, $60,000 revenue, $10,000 spend = 6.0x ROAS

Step 3: Create “no-affid” / “unattributed” category

• Marketing attribution: 100 sales
• Billing: 120 sales
• Create segment: “Unattributed / No-affid traffic” = 20 sales
• Report this segment separately
• Don’t ignore it - it’s real revenue

Step 4: Investigate only if percentage spikes

Weeks 1-10: 15-17% unattributed (normal)
Week 11: 35% unattributed (spike!)

→ Something broke in Week 11, investigate tracking

Don’t try to eliminate all unattributed sales - some level (5-20%) is normal and unavoidable.

---

The Facebook Consent Gap: Understanding Undercounting

Why Facebook Undercounts

Tracking requires user consent:

1. iOS 14.5+ ATT (App Tracking Transparency)

   • Users must opt-in to tracking
   • ~60-70% of iOS users decline
   • Facebook can’t track these conversions

2. GDPR Cookie Consent

   • European users must consent to cookies
   • Many decline
   • Facebook can’t track them

3. Ad Blockers

   • Block Facebook Pixel
   • Block conversion tracking
   • Facebook sees click but not conversion

4. Attribution Windows

   • Facebook defaults to 7-day click, 1-day view
   • Users who convert after window aren’t counted
   • Your system tracks them forever

Result: Facebook typically sees 50-80% of actual conversions

Adjusting for the Gap

Calculate your consent gap:

Consent Gap = (Your System Conversions - Facebook Conversions) / Your System Conversions

Example:
Your System: 100 conversions
Facebook: 60 conversions
Consent Gap: (100 - 60) / 100 = 40%

Facebook undercounts by 40% in your case

Apply adjustment when reading Facebook data:

Facebook shows:
- 60 conversions
- $30,000 revenue
- $10,000 ad spend
- 3.0x ROAS

Actual (with 40% adjustment):
- 100 conversions (60 / 0.6)
- $50,000 revenue ($30,000 / 0.6)
- $10,000 ad spend (same)
- 5.0x ROAS

Decision: Campaign is profitable at 5.0x ROAS, scale aggressively

Track this adjustment over time:

• Week 1: 35% gap
• Week 2: 40% gap
• Week 3: 45% gap (iOS update rolled out)
• Adjust expectations accordingly

---

Reconciliation Best Practices

Weekly/Bi-Weekly Reconciliation Process

Every week (or every two weeks), reconcile:

1. Export data from all systems

   • Facebook Ads: Conversions by campaign
   • PostHog: Conversions attributed to Facebook
   • Billing: Transactions with FB attribution

2. Compare at campaign level

Campaign 120236311971890504:

Facebook:     50 conv, $25,000 revenue
PostHog:      80 conv, $40,000 revenue
Billing:      75 conv, $37,500 revenue

Gap Analysis:
- FB undercount: 33% (expected, consent issues)
- PostHog overcount: 6% (minor, investigate)
- Reality: 75 conv, $37,500 (per billing)

3. Investigate discrepancies

   • PostHog 6% over → Check for duplicate events
   • FB 33% under → Normal consent gap, document

4. Adjust decision-making

   • Use billing numbers for ROAS calculation
   • Use PostHog for attribution to specific ads
   • Ignore Facebook numbers except for platform optimization

Monthly Deep Dive

Once a month, deep reconciliation:

1. Match transactions to users

   • Pull all billing transactions
   • Match each to PostHog user via email
   • Verify attribution is correct

2. Audit attribution accuracy

   • 95% of transactions should match to users
   • 95% of matched users should have attribution
   • If lower, fix tracking

3. Document system changes

   • Did conversion rate in PostHog suddenly change?
   • Check if schema evolved (new parameters added/removed)
   • Track impact of system updates

---

Key Takeaways

1. Never trust a single system - always cross-validate
2. Billing is ultimate truth - money doesn’t lie
3. Ad platforms undercount - consent tracking limitations (typically 20-50% gap)
4. Your attribution system is source of truth - first-party tracking sees more
5. Daily cross-check - FB vs PostHog vs Billing every morning
6. Understand the gaps - each system has known limitations
7. Adjust for consent - factor in Facebook’s ~40% undercount
8. Don’t panic on platform data - check your system first before stopping campaigns
9. Weekly reconciliation - keep systems in sync
10. Fix discrepancies - if your system disagrees with billing, investigate

Decision-making priority:

1. Billing system (revenue reality)
2. Attribution system (attribution + funnel insights)
3. Ad platform (directional guidance only)

Never make decisions on Facebook data alone. Always cross-validate.

---

The Micro-Macro Balance

You Need Both Perspectives

Micro level (event-by-event):

• Individual JSON footprints
• Parameter schemas
• User journeys
• Session recordings
• Raw event data

Macro level (system-by-system):

• 100 vs 110 vs 120 comparisons
• Ad system vs Marketing system vs Billing system
• Trend lines over time
• Percentage gaps between systems

You can’t just focus on one - both are critical:

Micro Without Macro = Missing the Big Picture

Problem: Deep in event parameters but missing system drift

Example:

• You’re analyzing individual events, checking footprints
• Events look correct, attribution seems right
• But you never compare totals across systems
• Marketing shows 100 sales, Billing shows 150 sales
• 50% of revenue is untracked and you don’t know

Result: Making decisions on incomplete data

Macro Without Micro = Blind to Root Causes

Problem: Comparing totals but don’t understand why they differ

Example:

• Marketing: 100 sales
• Billing: 120 sales
• You know there’s a 20% gap
• But you don’t check raw events to understand why
• Could be: duplicate events, test transactions, missing tracking, attribution bugs
• You can’t fix what you don’t understand

Result: Gap persists, gets worse over time

The Right Approach: Both Levels Always

Macro check (weekly):

Ad Platform:   80 conversions
Marketing:    100 conversions
Billing:      120 conversions

Gaps identified:
- Ad platform -20% (consent)
- Marketing -17% (unattributed)

Micro investigation (when gaps change):

Gap went from 17% → 35%

Drill down:
- Export 20 unattributed sales from billing
- Check their emails in PostHog
- Find: 15 have no PostHog record at all
- Check: Tracking pixel failed on checkout page
- Fix: Add server-side tracking backup

Both levels working together:

• Macro tells you “something changed”
• Micro tells you “here’s why and how to fix”

The Daily Practice

Every morning:

1. Macro check (2 minutes): FB vs PostHog vs Billing totals
2. Spot check 5-10 events (3 minutes): Verify parameters look right
3. If discrepancy found: Deep dive into micro (event analysis)

Every week:

1. Full macro reconciliation (10 minutes): All systems compared
2. Trend analysis (5 minutes): Are gaps growing or shrinking?
3. If gaps changed >5%: Deep micro investigation (30-60 minutes)

You’re always cautious about both levels - macro trends and micro details.

Warning Signs at Each Level

Macro warning signs:

• Gap between systems suddenly increases (15% → 30%)
• One system shows growth while others show decline
• Billing revenue doesn’t match marketing attribution revenue

Micro warning signs:

• Events missing expected parameters
• Parameter formats changed unexpectedly
• Duplicate events appearing
• Test emails in production events

Both matter. Watch both closely.