Integrated Datasets and Architecture of HERO

The HERO system organizes humanitarian data and its analytical components around a unified, ready-to-use structure. The framework combines various dimensions of risk (IPC, conflict, displacement, meteorological data, market prices, media signals, and vegetation health) by directly aligning them with food security assessment periods.

Systemic Data Missingness & Blackout Dynamics

The HERO analytics engine performs a two-dimensional missingness diagnosis across all 7 target data sources. By mapping missing values into binary shadow indicators (1 for NaN, 0 for observed), the pipeline uncovers two distinct failure patterns to guide targeted imputation strategies:

Random Point Missingness: Unsynchronized, stochastic dropouts where single, isolated values drop missing across datasets without a systemic cause—typically caused by minor sensor noise, momentary signal jitter, or transient network packet loss.
Spatial Coverage Blackouts: Extended, contiguous time blocks where all sensors across a specific geographic location drop offline simultaneously, creating complete "dark zones" across entire physical regions.

Shadow Matrix Analysis: Two-Step Analytical Workflow

Data unavailability is rarely random. This workflow isolates structural dependencies between sensors and tracks regional data degradation to prevent operational bias during modeling:

STEP 1: STRUCTURAL CORRELATION

Cross-Dataset Dependencies

Correlation Matrix Computation:
Calculates a pairwise Pearson correlation matrix across the 7 binary shadow variables to measure co-missingness independently of spatial location.
Co-Failure Pattern Identification:
High positive correlations (approaching +1.0) reveal systematic co-failing datasets—such as cases where conflict escalations cause simultaneous losses in both market prices and displacement data.
Bidirectional Heatmap Rendering:
Generates a correlation matrix to visually isolate interconnected clusters of missingness.

STEP 2: GEOGRAPHICAL BLACKOUT

Spatial Failure Patterns

Spatial Aggregation:
Groups the shadow matrix by Country and calculates the column mean to determine the exact percentage missingness rate per sensor within each national jurisdiction.
Severity Ranking & Scoring:
Sums missingness rates across all 7 indicators to compute a total_failure_score, sorting nations from highest to lowest overall data availability collapse.
Unidirectional Heatmap Rendering:
Generates a spatial heatmap to pinpoint geographical blackouts where multiple data streams systematically fail simultaneously.

Missingness Analysis: Two-Step Analytical Workflow

Data unavailability is rarely random. This workflow tests whether missing data points structurally depend on other observed features (MAR) or occur purely stochastically (MCAR), guiding the appropriate imputation strategy:

STEP 1: MULTIVARIATE MODELING

Logistic Regression & Predictability of Absence

Dummy Target Creation:
Maps data unavailability into a binary indicator (1 if missing/NaN, 0 if observed).
Classifier Training:
Trains a Logistic Regression model using all other numerical features to predict target missingness on a test split.
ROC AUC Evaluation:
An AUC > 0.6 demonstrates predictable missingness and rejects MCAR (supporting MAR/MNAR). An AUC near 0.5 indicates stochastic missingness.

STEP 2: UNIVARIATE VERIFICATION

Welch's T-Test Across Individual Features

Group Separation:
Splits the dataset for each predictor into two distinct cohorts: missing target group vs. observed target group.
Mean Difference Testing:
Executes an independent sample Welch's t-test (unequal variances) to isolate statistical shifts between both cohorts.
Statistical Significance:
A p-value < 0.05 highlights that a specific predictor directly influences missingness likelihood, supporting the MAR hypothesis.

Key Findings: Structurally Coupled Data Missingness

Data gaps in the HERO pipeline are structurally coupled rather than randomly distributed. When missingness occurs, multiple indicators collapse simultaneously due to shared real-world failure mechanisms.

1. Structural Topology (Correlation Heatmap)

The infrastructure collapses into isolated functional blocks:

Environmental Sensors

Strong Correlation (r = 0.92)

A high correlation of 0.92 between missing_NDVI and missing_CHIRPS. The failure of optical vegetation monitoring is directly accompanied by rainfall measurement blackouts.

Socio-Political Sensors

Critical Correlations (r = 0.60 - 0.65)

Critical correlations between ACLED and WFP (0.65) as well as ACLED and IDP (0.60). When tracking of armed conflicts is lost, market reporting and internally displaced person metrics collapse simultaneously.

## 2. Temporal Dynamics (Time Series)

Shadow vectors over time confirm distinct underlying causes for missingness:

Cyclic Environmental Pattern (MAR)

Periodic Natural Interference

NDVI and CHIRPS exhibit periodic and overlapping blackouts (missingness rates generally < 0.4), attributable to consistent atmospheric and optical barriers over time.

Synchronous Institutional Collapse (MNAR/MAR)

Catastrophic Pipeline Failures (Peaks at 1.0)

ACLED, WFP, and IDP time series record sudden, prolonged, and simultaneous infrastructure collapses. The lack of periodicity rules out natural noise and highlights severe failures in institutional data pipelines.

## 3. Geographic Fragmentation

Spatial distribution reveals structural gaps that standard imputation cannot address:

ACLED Isolation

Systemic Failure Rate 1.0 (17/18 Countries)

A systemic failure rate obliterates data across 17 out of 18 nations. Data survives exclusively in CAF (Central African Republic) and partially in KEN. This represents a purely MNAR dropout driven by geopolitical barriers.

IDP Blackouts

Deterministic Data Absence

A similar dynamic affects internally displaced persons data, which is deterministically missing across large dataset clusters (e.g., SLV, GTM, ZWE).