data trove us disasters mashup

Overview

Source: /home/coolhand/html/datavis/data_trove/data/wild/disasters/disasters_mashup.json

Saturn profiled 54,575 rows across 16 columns. The stats below are deterministic and machine-readable; the prose is a language-model interpretation of those stats (opt-in, added after the fact, never sees raw rows).

category categorical label

This column is a disaster/incident type label with exactly 4 categories: aviation_accident, storm, earthquake, and shipwreck. The distribution is notably skewed — aviation_accident dominates at 59.4% of all 54,575 rows (32,410 records), while earthquake and shipwreck are each underrepresented at roughly 6.7% apiece. The entropy ratio of 0.74 confirms meaningful but unbalanced spread across classes, which could bias classifiers trained on this target without resampling.

Treatment: Use as classification target; apply class-weighting or oversampling for minority classes (earthquake, shipwreck) before modelling.

latitude numeric feature

This column contains geographic latitude values ranging from -77.42 to 82.17, consistent with global coordinate data. The median of 38.38 and IQR of 9.12 suggest the bulk of records cluster around mid-latitude Northern Hemisphere locations (roughly US/Europe), but the negative minimum (-77.42) indicates some Southern Hemisphere entries. Highly surprising is the negative skew of -2.51 combined with extreme kurtosis of 15.97 and 4,302 outliers (7.88% of rows), pointing to a heavy tail of anomalous low-latitude or Southern Hemisphere observations that likely warrant geographic subsetting or anomaly review.

Treatment: Retain as-is for geo-spatial modelling; investigate the 4,302 outliers for data quality issues before binning or clustering by region.

longitude numeric feature

This column contains geographic longitude values, spanning from -179.28° to +178.83°, consistent with worldwide coordinates. The mean (-92.97°) and median (-92.81°) are tightly clustered in the central United States, suggesting the bulk of records are North American, yet 4,320 outliers (7.9% of rows) and an extreme kurtosis of 15.13 indicate a heavy-tailed distribution with a substantial minority of globally dispersed points. The positive skew of 2.84 confirms an asymmetric pull toward higher (less-negative or positive) longitude values, i.e., non-US locations.

Treatment: Retain as-is for geospatial modelling; consider pairing with latitude and clustering by region to handle the bimodal/heavy-tailed distribution before feeding into non-spatial models.

name text label

This column contains descriptive incident or event names, predominantly aviation accidents and natural disaster events (floods, tornadoes). The duplicate rate is strikingly high at 62.3% — with 33,988 duplicates across only 20,587 unique values out of 54,575 rows — largely driven by generic labels like 'Unnamed Wreck' (2,184 occurrences) and repeated aircraft model patterns (e.g., 'Aviation Accident - CESSNA 172' variants). While 86.6% of detected-language tokens are English, 14 other languages appear (French: 60, German: 58, Spanish: 46, Japanese: 32), indicating a multilingual dataset that may require language-aware processing.

Treatment: Normalize case variants (e.g., 'CESSNA 172' vs 'Cessna 172') before grouping or embedding; treat as a categorical label with high cardinality rather than free text.

date text timestamp

This column contains date strings in ISO-8601 format (YYYY-MM-DD), stored as text rather than a native date type. Nearly all top values fall on January 1st of their respective years (2002–2012), suggesting dates are truncated or snapped to year-start, which is analytically significant and likely not raw event timestamps. The duplicate rate is extremely high at 81.99%, consistent with annual granularity across 54,575 rows, and 9,264 unique values hint that some finer dates do exist beyond the dominant Jan-1 entries. Null rate is low at 5.74%.

Treatment: Parse to date type, investigate year-start snapping before using as a time feature, and consider extracting year as an ordinal variable.

subcategory categorical label

This column is a categorical event subcategory, most likely classifying incident or hazard reports across domains such as aviation, geophysical (seismic), meteorological (Tornado, Flash Flood, Thunderstorm Wind), and maritime events. 'aviation' dominates heavily at 59.4% of all 54,575 rows, creating pronounced class imbalance. A subtle data quality issue is present: some values use title case ('Tornado', 'Flash Flood', 'Thunderstorm Wind', 'Hail') while others are fully lowercase ('aviation', 'seismic', 'maritime'), suggesting records were ingested from at least two inconsistently formatted sources. Entropy ratio of 0.49 confirms the distribution is far from uniform.

Treatment: Normalize casing before use, then one-hot encode or target-encode accounting for the heavy 'aviation' majority (59.4%).

magnitude categorical feature

This column represents earthquake or seismic event magnitude, stored as a categorical/string type despite being a numeric measurement with 291 distinct decimal values (e.g., 4.5, 4.6, 4.7). Two signals demand attention: the null rate is extremely high at 80.09%, meaning only ~10,866 of 54,575 rows carry a value. The dominant value '0' accounts for 35.56% of non-null records (3,863 occurrences), which is likely a sentinel or placeholder rather than a true zero magnitude, since genuine zero-magnitude events would be vanishingly rare and the next most frequent values cluster around 4.5–5.1.

Treatment: Cast to float after replacing '0' sentinel values with NaN; investigate whether 80.09% nulls are structurally missing or data quality issues before imputing or dropping rows.

fatalities categorical feature

This column represents a fatality count per incident, stored as a categorical/string type despite being numeric in nature. The null rate is severe at 72.94%, meaning nearly three-quarters of records have no value recorded — this is the primary alert. Among non-null values, the distribution is heavily right-skewed: '0' dominates at 69.1% of non-null rows, with counts dropping sharply through 49 distinct values, indicating rare but high-fatality events exist in the tail.

Treatment: Cast to integer, treat nulls as unknown (not zero), then apply log1p-transform or use as-is for count-based modelling given heavy zero-inflation.

injuries categorical feature

This column represents an injury count per incident, stored as a categorical type despite containing integer values (0, 1, 2, …). The dominant concern is an extreme null rate of 72.94%, meaning nearly three-quarters of rows carry no injury data at all. Among non-null rows, the value '0' accounts for 68.14% of responses, indicating most recorded incidents involved no injuries, with a long tail reaching at least 178 distinct values — suggesting occasional high-casualty outliers.

Treatment: Cast to integer, impute or flag nulls explicitly, then consider log-transform or treat as a count target given heavy zero-inflation.

damage text feature

This column contains abbreviated monetary damage estimates (e.g., '2.5M', '250K', '0.00K') stored as free-form text, most likely representing financial loss or property damage figures from incident or insurance records. The null rate is extremely high at 72.94%, meaning nearly three-quarters of rows carry no damage value. The all-caps rate of 87.2% and one-word rate of 100% confirm a consistent but non-numeric encoding; the 1,014 unique values across 54,575 rows with a duplicate rate of 93.1% indicate a relatively coarse discrete scale. Analysts should note that string suffixes (K vs M) encode magnitude and must be parsed before any quantitative use.

Treatment: Parse magnitude suffixes (K=thousands, M=millions) and convert to a numeric column; impute or flag the 72.94% nulls before modelling.

state categorical feature

This column contains US state names (full uppercase spellings), acting as a geographic feature for records in the dataset. The critical issue is a 72.94% null rate, meaning nearly three-quarters of all 54,575 rows carry no state value — this is a severe missingness alert. Among non-null values, cardinality is 65 (slightly above 50 US states, suggesting territories or data anomalies), and distribution is moderately spread (entropy ratio 0.86) with Texas as the dominant value at 9.82% of non-null records.

Treatment: Investigate missingness mechanism before use; consider imputation or missingness indicator flag, and audit the 65 unique values to identify non-standard entries beyond the 50 states.

aircraft_type text label

This column contains aircraft make-and-model designations (e.g., 'Cessna 172', 'Piper PA-28-140') from what appears to be an aviation incident or registration dataset. Two major surprises: first, 40.6% of rows are null, indicating substantial missing coverage; second, case inconsistency is severe — 'CESSNA 172' (360 occurrences) and 'Cessna 172' (189 occurrences) are counted as distinct values despite being the same aircraft, with ~49.5% of values in all-caps, inflating n_unique (9,478) and the duplicate rate (70.8%) artificially. The top words confirm a GA-heavy dataset dominated by Cessna, Piper, and Beech.

Treatment: Normalize case (lowercase or title-case), then deduplicate/consolidate variant spellings before using as a categorical feature or grouping key.

event_id text foreign_key

This column is an aviation or safety incident event identifier — the 14-character format (e.g., '20010519X00967') encodes a date prefix followed by an alphanumeric case code, consistent with NTSB accident/incident IDs. Two signals are surprising: a null rate of 40.61% means nearly half of rows lack an event ID entirely, and the duplicate rate of 18.46% (5,983 duplicates across 26,427 unique values) indicates multiple rows share the same event ID, implying a one-to-many relationship where each event spawns several records. All values are exactly 14 characters and fully uppercase, confirming a tightly controlled format with no malformed entries.

Treatment: Left-join on this ID to an events dimension table; investigate and handle the 40.61% null rate before joining.

vessel_type categorical feature

This column categorizes the type of vessel involved in an incident or record, with 23 distinct values including 'ship', 'submarine', 'aircraft', and oddly 'car'. Two major data quality issues stand out: the null rate is extreme at 93.31%, meaning only ~3,700 of 54,575 rows carry any value, and the top recorded value is an empty string (3,311 occurrences), which inflates the apparent top_rate to 90.6% — suggesting the true fill rate is even lower than the null_rate implies. The long-tail alert is consistent with rare values like 'schooner' (2), 'sailboat' (2), and 'steamer' (1), while 'car' appearing as a vessel type signals potential data entry errors or schema misuse.

Treatment: Treat empty strings as nulls, impute or exclude before modelling given 93.31% missingness, and audit 'car' and 'aircraft' entries for schema validity.

cargo categorical feature

This column records the type of cargo carried by vessels or vehicles, with 17 distinct categories including 'human', 'timber', 'coal', 'fertilizer', and 'fish'. It is overwhelmingly sparse: 93.31% of rows are null, and among the non-null rows the top value is an empty string (3,632 occurrences), meaning genuinely populated values number only in the single digits each. The entropy ratio of 0.018 confirms near-total concentration, and the presence of a German-language entry ('Fischkutter (Stahl)') signals a language mix in the rare populated records.

Treatment: Exclude from modelling unless the non-null subset is the analytic focus; treat empty strings as nulls, consolidate language variants, and flag the 93.31% missingness as likely structural (field not applicable to most records).

depth_km unknown feature

This column represents earthquake or geological event depth in kilometres, a continuous numeric feature. The profiler skipped analysis entirely, so no distribution statistics, uniqueness counts, or range information are available. With 54,575 rows and a null rate of 0.0, the data is fully populated, but nothing can be said about skew, outliers, or value range from this evidence alone. An analyst should inspect the column directly before modelling.

Treatment: Manually profile for range and skew; apply log-transform if depth distribution is right-skewed before regression.