data trove bfro bigfoot sightings full scrape

Overview

Source: /home/coolhand/html/datavis/data_trove/data/wild/bigfoot_sightings.json

Saturn profiled 5,411 rows across 9 columns. The stats below are deterministic and machine-readable; the prose is a language-model interpretation of those stats (opt-in, added after the fact, never sees raw rows).

id numeric identifier

This column is a numeric row identifier: all 5,411 values are unique, there are no nulls, and the zero rate is 0.0, consistent with a primary key or surrogate ID. The IDs are not sequential (range 60–79,711 with a mean of ~23,288 and median of ~16,598), suggesting they originate from a larger parent table or were assigned non-contiguously. Mild positive skew (0.91) indicates more records cluster in lower ID ranges, but the near-zero kurtosis (−0.15) and absence of outliers confirm a broadly spread, roughly uniform-ish distribution rather than a tightly clustered one.

Treatment: Retain as row key for joins/lookups; exclude from any predictive model as a feature.

state categorical feature

This column contains US state names, with all 50 states likely represented plus Washington D.C. and possibly territories (53 unique values, 0 nulls across 5,411 rows). Washington dominates at 11.7% (631 records), roughly 1.5× California's share (431), suggesting a dataset with geographic bias toward the Pacific Northwest. Entropy ratio of 0.877 indicates reasonably broad distribution across states, though concentration in a handful of large/coastal states is apparent.

Treatment: One-hot encode for tree models or ordinal-encode by region grouping; investigate the Washington overrepresentation relative to population before modelling.

state_code categorical feature

This column contains US state abbreviations (plus possibly DC and territories, explaining the 53 distinct values vs. 50 states). Washington ('wa') is notably over-represented at 11.7% of 5,411 rows — roughly 1.5× California ('ca') and nearly 2× Ohio ('oh') — suggesting a geographic skew toward the Pacific Northwest rather than a nationally representative sample. Entropy ratio of 0.877 indicates reasonably broad distribution across states, but the top-heavy concentration in 'wa' is worth flagging.

Treatment: One-hot encode or target-encode for modelling; investigate why 'wa' is over-represented relative to population share before training.

county text label

This column contains US county names, functioning as a categorical geographic label with 1,022 unique values across 5,411 rows. The duplicate rate is high at 81.1%, which is expected for a county field where many records share the same geography. Notably, 338 rows (6.2%) have empty strings rather than nulls, masking true missingness since null_rate reports 0.0. The top counties — Pierce, Jefferson, Lewis, Washington, Snohomish — suggest a Pacific Northwest-heavy dataset, but the presence of Humboldt and Polk hints at multi-state coverage.

Treatment: Replace empty strings with NaN, then encode as categorical (ordinal or target-encode) for modelling or use as a group-by key for geographic aggregation.

url text identifier

This column contains unique URLs pointing to individual report pages on bfro.net (the Bigfoot Field Researchers Organization database), all following the pattern `https://www.bfro.net/gdb/show_report.asp?id=`. Every one of the 5,411 rows holds a distinct URL (duplicate_rate: 0.0, n_unique: 5411), with lengths tightly clustered between 46 and 49 characters (len_mean: 48.56, len_median: 49.0), reflecting only variation in the numeric report ID. This column is effectively a primary or foreign key into the BFRO report database — not a content feature — and carries no modelling signal on its own.

Treatment: Retain as a row identifier or use to left-join additional scraped report metadata; drop from any feature matrix.

month categorical feature

This column represents calendar month names, but with a cardinality of 32 instead of the expected 12, there are clearly duplicate or variant entries beyond the standard month labels — likely encoding errors, alternate spellings, or appended year/year-month combinations. The distribution is notably skewed toward summer and early-autumn months (August 634, October 632, July 618), with winter months dramatically underrepresented (December 233, January 228), suggesting seasonal bias in data collection. The entropy_ratio of 0.761 across 32 unique values rather than 12 is a strong flag that this field is dirty and needs normalisation before use.

Treatment: Audit and collapse the 32 distinct values down to 12 canonical month names, then encode as an ordered cyclic feature.

year numeric timestamp

This column represents calendar years for records in the dataset, spanning 1870 to 2025 with 99 distinct values. The distribution is left-skewed (skew = -0.974) with a mean of ~1998 and IQR of 22 years (1987–2009), meaning the bulk of records cluster in the late 20th to early 21st century while a thin tail extends back to 1870. The 49 outliers (0.9%) likely correspond to those historically distant records, and analysts should verify whether pre-20th-century entries are genuine or data quality issues.

Treatment: Treat as an ordinal temporal feature; investigate the 49 outlier records (pre-~1960s) for validity before modelling.

classification categorical label

This column is a three-level ordinal or nominal classification label applied to all 5,411 rows with no nulls. The distribution is nearly balanced between 'Class B' (2,722; 50.3%) and 'Class A' (2,655; 49.1%), but 'Class C' is severely underrepresented with only 34 instances (~0.6%), which would surprise any analyst expecting a balanced multi-class target and will require oversampling or class-weight adjustments if used as a target variable.

Treatment: Use as classification target; apply class-weighting or oversampling to address severe 'Class C' imbalance (34 of 5411 rows).

description text free_text

This column contains short free-text descriptions of reported sightings — most likely UFO or wildlife sighting reports, inferred from the high-frequency terms 'sighting', 'near', and 'possible'. With 5,407 unique values out of 5,411 rows and zero nulls, entries are nearly all distinct; the 4 duplicates (duplicate_rate 0.00074) are negligible. Mean length of ~67 characters and ~10 words per entry suggests structured-but-natural one-line summaries rather than long narratives. Flesch readability of 55.7 indicates plain, accessible prose with a vocabulary of 7,169 unique tokens across the corpus.

Treatment: Tokenize and embed (e.g., TF-IDF or sentence transformer) before modelling; near-uniqueness makes direct encoding unusable.