quirky ufo sightings 20260121

Overview

Source: /home/coolhand/html/datavis/data_trove/cache/quirky/ufo_sightings_20260121.parquet

Saturn profiled 147,890 rows across 13 columns. The stats below are deterministic and machine-readable; the prose is a language-model interpretation of those stats (opt-in, added after the fact, never sees raw rows).

Sighting numeric identifier

With n_unique equal to n (147890) and zero nulls, this looks like a per-row sighting identifier rather than a measured quantity. The distribution is near-uniform across 111 to 179773 (skew -0.013, kurtosis -1.13, mean 91984 close to median 91435), consistent with a sequential or randomly assigned ID rather than a feature with signal.

Treatment: drop from modelling; retain only as a join key.

Occurred text timestamp

Timestamp strings of the form 'YYYY-MM-DD HH:MM:SS Local', with length tightly clustered at 25 characters (mean 24.96, p95 25) and 'local' appearing in roughly 20000 of 147890 rows. Duplicate rate is 14.6% (21626 repeats), and several July 4th evening timestamps recur dozens of times, suggesting event clustering around holidays and round hours (22:00, 21:00, 23:00 dominate). 299 rows contain only the bare token 'Local', indicating missing date/time portions encoded as a stub rather than nulls.

Treatment: Parse to datetime, drop the trailing 'Local' suffix, and treat bare 'Local' entries as missing.

Location text metadata

Short city/state/country strings — top values like 'Phoenix, AZ, USA' (770) and dominant token 'usa' (17880) make this a geographic location field. Heavy duplication (74.9%, 110819 rows across 37070 uniques) is expected for city-level data, and entries are short (mean 20.4 chars, 3.57 words). The 'multilingual' alert is mostly noise from place names: 4481 detected as English with small counts in de (144), es (86), fr (89), and others.

Treatment: Parse into city/state/country components and use as categorical geo features.

Shape categorical feature

Categorical descriptor of an observed object's shape, with 39 distinct values across 147,890 rows. 'Light' dominates at 19.4% followed by 'Circle' (14,367) and 'Triangle' (13,086), and the vocabulary is noticeably noisy: 'Other' (10,062) and 'Unknown' (10,021) together rival the top class, plus near-synonyms like 'Sphere', 'Orb', and 'Circle' coexist. Null rate is 4.29% and entropy ratio of 0.74 indicates moderately spread mass rather than a single peak.

Treatment: Consolidate synonyms (Sphere/Orb/Circle), bucket Other/Unknown as missing, then one-hot encode.

Duration text feature

This is a free-text duration field, almost always a number paired with a unit like 'minutes' or 'seconds' (mean length 9.5 chars, word_median 2). Values are highly repetitive: 89% are duplicates and only 15,527 uniques exist across 147,890 rows, with '5 minutes' alone appearing 8,813 times. The vocabulary of 1,757 tokens and presence of variants like 'min' alongside 'minutes' suggest inconsistent unit spellings, and 4.83% are null.

Treatment: Parse into a numeric seconds column by normalizing unit tokens (minute/min, second/sec) before modelling.

No of observers numeric feature

This is a count of observers per record, with a typical value of 1-2 (median 2.0, IQR 1.0) but a mean of 4.6 dragged up by extreme outliers reaching 20000. Skew of 109.3 and kurtosis above 14000 confirm a heavy right tail, and 13.0% of values are flagged as outliers. The minimum of -10.0 is implausible for an observer count and suggests data entry errors, while 6.9% are zero and 4.5% are null.

Treatment: Clip negatives, investigate the 20000 tail, and log1p-transform before modelling.

Reported text timestamp

This is a 'Reported' timestamp column stored as text, with every one of the 147,890 values exactly 27 characters long and following a 'YYYY-MM-DD 00:00:00 Pacific' pattern. Despite being temporal, it was profiled as text, which explains the spurious 'multilingual' alert (4,649 'en', 336 'de', etc.) from language detection misreading date tokens. Duplicates are expected for a date field (7.7%, 11,418 rows), with 1999-11-16 and 1999-11-17 the heaviest days at 73 and 69 occurrences.

Treatment: Parse to datetime (strip the ' Pacific' suffix) and discard the language alert.

Posted categorical timestamp

`Posted` holds 626 unique date-stamped values (all at 00:00:00) across 147,890 rows with no nulls, so it functions as a posting date rather than a precise timestamp. The distribution is broad and high-entropy (entropy ratio 0.93); the most common date, 2020-06-25, accounts for only 1.24% of rows, and the top 10 dates span 1999 to 2023 with no obvious clustering. Worth noting: there are far fewer distinct dates (626) than would be expected if rows were spread daily across two-plus decades, suggesting posts cluster on specific days.

Treatment: Parse to date and derive year/month/day-of-week features before modelling.

Characteristics text feature

This is a categorical multi-label feature describing observed object characteristics (e.g., 'Lights on object', 'Aura or haze around object'), encoded as comma-joined tags rather than free text. Despite 147,890 rows, only 1,446 unique values and a vocab of 43 words exist, with a 98.6% duplicate rate and 28.2% nulls. Note the truncated token 'Changed Colo' appearing repeatedly, suggesting an upstream string-truncation bug.

Treatment: Split on comma and one-hot encode the tag set; fix the 'Changed Colo' truncation before encoding.

Summary text free_text

Free-form English text summaries, near-unique across 147,890 rows (144,208 distinct), with a median of 76 characters / 13 words but a long tail reaching 10,624 characters. Top tokens are stopwords plus 'light' (4,676), and readability averages 67.3 Flesch, suggesting conversational prose rather than structured codes. Surprises: 2,791 exact duplicates (1.9%) despite the near-unique flag, 1.8% all-caps entries, and a 28,632-word vocabulary — emojis and URLs are negligible.

Treatment: Tokenize and embed (or TF-IDF) before modelling; dedupe the 2,791 exact repeats first.

Text text free_text

Free-text field averaging 950 characters and 182 words per entry, with 127,124 unique values across 147,890 rows and a 65,106-token vocabulary — consistent with longish English prose (mean Flesch 69.6, fairly readable). First-person pronouns 'i' and 'my' rank in the top 10, suggesting personal narratives or reviews rather than formal documents. Notable signals: 11.28% null rate, 3.12% exact duplicates (4,091 rows), and a max length of 64,550 characters versus a p95 of 2,673 — a small set of extreme outliers.

Treatment: Tokenize and embed before modelling; dedupe the 4,091 exact repeats and clip extreme-length outliers first.

Location details text free_text

Free-text location descriptions, populated for fewer than 7% of rows (null_rate 0.931) and otherwise short prose averaging 38.7 characters / 6.9 words with a Flesch readability of 69.8. Among the 10,212 non-null entries there are 9,713 distinct strings and only 492 duplicates (4.8%), so values are near-unique and unlikely to act as a category. Top tokens are stopwords ('the', 'of', 'in', 'my', 'over'), confirming sentence-style annotations rather than place names or codes.

Treatment: Treat as optional free-text annotation; tokenize/embed if used, otherwise drop given the 93% null rate.

Explanation categorical label

This is a categorical 'Explanation' field labeling probable causes for sightings (e.g., 'Starlink - Probable', 'Rocket - Certain', 'Balloon - Possible'), with 58 distinct values combining a source and a confidence tier. It is almost entirely empty: null_rate is 0.9946, leaving only a small annotated subset where the top label 'Starlink - Probable' appears 78 times (9.7% of non-nulls). Entropy ratio of 0.82 indicates labels are spread fairly evenly across categories rather than dominated by one.

Treatment: Treat as a sparse annotation; split into source and confidence sub-labels and analyze only the ~0.5% labeled rows.