quirky fossils

Overview

Source: /home/coolhand/html/datavis/data_trove/data/quirky/fossils.json

Saturn profiled 22,043 rows across 21 columns. The stats below are deterministic and machine-readable; the prose is a language-model interpretation of those stats (opt-in, added after the fact, never sees raw rows).

name text label

This column holds taxonomic names of fossil organisms, dominated by dinosaur and conodont genera/clades like Theropoda (768), Dinosauria (512), and Palmatolepis (376). Values are short—mean length 15 characters and 58% are single words—so the Flesch readability score of -4.13 is a meaningless artifact of scientific Latin. With 4,660 uniques across 22,043 rows and a 78.9% duplicate rate, this behaves as a categorical taxon label rather than a free-text field.

Treatment: Treat as a high-cardinality categorical; group rare taxa or target-encode before modelling.

rank categorical feature

This column records the taxonomic rank of each record, with 18 distinct values and no nulls across 22,043 rows. 'Species' dominates at 41.2% (9,082 rows), followed by 'genus' (7,342) and 'unranked clade' (2,828); ranks below family drop off sharply, with 'subfamily' already at only 272. The presence of 'unranked clade' alongside formal Linnaean ranks is worth noting as a non-standard category.

Treatment: One-hot or ordinal-encode by taxonomic depth before modelling.

lat numeric feature

This is a latitude coordinate in degrees, ranging from -84.33 to 79.75 with a median of 41.70 and IQR of 11.61. The strong negative skew (-2.44) and kurtosis (7.05) suggest most points cluster in the northern hemisphere with a long tail of southern-hemisphere outliers (9.16% flagged). No nulls and 4,095 distinct values across 22,043 rows indicate repeated locations rather than per-row unique geocoding.

Treatment: Pair with a longitude column for spatial features; avoid log-transform and keep raw degrees.

lon numeric feature

This column captures longitude coordinates, with values spanning -176.67 to 177.07, consistent with the full global range. The distribution is right-skewed (skew 0.93) and centered on a median of -98.25, suggesting a heavy concentration of records in the Western Hemisphere (likely the Americas). Only 4,259 unique values across 22,043 rows indicate repeated location points, and just 3 outliers were flagged.

Treatment: Pair with latitude for geospatial features; consider binning or projecting rather than using raw degrees in linear models.

early_age_mya numeric feature

Numeric column representing the early bound of an age estimate in millions of years (mya), spanning 0.0117 to 538.8 across 22043 rows with no nulls. The distribution is right-skewed (skew 1.13) with median 110.1 well below mean 154.67, and saturn flags 2549 outlier rows (11.6%) — consistent with a long Paleozoic tail above the Q3 of 201.4. Only 164 unique values suggest ages are bucketed to standard stratigraphic boundaries rather than continuous measurements.

Treatment: Consider log-transform or binning to stratigraphic periods before modelling given the heavy right tail.

late_age_mya numeric feature

Numeric column representing the younger bound of a geological age in millions of years (Mya), with 156 distinct values across 22,043 rows and no nulls. Distribution is right-skewed (skew 1.17) with median 93.9 well below the mean of 147.5, ranging from 0 to 521 Mya, and 11.5% of values (2,535) flagged as outliers on the high end. The bounded discrete value set suggests entries snap to standardized stratigraphic stage boundaries rather than continuous measurements.

Treatment: Keep as-is or pair with early_age_mya to derive a midpoint; consider log or sqrt transform before regression given the right skew.

period categorical feature

This column records geologic time periods or stages, with 298 distinct values across 22,043 rows and no nulls. The distribution is moderately spread (entropy ratio 0.78), but 'Irvingtonian' dominates at 7.8% (1,723 rows), followed by 'Late Campanian' and various Paleocene/Mesozoic stages. The vocabulary mixes broad and finely-subdivided stage names (e.g., 'Late Maastrichtian' vs. 'Aptian'), so granularity is uneven.

Treatment: Group rare stages or map to coarser epochs before one-hot or target encoding.

late_interval categorical feature

Categorical geological stage marking the late end of a fossil/sample's age range, with 138 distinct stage names like Tithonian, Sinemurian, and Late Campanian. The column is dominated by empty strings (83.1% of 22,043 rows), leaving only ~3,724 records with an actual stage; entropy ratio of 0.22 reflects this sparsity. Among populated values, Tithonian (548) and Sinemurian (430) lead, suggesting Mesozoic specimens are over-represented.

Treatment: Treat empty string as missing and either impute from a paired early_interval or use as a sparse categorical with an explicit 'unknown' level.

phylum categorical label

Taxonomic phylum label with only 4 distinct values across 22,043 rows. Chordata dominates at 81.6% (17,993 rows), with Mollusca and Arthropoda each at exactly 2,000 — suggesting deliberate sampling caps on the non-Chordata classes. 50 rows carry an empty-string phylum that is not counted as null.

Treatment: Recode the 50 empty strings to null and treat as a low-cardinality categorical; expect class imbalance if used as a target.

class categorical label

Taxonomic class label for what appears to be a paleontological/zoological occurrence dataset, with 19 distinct values across 22,043 rows and no nulls. Mammalia dominates at 31.8% (7,015), followed by the dinosaur clades Saurischia (5,507) and Ornithischia (2,811), suggesting a fossil-heavy sample. Note two sentinel-style entries that should be cleaned: 60 empty strings and 26 'NO_CLASS_SPECIFIED' rows.

Treatment: Normalize the empty strings and 'NO_CLASS_SPECIFIED' to a single missing token, then one-hot or target-encode for modelling.

order categorical feature

Taxonomic order assignments for what appears to be a paleontology/biology specimen dataset, mixing extinct groups (Ammonitida, Ozarkodinida, Multituberculata, Phacopida) with extant mammalian orders (Rodentia, Artiodactyla, Carnivora). Coverage is poor: 32.3% are the sentinel 'NO_ORDER_SPECIFIED' and another 3019 rows are empty strings, so roughly 46% of records lack a real order. Across 22043 rows there are 99 distinct values with entropy ratio 0.586, indicating a few orders dominate the long tail.

Treatment: Normalise empty strings and 'NO_ORDER_SPECIFIED' into a single missing category before one-hot or target encoding.

family categorical feature

Taxonomic family classification for biological specimens, with 528 distinct families across 22043 rows and no nulls. The top value is the empty string at 15.5% (3418 rows), and 'NO_FAMILY_SPECIFIED' adds another 1996 rows—together roughly a quarter of records lack a real family assignment. Among populated values, families like Hadrosauridae (689), Grallatoridae (593), and Palmatolepidae (586) dominate, suggesting a paleontological dataset.

Treatment: Normalize empty strings and 'NO_FAMILY_SPECIFIED' to a single missing category, then target- or frequency-encode before modelling.

genus text feature

This column holds taxonomic genus names for fossil records — single-word Latinate identifiers like Palmatolepis, Polygnathus, and Grallator dominate, with one_word_rate at 0.989. Of 22,043 rows, 5,545 are empty strings and duplicate_rate is 0.88 across 2,608 unique values, so the field is heavily repeated and a quarter of rows carry no genus at all. Length stats (mean 8.2, max 33) and a vocab of 2,525 are consistent with controlled scientific nomenclature rather than free text.

Treatment: Treat as a high-cardinality categorical: normalize case, encode empties as missing, and target/frequency-encode rather than one-hot.

country categorical feature

Two-letter ISO country codes across 93 distinct values with no nulls in 22,043 rows. The distribution is heavily US-dominated at 50.9% (11,218 rows), with CA, CN, UK, and ES rounding out a long tail; entropy ratio of 0.50 confirms the concentration. Worth noting 'UK' is used rather than the ISO-standard 'GB', which may complicate joins against canonical country tables.

Treatment: Normalize codes (e.g., UK→GB) and group the long tail before one-hot or target encoding.

state categorical feature

Geographic subdivision (state/province/region) with 519 distinct values spanning US states, Canadian provinces, English regions, and Chinese provinces — indicating an international dataset rather than US-only. Wyoming leads at 8.6% (1903 rows), followed by Montana and an empty string with 1082 occurrences that null_rate=0 misses. Entropy ratio of 0.70 shows a fairly even spread across the long tail.

Treatment: Treat empty strings as missing, then group-encode or target-encode given high cardinality.

formation categorical other

The `formation` column is constant: all 22043 rows hold the empty string, giving cardinality 1 and entropy 0.0. It carries no information and the top_value being "" suggests the field was never populated rather than genuinely categorical.

Treatment: Drop; single constant value provides no signal.

collection categorical metadata

The 'collection' column contains a single value across all 22043 rows, and that value is the empty string. Cardinality is 1, entropy is 0, and null_rate is 0.0, meaning the field is technically populated but carries no information. This is likely a vestigial schema field that was never filled in.

Treatment: Drop; constant empty value provides no signal.

paleolat numeric feature

Paleolatitude in degrees, ranging from -86.16 to 89.2 with a median of 34.89 — consistent with reconstructed latitudes of fossil or geological samples. The distribution is left-skewed (skew -1.08) and ~6.97% of values flag as outliers (1503 records), suggesting a heavy southern-hemisphere tail against a northern-hemisphere mode. Null rate is low at 2.23% and 3214 unique values across 22043 rows indicate substantial repetition, likely from shared site coordinates.

Treatment: Use as-is for geographic modelling; consider binning by hemisphere or absolute latitude given the left skew.

paleolng numeric feature

This is paleolongitude — reconstructed longitudinal coordinates of fossil/sample locations on ancient continental configurations. Values span the full longitudinal range (-177.6 to 168.7) with a mean of -28.58 and median of -62.15, indicating a leftward (western) concentration and moderate positive skew (0.74). Null rate is 2.23% and only 3 outliers are flagged, so the distribution is well-behaved within plausible geographic bounds.

Treatment: Use as a geographic coordinate feature; pair with paleolat and consider cyclic encoding since longitude wraps at ±180.

reference_no text foreign_key

Short numeric reference codes (length 1-5, mean 4.17, all single-word) stored as text. Despite the name 'reference_no', it is far from unique: 22,043 rows collapse to 3,725 distinct values with an 83.1% duplicate rate, and the top code '4245' alone appears 794 times. The 89.8% allcaps_rate is a quirk of the detector treating digit-only strings as uppercase.

Treatment: Treat as a categorical foreign key and left-join to the reference dimension; do not assume uniqueness.

occurrence_no text identifier

This is almost certainly a primary-key style identifier: every one of the 22043 rows holds a unique single-token value (n_unique == n, one_word_rate 1.0), with no nulls or duplicates. Lengths range from 1 to 7 characters and the top words are all numeric strings like '164260' and '1439335', so the field is stored as text but contains integer occurrence numbers. The 'allcaps' alert is a quirk of the detector treating digit-only strings as uppercase and can be ignored.

Treatment: Use as a row key for joins; do not feed into modelling.