data trove fossils pbdb

Overview

Source: /home/coolhand/html/datavis/data_trove/data/quirky/fossils.json

Saturn profiled 22,043 rows across 21 columns. The stats below are deterministic and machine-readable; the prose is a language-model interpretation of those stats (opt-in, added after the fact, never sees raw rows).

name text label

This column contains taxonomic names of fossil organisms — dinosaurs (Theropoda, Sauropoda, Hadrosauridae), conodonts (Palmatolepis, Polygnathus, Icriodus), and other paleontological taxa — making it a biological classification label rather than a unique specimen identifier. The duplicate rate is extremely high at 78.9%, with only 4,660 unique values across 22,043 rows, reflecting that many specimens share the same taxon name. Over half of values (58.5%) are single words (genus or clade names), with a mean word count of 1.42, consistent with Linnaean binomial or higher-rank nomenclature. The top value 'Theropoda' alone appears 768 times, confirming this is a categorical grouping field, not a unique label.

Treatment: Encode as a categorical feature (ordinal or target-encode by frequency); consider hierarchical grouping by taxonomic rank for modelling.

rank categorical label

This column encodes taxonomic rank in a biological classification dataset, with 18 distinct levels spanning the Linnaean hierarchy from species up through class and beyond. 'Species' dominates at 41.2% (9,082 rows) and 'genus' follows at 7,342 rows, which is the expected shape for a taxonomy tree where leaf nodes vastly outnumber higher groupings. Notably, 'unranked clade' appears 2,828 times—making it the third-largest category—indicating a substantial portion of entries reflect modern phylogenetic classifications that don't fit traditional Linnaean ranks. Entropy ratio of 0.50 signals moderate concentration, not uniform distribution.

Treatment: Ordinal-encode with a defined taxonomic hierarchy or one-hot encode for modelling; treat 'unranked clade' as a separate nominal category since it has no fixed position in the hierarchy.

lat numeric feature

This column contains geographic latitude values, with a valid range from -84.33° to 79.75° and a median of 41.70°, consistent with mid-northern hemisphere locations (e.g., Europe or northern US). The distribution is strongly left-skewed (skew = -2.44) with high kurtosis (7.05), indicating a heavy tail of unusually southern or southern-hemisphere values — 9.2% of rows (2,019 records) are flagged as outliers. The -84.33° minimum is suspicious as it approaches the Antarctic Circle and may represent data quality issues or encoding errors.

Treatment: Validate extreme negative values (especially near -84.33°) as likely errors; use as-is or pair with longitude for geospatial features after outlier review.

lon numeric feature

This column contains geographic longitude values, spanning from -176.7° to 177.1°, consistent with near-global coverage. The median of -98.25° and mean of -47.2° indicate a strong concentration of records in the Americas (particularly North/Central America), which explains the positive skew of 0.93 — the distribution is pulled rightward toward European/African longitudes. The wide IQR of 114° and only 4,259 unique values across 22,043 rows suggests coordinates are rounded or snapped to coarse grid points rather than being fully precise.

Treatment: Use as-is or pair with latitude for geospatial modelling; consider binning into regions if cardinality reduction is needed.

early_age_mya numeric feature

This column represents the early (older) age boundary of a geological time range in millions of years ago (Mya), likely for fossil taxa or stratigraphic intervals. With only 164 unique values across 22,043 rows, most records share standardized geological stage boundaries rather than continuous age estimates. The distribution is right-skewed (skew 1.13) with a mean of ~154.7 Mya but a median of only 110.1 Mya, and 11.6% of values (2,549 rows) are flagged as outliers—driven by a long tail extending to 538.8 Mya representing Cambrian or older occurrences against a bulk of Mesozoic/Cenozoic records.

Treatment: Use as-is or bin into geological periods; log-transform if used in regression due to right skew and wide range (0.0117–538.8).

late_age_mya numeric feature

This column records the late (younger) age boundary of a fossil taxon's stratigraphic range in millions of years ago (Mya), a standard field in paleontological occurrence databases. With only 156 unique values across 22,043 rows, ages are drawn from a discrete set of stage/interval boundaries rather than continuous measurements. The distribution is right-skewed (skew 1.17, mean 147.5 vs. median 93.9) and 11.5% of rows are flagged as outliers, driven by a long tail extending to 521 Mya — likely Cambrian/Ordovician taxa pulling the upper end — while the near-zero minimum (0.0, ~0.1% of rows) represents taxa surviving to the Recent.

Treatment: Use as-is or bin into geological periods; consider log-transform or pair with early_age_mya to compute range duration before modelling.

period categorical label

This column represents geological time periods or North American Land Mammal Ages (NALMAs), with values like 'Irvingtonian', 'Torrejonian', 'Tiffanian', and 'Puercan' — terminology specific to paleontology and fossil occurrence datasets. With 298 unique values across 22,043 rows and zero nulls, the distribution is moderately concentrated: the top value 'Irvingtonian' accounts for only ~7.8% of rows, while entropy ratio of 0.78 suggests meaningful but not extreme skew. The mix of formal geological stages (e.g., 'Kimmeridgian', 'Aptian', 'Hettangian') alongside NALMA names signals that multiple classification schemes coexist in this column, which could complicate grouping or ordering without a lookup table.

Treatment: Map to a standardized chronostratigraphic hierarchy or numeric age range using a geological timescale reference before modelling or time-series analysis.

late_interval categorical feature

This column encodes the geological time interval representing the late (upper) bound of a fossil or stratigraphic occurrence, drawn from standard chronostratigraphic stage names (e.g., 'Tithonian', 'Sinemurian', 'Albian'). The most striking signal is that 83.1% of the 22,043 rows carry an empty string as the top value, meaning the late interval is unspecified for the vast majority of records — this is a heavily sparse categorical field despite a zero null rate. The remaining 138 distinct values span Mesozoic and Cenozoic stages with modest frequency, suggesting the dataset skews toward certain time periods (Tithonian and Sinemurian together account for ~4.4% of all rows).

Treatment: Treat empty strings as missing; encode valid stage names ordinally by geological age order or map to numeric Ma midpoints before modelling.

phylum categorical label

This column contains biological phylum classifications drawn from exactly 4 distinct values across 22,043 records, with no nulls. It is heavily dominated by 'Chordata' at 81.6% (17,993 rows), while 'Mollusca' and 'Arthropoda' each account for exactly 2,000 records — a suspiciously round number suggesting deliberate sampling or stratification. Notably, 50 records carry an empty string value, which acts as a de-facto null and should be treated as missing rather than a valid category.

Treatment: Encode as ordinal or one-hot; replace 50 empty-string records with NaN and impute or exclude before modelling.

class categorical label

This column contains the biological/taxonomic class of fossil or paleontological specimens, with 19 distinct values across 22,043 records and no nulls. The top value 'Mammalia' accounts for 31.8% of records, followed by dinosaur orders 'Saurischia' and 'Ornithischia' — notably these are clades, not true classes, mixed in with proper classes like Mammalia and Reptilia, suggesting taxonomic rank inconsistency. Two sentinel-style values ('NO_CLASS_SPECIFIED' with 26 occurrences and 60 empty-string entries) represent ~0.4% of records and should be treated as missing. Entropy ratio of 0.61 indicates moderate concentration rather than a uniform spread.

Treatment: Unify empty strings and 'NO_CLASS_SPECIFIED' into a single missing category, then encode as a categorical feature or target depending on modelling objective.

order categorical label

This column contains biological taxonomic order classifications (e.g., Rodentia, Carnivora, Artiodactyla), likely from a paleontological or natural history specimen dataset. Two sentinel/missing-value patterns dominate: 'NO_ORDER_SPECIFIED' accounts for 32.3% of rows (7,117) and an empty string accounts for a further 3,019 rows (~13.7%), meaning roughly 46% of records lack a meaningful order assignment. Despite 99 unique values and moderate entropy (3.89), the effective signal is skewed toward these two non-informative categories.

Treatment: Consolidate 'NO_ORDER_SPECIFIED' and empty string into a single null/unknown category, then encode remaining values as a nominal categorical feature (e.g., target or ordinal encoding) before modelling.

family categorical label

This column contains biological family-level taxonomic classifications for fossil or specimen records, with 528 distinct family names across 22,043 rows. The most surprising signal is that the top value is an empty string (3,418 occurrences, 15.5% of rows), and the second most frequent value is the sentinel 'NO_FAMILY_SPECIFIED' (1,996 occurrences) — together these two non-informative values account for roughly 24.6% of the dataset, indicating pervasive missing family assignments. Substantive values include well-known paleontological families such as Hadrosauridae, Grallatoridae, and Palmatolepidae, confirming a paleobiology or fossil-occurrence context.

Treatment: Unify missing indicators by mapping empty string and 'NO_FAMILY_SPECIFIED' to a single null/unknown category, then encode as a categorical feature or use as a grouping label.

genus text label

This column contains biological genus names from a paleontological or natural history dataset, as evidenced by taxa such as Palmatolepis (conodonts), Grallator/Eubrontes (dinosaur tracks), Baculites (ammonites), and Equus (horses). The duplicate rate is extremely high at 88.2% (19,435 of 22,043 rows share a value), which is expected for a categorical taxonomic label with only 2,608 unique genera. Notably, 5,545 rows (25.2% of the dataset) have an empty string rather than a null — a data quality issue that should be treated as missing. The vocabulary of 2,525 single-word tokens aligns tightly with the 2,608 unique values, confirming these are clean, single-word Latin genus names.

Treatment: Replace empty strings with NaN, then encode as a categorical feature (e.g. label or target encode) or use as a grouping key for taxonomic analysis.

country categorical feature

This column contains ISO-style two-letter country codes across 93 distinct countries, with zero nulls across 22,043 rows. The distribution is heavily US-dominated: 'US' alone accounts for 50.9% of all records (11,218 rows), while the next largest country 'CA' has only 1,830 — a roughly 6× drop-off. The entropy ratio of 0.50 confirms the pronounced imbalance, meaning models treating this as a uniform categorical feature will be misled by the long tail of 83 countries each with very small counts.

Treatment: One-hot encode top countries and bin the long tail into an 'Other' category, or embed via target-encoding given the severe US skew.

state categorical feature

This column represents a geographic state or province field, but its 519 unique values far exceed the 50 US states, revealing a mix of US states, Canadian provinces (Alberta), English regions (England), and Chinese provinces (Guangxi) — indicating international scope. The top value is 'Wyoming' at 8.6% of rows, which is disproportionately high for a state with a small population, suggesting dataset bias or a specific collection source. Notably, 1,082 rows (roughly 4.9%) contain an empty string rather than a null, meaning the null_rate of 0.0 understates true missingness. The entropy ratio of 0.70 confirms meaningful but imperfect spread across categories.

Treatment: Standardize empty strings to null, map to ISO region codes, and consider grouping by country before encoding.

formation categorical other

This column, 'formation', is a categorical field that is entirely empty: all 22,043 rows contain a blank string, giving it a cardinality of 1 and an entropy of 0. There is no null rate, meaning the field was populated with empty strings rather than true nulls. It carries zero information and should be dropped.

Treatment: Drop entirely; column contains only empty strings across all 22,043 rows and provides no analytical value.

collection categorical other

This column is a categorical field named 'collection' that is entirely empty strings across all 22,043 rows — it has cardinality of 1 and a top_value of '' with top_rate of 1.0. There are no nulls, meaning the field was populated with blank strings rather than left absent. The column carries zero information (entropy = 0.0) and is completely useless for analysis in its current state.

Treatment: Drop this column; it is a constant empty string with no analytical value.

paleolat numeric feature

This column represents paleolatitude — the reconstructed geographic latitude of a fossil or geological sample at the time of its deposition, ranging from -86.16° (near the South Pole) to 89.2° (near the North Pole). The distribution is moderately left-skewed (skew = -1.08) with a mean of 26.46° but a median of 34.89°, indicating a pull toward southern or equatorial values in the tail. Notably, 6.97% of records (1,503) are flagged as outliers, likely representing polar or high-latitude specimens that are genuinely rare in the fossil record rather than data errors. The 3,214 unique values across 22,043 records suggests coordinate rounding or discrete sampling grids.

Treatment: Retain as-is for spatial or paleogeographic modelling; consider binning into latitudinal zones (e.g., polar, temperate, tropical) if used as a categorical feature.

paleolng numeric feature

This column represents paleogeographic longitude — the reconstructed east-west position of a sample location at the time of fossil deposition, typically ranging from −180° to +180°. The values span −177.6 to 168.7, consistent with valid global longitude, and the 3,715 unique values across 22,043 records suggest discretized but reasonably fine-grained spatial resolution. The distribution is moderately right-skewed (skew 0.737) with a wide IQR of 97.35°, indicating samples are spread broadly but with more density in the western/negative hemisphere — the median of −62.15 is well below the mean of −28.58. Null rate of 2.23% is minor but worth flagging for paleogeographic reconstructions where completeness matters.

Treatment: Use as-is for spatial modelling; consider sin/cos encoding if used in circular/angular context to handle the −180°/+180° boundary.

reference_no text foreign_key

This column is a reference number field — short numeric-looking codes (mean length 4.17 characters, max 5) that identify some entity like a case, order, or account. Despite the name implying uniqueness, the duplicate rate is extremely high at 83.1%, with only 3,725 unique values across 22,043 rows, meaning the same reference numbers recur many times (e.g., '4245' appears 794 times, '6294' 743 times). The allcaps alert (89.8%) is likely a false positive triggered by numeric-only strings. This column appears to be a grouping or foreign-key identifier rather than a row-level unique reference.

Treatment: Use as a grouping key or left-join foreign key; do not treat as a unique row identifier.

occurrence_no text identifier

This column is a unique occurrence identifier — likely a numeric reference code stored as text, given the all-caps alert (which reflects purely digit characters) and mean length of ~5.76 characters. All 22,043 rows are non-null, non-duplicate, and every value is a single token, confirming it as a primary key-style field. The values range from short (min length 1) to 7 characters and appear to be plain integers (e.g., '164260', '1439335'), with no patterns suggesting a structured prefix scheme.

Treatment: Retain as a row identifier for joins or traceability; drop from any model feature matrix.