curated vs firehose

The query column is fully populated in curated (null_rate 0.0 across 279196 rows, 489 unique values led by 'firefaerie81.bsky.social' at 2.4%) but entirely null in firehose (null_rate 1.0 over 125645 rows, flagged all_null). This yields a null_rate_delta of 1.0 and a top_value_jaccard of 0.0, meaning the two slices share no overlap on this field. Curated also shows a healthy entropy of 7.68, while firehose has no distribution to compare. high · anthropic:claude-opus-4-7

The columns diverge completely: curated has a 0.0 null_rate across 279196 rows with url_rate 1.0, while firehose is entirely missing (null_rate 1.0 over 125645 rows, flagged all_empty). With null_rate_delta of 1.0 and top_value_jaccard at 0.0, there is no overlap to compare further. Curated also shows one_word_rate 1.0 and a duplicate_rate of 0.0665 (18560 duplicates), but firehose provides no values to contrast. high · anthropic:claude-opus-4-7

firehose alt_text is far dirtier and more skewed than curated: duplicate_rate jumps from 0.086 to 0.200 and the median length collapses from 127 to 84 even as the mean climbs to 281.9 and p95 nearly doubles to 1284, indicating a long tail of very long entries alongside many short repeats. Readability also craters (flesch_mean 67.3 vs 22.7) while url_rate (0.070 vs 0.010) and one_word_rate (0.063 vs 0.002) spike. The slices barely overlap on language mix (language_jaccard 0.35) or frequent values (top_value_jaccard 0.03), with firehose introducing ar, ceb, ko, he and others absent from curated. high · anthropic:claude-opus-4-7

firehose records carry a url_rate of 0.4649 versus curated's 0.1494, tripling link density and triggering its url_heavy alert. Language mix also diverges sharply (language_jaccard 0.25): firehose adds de, fi, fr, ja, ru, and tr on top of en/pt, earning the multilingual alert, while curated is effectively English-only (396 en vs 2 pt). firehose payloads are longer too (len_mean 1290.7 vs 1136.37, len_p95 2774 vs 2354) with a far more negative readability_flesch_mean of -422.28 vs -155.70, and its one_word_rate of 0.0242 dwarfs curated's 0.0005. high · anthropic:claude-opus-4-7

The image_thumb_url column is fully populated in curated (null_rate 0.0) but entirely missing in firehose (null_rate 1.0, all_empty alert), a null_rate_delta of 1.0. Curated also shows url_rate 1.0 across 279196 rows with 260636 unique values, while firehose contributes no values at all, yielding a top_value_jaccard of 0.0. The two slices share no overlap on this field. high · anthropic:claude-opus-4-7

The author_handle column is fully populated in curated (null_rate 0.0 across 279196 rows) but entirely missing in firehose, which triggers an all_null alert at null_rate 1.0 over 125645 rows. As a result, curated's 489 unique handles and top value 'firefaerie81.bsky.social' (top_rate 0.02446, entropy 7.6765) have no counterpart in firehose, yielding a top_value_jaccard of 0.0. This is a structural divergence: firehose is not carrying author_handle at all. high · anthropic:claude-opus-4-7