# 04_clone_matching — Discovery Demand Row-Clone Analysis (hashpack)

## Scope filter (cohort definition)
This pack analyzes **hashpack** documents where:

1. `filing_type = 'Demand or Request for Discovery'`
2. `case_id <> '27-CR-23-1886'`

Cohort view: `clone04_discovery_docs`

## What is being matched
Each document contains `hashpack.rows.rows[]`, where each element represents one cleaned pdftotext row:

- `text_clean` (human-readable cleaned text)
- `row_sha256` (hash of cleaned text)
- `page_num`, `row_in_page`
- `tags[]` (classification hints like header/boilerplate/digit-heavy)

The core duplication signal is: **the same `row_sha256` appears across multiple documents**.

## Outputs (CSV)

All exports are written under:
`reports/04_clone_matching/`

1. `04_discovery__doc_occurrences.csv`
   - One row == one document (occurrences table)
   - Includes: standard columns (case_id, cluster_name, cluster_id, filing_date, filing_type, pdf_filename)
   - Plus: total_rows, cloned_rows, unique_rows, and %s

2. `04_discovery__rowhash_stats_dupe.csv`
   - One row == one cloned row-hash (appears in 2+ docs)
   - Includes: first/last filing dates and span_days

3. `04_discovery__top_rowhash_text.csv`
   - Top-N most duplicated row hashes, with human-readable text

4. `04_discovery__rowhash_long_span.csv`
   - Row hashes whose first↔last appearance spans **>= 1 year**

5. `04_discovery__cluster_stats.csv`
   - Aggregation by `cluster_id/cluster_name`

6. `04_discovery__year_stats.csv`
   - Aggregation by filing year (doc-level clone % summaries)

7. `04_discovery__doc_pair_similarity_top.csv`
   - Top 500 most similar document pairs by Jaccard similarity over row-hash sets
   - Filters out ultra-common hashes to reduce boilerplate-driven pair explosions

8. `04_discovery__rowhash_tag_stats.csv`
   - Tag distribution across cloned row-hashes (useful for “boilerplate-like” vs “substantive”)

## How to run (DuckDB CLI)

```sql
.read duckdb/sql/core/bootstrap.sql
.read duckdb/sql/core/starter_views__v3_provenance.sql

.read duckdb/sql/topics/04_clone_matching/04_clone_matching__discovery_demand_rows__views.sql
.read duckdb/sql/topics/04_clone_matching/04_clone_matching__discovery_demand_rows__exports.sql
```

## Knobs you may want to tweak
Edit only `clone04_discovery_params`:

- `max_docs_per_rowhash_for_pairing`
- `top_n_rowhash_text_rows`