upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/index_tables_guide.md

@@ -1,10 +1,11 @@
 ---
 title: "Index Tables Guide for IDC"
 task: ""
+persona: "running"
 lineage_type: import
-upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/index_tables_guide.md
-upstream_sha: 9c9bd2e9
-imported_at: 2026-06-27
+upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/index_tables_guide.md
+upstream_sha: d661d27e
+imported_at: 2026-08-11
 prompt_class: prompt
 upstream_changes: accepted
 author: upstream
@@ -13,7 +14,7 @@ validated: false
 
 # Index Tables Guide for IDC
 
-**Tested with:** idc-index 0.11.14 (IDC data version v23)
+**Tested with:** idc-index 0.12.5 (IDC data version v24)
 
 This guide covers the structure and access patterns for IDC index tables: programmatic schema discovery, DataFrame access, and join column references. For the overview of available tables and their purposes, see the "Index Tables" section in the main SKILL.md.
 
@@ -30,9 +31,35 @@ For SQL query examples (filter discovery, finding annotations, size estimation),
 
 ## Prerequisites
 
-```bash
-pip install --upgrade idc-index
-```
+Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed
+version and prints the install command for the interpreter you are running.
+
+## Available Tables
+
+`SKILL.md` carries a compact map of the table families. This is the full inventory with row
+granularity and contents. Always call `client.fetch_index("table_name")` before querying any
+of them — it is safe and idempotent for all tables, including those loaded automatically at
+startup.
+
+| Table | Row Granularity | Description |
+|-------|-----------------|-------------|
+| `index` | 1 row = 1 DICOM series | Primary metadata for all current IDC data |
+| `version_metadata_index` | 1 row = 1 IDC release version | IDC version release timestamps; join on `idc_version` to correlate series with their release date |
+| `collections_index` | 1 row = 1 collection | Collection-level metadata and descriptions |
+| `analysis_results_index` | 1 row = 1 analysis result collection | Metadata about derived datasets (annotations, segmentations) |
+| `clinical_index` | 1 row = 1 (collection, table, column) triple | Dictionary mapping clinical data table columns to collections |
+| `sm_index` | 1 row = 1 slide microscopy series | Slide Microscopy (pathology) series metadata |
+| `sm_instance_index` | 1 row = 1 slide microscopy instance | Instance-level (SOPInstanceUID) metadata for slide microscopy |
+| `seg_index` | 1 row = 1 DICOM Segmentation series | Segmentation metadata: algorithm, segment count, reference to source image series |
+| `ann_index` | 1 row = 1 DICOM ANN series | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
+| `ann_group_index` | 1 row = 1 annotation group | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
+| `contrast_index` | 1 row = 1 series with contrast info | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
+| `volume_geometry_index` | 1 row = 1 CT/MR/PT series | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite `regularly_spaced_3d_volume` flag |
+| `rtstruct_index` | 1 row = 1 RTSTRUCT series | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |
+| `ct_index` | 1 row = 1 CT series | CT acquisition/reconstruction parameters: pixel spacing, slice thickness, kVp, convolution kernel, tube current (min/max for dose-modulated), exposure, spiral pitch, scan options |
+| `mr_index` | 1 row = 1 MR series | MR acquisition/sequence parameters: field strength, scanning sequence, TE (array for multi-echo), TR, flip angle, DiffusionBValue (array for DWI), pixel bandwidth, receive coil, number of temporal positions |
+| `pt_index` | 1 row = 1 PET series | PET acquisition/reconstruction/radiopharmaceutical parameters: series type, units, decay/scatter/attenuation correction, reconstruction method, radionuclide, injected dose, frame duration (array for dynamic PET) |
+| `prior_versions_index` | 1 row = 1 DICOM series | **Reproducibility only.** Contains series permanently removed from IDC (all `max_idc_version` < current version; zero overlap with `index`). Use ONLY when a user explicitly needs to reproduce work from a prior IDC version using data no longer in the current release. Do NOT use for version history or "what's new" questions — those use `series_init_idc_version`/`series_revised_idc_version` in the main `index` table. Column names `min_idc_version`/`max_idc_version` here are NOT equivalent to `series_init_idc_version`/`series_revised_idc_version` in `index`. |
 
 ## Accessing Index Tables
 
@@ -47,7 +74,7 @@ results = client.sql_query("SELECT * FROM index WHERE Modality = 'CT' LIMIT 10")
 
 # Fetch and query additional indices
 client.fetch_index("collections_index")
-collections = client.sql_query("SELECT collection_id, CancerTypes, TumorLocations FROM collections_index")
+collections = client.sql_query("SELECT collection_id, cancer_types, tumor_locations FROM collections_index")
 
 client.fetch_index("analysis_results_index")
 analysis = client.sql_query("SELECT * FROM analysis_results_index LIMIT 5")
@@ -100,6 +127,29 @@ schema = client.get_index_schema("index")
 # Returns same schema dict: {'table_description': ..., 'columns': [...]}
 ```
 
+### Finding which table contains a column
+
+The most common schema question is "where does `SliceThickness` live?" — the primary `index`
+holds series-level metadata only, so modality-specific acquisition parameters are in dedicated
+tables. Search the overview rather than guessing; neither call fetches anything:
+
+```python
+# Find which table(s) contain a specific column (no fetch required)
+target = "SliceThickness"
+for table_name, info in client.indices_overview.items():
+    if any(c["name"] == target for c in info["schema"]["columns"]):
+        print(f"'{target}' is in: {table_name}")
+# → 'SliceThickness' is in: ct_index
+
+# List all columns in a table from the schema (no fetch required)
+ct_cols = [c["name"] for c in client.indices_overview["ct_index"]["schema"]["columns"]]
+print("ct_index columns:", ct_cols)
+# → ['SeriesInstanceUID', 'PixelSpacing_row_mm', 'PixelSpacing_col_mm', 'Rows',
+#    'Columns', 'SliceThickness', 'KVP', 'ConvolutionKernel', ...]
+```
+
+Then `client.fetch_index("ct_index")` and join to `index` on `SeriesInstanceUID`.
+
 ## Key Columns Reference
 
 Most common columns in the primary `index` table (use `indices_overview` for complete list and descriptions):
@@ -143,6 +193,9 @@ Use this table to identify join columns between index tables. Always call `clien
 | `index` | `volume_geometry_index` | `index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID` |
 | `index` | `rtstruct_index` | `index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID` |
 | `rtstruct_index` | `index` (source images) | `rtstruct_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID` |
+| `index` | `ct_index` | `index.SeriesInstanceUID = ct_index.SeriesInstanceUID` |
+| `index` | `mr_index` | `index.SeriesInstanceUID = mr_index.SeriesInstanceUID` |
+| `index` | `pt_index` | `index.SeriesInstanceUID = pt_index.SeriesInstanceUID` |
 
 For complete query examples using these joins, see `references/sql_patterns.md`.