diff --git a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/.github/workflows/pr-skill-scan.yml b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/.github/workflows/pr-skill-scan.yml index 1992ae60..52951681 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/.github/workflows/pr-skill-scan.yml +++ b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/.github/workflows/pr-skill-scan.yml @@ -2,9 +2,9 @@ title: "Pr Skill Scan" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/991bd993/.github/workflows/pr-skill-scan.yml -upstream_sha: 991bd993 -imported_at: 2026-08-08 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/.github/workflows/pr-skill-scan.yml +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: unknown upstream_changes: accepted author: upstream @@ -86,6 +86,8 @@ jobs: if: steps.changed.outputs.skill_dirs != '' run: uv sync --python 3.13 + # Fork PRs do not receive SKILL_SCANNER_LLM_API_KEY. scan_pr_skills.py + # detects the missing key, writes an explanatory sticky comment, and exits 0. - name: Run scanner on changed skills if: steps.changed.outputs.skill_dirs != '' id: scan diff --git a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cli_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cli_guide.md index 91ef70a3..805952b9 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cli_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cli_guide.md @@ -2,9 +2,9 @@ title: "idc-index Command Line Interface Guide" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/cli_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/cli_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: catalogue upstream_changes: accepted author: upstream @@ -17,9 +17,8 @@ The `idc-index` package provides command-line tools for downloading DICOM data f ## Installation -```bash -pip install --upgrade idc-index -``` +Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed +version and prints the install command for the interpreter you are running. After installation, the `idc` command is available in your terminal. @@ -63,6 +62,22 @@ idc download manifest.txt --download-dir ./data ### Directory Template Variables +The same templates apply in Python, where the argument is `dirTemplate=` rather than the +`--dir-template` flag. The default is +`%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`: + +```python +# Simplified hierarchy (omit StudyInstanceUID level) +client.download_from_selection( + downloadDir="./data", + collection_id="tcga_luad", + dirTemplate="%collection_id/%PatientID/%Modality" +) +# Results in: ./data/tcga_luad/TCGA-05-4244/CT/ + +# dirTemplate="" disables the hierarchy, writing every file straight into downloadDir +``` + Use these variables in `--dir-template` to organize downloads: - `%collection_id` - Collection identifier diff --git a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/clinical_data_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/clinical_data_guide.md index e9cf874c..ec39fe2c 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/clinical_data_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/clinical_data_guide.md @@ -2,9 +2,9 @@ title: "Clinical Data Guide for IDC" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/clinical_data_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/clinical_data_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: catalogue upstream_changes: accepted author: upstream @@ -13,7 +13,7 @@ validated: false # Clinical Data Guide for IDC -**Tested with:** idc-index 0.11.7 (IDC data version v23) +**Tested with:** idc-index 0.12.5 (IDC data version v24) Clinical data (demographics, diagnoses, therapies, lab tests, staging) accompanies many IDC imaging collections. This guide covers how to discover, access, and integrate clinical data with imaging data using `idc-index`. @@ -29,9 +29,8 @@ For basic clinical data access, see the "Clinical Data Access" section in the ma ## Prerequisites -```bash -pip install --upgrade idc-index -``` +Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed +version and prints the install command for the interpreter you are running. No BigQuery credentials required - clinical data is packaged with `idc-index`. diff --git a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cloud_storage_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cloud_storage_guide.md index b8e58be3..d9ed2717 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cloud_storage_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/cloud_storage_guide.md @@ -2,9 +2,9 @@ title: "Cloud Storage Guide for IDC" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/cloud_storage_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/cloud_storage_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: catalogue upstream_changes: accepted author: upstream @@ -218,7 +218,7 @@ IDC releases new data versions every 2-4 months. The versioning system ensures r ### How Versioning Works -1. **Snapshots**: Each IDC version (v1, v2, ..., v23, etc.) represents a complete snapshot of all data at release time +1. **Snapshots**: Each IDC version (v1, v2, ..., v24, etc.) represents a complete snapshot of all data at release time 2. **UUID-based**: When data changes, new CRDC UUIDs are assigned; old UUIDs remain accessible 3. **Cumulative buckets**: All versions coexist in the same buckets—old series folders @@ -236,7 +236,7 @@ IDC releases new data versions every 2-4 months. The versioning system ensures r For querying version-specific metadata, BigQuery provides versioned tables. See `bigquery_guide.md` for details. - `bigquery-public-data.idc_current` — alias to latest version -- `bigquery-public-data.idc_v23` — specific version (replace 23 with desired version) +- `bigquery-public-data.idc_v24` — specific version (replace 24 with desired version) ### Reproducing a Previous Analysis diff --git a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/digital_pathology_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/digital_pathology_guide.md index 1421fdda..49e962f7 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/digital_pathology_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/catalogue/skills/imaging-data-commons/references/digital_pathology_guide.md @@ -2,9 +2,9 @@ title: "Digital Pathology Guide for IDC" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/digital_pathology_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/digital_pathology_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: catalogue upstream_changes: accepted author: upstream @@ -13,7 +13,7 @@ validated: false # Digital Pathology Guide for IDC -**Tested with:** IDC data version v23, idc-index 0.11.10 +**Tested with:** idc-index 0.12.5 (IDC data version v24) For general IDC queries and downloads, use `idc-index` (see main SKILL.md). This guide covers slide microscopy (SM) imaging, microscopy bulk simple annotations (ANN), and segmentations (SEG) in the context of digital pathology in IDC. @@ -264,12 +264,12 @@ client.sql_query(""" SELECT ar.analysis_result_id, ar.analysis_result_title, - ar.Modalities, - ar.Subjects, - ar.Collections + ar.modalities, + ar.subjects, + ar.collections FROM analysis_results_index ar - WHERE ar.Modalities LIKE '%ANN%' OR ar.Modalities LIKE '%SEG%' - ORDER BY ar.Subjects DESC + WHERE ar.modalities LIKE '%ANN%' OR ar.modalities LIKE '%SEG%' + ORDER BY ar.subjects DESC """) ``` diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/SKILL.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/SKILL.md new file mode 100644 index 00000000..d29ad941 --- /dev/null +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/SKILL.md @@ -0,0 +1,502 @@ +--- +lineage_type: import +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/SKILL.md +upstream_sha: d661d27e +imported_at: 2026-08-11 +prompt_class: prompt +upstream_changes: accepted +name: imaging-data-commons +description: Query and download public cancer imaging data from NCI Imaging Data Commons. Invoke for any question about IDC collections, cancer imaging datasets, DICOM data access, radiology (CT, MR, PET) or pathology AI training sets, metadata queries, visualization, or license checks — even when the user doesn't explicitly mention "IDC". No authentication required. +license: This skill is provided under the MIT License. IDC data itself has individual licensing (mostly CC-BY, some CC-NC) that must be respected when using the data. +metadata: + version: "1.5" + source-skill-version: 1.8.1 + skill-author: Andrey Fedorov, @fedorov + idc-index: "0.12.5" + idc-data-version: "v24" + repository: https://github.com/ImagingDataCommons/imaging-data-commons-skill +--- + +# Imaging Data Commons + +## Overview + +Query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access. + +**Expected network access:** IDC metadata is reachable three ways — a local DuckDB index shipped with the `idc-index` Python package (no network), or the hosted IDC service over MCP or REST (`api.imaging.datacommons.cancer.gov`, no authentication). File downloads use public GCS (`storage.googleapis.com`) and AWS S3 (`s3.amazonaws.com`) — no authentication required. DICOMweb access uses either the public IDC proxy (`proxy.imaging.datacommons.cancer.gov`, no auth) or the Google Cloud Healthcare API (`healthcare.googleapis.com`, requires GCP authentication). Optional BigQuery queries (`bigquery.googleapis.com`) also require GCP authentication. No credentials or environment variables are accessed by this skill. + +**Current IDC Data Version: v24** (always verify — see *Best Practices*) + +**Choose the access path first.** There is no single default: the cheapest correct path depends +on the session and the task. + +1. **Session already has the IDC MCP server?** Route discovery and metadata there — see *IDC + MCP Server*. +2. **Otherwise, is `idc-index` installed?** Run `python scripts/check_version.py`. If it passes, + use `idc-index` for everything. +3. **Not installed, and the task is read-only metadata** — counts, attribute values, collection + lookups, SQL under 10 000 rows, licenses, citations, viewer URLs? **Use the REST API over + `curl`; do not install anything.** Installing costs ~77 MB of packaged index data plus + pandas, pyarrow, and duckdb, which a metadata question does not need. See *Data Access + Options*. +4. **Not installed, and the task needs more than metadata** — downloading files, pandas or + plotting, pydicom/SimpleITK, pathology tiling, results past 10 000 rows, or a version-pinned + script the user re-runs? Install `idc-index`: `check_version.py` exits non-zero and prints + the exact install command for the running interpreter. Prefer a virtual environment, then + restart Python. + +`idc-index` ([GitHub](https://github.com/imagingdatacommons/idc-index)) is still the most +capable path and the only one that moves image bytes; the rule is just not to pay for it before +the task calls for it. `check_version.py` never installs anything itself — it also flags a newer +`idc-index` or skill release when one exists. + +**Setup for the `idc-index` path:** + +```python +from idc_index import IDCClient +client = IDCClient() + +# Verify IDC data version (should be "v24") +print(f"IDC data version: {client.get_idc_version()}") +``` + +**Core workflow:** query metadata with `client.sql_query()` → download with +`client.download_from_selection()` → visualize with `client.get_viewer_URL()`. Python examples +below assume this `client`; *Data Access Options* has the REST equivalents. For current data +scale, run the summary query in `references/sql_patterns.md` or `GET /v3/stats`. + +## IDC MCP Server + +IDC operates a hosted MCP server at `https://api.imaging.datacommons.cancer.gov/mcp` +(streamable HTTP, no authentication). Where it is available it complements — it does not +replace — the `idc-index` workflow below. + +**Identify it** by the MCP resource `idc://guide`, or by three or more of the tool names +`build_cohort`, `get_cohort_urls`, `list_analysis_results`, and `get_idc_version`. Generic +names such as `run_sql` are not evidence on their own. If identification is ambiguous, use +`idc-index`. + +**If this session has the server**, treat it as authoritative for discovery and metadata — +IDC version, counts, attribute values, cohort building, metadata SQL — and follow the +server's own instructions rather than re-deriving them from this file. Its data version is +whatever the server reports: call `get_idc_version` instead of relying on the version pinned +in this file. + +Return here for what the server does not do: downloading files, local pandas/notebook +analysis, DICOMweb, BigQuery, digital pathology tiling, and reproducible scripts. Hand off by +passing SeriesInstanceUIDs from the server to `client.download_from_selection(...)`, and run +`scripts/check_version.py` at that point. + +**If it is not available**, the identical service is reachable with no configuration as a REST +API at `https://api.imaging.datacommons.cancer.gov/v3` — use it for read-only metadata rather +than installing `idc-index`, per the routing gate in *Overview*. Suggest connecting the MCP +server at most once, only for repeated interactive discovery, and never change the user's +configuration yourself. + +See `references/mcp_guide.md` for the tool inventory, handoff patterns, and per-host notes. + +## When to Use This Skill + +- Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images +- Selecting image subsets by cancer type, modality, anatomical site, or other metadata +- Downloading DICOM data from IDC +- Checking data licenses before use in research or commercial applications +- Visualizing medical images in a browser without local DICOM viewer software + +## Quick Navigation + +Inline below: the MCP/REST routing rules, the IDC data model, the index tables and how they +join, the core API patterns (query, download, visualize, license, cite), best practices, and +troubleshooting. + +**Reference Guides (load on demand):** + +| Guide | When to Load | +|-------|--------------| +| `index_tables_guide.md` | Complex JOINs, schema discovery, DataFrame access | +| `use_cases.md` | End-to-end workflows: training datasets, batch downloads, DICOM reading with pydicom/SimpleITK, pipeline integration | +| `sql_patterns.md` | Quick SQL patterns for filter discovery, annotations, size estimation | +| `clinical_data_guide.md` | Clinical/tabular data, imaging+clinical joins, value mapping | +| `licensing_and_citation.md` | Commercial-use questions, mixed-license cohorts, citation formats | +| `cloud_storage_guide.md` | Direct S3/GCS access, versioning, UUID mapping | +| `dicomweb_guide.md` | DICOMweb endpoints, PACS integration | +| `digital_pathology_guide.md` | Slide microscopy (SM), annotations (ANN), pathology workflows | +| `bigquery_guide.md` | Full DICOM metadata, private elements (requires GCP) | +| `cli_guide.md` | Command-line tools (`idc download`, manifest files) | +| `parquet_access_guide.md` | Direct Parquet queries via GCS (no idc-index install needed) | +| `mcp_guide.md` | Hosted IDC MCP server: tool inventory, identification, handoff to `idc-index` | +| `rest_api_guide.md` | Hosted IDC REST API: endpoints, filter syntax, SQL over HTTP, manifests | + +## IDC Data Model + +IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance): + +- **collection_id**: Groups patients by disease, modality, or research focus (e.g., `tcga_luad`, `nlst`). A patient belongs to exactly one collection. +- **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections. Use it to find AI-generated or expert annotations, while `collection_id` finds original imaging data (which may itself include deposited annotations). + +**Key identifiers for queries:** +| Identifier | Scope | Use for | +|------------|-------|---------| +| `collection_id` | Dataset grouping | Filtering by project/study | +| `PatientID` | Patient | Grouping images by patient | +| `StudyInstanceUID` | DICOM study | Grouping of related series, visualization | +| `SeriesInstanceUID` | DICOM series | Grouping of related series, visualization | + +## Index Tables + +The `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames. The REST API exposes the same tables through `GET /tables` and `POST /sql`. + +**Important:** `client.indices_overview` is the authoritative source for current table descriptions, available columns, and their types — query it when writing SQL or exploring data structure. It also answers "which table contains column X"; see `references/index_tables_guide.md` for that search pattern and full schema discovery. + +### Available Tables + +Always call `client.fetch_index("table_name")` before querying any index table — it is safe and idempotent for all tables, including those loaded automatically at startup. + +| Family | Tables | Granularity | +|--------|--------|-------------| +| Core | `index` (primary metadata for all current data), `collections_index`, `analysis_results_index` | series / collection / analysis result | +| Modality acquisition parameters | `ct_index`, `mr_index`, `pt_index`, `contrast_index` | 1 row = 1 series of that modality | +| Derived objects | `seg_index`, `rtstruct_index`, `ann_index`, `ann_group_index` | 1 row = 1 series (or annotation group) | +| Microscopy | `sm_index`, `sm_instance_index` | 1 row = 1 SM series / instance | +| Geometry, clinical, history | `volume_geometry_index`, `clinical_index`, `version_metadata_index`, `prior_versions_index` | see guide | + +`references/index_tables_guide.md` has the full inventory with each table's columns and +contents — load it when you need to know what a specialized table actually holds. + +**`prior_versions_index` is for reproducibility only.** It contains series permanently *removed* +from IDC, with zero overlap with `index`. Use it only to reproduce work against a prior IDC +version. Do NOT use it for version history or "what's new" questions — those use +`series_init_idc_version` / `series_revised_idc_version` in the main `index` table, which are +not equivalent to this table's `min_idc_version` / `max_idc_version`. + +### Joining Tables + +**`SeriesInstanceUID` is the universal join key** for all series-level specialized tables: `sm_index`, `sm_instance_index`, `seg_index`, `ann_index`, `ann_group_index`, `contrast_index`, `volume_geometry_index`, `rtstruct_index`, `ct_index`, `mr_index`, `pt_index`. Always join these to `index` on `SeriesInstanceUID`. The exceptions below use different column names. + +| Join Column | Tables | Use Case | +|-------------|--------|----------| +| `collection_id` | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data | +| `analysis_result_id` | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) | +| `source_DOI` | index, analysis_results_index | Link by publication DOI | +| `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (`seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID`) | +| `referenced_SeriesInstanceUID` | ann_index → index, rtstruct_index → index | Link annotation or RTSTRUCT to its source image series | + +**Note:** `subjects`, `updated`, and `description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts). Joining `prior_versions_index` to `index` on `SeriesInstanceUID` always returns zero rows — see the warning above. + +For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see `references/index_tables_guide.md`. + +### Clinical Data Access + +Clinical (non-imaging) attributes — staging, demographics, therapy — live in per-collection +tables. `client.fetch_index("clinical_index")` loads the dictionary mapping columns to +collections; `client.get_clinical_table(name)` returns one table as a DataFrame. + +See `references/clinical_data_guide.md` for the discovery workflow, coded-value mapping, and +joining clinical data with imaging. + +## Data Access Options + +| Method | Auth | Best For | Reference | +|--------|------|----------|-----------| +| `idc-index` | No | Downloads, pandas analysis, unbounded queries — the most capable path | This document | +| IDC MCP server | No | Discovery, cohort building, metadata when the session already has it | `mcp_guide.md` | +| IDC REST API | No | Metadata with no install, from any language or shell — the default when `idc-index` is absent | `rest_api_guide.md` | +| Direct Parquet (GCS) | No | Version-pinned queries, or results past the REST row cap | `parquet_access_guide.md` | +| Cloud storage (S3/GCS) | No | Direct file access, bulk transfer, custom pipelines | `cloud_storage_guide.md` | +| DICOMweb via IDC proxy | No | Tool and PACS integration; daily quota, so testing and moderate use | `dicomweb_guide.md` | +| DICOMweb via Google Healthcare | Yes (GCP) | The same DICOMweb API at production volume, without the proxy quota | `dicomweb_guide.md` | +| SlicerIDCBrowser | No | 3D visualization and analysis in 3D Slicer | https://github.com/ImagingDataCommons/SlicerIDCBrowser | +| BigQuery | Yes (GCP) | Full DICOM metadata, private elements, SR measurements — last resort | `bigquery_guide.md` | + +**The IDC Portal (https://portal.imaging.datacommons.cancer.gov/) is interactive only** — +browser-based exploration, manual cohort selection, and download. Unlike every option above it +has no programmatic interface, so point a user there to browse or click through data +themselves; never use it as a step in a script or workflow. + +**REST API — the no-install metadata path** + +`https://api.imaging.datacommons.cancer.gov/v3`, no authentication: discovery, cohort counts and +manifests, read-only SQL, clinical tables, viewer URLs, licenses, citations. It is the same +service as the MCP server over plain HTTP, so it needs no configuration. It never moves image +bytes — switch to `idc-index` to download, to get a DataFrame, or for results past 10 000 rows. + +```bash +B=https://api.imaging.datacommons.cancer.gov/v3 +curl -s $B/version # idc_version, idc_index_data_version, api_version +curl -s $B/stats # collections, patients, studies, series, instances, size_TB +curl -s "$B/attributes/Modality/values?limit=5" # real filter values, with counts +curl -s $B/sql -H 'content-type: application/json' \ + -d '{"sql":"SELECT collection_id, COUNT(*) n FROM index GROUP BY 1 ORDER BY n DESC LIMIT 3"}' +curl -s $B/cohort/counts -H 'content-type: application/json' \ + -d '{"filters":{"terms":{"collection_id":["rider_pilot"]}}}' +``` + +**The filter object always goes under `filters`** — on `cohort/counts`, `cohort/manifest`, +`cohort/manifest.txt`, `licenses`, and `citations` alike. A bare filter or an unrecognized key is +a 422 naming the fix; an unfiltered series-enumerating request is a 400, not the whole archive. +Every filtered response echoes `filters_applied` and `warnings` — read them, because they name +any predicate the server dropped. A zero count with empty `warnings` therefore means the filter +matched nothing, not that a value was miscased; miscasing produces a warning that says so. + +`POST /sql` takes one read-only `SELECT`/`WITH` over the tables `idc-index` exposes plus +`clinical.`; `max_rows` defaults to 5 000, caps at 10 000, and `truncated` flags clipping. +`GET /attributes` lists the 19 filterable attributes — clinical values, segmented anatomy, and +acquisition parameters are not among them and need SQL. There is no rate limit or quota. **Use +v3 only:** V1 and V2 are superseded and scheduled for shutdown, so port any `/v1/`- or +`Modality_btw`-style example a user brings rather than extending it. + +Both sides build on `idc-index-data`, so compare the API's `idc_index_data_version` against local +`idc_index_data.__version__` before mixing them: the **major is the IDC data release** (`24.x.y` +serves `v24`), so differing minor/patch means the series are identical. If the API is a whole +release ahead, `idc-index` **cannot download the extra series** — it silently skips what its own +index does not list — so either upgrade it (run `scripts/check_version.py` for the right command) +or transfer directly from the bucket with `s5cmd --no-sign-request`. + +See `references/rest_api_guide.md` for the endpoint reference, filter grounding, limits, and the +manifest-based download flow. + +**Cloud storage organization** + +All DICOM files live in public buckets mirrored between AWS S3 and GCS, organized by CRDC UUIDs +(not DICOM UIDs) to support versioning, as `/.dcm`. Access +is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access; use the +`series_aws_url` column for S3 URLs. Note that `idc-open-data-cr` / `idc-open-cr` (~4% of data) +is commercial-use restricted (CC BY-NC). See `references/cloud_storage_guide.md` for the full +bucket list and UUID mapping. + +**DICOMweb access** + +IDC data is available via DICOMweb (Google Cloud Healthcare API) for PACS integration and +DICOMweb-compatible tools: a public proxy (no auth, daily quota) for testing and moderate +queries, or Google Healthcare (GCP auth) for production volumes. See +`references/dicomweb_guide.md`. + +**Direct Parquet access** + +The idc-index metadata tables are also published as Parquet on a public GCS bucket +(`idc-index-data-artifacts`), queryable with DuckDB or pandas. This needs DuckDB installed +and cannot reach the per-collection clinical tables, so prefer REST `/sql` for ad-hoc metadata; +choose Parquet to pin a data version or for results past the REST row cap. See +`references/parquet_access_guide.md`. + +## Core Capabilities + +The patterns below are the ones that go wrong when recalled from memory rather than checked. +Worked examples for each area live in the reference guides named inline. + +### 1. Discovery — enumerate values before filtering on them + +Filtering on a guessed `Modality` or `BodyPartExamined` string is the most common cause of an +empty result set. Enumerate first: + +```python +modalities = client.sql_query(""" + SELECT DISTINCT Modality, COUNT(*) as series_count + FROM index + GROUP BY Modality + ORDER BY series_count DESC +""") +print(modalities) +``` + +The same pattern works for any filter column, optionally narrowed by another — +`BodyPartExamined` within a `Modality`, `Manufacturer`, `collection_id`. On the REST path this +grounding is a single call — `GET /attributes/{attr}/values` returns values with counts — and the +cohort endpoints report a miscased value in `warnings` rather than as an empty result. + +Two indices carry curated collection-level metadata the primary `index` does not, both +requiring `client.fetch_index(...)` first: `collections_index` (cancer types, tumor locations, +species, subject counts) and `analysis_results_index` (derived datasets — AI segmentations, +expert annotations, radiomics — with their source collections and modalities). + +**Cancer type lives in `collections_index.cancer_types`, not in `index`** — filtering by +cancer type requires a join: + +```python +client.fetch_index("collections_index") +results = client.sql_query(""" + SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality + FROM index i + JOIN collections_index c ON i.collection_id = c.collection_id + WHERE c.cancer_types LIKE '%Breast%' + AND i.Modality = 'MR' + LIMIT 20 +""") +``` + +`client.sql_query()` returns a pandas DataFrame. Confirm column names with +`client.get_index_schema('index')` or `client.indices_overview` before writing a query rather +than assuming them. + +See `references/sql_patterns.md` for filter-value discovery, annotation and segmentation +queries, size estimation, clinical linking, and version tracking ("what's new in vX" — use +`series_init_idc_version` / `series_revised_idc_version` in `index`, never +`prior_versions_index`). + +### 2. Downloading DICOM files + +**The two download methods take their first two arguments in opposite order.** This is the +most common source of broken IDC code — check it rather than recalling it: + +| Method | First arg | Second arg | Use when | +|--------|-----------|------------|----------| +| `download_from_selection` | `downloadDir` (required) | filter kwargs (optional) | Filtering by collection, patient, study, or series | +| `download_dicom_series` | `seriesInstanceUID` (required) | `downloadDir` (required) | Downloading specific series by UID only | + +**`download_from_selection` takes filter keyword arguments, NOT a DataFrame.** The name +"from_selection" refers to filtering the IDC index by criteria — not to accepting a pandas +DataFrame. To download query results, extract the UIDs into a list first: + +```python +# Step 1: Query for series UIDs +series_df = client.sql_query(""" + SELECT SeriesInstanceUID + FROM index + WHERE Modality = 'CT' + AND BodyPartExamined = 'CHEST' + AND collection_id = 'nlst' + LIMIT 5 +""") + +# Step 2: Extract UIDs as a list from the DataFrame +uids = list(series_df['SeriesInstanceUID'].values) + +# Step 3: Pass the list to download_from_selection (NOT the DataFrame itself) +client.download_from_selection( + downloadDir="./data/lung_ct", + seriesInstanceUID=uids # list of strings, not a DataFrame +) + +# Alternative: download_dicom_series has seriesInstanceUID as FIRST arg (different order!) +client.download_dicom_series( + seriesInstanceUID=uids, # FIRST arg here + downloadDir="./data/lung_ct" +) + +# Whole collection: downloadDir is still the FIRST positional argument +client.download_from_selection(downloadDir="./data/rider", collection_id="rider_pilot") +``` + +Both methods default to AWS; pass `source_bucket_location="gcs"` to pull from Google Storage. + +**Downloaded files are named `.dcm`, not by SOPInstanceUID.** The DICOM +UIDs are preserved inside the file metadata, not in the filename. Use the `crdc_instance_uuid` +column to map files back to the series they came from. + +`idc download --download-dir ./data` does the same from a +shell. See `references/cli_guide.md` for the `dirTemplate` hierarchy options (Python default: +`%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`; `dirTemplate=""` +flattens), manifest downloads with resume, and dry-run size estimation. + +### 3. Visualizing IDC images + +```python +viewer_url = client.get_viewer_URL(seriesInstanceUID=uid) # one series +viewer_url = client.get_viewer_URL(studyInstanceUID=study_uid) # all series in a study +``` + +Returns a browser URL — nothing is downloaded. The method selects OHIF v3 for radiology or +SLIM for slide microscopy automatically. Viewing by study is useful when a single DICOM Study +holds several Series (T1, T2, and DWI from one MRI session). + +### 4. Licenses and citations — obligations, not optional steps + +IDC data carries license terms and attribution requirements that follow it into any downstream +publication or product, and neither is inferable from the pixel data. **Check the license +before use, and generate citations for whatever you download.** + +```python +# License breakdown for a selection +licenses = client.sql_query(""" + SELECT DISTINCT collection_id, license_short_name, + COUNT(DISTINCT SeriesInstanceUID) as series_count + FROM index GROUP BY collection_id, license_short_name +""") + +# Citations for the same selection you downloaded (APA by default) +for citation in client.citations_from_selection(collection_id="rider_pilot"): + print(citation) +``` + +About 97% of IDC data is CC BY (commercial use allowed with attribution) and about 3% is +CC BY-NC (non-commercial only). **Licenses attach to series, not collections** — 39 of 176 +collections carry more than one — so check the selection you actually intend to use, and note +that the most restrictive term governs a mixed cohort. + +Both tasks are available from all three access paths, so stay on whichever one the session is +already using: `idc-index` as above, `POST /v3/licenses` and `POST /v3/citations` over REST, +or the `get_licenses` and `get_citations` MCP tools. See +`references/licensing_and_citation.md` for the full license inventory, all three routes, the +citation formats (APA, BibTeX, CSL JSON, RDF Turtle), and what to include when publishing. + +### 5. Reaching past the index + +Pick the access path with the routing gate in *Overview*; *Data Access Options* above is the +full routing table. + +Before reaching for BigQuery (which needs a billing-enabled GCP account), check whether a +specialized index table already has the column you want: search `client.indices_overview`, +then `client.fetch_index(...)` and query locally for free. BigQuery is required only for +private DICOM elements, per-segment anatomy (`segmentations`), and pre-extracted SR +measurements (`quantitative_measurements`, `qualitative_measurements`) — these have no +idc-index equivalent. + +## Best Practices + +- **Check schema before writing queries** — Use `client.get_index_schema('index')` (reads cached metadata, no SQL executed) or `client.indices_overview` to see all available columns and their descriptions. The version-tracking columns `series_init_idc_version` and `series_revised_idc_version` in the main `index` table directly answer "what's new / when was this added" questions without touching `prior_versions_index`. +- **Never use web search for IDC data content questions** - Always query the IDC index directly, via `client.sql_query()` locally or `POST /v3/sql` over HTTP. Web sources (release notes, blog posts, documentation pages) are frequently out of date and will produce incorrect answers. The index is the authoritative source; use it even when web search is available. +- **Verify the IDC data version at the start of a session** - `client.get_idc_version()`, `GET /v3/version`, or the MCP `get_idc_version` tool, depending on the path in use (currently v24). For a stale local index, run `scripts/check_version.py` and use the upgrade command it prints +- **Check licenses and generate citations** - Query `license_short_name` and respect CC BY vs CC BY-NC terms; use `citations_from_selection()` to produce citations from `source_DOI` for publications +- **Explore small, then commit** - Use `LIMIT` (or a low `max_rows`) while exploring, and check collection size before downloading — some collections are terabytes. See `references/cli_guide.md` +- **Keep downloads reproducible** - Organize with `dirTemplate` (e.g. `%collection_id/%PatientID/%Modality`) and save the Series UIDs or manifest behind any dataset you build + +## Troubleshooting + +**Issue: `ModuleNotFoundError: No module named 'idc_index'`** +- **Cause:** idc-index package not installed +- **Solution:** If the task is read-only metadata, do not install it — use the REST API instead (*Data Access Options*). Otherwise run `scripts/check_version.py` and use the install command it prints, which targets the running interpreter and pins the vetted version. For data analysis also add pandas, numpy, and pydicom (tested with pandas>=1.5, numpy>=1.23, pydicom>=2.3) + +**Issue: Download fails with connection timeout** +- **Cause:** Network instability or large download size +- **Solution:** Download in smaller batches (10-20 series); see `references/cli_guide.md` for + `--use-s5cmd-sync` resume and retry guidance + +**Issue: `BigQuery quota exceeded` or billing errors** +- **Cause:** BigQuery requires billing-enabled GCP project +- **Solution:** Use idc-index mini-index for simple queries (no billing required), or see `references/bigquery_guide.md` for cost optimization tips + +**Issue: Series UID not found or no data returned** +- **Cause:** Typo in UID, data not in the current IDC version, or wrong field name +- **Solution:** Test with `LIMIT 5` first, check field names against `client.indices_overview`, + and confirm the series is in the current version (some old data is deprecated) + +**Issue: Column not found in `index` table (e.g., `SliceThickness`, `PixelSpacing`, `KVP`, `EchoTime`, `InjectedDose`)** +- **Cause:** The `index` table contains series-level metadata only; modality-specific acquisition and reconstruction parameters live in dedicated tables (`ct_index`, `mr_index`, `pt_index`) +- **Solution:** Search `client.indices_overview` for the column to find its table — the loop is under *Finding which table contains a column* in `references/index_tables_guide.md` — then fetch and join on `SeriesInstanceUID`: + ```python + client.fetch_index("ct_index") + result = client.sql_query(""" + SELECT i.SeriesInstanceUID, i.Modality, c.SliceThickness, c.KVP, c.PixelSpacing_row_mm + FROM index i + JOIN ct_index c USING (SeriesInstanceUID) + WHERE i.collection_id = 'your_collection' + """) + ``` + +**Issue: Downloaded DICOM files won't open** +- **Cause:** Corrupted download, or an object type the viewer does not handle — SEG, RTSTRUCT, + SR, and slide microscopy all need specialized tools +- **Solution:** Check `Modality` and `SOPClassUID` first, validate with + `pydicom.dcmread(file, force=True)`, try another viewer (3D Slicer, QuPath for pathology), + then re-download + +## Resources + +Reference guides and their decision triggers are listed in *Quick Navigation* above. + +- **IDC Portal**: https://portal.imaging.datacommons.cancer.gov/explore/ +- **Documentation**: https://learn.canceridc.dev/ — **Tutorials**: https://github.com/ImagingDataCommons/IDC-Tutorials +- **User Forum**: https://discourse.canceridc.dev/ — **idc-index**: https://github.com/ImagingDataCommons/idc-index +- **[indices_reference](https://idc-index.readthedocs.io/en/latest/indices_reference.html)** — external index-table docs (may be ahead of the installed version) +- **Citation**: Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence." RadioGraphics 43.12 (2023). https://doi.org/10.1148/rg.230180 +- **Skill updates**: [releases page](https://github.com/ImagingDataCommons/imaging-data-commons-skill/releases); watch the repository (Watch → Custom → Releases) diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/bigquery_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/bigquery_guide.md index 2ac6e84e..df3096d8 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/bigquery_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/bigquery_guide.md @@ -2,9 +2,9 @@ title: "BigQuery Guide for IDC" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/bigquery_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/bigquery_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: prompt upstream_changes: accepted author: upstream @@ -13,7 +13,7 @@ validated: false # BigQuery Guide for IDC -**Tested with:** IDC data version v23 +**Tested with:** `bigquery-public-data.idc_current` and idc-index 0.12.5 (IDC data version v24) For most queries and downloads, use `idc-index` (see main SKILL.md). This guide covers BigQuery for advanced use cases requiring full DICOM metadata or complex joins. @@ -661,7 +661,7 @@ ORDER BY collection_id - [Part 15 Appendix E - De-identification Profiles](https://dicom.nema.org/medical/dicom/current/output/chtml/part15/chapter_e.html) **Community Resources:** -- [NAMIC Wiki: DWI/DTI DICOM](https://www.na-mic.org/wiki/NAMIC_Wiki:DTI:DICOM_for_DWI_and_DTI) - comprehensive vendor comparison for diffusion imaging +- [NAMIC Wiki: DWI/DTI DICOM](https://web.archive.org/web/20260520044207/https://www.na-mic.org/wiki/NAMIC_Wiki:DTI:DICOM_for_DWI_and_DTI) - comprehensive vendor comparison for diffusion imaging (archived; original NA-MIC wiki retired) - [StandardizeBValue](https://github.com/nslay/StandardizeBValue) - tool to extract vendor b-values to standard tags ## Using Query Results with idc-index @@ -673,7 +673,7 @@ from google.cloud import bigquery from idc_index import IDCClient # Initialize BigQuery client -# Requires: pip install google-cloud-bigquery +# Requires: the Python google-cloud-bigquery package # Auth: gcloud auth application-default login # Project: needed for billing even on public datasets (free tier applies) bq_client = bigquery.Client(project="your-gcp-project-id") diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/dicomweb_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/dicomweb_guide.md index 6eb24782..02d930d9 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/dicomweb_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/dicomweb_guide.md @@ -2,9 +2,9 @@ title: "DICOMweb Guide for IDC" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/dicomweb_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/dicomweb_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: prompt upstream_changes: accepted author: upstream @@ -52,7 +52,7 @@ Replace `{VERSION}` with the IDC release number. To find the current version: ```python from idc_index import IDCClient client = IDCClient() -print(client.get_idc_version()) # e.g., "23" for v23 +print(client.get_idc_version()) # e.g., "v24" for current version ``` - **~96% data coverage** - Only replicates data from `idc-open-data` bucket (missing ~4% from other buckets) @@ -347,7 +347,7 @@ credentials, project = default() credentials.refresh(Request()) # Build authenticated request -base_url = "https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v23/dicomWeb" +base_url = "https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v24/dicomWeb" response = requests.get( f"{base_url}/studies", diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/index_tables_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/index_tables_guide.md index d0bc9696..a7533b52 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/index_tables_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/index_tables_guide.md @@ -1,10 +1,11 @@ --- title: "Index Tables Guide for IDC" task: "" +persona: "running" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/index_tables_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/index_tables_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: prompt upstream_changes: accepted author: upstream @@ -13,7 +14,7 @@ validated: false # Index Tables Guide for IDC -**Tested with:** idc-index 0.11.14 (IDC data version v23) +**Tested with:** idc-index 0.12.5 (IDC data version v24) This guide covers the structure and access patterns for IDC index tables: programmatic schema discovery, DataFrame access, and join column references. For the overview of available tables and their purposes, see the "Index Tables" section in the main SKILL.md. @@ -30,9 +31,35 @@ For SQL query examples (filter discovery, finding annotations, size estimation), ## Prerequisites -```bash -pip install --upgrade idc-index -``` +Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed +version and prints the install command for the interpreter you are running. + +## Available Tables + +`SKILL.md` carries a compact map of the table families. This is the full inventory with row +granularity and contents. Always call `client.fetch_index("table_name")` before querying any +of them — it is safe and idempotent for all tables, including those loaded automatically at +startup. + +| Table | Row Granularity | Description | +|-------|-----------------|-------------| +| `index` | 1 row = 1 DICOM series | Primary metadata for all current IDC data | +| `version_metadata_index` | 1 row = 1 IDC release version | IDC version release timestamps; join on `idc_version` to correlate series with their release date | +| `collections_index` | 1 row = 1 collection | Collection-level metadata and descriptions | +| `analysis_results_index` | 1 row = 1 analysis result collection | Metadata about derived datasets (annotations, segmentations) | +| `clinical_index` | 1 row = 1 (collection, table, column) triple | Dictionary mapping clinical data table columns to collections | +| `sm_index` | 1 row = 1 slide microscopy series | Slide Microscopy (pathology) series metadata | +| `sm_instance_index` | 1 row = 1 slide microscopy instance | Instance-level (SOPInstanceUID) metadata for slide microscopy | +| `seg_index` | 1 row = 1 DICOM Segmentation series | Segmentation metadata: algorithm, segment count, reference to source image series | +| `ann_index` | 1 row = 1 DICOM ANN series | Microscopy Bulk Simple Annotations series metadata; references annotated image series | +| `ann_group_index` | 1 row = 1 annotation group | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm | +| `contrast_index` | 1 row = 1 series with contrast info | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) | +| `volume_geometry_index` | 1 row = 1 CT/MR/PT series | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite `regularly_spaced_3d_volume` flag | +| `rtstruct_index` | 1 row = 1 RTSTRUCT series | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID | +| `ct_index` | 1 row = 1 CT series | CT acquisition/reconstruction parameters: pixel spacing, slice thickness, kVp, convolution kernel, tube current (min/max for dose-modulated), exposure, spiral pitch, scan options | +| `mr_index` | 1 row = 1 MR series | MR acquisition/sequence parameters: field strength, scanning sequence, TE (array for multi-echo), TR, flip angle, DiffusionBValue (array for DWI), pixel bandwidth, receive coil, number of temporal positions | +| `pt_index` | 1 row = 1 PET series | PET acquisition/reconstruction/radiopharmaceutical parameters: series type, units, decay/scatter/attenuation correction, reconstruction method, radionuclide, injected dose, frame duration (array for dynamic PET) | +| `prior_versions_index` | 1 row = 1 DICOM series | **Reproducibility only.** Contains series permanently removed from IDC (all `max_idc_version` < current version; zero overlap with `index`). Use ONLY when a user explicitly needs to reproduce work from a prior IDC version using data no longer in the current release. Do NOT use for version history or "what's new" questions — those use `series_init_idc_version`/`series_revised_idc_version` in the main `index` table. Column names `min_idc_version`/`max_idc_version` here are NOT equivalent to `series_init_idc_version`/`series_revised_idc_version` in `index`. | ## Accessing Index Tables @@ -47,7 +74,7 @@ results = client.sql_query("SELECT * FROM index WHERE Modality = 'CT' LIMIT 10") # Fetch and query additional indices client.fetch_index("collections_index") -collections = client.sql_query("SELECT collection_id, CancerTypes, TumorLocations FROM collections_index") +collections = client.sql_query("SELECT collection_id, cancer_types, tumor_locations FROM collections_index") client.fetch_index("analysis_results_index") analysis = client.sql_query("SELECT * FROM analysis_results_index LIMIT 5") @@ -100,6 +127,29 @@ schema = client.get_index_schema("index") # Returns same schema dict: {'table_description': ..., 'columns': [...]} ``` +### Finding which table contains a column + +The most common schema question is "where does `SliceThickness` live?" — the primary `index` +holds series-level metadata only, so modality-specific acquisition parameters are in dedicated +tables. Search the overview rather than guessing; neither call fetches anything: + +```python +# Find which table(s) contain a specific column (no fetch required) +target = "SliceThickness" +for table_name, info in client.indices_overview.items(): + if any(c["name"] == target for c in info["schema"]["columns"]): + print(f"'{target}' is in: {table_name}") +# → 'SliceThickness' is in: ct_index + +# List all columns in a table from the schema (no fetch required) +ct_cols = [c["name"] for c in client.indices_overview["ct_index"]["schema"]["columns"]] +print("ct_index columns:", ct_cols) +# → ['SeriesInstanceUID', 'PixelSpacing_row_mm', 'PixelSpacing_col_mm', 'Rows', +# 'Columns', 'SliceThickness', 'KVP', 'ConvolutionKernel', ...] +``` + +Then `client.fetch_index("ct_index")` and join to `index` on `SeriesInstanceUID`. + ## Key Columns Reference Most common columns in the primary `index` table (use `indices_overview` for complete list and descriptions): @@ -143,6 +193,9 @@ Use this table to identify join columns between index tables. Always call `clien | `index` | `volume_geometry_index` | `index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID` | | `index` | `rtstruct_index` | `index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID` | | `rtstruct_index` | `index` (source images) | `rtstruct_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID` | +| `index` | `ct_index` | `index.SeriesInstanceUID = ct_index.SeriesInstanceUID` | +| `index` | `mr_index` | `index.SeriesInstanceUID = mr_index.SeriesInstanceUID` | +| `index` | `pt_index` | `index.SeriesInstanceUID = pt_index.SeriesInstanceUID` | For complete query examples using these joins, see `references/sql_patterns.md`. diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/licensing_and_citation.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/licensing_and_citation.md new file mode 100644 index 00000000..8afdf58e --- /dev/null +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/licensing_and_citation.md @@ -0,0 +1,243 @@ +--- +title: "Licensing and Citation Guide for IDC" +task: "" +lineage_type: import +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/licensing_and_citation.md +upstream_sha: d661d27e +imported_at: 2026-08-11 +prompt_class: prompt +upstream_changes: accepted +author: upstream +validated: false +--- + +# Licensing and Citation Guide for IDC + +## When to Use This Guide + +Load this guide when: +- A user asks whether IDC data can be used commercially, redistributed, or included in a product +- You are assembling a cohort that mixes collections and need to know which terms govern it +- A user is publishing results and needs formatted citations (APA, BibTeX, CSL JSON, RDF Turtle) +- You need the parameters or output formats for citation generation + +The obligation summary — check the license, generate citations — lives in `SKILL.md`. This +guide holds the detail behind it. + +**These are the two IDC tasks least tied to any one access path.** Licenses and citations are +available identically from `idc-index`, the REST API, and the hosted MCP server. Use whichever +route the session is already on rather than installing Python to answer a licensing question, +or dropping out of an MCP session to run a script. + +| Task | `idc-index` (Python) | REST API | MCP server | +|------|----------------------|----------|------------| +| License breakdown for a selection | `sql_query` on `license_short_name` | `POST /v3/licenses` | `get_licenses` | +| Citations for a selection | `citations_from_selection()` | `POST /v3/citations` | `get_citations` | + +Route-specific detail lives in `references/rest_api_guide.md` (endpoint reference, filter +syntax, and the body-shape pitfall that makes a mis-shaped filter return all of IDC) and +`references/mcp_guide.md` (tool inventory). The license semantics below apply to all three. + +## Licenses in IDC + +Every DICOM file in IDC is tagged with its license in the file metadata, and every row in the +`index` table carries a `license_short_name` column. There is no single IDC-wide license. + +| License | Share of data | Commercial use | Attribution required | +|---------|---------------|----------------|----------------------| +| CC BY 4.0 | 74.7% | Yes | Yes | +| CC BY 3.0 | 22.1% | Yes | Yes | +| CC BY-NC 4.0 | 2.1% | **No** | Yes | +| CC BY-NC 3.0 | 0.8% | **No** | Yes | +| NLM Terms and Conditions | 0.3% | Read the terms | Yes | + +About 97% of IDC data by size permits commercial reuse; just under 3% is non-commercial. Treat +any `license_short_name` that is not a recognizable Creative Commons string as custom, and +report the exact value to the user rather than assuming it permits reuse. + +**Licenses attach to individual series, not to whole collections.** 39 of IDC's 176 collections +carry more than one license — analysis results and original images within one collection can +differ, as can series from different sources. Never conclude that a collection is +commercially usable from one series, or from the collection's headline license: group by +`license_short_name` over the exact selection you intend to use. + +**When a cohort mixes licenses, the most restrictive term governs the combined dataset.** If a +selection contains any CC BY-NC series, either drop those series or tell the user the whole +derived dataset is non-commercial. + +Commercially restricted data is also physically separated in cloud storage: the +`idc-open-data-cr` (AWS) / `idc-open-cr` (GCS) buckets hold the CC BY-NC collections. See +`references/cloud_storage_guide.md` for bucket details. + +## Checking licenses + +### Via `idc-index` + +```python +from idc_index import IDCClient +client = IDCClient() + +# Licenses across all collections +licenses = client.sql_query(""" + SELECT DISTINCT + collection_id, + license_short_name, + COUNT(DISTINCT SeriesInstanceUID) as series_count + FROM index + GROUP BY collection_id, license_short_name + ORDER BY collection_id +""") +print(licenses) +``` + +```python +# Licenses present in one specific cohort — run this before handing a dataset to a user +cohort_licenses = client.sql_query(""" + SELECT license_short_name, COUNT(DISTINCT SeriesInstanceUID) as series_count + FROM index + WHERE Modality = 'MR' AND BodyPartExamined = 'BREAST' + GROUP BY license_short_name +""") +print(cohort_licenses) +``` + +```python +# Commercial-safe subset: exclude non-commercial collections outright +commercial_ok = client.sql_query(""" + SELECT collection_id, SeriesInstanceUID + FROM index + WHERE Modality = 'CT' + AND license_short_name NOT LIKE '%NC%' + LIMIT 20 +""") +``` + +### Via the REST API + +`POST /v3/licenses` takes the filter object **directly** (not wrapped in a `filters` key) and +returns the per-license breakdown with series counts and sizes: + +```bash +B=https://api.imaging.datacommons.cancer.gov/v3 +curl -s $B/licenses \ + -H 'content-type: application/json' \ + -d '{"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}' +``` + +Response shape: `licenses[{license_short_name, series, size_TB}]`. A collection's licenses are +also included in `GET /v3/collections/{id}`. + +### Via the MCP server + +Call `get_licenses` with the same selection you built with `build_cohort`. The result carries +the same per-license breakdown; the CC BY vs CC BY-NC distinction above applies unchanged. + +## Citations and attribution + +The `source_DOI` column links to the publications describing how each dataset was generated. +All three routes turn a selection into formatted citations that satisfy the attribution +requirement common to every IDC license. + +Generate citations from the *same* selection you downloaded, not from the collection as a +whole — a five-series subset of a collection that spans several source publications should +cite only the publications it actually draws on. + +### Via `idc-index` + +```python +# Citations for a collection (APA is the default format) +citations = client.citations_from_selection(collection_id="rider_pilot") +for citation in citations: + print(citation) +``` + +```python +# Citations for a specific set of series — matches what you actually downloaded +results = client.sql_query(""" + SELECT SeriesInstanceUID FROM index + WHERE collection_id = 'tcga_luad' LIMIT 5 +""") +citations = client.citations_from_selection( + seriesInstanceUID=list(results['SeriesInstanceUID'].values) +) +``` + +```python +# BibTeX, for LaTeX manuscripts +bibtex_citations = client.citations_from_selection( + collection_id="tcga_luad", + citation_format=IDCClient.CITATION_FORMAT_BIBTEX +) +``` + +`citations_from_selection()` takes the same selection filters as the download methods — +`collection_id`, `patientId`, `studyInstanceUID`, `seriesInstanceUID` — plus `citation_format`. + +### Via the REST API + +`POST /v3/citations` **wraps** the filter in a `filters` key (unlike `/v3/licenses` — this +asymmetry is the single most common REST mistake; see `references/rest_api_guide.md`): + +```bash +curl -s $B/citations \ + -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}, "citation_format": "bibtex"}' +``` + +The response separates the per-dataset `citations[]` from `idc_acknowledgment` (the IDC paper) +and `recommendation`. Include both parts — see *What to include when publishing* below. + +### Via the MCP server + +Call `get_citations` for a selection. It returns the per-dataset citations plus the IDC paper +to acknowledge IDC itself, matching the REST response. + +### Citation formats + +| `idc-index` constant | REST / MCP `citation_format` | Output | +|----------------------|------------------------------|--------| +| `IDCClient.CITATION_FORMAT_APA` | `apa` (default) | APA string | +| `IDCClient.CITATION_FORMAT_BIBTEX` | `bibtex` | BibTeX entry, for LaTeX | +| `IDCClient.CITATION_FORMAT_JSON` | `csl-json` | CSL JSON | +| `IDCClient.CITATION_FORMAT_TURTLE` | `turtle` | RDF Turtle | + +## What to include when publishing + +1. **The dataset citations** for every collection or series set used. +2. **The IDC data version** — `client.get_idc_version()`, `GET /v3/version`, or the MCP + `get_idc_version` tool. IDC releases are versioned and series are added and revised between + them, so the version is what makes the selection reproducible. +3. **The IDC platform citation**, to acknowledge IDC itself. The REST and MCP routes return + this as `idc_acknowledgment`; when using `idc-index`, add it yourself: + + > Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, + > Reproducibility, and Scalability in Imaging Artificial Intelligence." *RadioGraphics* 43.12 + > (2023). https://doi.org/10.1148/rg.230180 + +4. **The series manifest** — save the `SeriesInstanceUID` list alongside the analysis so the + exact cohort can be rebuilt. + +## Troubleshooting + +### Issue: Fewer citations returned than collections selected + +- **Cause:** Citations are derived from `source_DOI`, and several collections can share one + DOI, so a multi-collection selection may legitimately produce a shorter list. +- **Solution:** Query `SELECT DISTINCT collection_id, source_DOI FROM index WHERE ...` to see + the mapping directly. + +### Issue: `POST /v3/citations` returns citations for all of IDC + +- **Cause:** The filter was passed directly instead of wrapped in `filters`. `/v3/licenses` + takes the filter directly; `/v3/citations` wraps it. A mis-shaped body is not an error — it + is treated as an empty filter. +- **Solution:** Check the response counts against a `POST /v3/cohort/counts` for the same + selection. See `references/rest_api_guide.md`. + +## Resources + +- **IDC Portal** — https://portal.imaging.datacommons.cancer.gov/ +- **IDC data licensing documentation** — https://learn.canceridc.dev/data/licensing +- **`references/rest_api_guide.md`** — `/v3/licenses` and `/v3/citations` endpoint reference +- **`references/mcp_guide.md`** — `get_licenses` and `get_citations` tool inventory +- **`references/cloud_storage_guide.md`** — bucket separation for commercially restricted data diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/mcp_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/mcp_guide.md new file mode 100644 index 00000000..da825372 --- /dev/null +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/mcp_guide.md @@ -0,0 +1,194 @@ +--- +title: "IDC MCP Server Guide" +task: "" +lineage_type: import +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/mcp_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 +prompt_class: prompt +upstream_changes: accepted +author: upstream +validated: false +--- + +# IDC MCP Server Guide + +IDC operates a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server that +exposes IDC discovery and metadata as agent tools. This guide covers how to recognize it, how +to divide work between it and `idc-index`, and what to hand off across that boundary. + +The server is optional. Everything in `SKILL.md` works without it. + +## Endpoint + +| Property | Value | +|----------|-------| +| URL | `https://api.imaging.datacommons.cancer.gov/mcp` | +| Transport | Streamable HTTP (`streamable-http`, sometimes spelled `http`) | +| Authentication | None | +| Server identity | `IDC (Imaging Data Commons)` | + +## Identifying the server + +Tool names and resource URIs are defined by the server, so they are the same on every host. +Use them, not host-specific naming conventions, to decide whether the server is present. + +**Strongest signal — resource URIs.** The server publishes two resources under an `idc://` +scheme: + +| URI | Content | +|-----|---------| +| `idc://guide` | Data model and recommended workflow (Markdown) | +| `idc://tables` | Tables available to `run_sql`, with descriptions and column counts (JSON) | + +If the host can enumerate MCP resources, a resource with URI `idc://guide` identifies the +server unambiguously. + +**Fallback — tool-name fingerprint.** Require three or more of `build_cohort`, +`get_cohort_urls`, `list_analysis_results`, and `get_idc_version`. Do not treat `run_sql`, +`get_stats`, `list_tables`, or `get_citations` as evidence on their own; those names are +generic enough that another server could expose them. + +**This is disambiguation, not authentication.** No runtime check can prove the server on the +other end is operated by NCI — a hostile server could serve `idc://guide` and name its tools +anything. The trust anchor is the URL the user configured plus TLS, which is established when +the server is added, not when the skill runs. That is sufficient for routing: the check only +has to distinguish IDC from the user's other installed servers. + +**Fail soft.** If identification is ambiguous, or a tool call fails, fall back rather than +reporting an error — to the REST API (`rest_api_guide.md`) for read-only metadata, which is the +same service with no configuration, or to `idc-index` when it is already installed or the task +needs downloads or local analysis. Tool names may change as the server matures. + +## Tool inventory + +Verified against server version `3.0.0b3`. Treat this as a snapshot, not a contract — call +the server's own listing rather than assuming this list is current. + +| Group | Tools | +|-------|-------| +| Version and scale | `get_idc_version`, `get_stats` | +| Collections | `list_collections`, `get_collection`, `list_analysis_results` | +| Attribute grounding | `list_attributes`, `get_attribute_values` | +| Cohorts | `build_cohort`, `get_cohort_urls` | +| SQL | `list_tables`, `get_table_schema`, `run_sql` | +| Clinical data | `list_clinical_tables`, `get_clinical_table_schema`, `get_clinical_table` | +| Attribution | `get_citations`, `get_licenses` | +| Visualization | `get_viewer_url` | + +The server ships its own usage instructions, which most hosts inject automatically. Follow +those instructions for tool sequencing (ground with `list_attributes` / +`get_attribute_values` before filtering; check `list_tables` before writing SQL). Do not +re-derive that workflow from `SKILL.md` — the two would drift apart on the server's next +release. + +**Cohort results report their own filters.** `build_cohort` and `get_cohort_urls` require at +least one filter predicate and fail cleanly without one, rather than returning the whole archive; +results echo the filters actually applied along with warnings for any predicate that was dropped +or any value whose casing did not match. Read those warnings before reporting a count — a zero +with no warning means the filter matched nothing, which is a real answer. Same contract as the +REST endpoints they wrap; see `rest_api_guide.md`. + +## Division of labor + +The server and `idc-index` overlap on metadata queries and diverge everywhere else. + +| Task | Use | +|------|-----| +| IDC data version, collection and series counts | Server (`get_idc_version`, `get_stats`) | +| Valid filter values before building a query | Server (`get_attribute_values`) | +| Cohort selection by attribute filters | Server (`build_cohort`) | +| One-off metadata SQL, answer consumed as prose | Server (`run_sql`) | +| Metadata SQL whose result feeds local Python | `idc-index` (`client.sql_query`) | +| Downloading DICOM files | `idc-index` (`client.download_from_selection`) | +| pandas / notebook analysis, plotting | `idc-index` | +| Reading pixel data (pydicom, SimpleITK) | `idc-index` + local files | +| DICOMweb, BigQuery, direct S3/GCS, Parquet | `idc-index` and the relevant reference guide | +| Digital pathology tiling and annotation workflows | `idc-index` + `digital_pathology_guide.md` | +| Reproducible scripts a user will re-run | `idc-index` (a script outlives the session) | + +Two rules resolve the overlap: + +- **Prefer the server for discovery.** It is hosted against a current IDC release, so it does + not depend on the `idc-index` version pinned in `SKILL.md`. +- **Prefer `idc-index` when the result must become a Python object.** Round-tripping a + DataFrame through tool output wastes context and loses types. + +## Handing off from the server to `idc-index` + +The boundary artifact is a list of `SeriesInstanceUID` values. + +```python +# UIDs obtained from the MCP server's build_cohort / run_sql output +series_uids = [ + "1.3.6.1.4.1.14519.5.2.1.7009.2403.334240657131972136850343327463", + # ... +] + +from idc_index import IDCClient +client = IDCClient() + +# Confirm size before downloading — the server reports size_TB, but re-check locally +sizes = client.sql_query(f""" + SELECT COUNT(*) AS series, SUM(series_size_MB)/1000 AS size_GB + FROM index + WHERE SeriesInstanceUID IN ({','.join(f"'{u}'" for u in series_uids)}) +""") +print(sizes) + +client.download_from_selection( + downloadDir="./data", + seriesInstanceUID=series_uids, # a list, not a DataFrame + dirTemplate="%collection_id/%PatientID/%Modality", +) +``` + +Run `python scripts/check_version.py` before the first `idc-index` call in a session, even if +discovery happened server-side — the two components version independently. + +`get_cohort_urls` also returns ready-made `idc` CLI commands. Those are the better handoff +when the user wants a shell command they can re-run outside the session; see +`references/cli_guide.md`. + +Going the other direction, `idc-index` results are already local, so there is rarely a reason +to send them back to the server. + +## Version authority + +When the server is present, it is the authority on the IDC data version: call +`get_idc_version` rather than quoting the `idc-data-version` value in the `SKILL.md` +frontmatter, which records the release the skill was last verified against. + +If the server and a locally installed `idc-index` report different versions, say so and name +both. The mismatch is real — the hosted server tracks IDC releases independently of the user's +installed package — and it changes which answers about "what's new" are correct. + +## Host-specific notes + +Everything above is portable. The items below are not, and apply only to specific agent +environments. + +### Claude Code + +- **Tool naming.** MCP tools are exposed as `mcp____`, where `` is the + configured server name with every character outside `A-Za-z0-9_-` replaced by `_`. A CLI + install named `idc` yields `mcp__idc__build_cohort`; a claude.ai connector named + `IDC MCP prod` yields `mcp__claude_ai_IDC_MCP_prod__build_cohort`. +- **Enumerating resources.** `ListMcpResourcesTool` returns each resource with a `server` + field, which is how to find the `idc://guide` resource and the owning server name in one + call. +- **Adding the server.** + `claude mcp add --transport http idc https://api.imaging.datacommons.cancer.gov/mcp` +- **Permission rules.** Allow rules need a literal, glob-free server segment: `mcp__idc__*` + works, `mcp__*` does not. Connector installs need their own + `mcp__claude_ai___*` rule, so the rule differs by install path. +- **Detecting via the CLI does not work.** `claude mcp list` reads only file-based + configuration (`~/.claude.json`, `.mcp.json`). It reports "No MCP servers configured" for a + claude.ai connector that is connected and working in the same session, so it cannot be used + as a presence check. + +### Other hosts + +Any agent that supports MCP over streamable HTTP can use the server. Consult that agent's +documentation for how servers are registered and how tool names are namespaced; the endpoint +URL and the absence of authentication are all the configuration it needs. diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/parquet_access_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/parquet_access_guide.md index 2e233250..bfd70cac 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/parquet_access_guide.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/parquet_access_guide.md @@ -2,9 +2,9 @@ title: "Direct Parquet Access Guide for IDC" task: "" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/parquet_access_guide.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/parquet_access_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: prompt upstream_changes: accepted author: upstream @@ -13,20 +13,23 @@ validated: false # Direct Parquet Access Guide for IDC -**Tested with:** idc-index-data 23.10.1, DuckDB 1.x +**Tested with:** idc-index-data 24.2.2 (IDC data version v24), DuckDB 1.5 -All idc-index metadata tables are published as Parquet files to a public GCS bucket with unrestricted CORS access. This enables metadata queries with DuckDB or pandas without installing idc-index — useful for quick exploration or environments where pip install is unavailable. +All idc-index metadata tables are published as Parquet files to a public GCS bucket with unrestricted CORS access. This enables metadata queries with DuckDB or pandas without installing idc-index. **Limitation:** download helpers (`download_from_selection()`), viewer URLs (`get_viewer_URL()`), and citation generation require the idc-index client and are not available from raw Parquet files. +**This is not the first no-install option to reach for.** It still needs DuckDB installed, and the per-collection clinical tables are not published here — only the `clinical_index` dictionary. For ad-hoc metadata with nothing installed, the REST API (`rest_api_guide.md`) needs no install at all and reaches `clinical.
` through `POST /sql`. + ## When to Use This Guide Load this guide when you need to: -- Query IDC metadata without installing idc-index -- Run ad-hoc DuckDB queries against the latest index files -- Access `volume_geometry_index` or `rtstruct_index` for geometry validation or RT structure queries +- Pin queries to a specific IDC data version (see *Pinning to a Specific Version* below) rather than whatever the hosted API currently serves +- Return more rows than the REST `/sql` ceiling of 10 000 +- Run heavy or repeated local DuckDB analysis without driving the hosted API +- Query IDC metadata where DuckDB is available but idc-index is not -For full API access (downloads, viewer, citations), use idc-index as documented in the main SKILL.md. +For downloads, viewer URLs, and citations, use idc-index as documented in the main SKILL.md. ## URL Pattern @@ -51,16 +54,17 @@ https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifact | `collections_index.parquet` | — | Collection-level metadata | | `analysis_results_index.parquet` | — | Derived dataset metadata | | `clinical_index.parquet` | ~0.2 MB | Clinical data column dictionary | +| `ct_index.parquet` | — | CT acquisition/reconstruction parameters | +| `mr_index.parquet` | — | MR sequence/acquisition parameters | +| `pt_index.parquet` | — | PET acquisition/radiopharmaceutical parameters | | `prior_versions_index.parquet` | — | Series from previous IDC releases | **Note:** the main index file is named `idc_index.parquet`, not `index.parquet`. Reference it with an alias in SQL queries (e.g., `FROM read_parquet(...) AS index`). ## Prerequisites -```bash -pip install duckdb -# or: uv add duckdb -``` +Install the Python `duckdb` package, using whatever installer manages the environment you are +running in. DuckDB reads Parquet directly from HTTPS URLs using HTTP range requests — no GCS client library or authentication required. diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/rest_api_guide.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/rest_api_guide.md new file mode 100644 index 00000000..037bc4ef --- /dev/null +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/rest_api_guide.md @@ -0,0 +1,626 @@ +--- +title: "IDC REST API Guide" +task: "" +persona: "running" +lineage_type: import +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/rest_api_guide.md +upstream_sha: d661d27e +imported_at: 2026-08-11 +prompt_class: prompt +upstream_changes: accepted +author: upstream +validated: false +--- + +# IDC REST API Guide + +**Tested with:** API `3.0.0b3` (build `0640860`), IDC data version v24, `idc_index_data_version` 24.2.2 + +IDC operates a hosted REST API that exposes discovery, cohort building, metadata SQL, and +download manifests over plain HTTP. No authentication, account, or credentials are required — +every example on this page can be run from any terminal with `curl`. + +The API and the [MCP server](mcp_guide.md) are the same service behind two transports: the MCP +tools wrap these endpoints. Both are hosted against a current IDC release, independently of the +`idc-index` version installed locally. + +## When to Use This Guide + +Load this guide when you need to: +- Query IDC from a language or environment with no `idc-index` install (shell, R, Java, JS, a + notebook without pip access) +- Build cohorts and manifests over HTTP for a pipeline or web application +- Run metadata SQL against a current IDC release without downloading local index files +- Hand a user copy-pasteable `curl` commands they can run anywhere + +For downloads, pandas/notebook analysis, reading pixel data, DICOMweb, BigQuery, or digital +pathology tiling, use `idc-index` as documented in `SKILL.md`. The API never moves image bytes: +it returns public `s3://` URLs and manifests, and the transfer happens directly from cloud +storage to the client. + +**Choosing among the three interfaces:** + +| Situation | Use | +|-----------|-----| +| Session already has the hosted MCP server | MCP tools (`mcp_guide.md`) — same data, no HTTP plumbing | +| `idc-index` already installed, results feed pandas / downloads | `idc-index` (`SKILL.md`) | +| `idc-index` **not** installed and the task is read-only metadata | This REST API — do not install to answer a metadata question | +| No Python, or a non-Python client, or shell commands the user re-runs | This REST API | +| Installed `idc-index` is a whole IDC data release behind and cannot be upgraded here | REST API for the query, **direct bucket transfer** may be needed for the download — `idc-index` cannot fetch what its index does not list | + +Being hosted, the API always serves a current IDC release — but so does an up-to-date +`idc-index`. "I want the newest data" is not on its own a reason to prefer the API: the normal +fix for a stale local index is to upgrade it. Reach for the API on version grounds only when +upgrading is not an option in that environment. Avoiding the install *is* a reason, though: the +packaged index data is ~77 MB before pandas, pyarrow, and duckdb, which a metadata question does +not need. + +## Endpoint and Versioning + +| Property | Value | +|----------|-------| +| Base URL | `https://api.imaging.datacommons.cancer.gov/v3` | +| Authentication | None | +| Content type | `application/json` (request and response), except `cohort/manifest.txt` → `text/plain` | +| Interactive docs | https://api.imaging.datacommons.cancer.gov/v3/docs (Swagger UI) | +| OpenAPI spec | https://api.imaging.datacommons.cancer.gov/v3/openapi.json | +| Source | https://github.com/ImagingDataCommons/IDC-REST-MCP | + +**v3 is in beta.** The contract may still change before the final `3.0.0` release, so verify the +running build rather than assuming the values in this guide: + +```bash +curl -s https://api.imaging.datacommons.cancer.gov/v3/version +# {"idc_version":"v24","idc_index_data_version":"24.2.2","api_version":"3.0.0b3","build":"0640860"} +``` + +`idc_version` is the IDC data release the API serves and is the authority when the API is in +use — prefer it over the `idc-data-version` pinned in the `SKILL.md` frontmatter, which records +the release the skill was last verified against. + +### Checking the API against a local idc-index + +The API and `idc-index` are built on the same `idc-index-data` package, and both report its +version, so consistency is an exact check rather than a guess: + +| Side | Data release (coarse) | `idc-index-data` version (exact) | +|------|-----------------------|----------------------------------| +| API | `GET /v3/version` → `idc_version` | `GET /v3/version` → `idc_index_data_version` | +| Local | `IDCClient().get_idc_version()` | `idc_index_data.__version__` | + +```python +import idc_index_data +import requests + +api = requests.get("https://api.imaging.datacommons.cancer.gov/v3/version", timeout=30).json() +api_version, local_version = api["idc_index_data_version"], idc_index_data.__version__ + +if api_version.split(".")[0] != local_version.split(".")[0]: + print(f"Different IDC data release: API {api_version}, local {local_version} — upgrade") +elif api_version != local_version: + print(f"Same data release, different index build: API {api_version}, local {local_version}") + # → Same data release, different index build: API 24.2.2, local 24.2.0 +``` + +**The major is the IDC data release; the rest is the index build.** `idc-index-data` `24.x.y` +serves IDC `v24` — 24.0.0 shipped with the v24 release, and 24.1.0 / 24.2.x are later builds of +the *same* release. What differs between them is the index itself (added tables and columns, +corrected metadata), never which series IDC contains. + +So read a mismatch by its position: + +| Difference | Means | Consequence | +|------------|-------|-------------| +| Major (24.x.y vs 25.x.y) | Different IDC data release | Series added, revised, or removed. Counts legitimately differ, and `idc-index` **cannot download** what its index does not list | +| Minor or patch (24.2.0 vs 24.2.2) | Same data release, different index build | Same series everywhere; downloads are unaffected. A metadata query can still differ if it touches a column that was added or corrected | + +Comparing `idc_version` alone cannot make this distinction in the other direction either — the +`vNN` label is exactly the major, so matching `v24` on both sides tells you the release agrees +but says nothing about the index build. + +When the two disagree, say so and name both versions, then reconcile rather than mixing +results: upgrading `idc-index` brings the local side to the newer `idc-index-data`, and +`python scripts/check_version.py` reports whether an upgrade is available and prints the +command for the interpreter you are running. Do not present API-derived and locally-derived counts side by side as if they came +from one index. + +**A major behind also breaks downloads.** `idc-index` resolves every `s3://` URL it is given +against its *own* index, so a manifest built from a newer IDC data release can name series it +has never heard of — see *When the local index is a data release behind the API* under +**Getting the Data**. + +### Use v3 only — V1 and V2 are being retired + +**Do not write new code against the V1 or V2 IDC APIs, and do not follow V1/V2 examples found +in older tutorials, notebooks, blog posts, or forum answers.** Both are superseded by v3 and +are scheduled to be deprecated and shut down; code written against them will stop working. If +a user brings V1 or V2 code, say so and port it to v3 rather than extending it. + +Two signals that a snippet is V1/V2 rather than v3, both of which will fail against `/v3`: + +- a base URL other than `https://api.imaging.datacommons.cancer.gov/v3` (for example a + `/v1/` or `/v2/` path segment) +- per-attribute filter suffixes such as `Modality_btw` or `_lt` / `_gt`, instead of v3's + separate `terms` and `ranges` objects + +V1/V2 documentation survives only in the [IDC docs archive](https://learn.canceridc.dev/archive/archive) +for historical reference. Treat it as read-only history, not as a source of working examples. + +## The Query Surfaces + +The endpoints group into five surfaces that build on each other: + +| Surface | Answers | Endpoints | +|---------|---------|-----------| +| Discovery | What exists? What can I filter on? | `GET /version`, `/stats`, `/collections`, `/collections/{id}`, `/analysis_results`, `/attributes`, `/attributes/{attr}/values` | +| Cohort | How big is my selection, and what's in it? | `POST /cohort/counts`, `POST /cohort/manifest` | +| Retrieval | Give me the download links | `POST /cohort/manifest.txt` | +| SQL | Anything relational or aggregate | `GET /tables`, `GET /tables/{table}`, `POST /sql` | +| Side tools | View / cite / license-check a cohort | `GET /viewer-url`, `POST /citations`, `POST /licenses` | + +Discovery supplies the vocabulary (attribute names and their valid values) that the cohort +filters consume. SQL is the escape hatch for questions structured filters cannot express. + +Clinical data has its own discovery endpoints (`GET /clinical/tables`, +`/clinical/tables/{table}`, `/clinical/tables/{table}/rows`) and is filtered or joined through +SQL against the `clinical` schema. + +## Endpoint Reference + +All paths are relative to `https://api.imaging.datacommons.cancer.gov/v3`. + +| Method & path | Purpose | Key response fields | +|---------------|---------|---------------------| +| `GET /version` | IDC data release + API build | `idc_version`, `idc_index_data_version`, `api_version`, `build` | +| `GET /stats` | Headline totals | `collections`, `patients`, `studies`, `series`, `instances`, `size_TB` | +| `GET /collections` | List collections (JSON array) | `collection_id`, `collection_name`, `cancer_types`, `tumor_locations`, `species`, `subjects`, `description` | +| `GET /collections/{id}` | Collection detail | the above plus `patients`, `studies`, `series`, `instances`, `size_TB`, `modalities[]`, `licenses[]` | +| `GET /analysis_results` | Derived datasets (JSON array) | `analysis_result_id`, `analysis_result_title`, `source_DOI`, `subjects`, `collections`, `modalities`, `license_short_name` | +| `GET /attributes` | Filterable attributes | `name`, `table`, `data_type`, `kind` (`term` \| `range`), `categorical`, `description` | +| `GET /attributes/{attr}/values?limit=` | Distinct values with counts | `attribute`, `values[{value,count}]`, `truncated`, `note` | +| `GET /tables` | Tables available to SQL | `tables[{name,description,column_count}]` | +| `GET /tables/{table}` | Column schema | `name`, `description`, `columns[{name,type,description}]` | +| `GET /clinical/tables?collection_id=` | Clinical tables, optionally one collection | `tables[{table_name,sql_path,collection_id,column_count}]` | +| `GET /clinical/tables/{table}` | Clinical columns + labels | `name`, `columns[{name,type,description}]` | +| `GET /clinical/tables/{table}/rows?max_rows=` | Clinical rows (capped) | `columns`, `rows`, `row_count`, `truncated`, `max_rows` | +| `POST /cohort/counts` | Distinct counts for a filter (cheap) | `patients`, `studies`, `series`, `instances`, `size_TB` | +| `POST /cohort/manifest` | Counts + page of series + download payload | `counts`, `page`, `page_size`, `returned`, `total_series`, `series[]`, `download` | +| `POST /cohort/manifest.txt` | Full manifest as `text/plain` | one `s3://…/*` URL per line | +| `POST /sql` | Guarded read-only SQL (DuckDB) | `columns`, `rows`, `row_count`, `truncated`, `max_rows` | +| `GET /viewer-url` | OHIF / Slim viewer link | `viewer_url`, `viewer`, `study_instance_uid`, `series_instance_uid` | +| `POST /citations` | Citations for a cohort | `format`, `citations[]`, `idc_acknowledgment`, `recommendation` | +| `POST /licenses` | License breakdown for a cohort | `licenses[{license_short_name,series,size_TB}]` | + +`GET /health` and `GET /v3` (API root) also exist for liveness checks. + +## Filter Syntax + +Cohort filters are shared by `cohort/counts`, `cohort/manifest`, `cohort/manifest.txt`, +`licenses`, and `citations`. **The filter object always goes under a `filters` key**, on every one +of them. It has two parts: + +- **`terms`** — `{attribute: [values]}` for equality/membership. Values are **OR**'d within an + attribute and **AND**'d across attributes. +- **`ranges`** — `{attribute: {"gte": x, "lte": y}}` for numeric and date attributes. Either + bound may be omitted for an open-ended range. + +```json +{ + "filters": { + "terms": {"Modality": ["CT"], "collection_id": ["nlst"]}, + "ranges": {"instanceCount": {"gte": 100, "lte": 200}} + } +} +``` + +Filters operate only on the `index` table's filterable attributes — 19 of them as of `3.0.0b3`: + +| Kind | Attributes | +|------|------------| +| `term` | `collection_id`, `analysis_result_id`, `PatientID`, `StudyInstanceUID`, `SeriesInstanceUID`, `Modality`, `BodyPartExamined`, `Manufacturer`, `ManufacturerModelName`, `PatientSex`, `sop_class_name`, `license_short_name`, `source_DOI` | +| `range` | `instanceCount`, `series_size_MB`, `series_init_idc_version`, `series_revised_idc_version`, `StudyDate`, `SeriesDate` | + +`SeriesInstanceUID`, `StudyInstanceUID`, and `PatientID` being filterable is what makes the +side tools work at any granularity — the licenses or citations for a single series are just a +one-value filter. + +Anything outside this list — clinical values, segmented anatomy, per-modality acquisition +parameters — is not filterable here; use the SQL surface instead. + +**Ground values before filtering.** Call `GET /attributes` for what is filterable and whether +it is a term or a range, then `GET /attributes/{attr}/values` for real values and their casing — +values are matched case-sensitively. + +### The server reports what it filtered on + +Every filtered response echoes `filters_applied` and `warnings`, and misuse is refused rather +than ignored. Together these make a result self-describing, so you do not have to sanity-check a +count against a number you happen to remember. + +| Mistake | Response | +|---------|----------| +| Bare filter object, no `filters` key | `422` naming the correct shape | +| Unrecognized key at any depth (`term` for `terms`, a range bound spelled `min`) | `422` pointing at the key | +| Unknown filter attribute, or a range attribute used as a term | `400` naming the discovery call | +| Unfiltered `cohort/manifest` or `manifest.txt` | `400` — it will not enumerate the whole archive | +| Unfiltered `cohort/counts` or `licenses` | `200` plus an explicit "ENTIRE IDC archive" warning | +| A predicate that constrains nothing (`{"collection_id": []}`, `{"instanceCount": {}}`) | `200`, and `warnings` names the ignored predicate | +| Miscased value (`mr` for `MR`) | `200`, zero counts, and a warning naming the casing that exists | + +```bash +curl -s $B/cohort/counts -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}}' +# {"patients":8,"studies":154,"series":774,"instances":21111,"size_TB":0.011, +# "filters_applied":{"terms":{"collection_id":["rider_pilot"]},"ranges":{}},"warnings":[]} +``` + +**Read `warnings` before reporting any count.** A zero count with `warnings: []` means the filter +applied and matched nothing — that is a real answer. A zero count *with* a casing warning means +the filter was wrong. And `filters_applied` covers the case no shape check can: a request +carrying one good predicate plus one that constrains nothing returns a perfectly plausible +number, and only `warnings` reveals the dropped half. + +`cohort/manifest.txt` returns `text/plain` and so cannot carry these fields; there the +required-predicate `400` does the same job, appending the ignored-predicate reason to its message. + +## Worked Examples + +### Discovery + +```bash +B=https://api.imaging.datacommons.cancer.gov/v3 + +curl -s $B/version # data release + API build +curl -s $B/stats # headline totals +curl -s $B/collections # all 176 collections +curl -s $B/collections/rider_pilot # one collection: counts, modalities, licenses +curl -s $B/analysis_results # derived datasets (segmentations, annotations) +curl -s $B/attributes # what can be filtered, and how +curl -s "$B/attributes/Modality/values?limit=10" +``` + +### Cohort building + +Check size first — `counts` is cheap and answers "is this download sane?": + +```bash +curl -s $B/cohort/counts \ + -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}}' +# {"patients":3718,"studies":5689,"series":47986,"instances":6493262,"size_TB":2.421, +# "filters_applied":{...},"warnings":[]} +``` + +Then request a page of series plus the download payload — same filter, same `filters` key. + +```bash +curl -s $B/cohort/manifest \ + -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}, + "page": 0, "page_size": 3}' +``` + +Each `series[]` row carries `collection_id`, `PatientID`, `StudyInstanceUID`, +`SeriesInstanceUID`, `Modality`, `SeriesDescription`, `instanceCount`, `series_size_MB`, +`aws_bucket`, `crdc_series_uuid`, and `series_aws_url`. Set `"include_rows": false` to get +counts and the download payload without the rows. + +### SQL + +```bash +curl -s $B/sql \ + -H 'content-type: application/json' \ + -d '{"sql": "SELECT Modality, count(*) n FROM index GROUP BY 1 ORDER BY n DESC", "max_rows": 20}' +``` + +### Viewer, licenses, citations + +```bash +curl -s "$B/viewer-url?study_instance_uid=1.3.6.1.4.1.14519.5.2.1.7695.4164.129908397467389975396031099306" + +curl -s $B/licenses \ + -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}}' + +curl -s $B/citations \ + -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}, "citation_format": "bibtex"}' +``` + +`viewer-url` takes `series_instance_uid` or `study_instance_uid` (and an optional `viewer` +override); it picks OHIF v3 for radiology and Slim for slide microscopy automatically. + +`citation_format` is one of `apa` (default), `bibtex`, `csl-json`, `turtle`. The response +separates the per-dataset `citations[]` from `idc_acknowledgment`, the IDC paper — include +both when publishing. + +### Python client + +```python +import requests + +BASE = "https://api.imaging.datacommons.cancer.gov/v3" +session = requests.Session() + +def get(path, **params): + r = session.get(f"{BASE}{path}", params=params, timeout=60) + r.raise_for_status() + return r.json() + +def post(path, payload): + r = session.post(f"{BASE}{path}", json=payload, timeout=120) + r.raise_for_status() + return r.json() + +# 1. Confirm which IDC release the API is serving +print(get("/version")["idc_version"]) + +# 2. Ground the filter values before using them +modalities = {v["value"] for v in get("/attributes/Modality/values", limit=1000)["values"]} +assert "MR" in modalities + +# 3. Size the cohort, then page through it — the filter always goes under `filters` +filters = {"terms": {"collection_id": ["rider_pilot"], "Modality": ["CT"]}} +counts = post("/cohort/counts", {"filters": filters}) +assert not counts["warnings"], counts["warnings"] # nothing was silently dropped +print(counts) + +manifest = post("/cohort/manifest", {"filters": filters, "page": 0, "page_size": 100}) +uids = [row["SeriesInstanceUID"] for row in manifest["series"]] + +# 4. Hand off to idc-index for the download (see "Handing Off to idc-index" below) +``` + +Error responses raise through `raise_for_status()`; read `r.json()["error"]["message"]` for the +reason before retrying. + +## The SQL Surface + +`POST /sql` runs read-only DuckDB SQL against the same tables `idc-index` exposes locally. +Ground the schema with `GET /tables` and `GET /tables/{table}` — do not guess table or column +names. + +**Guardrails** (verified against `3.0.0b3`): + +- Only single read-only `SELECT` / `WITH … SELECT` statements are accepted. Anything else is + rejected with `{"error": {"code": "invalid_query", "message": "Only read-only SELECT (or WITH ... SELECT) statements are allowed."}}` and HTTP 400. +- A server row cap and per-query timeout apply. `max_rows` defaults to **5000** and is clamped + to **10000**; results carry `truncated: true` when the cap was hit, and echo the `max_rows` + actually applied. +- Invalid SQL returns DuckDB's own error text, including its "Candidate bindings" suggestions — + useful for fixing a misspelled column without another schema round-trip. + +**Tables reachable from SQL** are the `index` table plus the specialized indices documented in +`SKILL.md` (`collections_index`, `analysis_results_index`, `version_metadata_index`, +`prior_versions_index`, `seg_index`, `ann_index`, `ann_group_index`, `rtstruct_index`, +`ct_index`, `mr_index`, `pt_index`, `sm_index`, `sm_instance_index`, `contrast_index`, +`volume_geometry_index`, `clinical_index`). Join them to `index` on `SeriesInstanceUID`, except +where `SKILL.md` documents a different key (`segmented_SeriesInstanceUID`, +`referenced_SeriesInstanceUID`, `collection_id`, `analysis_result_id`). + +The SQL patterns in `references/sql_patterns.md` are written for `client.sql_query()` but the +SQL itself transfers unchanged — send it as the `sql` field. + +**Array columns:** columns typed `STRING[]` (e.g. `SegmentedPropertyType_CodeMeanings`) hold a +list per row. Match with `list_contains(col, 'value')`, not `=` or `LIKE`: + +```bash +curl -s $B/sql -H 'content-type: application/json' -d '{ + "sql": "SELECT i.collection_id, count(DISTINCT i.SeriesInstanceUID) AS slides FROM index i JOIN seg_index seg ON seg.segmented_SeriesInstanceUID = i.SeriesInstanceUID WHERE i.Modality = '\''SM'\'' AND list_contains(seg.SegmentedPropertyType_CodeMeanings, '\''Nucleus'\'') GROUP BY 1 ORDER BY slides DESC", + "max_rows": 20}' +``` + +**A SQL result can be a manifest.** Selecting `series_aws_url` gives download URLs directly, +and unlike `cohort/manifest.txt` you can carry extra columns alongside them — a per-row +`license_short_name`, for instance, which the plain manifest does not include: + +```sql +SELECT SeriesInstanceUID, license_short_name, series_aws_url +FROM index WHERE collection_id = 'rider_pilot' +``` + +For bulk series, still prefer `cohort/manifest.txt` — it is not subject to the SQL row cap. + +### Clinical data + +Clinical data comes in two layers: + +- **`clinical_index`** — a data dictionary, one row per (collection, table, column) with a + human-readable `column_label` and coded `values`. It is an ordinary SQL table and joins to + `index` on `collection_id`. +- **Per-collection clinical tables** (e.g. `nlst_canc`) — the actual rows. There are ~150, so + they are kept out of `GET /tables` and discovered through the `/clinical/tables` endpoints + instead. In SQL they live under a separate schema and are addressed as `clinical.
`. + +Clinical tables join to imaging on **`dicom_patient_id = index.PatientID`**, not on a series +UID. Clinical data is not harmonized across collections — table and column names vary, so +always discover before querying. + +```bash +curl -s "$B/clinical/tables?collection_id=nlst" # which tables this collection has +curl -s $B/clinical/tables/nlst_canc # columns + labels +curl -s "$B/clinical/tables/nlst_canc/rows?max_rows=100" # rows, capped + +curl -s $B/sql -H 'content-type: application/json' -d '{ + "sql": "SELECT count(DISTINCT i.PatientID) AS patients FROM index i JOIN clinical.nlst_canc c ON c.dicom_patient_id = i.PatientID WHERE i.collection_id = '\''nlst'\'' AND i.Modality = '\''CT'\'' AND c.clinical_stag = '\''400'\''"}' +``` + +See `references/clinical_data_guide.md` for value mapping and the wider clinical data model. + +## Getting the Data + +The API returns manifests; a client moves the bytes. Every URL points at public AWS S3 or GCS +buckets and needs no credentials. + +```bash +# save the full manifest +curl -s $B/cohort/manifest.txt \ + -H 'content-type: application/json' \ + -d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}}' > idc_manifest.txt + +# download it (needs idc-index installed) +idc download-from-manifest idc_manifest.txt --download-dir ./idc-data +``` + +For a filter that is a single `collection_id`, the `download` payload of `cohort/manifest` +emits the simpler `idc download --download-dir ./idc-data` form. + +### When the local index is a data release behind the API + +This applies when the two `idc-index-data` **majors** differ — the API serving `25.x.y` against +a local `24.x.y`, say. A newer build of the same release (24.2.2 vs 24.2.0) covers the same +series, so manifests from it resolve locally and downloads are unaffected. + +`idc download-from-manifest` does not simply hand the URLs to a transfer client: it extracts +each `crdc_series_uuid` from the manifest and joins it against the **local** index (then +against `prior_versions_index`) to compute sizes and build the output hierarchy. A manifest +produced by an API serving a newer IDC data release can therefore contain series the local +index has never heard of. + +Those rows are **not** downloaded, and the command does not fail — it logs, then continues +with the rest: + +``` +The total of N copy commands are not recognized as referencing any associated series in the +main index. ... they may correspond to files available in a release of IDC different from v24 +used in this version of idc-index. +... +The corresponding files could not be downloaded. +``` + +The result is a partial download that otherwise looks successful. `download_from_selection(seriesInstanceUID=…)` +has the same blind spot from the other direction: it filters the local index, so UIDs it does +not contain are silently dropped from the selection. + +**Fix it one of two ways:** + +1. **Upgrade** — upgrade `idc-index` (`python scripts/check_version.py` prints the command), + then re-run. This is the right answer + whenever it is possible; it restores the hierarchy, size checks, and progress reporting. +2. **Bypass the index** — transfer directly from the bucket. The manifest URLs are + self-contained (`s3:////*`), so no index is needed at all: + +```bash +# one directory per series, named by crdc_series_uuid +awk -F/ '{print "cp " $0 " ./idc-data/" $4 "/"}' idc_manifest.txt > s5cmd_commands.txt +s5cmd --no-sign-request run s5cmd_commands.txt + +# for source=gcs manifests, add the GCS endpoint +s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run s5cmd_commands.txt +``` + +`s5cmd run` expects one *command* per line, which is why the bare URLs are rewritten as `cp` +commands; drop the `$4` segment to land every file flat in `./idc-data/`. `aws s3 cp +--no-sign-request --recursive` works the same way per URL. + +What you give up by bypassing `idc-index` is convenience, not data: files are laid out under +CRDC UUIDs instead of `%collection_id/%PatientID/%Modality`, and there is no local size or +disk-space check — so read `size_TB` from `cohort/counts` first. See +`references/cloud_storage_guide.md` for bucket layout, `aws`/`gsutil` equivalents, and +UUID-to-DICOM-UID mapping. + +**`source`: `aws` (default) vs `gcs`.** Both return `s3://` URLs — GCS is reached through its +S3-compatible endpoint, never a `gs://` URL. That is why `idc download-from-manifest` only +recognizes `s3://` lines. Driving `s5cmd` yourself, use `--no-sign-request`, and for +`source=gcs` add `--endpoint-url https://storage.googleapis.com`. + +IDC is ~99 TB across 176 collections. Always report `series` and `size_TB` from +`cohort/counts` and confirm with the user before starting a broad download. + +## Limits, Defaults, and Errors + +Measured against `3.0.0b3`. Values above a cap are silently clamped — the response echoes the +value actually used (`max_rows`, `page_size`), so read it back rather than assuming the request +was honored. + +| Endpoint | Parameter | Default | Cap | +|----------|-----------|---------|-----| +| `GET /attributes/{attr}/values` | `limit` | 100 | 10000 | +| `POST /sql` | `max_rows` | 5000 | 10000 | +| `GET /clinical/tables/{table}/rows` | `max_rows` | 5000 | 100000 | +| `POST /cohort/manifest` | `page_size` | 100 | 5000 | +| `POST /cohort/manifest` | `page` | 0 | — | +| `POST /cohort/manifest` | `include_rows` | `true` | — | +| `POST /cohort/manifest.txt` | `limit` | 100000 | — | +| `POST /cohort/manifest.txt` | `source` | `aws` | — | +| `POST /citations` | `citation_format` | `apa` | — | + +`cohort/manifest.txt` is the surface that is not *row*-capped the way `/sql` is — it returned all +774 lines for `rider_pilot` with no `limit` set, and enumerates up to 100 000 series. That is why +bulk series belong there rather than in a `/sql` dump. + +There is **no per-caller rate limit or quota** and no `429`. What is bounded is the individual +request: a 30 s SQL statement timeout, 4 GB query memory, and the caps above. A burst is absorbed +by autoscaling and surfaces as slower responses or a `503` — back off and retry rather than +treating it as permanent. For sustained heavy metadata access, query the `idc-index` Parquet files +(`references/parquet_access_guide.md`) or BigQuery instead of driving this API hard. + +Size-capped responses carry a `truncated` boolean: `false` means the result is complete, `true` +means raise the limit or aggregate/narrow instead. Explore narrow, then widen. + +Errors come in two shapes. Semantic problems are HTTP 400 with a uniform body: + +```json +{"error": {"code": "invalid_query", "message": "Unknown or non-term filter attribute: 'NotAnAttribute'. Use list_attributes to see valid attributes."}} +``` + +Request-shape problems are HTTP 422 with FastAPI's `detail[]` array, which names the offending +key — a bare filter object, an unrecognized key, or a misspelled range bound all land here. Both +kinds are actionable: an unknown attribute names the discovery call to make, and a bad column name +carries DuckDB's candidate bindings. Read the message and fix the request rather than retrying it +unchanged. + +What does **not** produce an error is a filter that is valid but empty or over-broad. Those return +HTTP 200 with a `warnings` entry saying so — see *The server reports what it filtered on*. Read +`warnings`; do not infer from the count alone. + +## Handing Off to idc-index + +The boundary artifact is a list of `SeriesInstanceUID` values (or a saved `manifest.txt`). + +```python +# UIDs from a cohort/manifest or /sql response +series_uids = [row["SeriesInstanceUID"] for row in manifest["series"]] + +from idc_index import IDCClient +client = IDCClient() + +client.download_from_selection( + downloadDir="./data", + seriesInstanceUID=series_uids, # a list, not a DataFrame + dirTemplate="%collection_id/%PatientID/%Modality", +) +``` + +Run `python scripts/check_version.py` before the first `idc-index` call in a session, even when +discovery happened over the API — the two version independently. Compare +`idc_index_data_version` on both sides first (see *Checking the API against a local +idc-index*). + +**This handoff is only valid while the two are on the same IDC data release** — the same +`idc-index-data` major. `idc-index` can only download series its own index lists, so when the +API is a release ahead, UIDs and manifest URLs it returned may resolve to nothing locally: +`download_from_selection` silently drops them and `download-from-manifest` logs them as +unrecognized and skips them. Do not report that as "no data": name both versions, then either +upgrade `idc-index` or download straight from the bucket, as described in *When the local index +is a data release behind the API*. + +## What the API Does Not Cover + +- **Image bytes.** The API returns URLs and manifests only; files transfer from S3/GCS. +- **Pixel data access and DICOMweb.** Use `references/dicomweb_guide.md`. +- **Full DICOM metadata, per-segment detail, SR quantitative/qualitative measurements, private + DICOM elements.** Still BigQuery-only — see `references/bigquery_guide.md`. +- **Writes.** The service is read-only by construction; there are no POST endpoints that mutate + state, and the SQL connection rejects anything but `SELECT`. +- **Local analysis.** DataFrames, plotting, pydicom/SimpleITK, pathology tiling all stay with + `idc-index`. + +## Related Documentation + +- IDC REST API docs: https://learn.canceridc.dev/rest-api/api +- Swagger UI: https://api.imaging.datacommons.cancer.gov/v3/docs +- API and MCP server source: https://github.com/ImagingDataCommons/IDC-REST-MCP +- `references/mcp_guide.md` — the same capabilities as agent tools +- `references/sql_patterns.md` — SQL that transfers unchanged to `POST /sql` +- `references/cli_guide.md` — `idc download-from-manifest` and the rest of the CLI diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/sql_patterns.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/sql_patterns.md new file mode 100644 index 00000000..ab349fa7 --- /dev/null +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/sql_patterns.md @@ -0,0 +1,476 @@ +--- +title: "SQL Query Patterns for IDC" +task: "" +persona: "running" +lineage_type: import +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/sql_patterns.md +upstream_sha: d661d27e +imported_at: 2026-08-11 +prompt_class: prompt +upstream_changes: accepted +author: upstream +validated: false +--- + +# SQL Query Patterns for IDC + +**Tested with:** idc-index 0.12.5 (IDC data version v24) + +Quick reference for common SQL query patterns when working with IDC data. For detailed examples with context, see the "Core Capabilities" section in the main SKILL.md. + +## When to Use This Guide + +Load this guide when you need quick-reference SQL patterns for: +- Discovering available filter values (modalities, body parts, manufacturers) +- Finding annotations and segmentations across collections +- Querying slide microscopy and annotation data +- Estimating download sizes before download +- Linking imaging data to clinical data +- Filtering by 3D volume geometry validity (volume_geometry_index) +- Finding RT Structure Set series and ROI metadata (rtstruct_index) +- Filtering by CT/MR/PET acquisition parameters (ct_index, mr_index, pt_index) + +For table schemas, DataFrame access, and join column references, see `references/index_tables_guide.md`. + +## Prerequisites + +Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed +version and prints the install command for the interpreter you are running. + +```python +from idc_index import IDCClient +client = IDCClient() +``` + +## Overall Data Scale + +Counts and total size across all of IDC — useful for orienting a user, and for sanity-checking +that the index loaded the release you expect: + +```python +stats = client.sql_query(""" + SELECT + COUNT(DISTINCT collection_id) as collections, + COUNT(DISTINCT analysis_result_id) as analysis_results, + COUNT(DISTINCT PatientID) as patients, + COUNT(DISTINCT StudyInstanceUID) as studies, + COUNT(DISTINCT SeriesInstanceUID) as series, + SUM(instanceCount) as instances, + SUM(series_size_MB)/1000000 as size_TB + FROM index +""") +print(stats) +``` + +### Per-collection breakdown + +```python +# Get summary statistics from primary index +collections_summary = client.sql_query(""" + SELECT collection_id, + COUNT(DISTINCT PatientID) as patients, + COUNT(DISTINCT SeriesInstanceUID) as series, + SUM(series_size_MB) as size_mb + FROM index + GROUP BY collection_id + ORDER BY patients DESC +""") +``` + +For richer per-collection metadata — cancer types, tumor locations, species, supporting data — +query `collections_index` instead; for derived datasets, `analysis_results_index`. Both need +`client.fetch_index(...)` first: + +```python +client.fetch_index("collections_index") +collections_info = client.sql_query(""" + SELECT collection_id, cancer_types, tumor_locations, species, subjects, supporting_data + FROM collections_index +""") + +client.fetch_index("analysis_results_index") +analysis_info = client.sql_query(""" + SELECT analysis_result_id, analysis_result_title, subjects, collections, modalities + FROM analysis_results_index +""") +``` + +## Discover Available Filter Values + +```python +# What modalities exist? +client.sql_query("SELECT DISTINCT Modality FROM index") + +# What body parts for a specific modality? +client.sql_query(""" + SELECT DISTINCT BodyPartExamined, COUNT(*) as n + FROM index WHERE Modality = 'CT' AND BodyPartExamined IS NOT NULL + GROUP BY BodyPartExamined ORDER BY n DESC +""") + +# What manufacturers for MR? +client.sql_query(""" + SELECT DISTINCT Manufacturer, COUNT(*) as n + FROM index WHERE Modality = 'MR' + GROUP BY Manufacturer ORDER BY n DESC +""") +``` + +## Find Annotations and Segmentations + +**Note:** Not all image-derived objects belong to analysis result collections. Some annotations are deposited alongside original images. Use DICOM Modality or SOPClassUID to find all derived objects regardless of collection type. + +```python +# Find ALL segmentations and structure sets by DICOM Modality +# SEG = DICOM Segmentation, RTSTRUCT = Radiotherapy Structure Set +client.sql_query(""" + SELECT collection_id, Modality, COUNT(*) as series_count + FROM index + WHERE Modality IN ('SEG', 'RTSTRUCT') + GROUP BY collection_id, Modality + ORDER BY series_count DESC +""") + +# Find segmentations for a specific collection (includes non-analysis-result items) +client.sql_query(""" + SELECT SeriesInstanceUID, SeriesDescription, analysis_result_id + FROM index + WHERE collection_id = 'tcga_luad' AND Modality = 'SEG' +""") + +# List analysis result collections (curated derived datasets) +client.fetch_index("analysis_results_index") +client.sql_query(""" + SELECT analysis_result_id, analysis_result_title, collections, modalities + FROM analysis_results_index +""") + +# Find analysis results for a specific source collection +client.sql_query(""" + SELECT analysis_result_id, analysis_result_title + FROM analysis_results_index + WHERE Collections LIKE '%tcga_luad%' +""") + +# Use seg_index for detailed DICOM Segmentation metadata +client.fetch_index("seg_index") + +# Get segmentation statistics by algorithm +client.sql_query(""" + SELECT AlgorithmName, AlgorithmType, COUNT(*) as seg_count + FROM seg_index + WHERE AlgorithmName IS NOT NULL + GROUP BY AlgorithmName, AlgorithmType + ORDER BY seg_count DESC + LIMIT 10 +""") + +# Find segmentations for specific source images (e.g., chest CT) +client.sql_query(""" + SELECT + s.SeriesInstanceUID as seg_series, + s.AlgorithmName, + s.total_segments, + s.segmented_SeriesInstanceUID as source_series + FROM seg_index s + JOIN index src ON s.segmented_SeriesInstanceUID = src.SeriesInstanceUID + WHERE src.Modality = 'CT' AND src.BodyPartExamined = 'CHEST' + LIMIT 10 +""") + +# Find TotalSegmentator results with source image context +client.sql_query(""" + SELECT + seg_info.collection_id, + COUNT(DISTINCT s.SeriesInstanceUID) as seg_count, + SUM(s.total_segments) as total_segments + FROM seg_index s + JOIN index seg_info ON s.SeriesInstanceUID = seg_info.SeriesInstanceUID + WHERE s.AlgorithmName LIKE '%TotalSegmentator%' + GROUP BY seg_info.collection_id + ORDER BY seg_count DESC +""") + +# Use ann_index and ann_group_index for Microscopy Bulk Simple Annotations +# ann_group_index has AnnotationGroupLabel, GraphicType, NumberOfAnnotations, AlgorithmName +client.fetch_index("ann_index") +client.fetch_index("ann_group_index") +client.sql_query(""" + SELECT g.AnnotationGroupLabel, g.GraphicType, g.NumberOfAnnotations, i.collection_id + FROM ann_group_index g + JOIN ann_index a ON g.SeriesInstanceUID = a.SeriesInstanceUID + JOIN index i ON a.SeriesInstanceUID = i.SeriesInstanceUID + WHERE g.AlgorithmName IS NOT NULL + LIMIT 10 +""") +# See references/digital_pathology_guide.md for AnnotationGroupLabel filtering, SM+ANN joins, and more +``` + +## Query Slide Microscopy and Annotation Data + +Use `sm_index` for slide microscopy metadata and `ann_index`/`ann_group_index` for annotations on slides (DICOM ANN objects). Filter annotation groups by `AnnotationGroupLabel` to find annotations by name. + +```python +client.fetch_index("sm_index") +client.fetch_index("ann_index") +client.fetch_index("ann_group_index") + +# Example: find annotation groups by label within a collection +client.sql_query(""" + SELECT g.AnnotationGroupLabel, g.GraphicType, g.NumberOfAnnotations + FROM ann_group_index g + JOIN index i ON g.SeriesInstanceUID = i.SeriesInstanceUID + WHERE i.collection_id = 'your_collection_id' + AND LOWER(g.AnnotationGroupLabel) LIKE '%keyword%' +""") +``` + +See `references/digital_pathology_guide.md` for SM queries, ANN filtering patterns, SM+ANN cross-references, and join examples. + +## Estimate Download Size + +```python +# Size for specific criteria +client.sql_query(""" + SELECT SUM(series_size_MB) as total_mb, COUNT(*) as series_count + FROM index + WHERE collection_id = 'nlst' AND Modality = 'CT' +""") +``` + +## Link to Clinical Data + +```python +client.fetch_index("clinical_index") + +# Find collections with clinical data and their tables +client.sql_query(""" + SELECT collection_id, table_name, COUNT(DISTINCT column_label) as columns + FROM clinical_index + GROUP BY collection_id, table_name + ORDER BY collection_id +""") +``` + +See `references/clinical_data_guide.md` for complete patterns including value mapping and patient cohort selection. + +## Version Tracking — "What's New in IDC vX?" + +Use `series_init_idc_version` and `series_revised_idc_version` in the main `index` table. Do NOT +use `prior_versions_index` for this — it contains only removed series. + +```python +VERSION = 24 # Replace with target version + +# Series added for the first time in vVERSION +client.sql_query(f""" + SELECT collection_id, + COUNT(DISTINCT SeriesInstanceUID) as new_series, + ROUND(SUM(series_size_MB)/1000, 2) as size_GB + FROM index + WHERE series_init_idc_version = {VERSION} + GROUP BY collection_id + ORDER BY new_series DESC +""") + +# Series revised (updated content) in vVERSION but originally added earlier +client.sql_query(f""" + SELECT collection_id, + COUNT(DISTINCT SeriesInstanceUID) as revised_series + FROM index + WHERE series_revised_idc_version = {VERSION} + AND series_init_idc_version < {VERSION} + GROUP BY collection_id + ORDER BY revised_series DESC +""") + +# When was each collection first added to IDC? +client.fetch_index("version_metadata_index") +client.sql_query(""" + WITH first_versions AS ( + SELECT collection_id, MIN(series_init_idc_version) as first_version + FROM index + GROUP BY collection_id + ) + SELECT f.collection_id, f.first_version, v.version_timestamp as first_release_date + FROM first_versions f + JOIN version_metadata_index v ON f.first_version = v.idc_version + ORDER BY f.first_version DESC +""") +``` + +## Troubleshooting + +**Issue:** Query returns error "table not found" +- **Cause:** Index not fetched before query +- **Solution:** Call `client.fetch_index("table_name")` before using tables other than the primary `index` + +**Issue:** LIKE pattern not matching expected results +- **Cause:** Case sensitivity or whitespace +- **Solution:** Use `LOWER(column)` for case-insensitive matching, `TRIM()` for whitespace + +**Issue:** JOIN returns fewer rows than expected +- **Cause:** NULL values in join columns or no matching records +- **Solution:** Use `LEFT JOIN` to include rows without matches, check for NULLs with `IS NOT NULL` + +## Volume Geometry Validation + +`volume_geometry_index` covers single-frame CT, MR, and PT series. Fetch it before querying. + +```python +client.fetch_index("volume_geometry_index") + +# Series that form a regularly-spaced 3D volume (no resampling needed) +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, i.BodyPartExamined, + v.obliquity_degrees + FROM index i + JOIN volume_geometry_index v ON i.SeriesInstanceUID = v.SeriesInstanceUID + WHERE i.Modality = 'CT' + AND v.regularly_spaced_3d_volume = TRUE + LIMIT 10 +""") + +# Fraction of 3D-valid CT per collection +client.sql_query(""" + SELECT i.collection_id, + COUNT(*) as total_ct, + SUM(CASE WHEN v.regularly_spaced_3d_volume THEN 1 ELSE 0 END) as valid_3d, + ROUND(100.0 * SUM(CASE WHEN v.regularly_spaced_3d_volume THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_valid + FROM index i + JOIN volume_geometry_index v ON i.SeriesInstanceUID = v.SeriesInstanceUID + WHERE i.Modality = 'CT' + GROUP BY i.collection_id + ORDER BY total_ct DESC + LIMIT 10 +""") +``` + +Key columns: `regularly_spaced_3d_volume` (composite flag), `obliquity_degrees` (0 = pure axial/sagittal/coronal), plus individual boolean checks: `single_orientation`, `orthogonal_orientation`, `unique_slice_positions`, `consistent_pixel_spacing`, `consistent_image_dimensions`, `uniform_slice_spacing`. + +## RT Structure Sets + +`rtstruct_index` has one row per RTSTRUCT series. Array columns (`ROINames`, `ROIGenerationAlgorithms`, `RTROIInterpretedTypes`) are stored as strings. + +```python +client.fetch_index("rtstruct_index") + +# RTSTRUCT series with ROI counts and names +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, + r.total_rois, r.ROINames, r.RTROIInterpretedTypes, + r.referenced_SeriesInstanceUID + FROM index i + JOIN rtstruct_index r ON i.SeriesInstanceUID = r.SeriesInstanceUID + LIMIT 10 +""") + +# Collections with the most RTSTRUCT series +client.sql_query(""" + SELECT i.collection_id, + COUNT(*) as rtstruct_series, + ROUND(AVG(r.total_rois), 1) as avg_rois + FROM index i + JOIN rtstruct_index r ON i.SeriesInstanceUID = r.SeriesInstanceUID + GROUP BY i.collection_id + ORDER BY rtstruct_series DESC + LIMIT 10 +""") + +# Find source CT series for a given RTSTRUCT +client.sql_query(""" + SELECT r.SeriesInstanceUID as rtstruct_uid, + r.total_rois, r.ROINames, + src.SeriesInstanceUID as source_ct_uid, + src.collection_id, src.BodyPartExamined + FROM rtstruct_index r + JOIN index src ON r.referenced_SeriesInstanceUID = src.SeriesInstanceUID + LIMIT 10 +""") +``` + +## Modality Acquisition Parameters + +`ct_index`, `mr_index`, and `pt_index` (added in idc-index 0.12.3) expose acquisition and reconstruction parameters for CT, MR, and PET series. All join on `SeriesInstanceUID`. Dose-modulated CT acquisitions have `_min`/`_max` columns for tube current, exposure, and exposure time. + +```python +client.fetch_index("ct_index") +client.fetch_index("mr_index") +client.fetch_index("pt_index") + +# CT: thin-slice series (≤2mm) with standard reconstruction +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, i.BodyPartExamined, + c.SliceThickness, c.ConvolutionKernel, c.KVP + FROM index i + JOIN ct_index c ON i.SeriesInstanceUID = c.SeriesInstanceUID + WHERE c.SliceThickness <= 2.0 + AND c.ConvolutionKernel IS NOT NULL + LIMIT 10 +""") + +# CT: dose-modulated acquisitions (tube current varies across slices) +client.sql_query(""" + SELECT i.collection_id, c.SeriesInstanceUID, + c.XRayTubeCurrent_min, c.XRayTubeCurrent_max, c.SliceThickness + FROM ct_index c + JOIN index i ON c.SeriesInstanceUID = i.SeriesInstanceUID + WHERE c.XRayTubeCurrent_min != c.XRayTubeCurrent_max + LIMIT 10 +""") + +# MR: DWI series (have non-null DiffusionBValue) at 3T +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, i.SeriesDescription, + m.MagneticFieldStrength, m.DiffusionBValue + FROM index i + JOIN mr_index m ON i.SeriesInstanceUID = m.SeriesInstanceUID + WHERE m.DiffusionBValue IS NOT NULL + AND m.MagneticFieldStrength >= 2.9 + LIMIT 10 +""") + +# MR: multi-echo series (EchoTime stored as array with multiple values) +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, + m.EchoTime, m.EchoTrainLength, m.ScanningSequence + FROM index i + JOIN mr_index m ON i.SeriesInstanceUID = m.SeriesInstanceUID + WHERE m.EchoTrainLength > 1 + LIMIT 10 +""") + +# PET: FDG studies with specific reconstruction method +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, + p.RadionuclideCodeMeaning, p.ReconstructionMethod, + p.Units, p.DecayCorrection + FROM index i + JOIN pt_index p ON i.SeriesInstanceUID = p.SeriesInstanceUID + WHERE p.RadionuclideCodeMeaning LIKE '%fluorodeoxyglucose%' + LIMIT 10 +""") + +# PET: dynamic acquisitions (ActualFrameDuration is array with multiple values) +client.sql_query(""" + SELECT i.collection_id, i.SeriesInstanceUID, + p.NumberOfTimeSlices, p.ActualFrameDuration + FROM index i + JOIN pt_index p ON i.SeriesInstanceUID = p.SeriesInstanceUID + WHERE p.NumberOfTimeSlices > 1 + LIMIT 10 +""") +``` + +Key columns by table (use `client.indices_overview["ct_index"]["schema"]` for the full list): +- **ct_index**: `SliceThickness`, `KVP`, `ConvolutionKernel`, `SpiralPitchFactor`, `XRayTubeCurrent_min/max`, `Exposure_min/max`, `PixelSpacing_row_mm/col_mm`, `Rows`, `Columns` +- **mr_index**: `MagneticFieldStrength`, `ScanningSequence`, `SequenceVariant`, `MRAcquisitionType`, `EchoTime` (array), `RepetitionTime`, `FlipAngle`, `DiffusionBValue` (array), `NumberOfTemporalPositions`, `ReceiveCoilName` +- **pt_index**: `RadionuclideCodeMeaning`, `Radiopharmaceutical`, `RadionuclideTotalDose`, `ReconstructionMethod`, `DecayCorrection`, `AttenuationCorrectionMethod`, `ActualFrameDuration` (array), `NumberOfTimeSlices` + +## Resources + +- `references/index_tables_guide.md` for table schemas, DataFrame access, and join column references +- `references/clinical_data_guide.md` for clinical data patterns and value mapping +- `references/digital_pathology_guide.md` for pathology-specific queries +- `references/bigquery_guide.md` for advanced queries requiring full DICOM metadata +- `references/parquet_access_guide.md` for direct Parquet queries without installing idc-index diff --git a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/use_cases.md b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/use_cases.md index 4e8064c7..a125f200 100644 --- a/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/use_cases.md +++ b/upstream/K-Dense-AI-scientific-agent-skills/skills/imaging-data-commons/references/use_cases.md @@ -1,10 +1,11 @@ --- title: "Common Use Cases for IDC" task: "" +persona: "running" lineage_type: import -upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/9c9bd2e9/skills/imaging-data-commons/references/use_cases.md -upstream_sha: 9c9bd2e9 -imported_at: 2026-06-27 +upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/d661d27e/skills/imaging-data-commons/references/use_cases.md +upstream_sha: d661d27e +imported_at: 2026-08-11 prompt_class: prompt upstream_changes: accepted author: upstream @@ -13,7 +14,7 @@ validated: false # Common Use Cases for IDC -**Tested with:** idc-index 0.11.9 (IDC data version v23) +**Tested with:** idc-index 0.12.5 (IDC data version v24) This guide provides complete end-to-end workflow examples for common IDC use cases. Each use case demonstrates the full workflow from query to download with best practices. @@ -29,9 +30,8 @@ For core API patterns (query, download, visualize, citations), see the "Core Cap ## Prerequisites -```bash -pip install --upgrade idc-index -``` +Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed +version and prints the install command for the interpreter you are running. ## Use Case 1: Find and Download Lung CT Scans for Deep Learning @@ -191,6 +191,98 @@ client.download_from_selection( cc_by_data.to_csv('commercial_dataset_manifest_CC-BY_ONLY.csv', index=False) ``` +## Use Case 5: Batch Download with Filtering + +**Objective:** Download a large filtered dataset in batches to avoid timeouts + +**Steps:** +```python +from idc_index import IDCClient +import pandas as pd + +client = IDCClient() + +# Find chest CT scans from GE scanners with a permissive license +query = """ +SELECT + SeriesInstanceUID, + PatientID, + collection_id, + ManufacturerModelName +FROM index +WHERE Modality = 'CT' + AND BodyPartExamined = 'CHEST' + AND Manufacturer = 'GE MEDICAL SYSTEMS' + AND license_short_name = 'CC BY 4.0' +LIMIT 100 +""" + +results = client.sql_query(query) + +# Save manifest for reproducibility +results.to_csv('lung_ct_manifest.csv', index=False) + +# Download in batches to avoid timeout +batch_size = 10 +for i in range(0, len(results), batch_size): + batch = results.iloc[i:i+batch_size] + client.download_from_selection( + seriesInstanceUID=list(batch['SeriesInstanceUID'].values), + downloadDir=f"./data/batch_{i//batch_size}" + ) +``` + +## Use Case 6: Integration with Analysis Pipelines + +**Objective:** Load downloaded DICOM files into Python for processing + +**Read individual DICOM files with pydicom:** +```python +import pydicom +import os + +series_dir = "./data/rider/rider_pilot/RIDER-1007893286/CT_1.3.6.1..." + +dicom_files = [os.path.join(series_dir, f) for f in os.listdir(series_dir) + if f.endswith('.dcm')] + +ds = pydicom.dcmread(dicom_files[0]) +print(f"Patient ID: {ds.PatientID}") +print(f"Modality: {ds.Modality}") +print(f"Image shape: {ds.pixel_array.shape}") +``` + +**Build 3D volume from CT series:** +```python +import pydicom +import numpy as np +from pathlib import Path + +def load_ct_series(series_path): + files = sorted(Path(series_path).glob('*.dcm')) + slices = [pydicom.dcmread(str(f)) for f in files] + slices.sort(key=lambda x: float(x.ImagePositionPatient[2])) + volume = np.stack([s.pixel_array for s in slices]) + return volume, slices[0] + +volume, metadata = load_ct_series("./data/lung_ct/series_dir") +print(f"Volume shape: {volume.shape}") # (z, y, x) +``` + +**Load DICOM series with SimpleITK (recommended for correct geometry):** +```python +import SimpleITK as sitk + +series_path = "./data/ct_series" +reader = sitk.ImageSeriesReader() +dicom_names = reader.GetGDCMSeriesFileNames(series_path) +reader.SetFileNames(dicom_names) +image = reader.Execute() + +smoothed = sitk.CurvatureFlow(image1=image, timeStep=0.125, numberOfIterations=5) +sitk.WriteImage(smoothed, "processed_volume.nii.gz") +``` + ## Resources - Main SKILL.md for core API patterns (query, download, visualize)