[Upstream sync] K-Dense-AI/scientific-agent-skills (github) — 45 added, 56 modified #33
+310
@@ -0,0 +1,310 @@
|
||||
---
|
||||
title: "Core Workflow Patterns"
|
||||
task: ""
|
||||
lineage_type: import
|
||||
upstream_source: https://github.com/K-Dense-AI/scientific-agent-skills/blob/991bd993/skills/cellxgene-census/references/core_workflow_patterns.md
|
||||
upstream_sha: 991bd993
|
||||
imported_at: 2026-08-08
|
||||
prompt_class: prompt
|
||||
upstream_changes: accepted
|
||||
author: upstream
|
||||
validated: false
|
||||
---
|
||||
|
||||
# Core Workflow Patterns
|
||||
|
||||
The eight patterns in full, with code: opening the Census, exploring Census information,
|
||||
querying expression data at small to medium scale, large-scale out-of-core queries,
|
||||
machine learning with PyTorch, spatial Census data, Scanpy integration, and
|
||||
multi-dataset integration.
|
||||
|
||||
## Core Workflow Patterns
|
||||
|
||||
### 1. Opening the Census
|
||||
|
||||
Always use the context manager to ensure proper resource cleanup:
|
||||
|
||||
```python
|
||||
import cellxgene_census
|
||||
|
||||
# Open latest stable version
|
||||
with cellxgene_census.open_soma() as census:
|
||||
# Work with census data
|
||||
|
||||
# Open the current LTS version for reproducibility
|
||||
with cellxgene_census.open_soma(census_version="2025-11-08") as census:
|
||||
# Work with census data
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Use context manager (`with` statement) for automatic cleanup
|
||||
- Specify `census_version` for reproducible analyses
|
||||
- `stable` opens the current LTS Census release; `latest` opens the newest weekly release retained for a shorter period
|
||||
|
||||
### 2. Exploring Census Information
|
||||
|
||||
Before querying expression data, explore available datasets and metadata.
|
||||
|
||||
**Access summary information:**
|
||||
```python
|
||||
# Get summary statistics as label/value rows
|
||||
summary = census["census_info"]["summary"].read().concat().to_pandas()
|
||||
summary_values = summary.set_index("label")["value"]
|
||||
print(f"Total cells: {int(summary_values['total_cell_count']):,}")
|
||||
print(f"Unique cells: {int(summary_values['unique_cell_count']):,}")
|
||||
|
||||
# Get all datasets
|
||||
datasets = census["census_info"]["datasets"].read().concat().to_pandas()
|
||||
|
||||
# Get precomputed counts by organism, cell type, tissue, disease, and assay
|
||||
summary_counts = census["census_info"]["summary_cell_counts"].read().concat().to_pandas()
|
||||
tissue_counts = summary_counts[summary_counts["category"].eq("tissue_general")]
|
||||
```
|
||||
|
||||
**Query cell metadata to understand available data:**
|
||||
```python
|
||||
# Get unique cell types in a tissue
|
||||
cell_metadata = cellxgene_census.get_obs(
|
||||
census,
|
||||
"homo_sapiens",
|
||||
value_filter="tissue_general == 'brain' and is_primary_data == True",
|
||||
column_names=["cell_type"]
|
||||
)
|
||||
unique_cell_types = cell_metadata["cell_type"].unique()
|
||||
print(f"Found {len(unique_cell_types)} cell types in brain")
|
||||
|
||||
# Count cells by tissue
|
||||
tissue_metadata = cellxgene_census.get_obs(
|
||||
census,
|
||||
"homo_sapiens",
|
||||
value_filter="is_primary_data == True",
|
||||
column_names=["tissue_general"],
|
||||
)
|
||||
tissue_counts = tissue_metadata["tissue_general"].value_counts()
|
||||
```
|
||||
|
||||
**Important:** Always filter for `is_primary_data == True` to avoid counting duplicate cells unless specifically analyzing duplicates.
|
||||
|
||||
### 3. Querying Expression Data (Small to Medium Scale)
|
||||
|
||||
For queries returning < 100k cells that fit in memory, use `get_anndata()`:
|
||||
|
||||
```python
|
||||
# Basic query with cell type and tissue filters
|
||||
adata = cellxgene_census.get_anndata(
|
||||
census=census,
|
||||
organism="Homo sapiens", # or "Mus musculus"
|
||||
obs_value_filter="cell_type == 'B cell' and tissue_general == 'lung' and is_primary_data == True",
|
||||
obs_column_names=["assay", "disease", "sex", "donor_id"],
|
||||
)
|
||||
|
||||
# Query specific genes with multiple filters
|
||||
adata = cellxgene_census.get_anndata(
|
||||
census=census,
|
||||
organism="Homo sapiens",
|
||||
var_value_filter="feature_name in ['CD4', 'CD8A', 'CD19', 'FOXP3']",
|
||||
obs_value_filter="cell_type == 'T cell' and disease == 'COVID-19' and is_primary_data == True",
|
||||
obs_column_names=["cell_type", "tissue_general", "donor_id"],
|
||||
)
|
||||
```
|
||||
|
||||
**Filter syntax:**
|
||||
- Use `obs_value_filter` for cell filtering
|
||||
- Use `var_value_filter` for gene filtering
|
||||
- Combine conditions with `and`, `or`
|
||||
- Use `in` for multiple values: `tissue in ['lung', 'liver']`
|
||||
- Select only needed columns with `obs_column_names`
|
||||
- In current LTS releases, `disease` and `disease_ontology_term_id` may contain ` || `-delimited multiple values; inspect available values before relying on exact equality filters for disease cohorts
|
||||
|
||||
**Getting metadata separately:**
|
||||
```python
|
||||
# Query cell metadata
|
||||
cell_metadata = cellxgene_census.get_obs(
|
||||
census, "homo_sapiens",
|
||||
value_filter="disease == 'COVID-19' and is_primary_data == True",
|
||||
column_names=["cell_type", "tissue_general", "donor_id"]
|
||||
)
|
||||
|
||||
# Query gene metadata
|
||||
gene_metadata = cellxgene_census.get_var(
|
||||
census, "homo_sapiens",
|
||||
value_filter="feature_name in ['CD4', 'CD8A']",
|
||||
column_names=["feature_id", "feature_name", "feature_length"]
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Large-Scale Queries (Out-of-Core Processing)
|
||||
|
||||
For queries exceeding available RAM, use `axis_query()` with iterative processing:
|
||||
|
||||
```python
|
||||
import tiledbsoma as soma
|
||||
|
||||
# Create axis query
|
||||
with census["census_data"]["homo_sapiens"].axis_query(
|
||||
measurement_name="RNA",
|
||||
obs_query=soma.AxisQuery(
|
||||
value_filter="tissue_general == 'brain' and is_primary_data == True"
|
||||
),
|
||||
var_query=soma.AxisQuery(
|
||||
value_filter="feature_name in ['FOXP2', 'TBR1', 'SATB2']"
|
||||
),
|
||||
) as query:
|
||||
# Iterate through expression matrix in chunks
|
||||
iterator = query.X("raw").tables()
|
||||
for batch in iterator:
|
||||
# batch is a pyarrow.Table with columns:
|
||||
# - soma_data: expression value
|
||||
# - soma_dim_0: cell (obs) coordinate
|
||||
# - soma_dim_1: gene (var) coordinate
|
||||
process_batch(batch)
|
||||
```
|
||||
|
||||
**Computing incremental statistics:**
|
||||
```python
|
||||
import tiledbsoma as soma
|
||||
|
||||
# Example: Calculate mean expression
|
||||
n_observations = 0
|
||||
sum_values = 0.0
|
||||
|
||||
with census["census_data"]["homo_sapiens"].axis_query(
|
||||
measurement_name="RNA",
|
||||
obs_query=soma.AxisQuery(value_filter="tissue_general == 'brain' and is_primary_data == True"),
|
||||
var_query=soma.AxisQuery(value_filter="feature_name in ['FOXP2', 'TBR1', 'SATB2']"),
|
||||
) as query:
|
||||
iterator = query.X("raw").tables()
|
||||
for batch in iterator:
|
||||
values = batch["soma_data"].to_numpy()
|
||||
n_observations += len(values)
|
||||
sum_values += values.sum()
|
||||
|
||||
mean_expression = sum_values / n_observations
|
||||
```
|
||||
|
||||
### 5. Machine Learning with PyTorch
|
||||
|
||||
For training models, use TileDB-SOMA-ML. The former `cellxgene_census.experimental.ml` PyTorch loaders are deprecated and scheduled for removal.
|
||||
|
||||
```python
|
||||
import tiledbsoma as soma
|
||||
from tiledbsoma_ml import ExperimentDataset, experiment_dataloader
|
||||
|
||||
with cellxgene_census.open_soma() as census:
|
||||
experiment = census["census_data"]["homo_sapiens"]
|
||||
with experiment.axis_query(
|
||||
measurement_name="RNA",
|
||||
obs_query=soma.AxisQuery(
|
||||
value_filter="tissue_general == 'liver' and is_primary_data == True"
|
||||
),
|
||||
) as query:
|
||||
dataset = ExperimentDataset(
|
||||
query=query,
|
||||
layer_name="raw",
|
||||
obs_column_names=["cell_type"],
|
||||
batch_size=128,
|
||||
shuffle=True,
|
||||
)
|
||||
dataloader = experiment_dataloader(dataset)
|
||||
|
||||
# Training loop
|
||||
for epoch in range(num_epochs):
|
||||
dataset.set_epoch(epoch)
|
||||
for X, obs in dataloader:
|
||||
labels = obs["cell_type"]
|
||||
|
||||
# Forward pass
|
||||
outputs = model(X)
|
||||
loss = criterion(outputs, labels)
|
||||
|
||||
# Backward pass
|
||||
optimizer.zero_grad()
|
||||
loss.backward()
|
||||
optimizer.step()
|
||||
```
|
||||
|
||||
**Train/test splitting:**
|
||||
```python
|
||||
train_dataset, test_dataset = dataset.random_split(0.8, 0.2, seed=42)
|
||||
train_loader = experiment_dataloader(train_dataset, num_workers=2)
|
||||
test_loader = experiment_dataloader(test_dataset, num_workers=2)
|
||||
```
|
||||
|
||||
Use `batch_size` and `shuffle` on `ExperimentDataset`, not on `torch.utils.data.DataLoader`; `experiment_dataloader()` rejects DataLoader-level `batch_size`, `shuffle`, `sampler`, and `batch_sampler` arguments.
|
||||
|
||||
### 6. Spatial Census Data
|
||||
|
||||
Spatial data is available for supported Census releases in a separate `census_spatial_sequencing` collection. Use the spatial extra and a current TileDB-SOMA version when querying Visium or Slide-seq V2 data:
|
||||
|
||||
```python
|
||||
import cellxgene_census
|
||||
import tiledbsoma as soma
|
||||
|
||||
with cellxgene_census.open_soma(census_version="2025-11-08") as census:
|
||||
spatial_experiment = census["census_spatial_sequencing"]["homo_sapiens"]
|
||||
with spatial_experiment.axis_query(
|
||||
measurement_name="RNA",
|
||||
obs_query=soma.AxisQuery(
|
||||
value_filter="dataset_id == '4cceac62-9513-42a4-90e5-2878dbb0192c'"
|
||||
),
|
||||
) as query:
|
||||
sdata = query.to_spatialdata(X_name="raw")
|
||||
```
|
||||
|
||||
### 7. Integration with Scanpy
|
||||
|
||||
Seamlessly integrate Census data with scanpy workflows:
|
||||
|
||||
```python
|
||||
import scanpy as sc
|
||||
|
||||
# Load data from Census
|
||||
adata = cellxgene_census.get_anndata(
|
||||
census=census,
|
||||
organism="Homo sapiens",
|
||||
obs_value_filter="cell_type == 'neuron' and tissue_general == 'cortex' and is_primary_data == True",
|
||||
)
|
||||
|
||||
# Standard scanpy workflow
|
||||
sc.pp.normalize_total(adata, target_sum=1e4)
|
||||
sc.pp.log1p(adata)
|
||||
sc.pp.highly_variable_genes(adata, n_top_genes=2000)
|
||||
|
||||
# Dimensionality reduction
|
||||
sc.pp.pca(adata, n_comps=50)
|
||||
sc.pp.neighbors(adata)
|
||||
sc.tl.umap(adata)
|
||||
|
||||
# Visualization
|
||||
sc.pl.umap(adata, color=["cell_type", "tissue", "disease"])
|
||||
```
|
||||
|
||||
### 8. Multi-Dataset Integration
|
||||
|
||||
Query and integrate multiple datasets:
|
||||
|
||||
```python
|
||||
# Strategy 1: Query multiple tissues separately
|
||||
tissues = ["lung", "liver", "kidney"]
|
||||
adatas = []
|
||||
|
||||
for tissue in tissues:
|
||||
adata = cellxgene_census.get_anndata(
|
||||
census=census,
|
||||
organism="Homo sapiens",
|
||||
obs_value_filter=f"tissue_general == '{tissue}' and is_primary_data == True",
|
||||
)
|
||||
adata.obs["tissue"] = tissue
|
||||
adatas.append(adata)
|
||||
|
||||
# Concatenate with AnnData's current API
|
||||
import anndata as ad
|
||||
combined = ad.concat(adatas, label="tissue", keys=tissues)
|
||||
|
||||
# Strategy 2: Query multiple datasets directly
|
||||
adata = cellxgene_census.get_anndata(
|
||||
census=census,
|
||||
organism="Homo sapiens",
|
||||
obs_value_filter="tissue_general in ['lung', 'liver', 'kidney'] and is_primary_data == True",
|
||||
)
|
||||
```
|
||||
Reference in New Issue
Block a user