Tutorials

Learn DataJoint by building real pipelines.

These tutorials guide you through building data pipelines step by step. Each tutorial is a Jupyter notebook, published here with its code and its executed outputs, so you can follow the whole pipeline without leaving the page. Start with the basics and progress to domain-specific and advanced topics.

Want to run them yourself? Every notebook can be downloaded and executed against your own MySQL or PostgreSQL database. Installation covers the setup: DataJoint, a database, Jupyter, and getting the notebooks.

Learning Paths

Choose your learning path based on your goals:

🌱 New to DataJoint

Goal: Understand core concepts and build your first pipeline

Path:

1. First Pipeline — 30 min — Tables, queries, four core operations
2. Schema Design — 45 min — Primary keys, relationships, table tiers
3. Data Entry — 30 min — Inserting and managing data
4. Queries — 45 min — Operators, restrictions, projections
5. Try an example: University Database — Complete pipeline with realistic data

Next: Read Relational Workflow Model to understand the conceptual foundation.

---

🚀 Building Production Pipelines

Goal: Create automated, scalable data processing workflows

Prerequisites: Complete basics above or have equivalent experience

Path:

1. Computation — Automated processing with Imported/Computed tables
2. Object Storage — Handle large data (arrays, files, images)
3. Distributed Computing — Multi-worker parallel execution
4. Practice: Fractal Pipeline or Blob Detection

Next:

• Run Computations — populate() usage patterns
• Distributed Computing — Cluster deployment
• Handle Errors — Job management and recovery

---

🧪 Domain-Specific Applications

Goal: Build scientific data pipelines for your field

Prerequisites: Complete basics, understand computation model

Production Software: DataJoint Elements

Standard pipelines for neurophysiology experiments, actively used in many labs worldwide. These are not tutorials—they are production-ready modular pipelines for calcium imaging, electrophysiology, array ephys, optogenetics, and more.

Learning tutorials (neuroscience):

• Calcium Imaging — Import movies, segment cells, extract traces
• Electrophysiology — Import recordings, spike detection, waveforms
• Allen CCF — Hierarchical brain atlas ontology

Complete demo pipeline:

• LC-MS Demo — Liquid chromatography-mass spectrometry pipeline showcasing DataJoint best practices with PostgreSQL: sample tracking, scan acquisition, mass spectral analysis, and parameterized peak detection

General patterns:

• Hotel Reservations — Booking systems with resource management
• Languages & Proficiency — Many-to-many relationships

---

🔧 Extending DataJoint

Goal: Customize DataJoint for specialized needs

Prerequisites: Proficient with basics and production pipelines

Path:

1. Custom Codecs — Create domain-specific data types
2. JSON Data Type — Semi-structured data patterns
3. SQL Comparison — Understand DataJoint's query algebra

Next:

• Codec API — Complete codec specification
• Create Custom Codec — Step-by-step codec development

---

Basics

Core concepts for getting started with DataJoint:

1. First Pipeline — Tables, queries, and the four core operations
2. Schema Design — Primary keys, relationships, and table tiers
3. Data Entry — Inserting and managing data
4. Queries — Operators and fetching results
5. Computation — Imported and Computed tables
6. Object Storage — Blobs, attachments, and object stores

Examples

Complete pipelines demonstrating DataJoint patterns:

• University Database — Academic records with students, courses, and grades
• Hotel Reservations — Booking system with rooms, guests, and reservations
• Languages & Proficiency — Language skills tracking with many-to-many relationships
• Fractal Pipeline — Iterative computation and parameter sweeps
• Blob Detection — Image processing with automated computation

Domain Tutorials

Real-world scientific pipelines:

• Calcium Imaging — Import TIFF movies, segment cells, extract fluorescence traces
• Electrophysiology — Import recordings, detect spikes, extract waveforms
• Electrophysiology with Object Storage — Neural data with <npy@> lazy loading
• Allen CCF — Brain atlas with hierarchical region ontology

Advanced Topics

Extending DataJoint for specialized use cases:

• SQL Comparison — DataJoint for SQL users
• JSON Data Type — Semi-structured data in tables
• Distributed Computing — Multi-process and cluster workflows
• Custom Codecs — Extending the type system