Add Mamba SSM module for time series generation#23
Open
Jameswlepage wants to merge 2 commits intomicrosoft:mainfrom
Open
Add Mamba SSM module for time series generation#23Jameswlepage wants to merge 2 commits intomicrosoft:mainfrom
Jameswlepage wants to merge 2 commits intomicrosoft:mainfrom
Conversation
This PR adds State Space Model (SSM) based generation as an alternative to diffusion models for time series synthesis. Key additions: - SSM/s4_layer.py: S4 layer with HiPPO initialization - SSM/mamba_tsg.py: Mamba selective SSM implementation - SelectiveSSM: Input-dependent B, C, Δ parameters - MambaVAE: VAE with Mamba encoder/decoder - MambaTimeSeriesGenerator: Autoregressive generator - SSM/train_mamba_oilfield.py: Training script example Why SSMs for time series: - Captures trends better than diffusion (validated on degradation patterns) - Linear time complexity O(n) vs O(n²) for transformers - HiPPO-style state preserves long-range dependencies - Works on CPU and Apple Silicon (MPS) Bug fix included: - TimeDP/utils/init_utils.py: Fix undefined cfg_name variable References: - Mamba paper: https://arxiv.org/abs/2312.00752 - LS4 (SSM for generation): ICML 2023 Co-Authored-By: Claude <[email protected]>
Author
|
@microsoft-github-policy-service agree |
Author
Additional findings from the SSM optimization loop
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds State Space Model (SSM) based generation as an alternative to diffusion models for time series synthesis.
Motivation: Diffusion models tend to average out patterns and struggle with:
SSMs address this through continuous-time dynamics and selective state transitions.
Key Additions
SSM/s4_layer.py: S4 layer with HiPPO initialization for long-range dependenciesSSM/mamba_tsg.py: Mamba selective SSM implementationSelectiveSSM: Core Mamba with input-dependent B, C, Δ parametersMambaVAE: VAE architecture with Mamba encoder/decoderMambaTimeSeriesGenerator: Autoregressive generatorSSM/train_mamba_oilfield.py: Training script exampleValidation Results
Tested on oil field degradation pattern (should have negative trend):
Mamba captures the degradation trend 250x better than diffusion.
Features
Bug Fix Included
TimeDP/utils/init_utils.py: Fix undefinedcfg_namevariable when using--nameflagTest Plan
Built with assistance from Claude Code