Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 1ccc3f2

Browse files
piotrlaczkowskipiotrlaczkowski
committed
docs(KDP): refactoring documentation
1 parent f98dc2c commit 1ccc3f2

File tree

147 files changed

+1019
-975
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

147 files changed

+1019
-975
lines changed

‎Makefile‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,10 @@ generate_doc_content:
6565
mkdir -p docs/generated/api
6666
poetry run python scripts/generate_docstring_docs.py
6767
@echo "Generating model architecture diagrams"
68-
mkdir -p docs/imgs/architectures
69-
poetry run python scripts/generate_model_architectures.py
68+
mkdir -p docs/features/imgs/models
69+
poetry run python scripts/generate_model_diagrams.py
70+
@echo "Organizing documentation images"
71+
./scripts/organize_docs_images.sh
7072
@echo "Documentation content generation complete"
7173

7274
.PHONY: docs_deploy
@@ -107,9 +109,26 @@ serve_doc: generate_doc_content
107109
# Clean All
108110
# ------------------------------------
109111

112+
.PHONY: identify_unused_diagrams
113+
## Identify potentially unused diagram files
114+
identify_unused_diagrams:
115+
@echo "Scanning for potentially unused diagram files"
116+
poetry run python scripts/cleanup_unused_diagrams.py
117+
@echo "Scan complete. Check unused_diagrams_report.txt for details"
118+
119+
.PHONY: clean_old_diagrams
120+
## Remove obsolete diagram directories and unused diagram files
121+
clean_old_diagrams: identify_unused_diagrams
122+
@echo "Cleaning up old diagram directories"
123+
rm -rf docs/imgs/architectures
124+
@echo "Organizing documentation images"
125+
./scripts/organize_docs_images.sh
126+
@echo "Old diagram directories cleaned up"
127+
@echo "NOTE: Review unused_diagrams_report.txt to identify additional files to remove manually"
128+
110129
.PHONY: clean
111130
## Remove cache, built package, and docs directories after build or installation
112-
clean:
131+
clean: clean_old_diagrams
113132
find . -type d -name dist -exec rm -r {} +
114133
find . -type f -name '*.rst' ! -name 'index.rst' -delete
115134
find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete

‎README.md‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# 🌟 Keras Data Processor (KDP) - Powerful Data Preprocessing for TensorFlow
22

33
<p align="center">
4-
<img src="docs/kdp_logo.png" width="350"/>
4+
<img src="docs/getting-started/imgs/kdp_logo.png" width="350"/>
55
</p>
66

77
**Transform your raw data into ML-ready features with just a few lines of code!**
@@ -90,22 +90,33 @@ We've built an extensive documentation system to help you get the most from KDP:
9090
Your preprocessing pipeline is built as a Keras model that can be used independently or as the first layer of any model:
9191

9292
<p align="center">
93-
<img src="docs/imgs/Model_Architecture.png" width="800"/>
93+
<img src="docs/features/imgs/models/all_basic_types.png" width="800"/>
9494
</p>
9595

9696
## 📊 Performance
9797

9898
KDP outperforms alternative preprocessing approaches, especially as data size increases:
9999

100100
<p align="center">
101-
<img src="docs/imgs/time_vs_nr_data.png" width="400"/>
102-
<img src="docs/imgs/time_vs_nr_features.png" width="400"/>
101+
<img src="docs/getting-started/imgs/time_vs_nr_data.png" width="400"/>
102+
<img src="docs/getting-started/imgs/time_vs_nr_features.png" width="400"/>
103103
</p>
104104

105105
## 🤝 Contributing
106106

107107
We welcome contributions! Please check out our [Contributing Guide](docs/contributing.md) for guidelines on how to proceed.
108108

109+
## 🛠️ Development Tools
110+
111+
KDP includes tools to help developers:
112+
113+
- **Documentation Generation**: Automatically generate API docs from docstrings
114+
- **Model Diagram Generation**: Visualize model architectures with `make generate_doc_content` or run:
115+
```bash
116+
python scripts/generate_model_diagrams.py
117+
```
118+
This creates diagram images in `docs/features/imgs/models/` for all feature types and configurations.
119+
109120
## 📄 License
110121

111122
This project is licensed under the MIT License - see the LICENSE file for details.

‎docs/README.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# KDP Documentation
2+
3+
This directory contains the documentation for the Keras Data Processor (KDP) project.
4+
5+
## Directory Structure
6+
7+
- `assets/` - Shared assets used across multiple sections
8+
- `images/` - Central repository for all documentation images
9+
- `common/` - Common images (logos, performance charts, etc.)
10+
- `features/` - Feature-specific images
11+
- `advanced/` - Advanced feature images
12+
- `code/` - Code examples
13+
- `js/` - JavaScript files for documentation
14+
- `api/` - API documentation
15+
- `advanced/` - Advanced features documentation
16+
- `examples/` - Example usage and tutorials
17+
- `features/` - Feature-specific documentation
18+
- `imgs/` - Feature-specific images
19+
- `imgs/models/` - Generated model diagrams
20+
- `generated/` - Auto-generated documentation
21+
- `getting-started/` - Quick start guides
22+
- `imgs/` - Getting started images
23+
- `integrations/` - Integration with other tools
24+
- `optimization/` - Performance optimization guidance
25+
- `reference/` - Reference material
26+
- `contributing/` - Contribution guidelines
27+
28+
## Image Organization
29+
30+
Images in the documentation follow a section-specific organizational pattern:
31+
32+
1. **Section-Specific Storage**: Each documentation section has its own `imgs/` directory with relevant images
33+
2. **Model Diagrams**: All model diagrams are generated automatically and stored in `features/imgs/models/`
34+
3. **Common Images**: Common images like logos and performance charts are copied to each section where they're needed
35+
36+
This approach ensures:
37+
- Clear organization of images by documentation section
38+
- Easy updates when images change
39+
- Improved clarity about which images are used where
40+
- Simplified maintenance
41+
42+
For more details about image organization, see `contributing/development/images-organization.md`.
43+
44+
## Building Documentation
45+
46+
Documentation is built using MkDocs. To build and preview:
47+
48+
```bash
49+
# Generate all documentation content (API docs, model diagrams, etc.)
50+
make generate_doc_content
51+
52+
# Serve the documentation locally
53+
make serve_doc
54+
```
55+
56+
### Generated Content
57+
58+
Some documentation content is generated automatically:
59+
60+
1. API documentation from docstrings:
61+
```bash
62+
python scripts/generate_docstring_docs.py
63+
```
64+
65+
2. Model architecture diagrams:
66+
```bash
67+
python scripts/generate_model_diagrams.py
68+
```
69+
70+
3. Image organization:
71+
```bash
72+
./scripts/organize_docs_images.sh
73+
```

‎docs/advanced/distribution-aware-encoding.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,16 @@ preprocessor = PreprocessingModel(
207207
)
208208
```
209209

210+
## 📊 Model Architecture
211+
212+
The distribution-aware encoder architecture automatically adapts to your data's distribution:
213+
214+
![Distribution-Aware Architecture](imgs/distribution_aware.png)
215+
216+
The diagram shows how distribution-aware encoding transforms numerical features to better match their underlying distribution, improving model performance.
217+
218+
## 💡 How to Enable
219+
210220
## 🔗 Related Topics
211221

212222
- [Numerical Features](../features/numerical-features.md)

‎docs/advanced/feature-moe.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,13 +41,25 @@ model = result["model"]
4141

4242
KDP's Feature MoE uses a "divide and conquer" approach with smart routing:
4343

44-
![Feature MoE Architecture](imgs/feature_moe_architecture.png)
44+
![Feature MoE Architecture](imgs/feature_moe.png)
4545

4646
1. **Expert Networks**: Each expert is a specialized neural network that processes features in its own unique way.
4747
2. **Router Network**: Determines which experts should process each feature.
4848
3. **Adaptive Weighting**: Features can use multiple experts with different weights.
4949
4. **Residual Connections**: Preserve the original feature information while adding expert insights.
5050

51+
## 📊 Model Architecture
52+
53+
The Feature-wise Mixture of Experts in KDP works by:
54+
55+
1. Applying different specialized "expert" networks to each feature
56+
2. Using a gating mechanism to determine how much each expert contributes
57+
3. Combining the expert outputs to produce the final feature representation
58+
59+
![Feature MoE Architecture](imgs/feature_moe.png)
60+
61+
The diagram illustrates how different features are processed by specialized expert networks, with a gating mechanism determining which experts handle each feature.
62+
5163
## ⚙️ Configuration Options
5264

5365
Customize Feature MoE behavior with these parameters:

‎docs/advanced/imgs/advanced_numerical_embedding.png‎

77.5 KB
Loading

‎docs/advanced/imgs/cat_feature_pipeline.png‎

-85.5 KB
Binary file not shown.

‎docs/advanced/imgs/categorical_example_model.png‎

-322 KB
Binary file not shown.

‎docs/advanced/imgs/cross_features.png‎

-93.9 KB
Binary file not shown.

‎docs/advanced/imgs/custom_feature_pipeline.png‎

-54.6 KB
Binary file not shown.

0 commit comments

Comments
 (0)