Conversation
- Remove var savingsPercent from ToonMetrics case class, make it a computed def - Remove debugging println statements from production code - Remove unused scala.util.Try import from ToonUDFs - Rename HeadroomPercent.None to NoHeadroom to avoid shadowing Option.None - Update documentation to clarify unsupported type handling
- Add SparkDatasetOps with Dataset[T] extension methods - Provide compile-time type safety via Spark Encoders - Support toToon(), toonMetrics(), fromToon() for typed Datasets - Automatic key inference from type parameter - Comprehensive scaladoc with usage examples - Dataset tests created (WIP: needs Spark implicits refactor) This enables users to work with type-safe Datasets instead of untyped DataFrames while maintaining all TOON encoding capabilities.
713ef26 to
aa00259
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a comprehensive Apache Spark integration for the TOON format library, enabling efficient DataFrame/Dataset encoding for LLM processing with a focus on production readiness and monitoring.
Summary: The integration adds Spark-specific TOON encoding capabilities with intelligent schema alignment detection, adaptive chunking to optimize token costs, and production monitoring utilities. It includes Delta Lake CDC streaming and Apache Iceberg time travel support for temporal analytics use cases.
Key Changes:
- Spark DataFrame/Dataset to TOON conversion with chunking support and error handling
- LLM client abstraction aligned with llm4s patterns for future compatibility
- Delta Lake Change Data Feed and Apache Iceberg time travel integrations
- Production monitoring framework with health checks, telemetry, and readiness reports
Reviewed changes
Copilot reviewed 27 out of 28 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
spark-integration/src/test/scala/io/toonformat/toon4s/spark/llm/LlmClientTest.scala |
Test suite for LLM client abstractions including messages, conversations, completions, and error handling |
spark-integration/src/test/scala/io/toonformat/toon4s/spark/ToonMetricsTest.scala |
Tests for token estimation, metrics calculation, and cost savings analysis |
spark-integration/src/test/scala/io/toonformat/toon4s/spark/SparkToonOpsTest.scala |
Tests for DataFrame-TOON conversion, chunking, round-trip encoding/decoding |
spark-integration/src/test/scala/io/toonformat/toon4s/spark/SparkJsonInteropTest.scala |
Tests for Row-JsonValue conversion with various data types |
spark-integration/src/main/scala/.../monitoring/ToonMonitoring.scala |
Production monitoring utilities including health assessment and telemetry collection |
spark-integration/src/main/scala/.../llm/package.scala |
LLM integration type aliases and helpers aligned with llm4s patterns |
spark-integration/src/main/scala/.../llm/MockLlmClient.scala |
Mock LLM client implementation for testing |
spark-integration/src/main/scala/.../llm/Message.scala |
Message types for LLM conversations (System, User, Assistant) |
spark-integration/src/main/scala/.../llm/LlmError.scala |
Error hierarchy for LLM operations with retry logic |
spark-integration/src/main/scala/.../llm/LlmClient.scala |
LLM client trait and configuration types |
spark-integration/src/main/scala/.../llm/Completion.scala |
Completion response types with token usage tracking |
spark-integration/src/main/scala/.../integrations/IcebergTimeTravel.scala |
Apache Iceberg time travel integration for historical snapshots |
spark-integration/src/main/scala/.../integrations/DeltaLakeCDC.scala |
Delta Lake CDC integration for real-time streaming |
spark-integration/src/main/scala/.../error/SparkToonError.scala |
Error types for Spark integration failures |
spark-integration/src/main/scala/.../ToonUDFs.scala |
User-defined functions for Spark SQL |
spark-integration/src/main/scala/.../ToonMetrics.scala |
Token metrics and savings calculations |
spark-integration/src/main/scala/.../ToonAlignmentAnalyzer.scala |
Schema alignment detection based on benchmark data |
spark-integration/src/main/scala/.../SparkToonOps.scala |
Core DataFrame extension methods for TOON operations |
spark-integration/src/main/scala/.../SparkJsonInterop.scala |
Conversion utilities between Spark Row and JsonValue |
spark-integration/src/main/scala/.../SparkDatasetOps.scala |
Type-safe Dataset[T] extension methods |
spark-integration/src/main/scala/.../LlmClient.scala.old |
Legacy LLM client (marked as .old file) |
spark-integration/src/main/scala/.../AdaptiveChunking.scala |
Adaptive chunking strategy to optimize prompt tax |
spark-integration/examples/README.md |
Comprehensive examples documentation with production checklist |
spark-integration/examples/ProductionMonitoringExample.scala |
Example demonstrating pre-deployment validation and monitoring |
spark-integration/examples/IcebergTrendAnalysisExample.scala |
Example for quarterly trend analysis using Iceberg time travel |
spark-integration/examples/DatabricksStreamingExample.scala |
Example for real-time fraud detection with Delta CDC |
spark-integration/README.md |
Main documentation for Spark integration features and usage |
build.sbt |
Build configuration adding spark-integration module |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
Dataset encoder resolution doesn't work in test framework due to implicits scope. Production code works fine. Added compile-time verification tests instead.
Added component diagrams, data flow, and design principles. Includes benchmark-driven decisions and real-world use cases.
Add toon4s-spark to CI checks and release workflow. Both modules use unified versioning via sbt-dynver.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added
Changed
Removed