Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@MikeGoldsmith
Copy link
Contributor

@MikeGoldsmith MikeGoldsmith commented Oct 31, 2025

Summary

This PR integrates two complementary approaches for handling multiple Honeycomb environments:

  1. Per-path API key lookup (original work): Computes TeamHeaderValue per pipeline by looking up API keys from userdata based on Environment IDs
  2. Router/SetEnvironment components (from mike/multiple-environments): Adds routing infrastructure and environment-specific pipeline transformations

The integration ensures that both SamplingSequencer and HoneycombExporter components in the same pipeline use the correct API key for their target environment.

Problem Being Solved

Previously, there was an architectural issue where:

  • Storing a single TeamHeaderValue globally didn't work for multiple pipelines with different environments
  • Path traversal would terminate at connection type changes (e.g., OTelTraces → SampleData at SamplingSequencer)
  • This caused split paths where the SamplingSequencer and HoneycombExporter were in separate paths, preventing them from sharing API key information

Solution Approach

1. Per-Path API Key Lookup

  • Added findTeamHeaderValueForPath() to look up API keys from userdata["APIKeys"] map based on Environment ID
  • Stores computed TeamHeaderValue in pathUserdata for all components in a pipeline
  • Ensures SamplingSequencer gets the correct x-honeycomb-team header for Refinery to determine which ruleset to use

2. Router/SetEnvironment Integration

  • Merged mike/multiple-environments branch which adds Router and SetEnvironment components
  • Updated path traversal to follow Environment connections while maintaining original signal types
  • Added pipeline transformation logic for routing connectors
  • Supports environment-specific API key variable injection

3. Latest Main Branch

  • Merged latest main to get SamplingSequencer 25-port update and other recent fixes

Key Changes

Core Translation Logic (pkg/translator/translator.go):

  • Added findTeamHeaderValueForPath() method that extracts API keys using this precedence:
    1. Environment ID lookup in userdata["APIKeys"] map
    2. User-defined APIKey property
    3. Template default value
  • Integration in GenerateConfig() to populate pathUserdata["TeamHeaderValue"]
  • Restored test utility functions (Inspect(), componentVersionSupported(), predicates)

Component Templates:

  • pkg/data/components/SamplingSequencer.yaml: Added template fallback {{ or .User.TeamHeaderValue "${HTP_EXPORTER_APIKEY}" }}
  • pkg/data/components/HoneycombExporter.yaml: Uses .User.TeamHeaderValue from pathUserdata
  • Added pkg/data/components/Router.yaml and pkg/data/components/SetEnvironment.yaml

Path Traversal (pkg/hpsf/hpsfTypes.go):

  • Updated FindAllPaths() to follow Environment connections
  • Added PipelineName field to PathWithConnections for custom pipeline naming

Test Data:

  • Fixed connection types: HoneycombHoneycombEvents
  • Updated golden files to expect fallback API key for split-path scenarios
  • Added Router and SetEnvironment test cases

Testing

Test Results: ~95% passing (most tests green)

Passing ✅

  • All per-path API key lookup functionality
  • All Inspect and utility function tests
  • Default HPSF configuration tests
  • Component validation tests
  • Router and SetEnvironment tests

Known Failures ⚠️

(3 tests, pre-existing from mike/multiple-environments branch):

  1. TestSamplingSequencerRuleOrdering - Paths ordered lexicographically (Rule 1, Rule 10, ...) instead of numerically (Rule 1, Rule 2, ...)
  2. TestTranslator_ValidateBadConfigs - SamplingSequencer version compatibility issue
  3. TestConditions - Related to SamplingSequencer version

Root Cause of Test Failures

The TestSamplingSequencerRuleOrdering failure analysis:

  • The orderPaths() and getFirstConnectionPortIndex() functions are identical to main (where test passes)
  • SamplingSequencer component has correct port indices (1-25)
  • The issue stems from how FindAllPaths() was modified in the multiple-environments branch to handle Environment connections
  • This changes the initial order of paths before orderPaths() is called
  • The Environment connection traversal logic appears to interfere with proper port-index-based ordering

This is a pre-existing issue in the mike/multiple-environments branch, not introduced by this API key lookup integration.

Migration Notes

For users wanting to use environment-specific API keys:

  1. Provide userdata["APIKeys"] as a map of environment IDs to API keys
  2. Set the Environment property on HoneycombExporter components
  3. The translator will automatically look up and inject the correct API key per pipeline

Example:

userdata := map[string]any{
    "APIKeys": map[string]string{
        "hny_env_prod":    "prod_api_key_xyz",
        "hny_env_staging": "staging_api_key_abc",
    },
}

Future Work

  • Investigate the interaction between FindAllPaths() Environment connection handling and path ordering to fix the remaining test failures
  • Consider whether the split-path behavior for SamplingSequencer should be addressed at the path traversal level

Breaking Changes

None. The changes are additive and maintain backward compatibility.

Verification

To test the core functionality:

# Verify compilation
go build ./pkg/translator/

# Run tests (expect 3 known failures)
go test ./pkg/translator/

# Test specific functionality
go test ./pkg/translator/ -run TestDefaultHPSF  # Should pass
go test ./pkg/translator/ -run TestInspect       # Should pass

@MikeGoldsmith MikeGoldsmith self-assigned this Oct 31, 2025
MikeGoldsmith and others added 13 commits November 3, 2025 13:32
Add a features parameter to GenerateConfig to enable client capability
negotiation. This allows HPSF to generate different configs based on what
the client (e.g., Beekeeper) supports, maintaining backward compatibility
while enabling new features.

Key changes:
- Add Feature type and Features wrapper in pkg/hpsftypes
- Add ParseFeaturesFromString() and ParseFeatures() helper functions
- Update GenerateConfig signature to accept features parameter
- Update all call sites to pass nil features for backward compatibility
- Add comprehensive tests for feature infrastructure

No features are exposed yet - this is the foundation for future features
like environment ID substitution.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@MikeGoldsmith MikeGoldsmith changed the base branch from main to mike/backward-compat-features November 3, 2025 13:43
Changed approach from feature flag-based substitution to userdata-based
API key lookup. This simplifies the integration and removes the need for
feature negotiation.

New approach:
- Hound/Beekeeper passes API keys in userdata: {"APIKeys": {"hny_xxx": "key"}}
- HPSF looks up Environment IDs in the map and uses the API key directly
- Falls back to APIKey property if Environment not in map

Key changes:
- Updated findTeamHeaderValue() to accept userdata and look up Environment
  IDs in the APIKeys map
- Removed features parameter from GenerateConfig() signature
- Updated all tests to pass APIKeys map in userdata
- Updated golden files to expect actual API keys instead of __ENV:...__ format
- Removed FeatureEnvIDSubstitution constant (keeping infrastructure for future)

Priority order:
1. Environment ID (looked up in APIKeys map) → use API key from map
2. User-defined APIKey property → use as-is
3. Default APIKey from template → use as-is

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@MikeGoldsmith MikeGoldsmith changed the base branch from mike/backward-compat-features to main November 4, 2025 15:32
MikeGoldsmith and others added 3 commits November 4, 2025 15:33
Add clear documentation about the current single-environment limitation
where all components (including SamplingSequencer) share the same
TeamHeaderValue from userdata.

Document future solution: SamplingSequencer should have its own Environment
property to explicitly couple it with its paired HoneycombExporter, enabling
multi-environment support while maintaining the requirement that a
SamplingSequencer uses the same Environment as its downstream exporter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Implements environment ID to API key lookup using per-pipeline userdata,
replacing the previous global approach. This enables proper multi-environment
support in the future while maintaining single-environment compatibility.

Key changes:
- Added findTeamHeaderValueForPath() to compute TeamHeaderValue per-pipeline
- Each pipeline gets its own pathUserdata with computed TeamHeaderValue
- Environment ID (e.g., hny_abc123) is looked up in APIKeys map from userdata
- Falls back to APIKey property, then APIKey default value
- SamplingSequencer and HoneycombExporter share the same API key per pipeline
- For single-environment setups, falls back to any HoneycombExporter globally

The lookup priority is:
1. Environment ID looked up in userdata["APIKeys"] map
2. User-defined APIKey property
3. APIKey default value from template (${HTP_EXPORTER_APIKEY})

This approach works for current single-environment configurations and
doesn't block future multi-environment support with Router components.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@MikeGoldsmith MikeGoldsmith changed the title feat: Add support for setting an environment ID in HoneycombExporter & SamplingSequencer feat: implement per-pipeline API key lookup from environment IDs Nov 4, 2025
MikeGoldsmith and others added 5 commits November 4, 2025 16:44
The previous test had SamplingSequencer in a path without a connected
HoneycombExporter, which is not a valid configuration since
SamplingSequencer logically depends on HoneycombExporter existing.

Changes:
- Fixed signal type from "Honeycomb" to "SampleData" for Rule 1 port
- Updated comment to reflect the full connection chain
- Configuration now properly shows: otlp → refinery → sampler → honeycomb

This ensures the test validates a realistic configuration where the
SamplingSequencer and HoneycombExporter are in the same logical pipeline.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Add template fallback to use ${HTP_EXPORTER_APIKEY} when TeamHeaderValue is empty
- Fix connection types in test files (Honeycomb → HoneycombEvents)
- Update golden test files to expect fallback for split-path scenarios
- Remove unused h parameter from findTeamHeaderValueForPath

This is an intermediate commit before merging mike/multiple-environments
to get complete pipeline traversal solution.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
This commit integrates two complementary approaches to environment handling:

1. **Router/SetEnvironment components** (from multiple-environments branch):
   - Adds Router and SetEnvironment components for multi-environment routing
   - Updates path traversal to follow Environment connections
   - Adds pipeline transformation logic for routing connectors
   - Injects environment-specific API key variables (${HTP_EXPORTER_APIKEY_<ENV>})

2. **Per-path API key lookup** (from hny-exporter-env branch):
   - Adds findTeamHeaderValueForPath() to compute TeamHeaderValue per pipeline
   - Looks up actual API keys from userdata["APIKeys"] map based on Environment ID
   - Stores TeamHeaderValue in pathUserdata for all components in a pipeline
   - Ensures SamplingSequencer and HoneycombExporter use same API key

**Integration approach:**
- Resolved merge conflicts by keeping all functions from both branches
- Added findTeamHeaderValueForPath() integration into GenerateConfig()
- Restored missing Inspect() and utility functions needed by tests
- Added componentVersionSupported() for test compatibility

**Files modified:**
- pkg/translator/translator.go: Core translation logic with both approaches
- pkg/hpsf/hpsfTypes.go: Path traversal with Environment connection support
- pkg/data/components/: Added Router.yaml, SetEnvironment.yaml
- pkg/config/: Updated to support routing connectors and environment handling
- testdata/: Added router and setenvironment test cases

**Known issue:**
TestSamplingSequencerRuleOrdering fails due to changes from multiple-environments
branch. This test passes on the original hny-exporter-env branch. The issue appears
to be related to port index extraction or sorting, not the API key lookup feature.
This needs investigation but doesn't block the core functionality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
The previous merge commit accidentally lost the Inspect() method and related
test utility functions (Exporters, Processors, Receivers, Samplers predicates,
ComponentInfo, InspectionResult types, and componentVersionSupported).

These functions are required by the test suite and were in the original
hny-exporter-env branch but got lost during the merge resolution.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
The findTeamHeaderValueForPath function was accidentally removed during the
merge commit. This function is essential for per-path API key lookup.

Added back:
- findTeamHeaderValueForPath() method definition
- Integration call in GenerateConfig() to populate pathUserdata["TeamHeaderValue"]

This fixes the issue where HoneycombExporter was getting <no value> for
x-honeycomb-team header instead of the proper API key.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
MikeGoldsmith and others added 6 commits November 4, 2025 18:50
Simplify pipeline generation by removing custom pipeline naming in favor of
generated pipeline names based on path hashing. This change removes:

- EnvironmentName property from HoneycombExporter component definition
- PipelineName field from PathWithConnections struct
- Custom pipeline name assignments in translator (intake, environment-based)
- EnvironmentName references in router test files

Pipeline names are now consistently generated using GetID() which creates
a hash of component names and connection type. Environment-specific API
keys are still looked up correctly via Environment IDs.

Test results:
- 48/51 tests passing (~94%)
- 3 pre-existing test failures remain (documented in PR #246)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Remove EnvironmentName property from HoneycombExporter and all related code
that built environment maps from this property. This simplifies pipeline
generation and standardizes on using Environment IDs for lookups.

Changes:
- Remove EnvironmentName property validation functions
- Remove buildExporterEnvironmentMap() function
- Remove injectEnvironmentAPIKeys() function
- Remove generateRefineryRulesWithRouter() EnvironmentName traversal logic
- Remove SetEnvironment EnvironmentName lookup (property doesn't exist)
- Remove dynamic routing connector generation based on EnvironmentName
- Update test files to remove EnvironmentName property references

The system now relies on Environment IDs in the Environment property for
API key lookups. Environment slug/name information for refinery rules will
need to be provided through userdata in future updates.

Test results: 48/51 tests passing (~94%)
- 3 pre-existing failures remain (documented in PR #246)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Fixes TestSamplingSequencerRuleOrdering by ensuring connections are
sorted numerically by port index rather than lexicographically by name.

Changes:
- Add Index field to Port struct to store explicit port ordering
- Add PopulateComponentPorts method to load port metadata from templates
- Implement getPortIndex helper for retrieving port indices
- Sort connections by port index in FindAllPaths before traversal
- Update getFirstConnectionPortIndex to check path components first

This ensures "Rule 2" comes before "Rule 10" in path traversal order,
fixing the ordering regression introduced by Environment connection
routing logic.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Add isComponentVersionCompatible helper function to eliminate duplicate
  version checking code between MakeConfigComponent and
  getMatchingTemplateComponents
- Remove redundant exact version match check (tc.Version == component.Version)
  as artifactVersionSupported already handles exact matches
- Implement intelligent version inference: when min/max not set, accept
  same major version with version <= component version
- Maintains backward compatibility (v0.1.0 works with v0.2.0 templates)
- Rejects invalid versions (e.g., v999999.1.0)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
…Sequencer

When a SamplingSequencer and HoneycombExporter are in different signal paths
(e.g., traces path vs sampling/events path), the SamplingSequencer needs to
use the same APIKey as the downstream HoneycombExporter.

- Add findDownstreamHoneycombExporter to search for HoneycombExporter
  components downstream from a SamplingSequencer
- Only propagate explicitly set APIKey properties (not Environment-based
  lookups or defaults) to ensure correct behavior
- Update test expectation to match YAML parser behavior (strips ${} wrapper
  when parsing config)

This ensures that user-defined API keys in HoneycombExporter are correctly
used by SamplingSequencer exporters that send data to Refinery.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants