Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/.vitepress/config.mts
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ export default defineConfig({
text: "Promise Operator",
link: "/getting-started/resources/promise",
},
{ text: "HTTP Request Validations", link: "/getting-started/resources/http-request-validations" },
{ text: "Skip Conditions", link: "/getting-started/resources/skip" },
{
text: "Preflight Validations",
Expand Down
6 changes: 6 additions & 0 deletions docs/getting-started/configuration/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ This section contains the agent settings that will be used to build the agent's

```apl
agentSettings {
timezone = "Etc/UTC"
installAnaconda = false
condaPackages { ... }
pythonPackages { ... }
Expand All @@ -142,6 +143,11 @@ agentSettings {
}
```

#### Timezone Settings

Configure the `timezone` setting with a valid tz database identifier (e.g., `America/New_York`) for the Docker image;
see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones for valid identifiers.

#### Enabling Anaconda

- **`installAnaconda`**: **"The Operating System for AI"**, [Anaconda](https://www.anaconda.com), will be installed when
Expand Down
126 changes: 126 additions & 0 deletions docs/getting-started/resources/http-request-validations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
outline: deep
---

# HTTP Request Validations

HTTP request validations are a critical mechanism for ensuring that incoming HTTP requests meet specific criteria before a resource action is executed. These validations verify the request's HTTP method, URL path, headers, and query parameters against predefined restrictions.

These checks safeguard system integrity, enforce security policies, and streamline workflows by skipping actions that do not comply with the specified requirements. They are particularly relevant when operating in API server mode (`APIServerMode` enabled).

HTTP request validations operate using an `AND` logic, meaning all specified conditions must be satisfied for the action to proceed. If any validation fails, the action is skipped, and a log entry is recorded to indicate the reason for skipping.

## Why HTTP Request Validations Matter

- **Enforce Request Compliance:** Validations ensure that only requests with permitted methods, paths, headers, and parameters are processed, reducing the risk of unauthorized or malformed requests.
- **Early Action Skipping:** By validating requests before execution, non-compliant actions are skipped early, saving system resources and preventing unintended behavior.
- **Improved Debugging:** When an action is skipped due to a validation failure, detailed log messages help diagnose the issue, such as identifying an invalid HTTP method or path.

## Defining HTTP Request Validations

HTTP request validations are defined in the `run` block of a resource configuration and are enforced only when `APIServerMode` is enabled. They consist of four key fields:

- `restrictToHTTPMethods`: Specifies the HTTP methods (e.g., `GET`, `POST`) required for the request.
- `restrictToRoutes`: Specifies the URL paths (e.g., `/api/v1/whois`) required for the request.
- `allowedHeaders`: Specifies the HTTP headers permitted in the request.
- `allowedParams`: Specifies the query parameters permitted in the request.

If any of these fields are empty, all corresponding values are permitted (e.g., an empty `restrictToHTTPMethods` allows all HTTP methods). If a validation fails, the action is skipped, and no further processing (e.g., `Exec`, `Python`, `Chat`, or `HTTPClient` steps) occurs for that action.

Here’s an example of how to configure HTTP request validations:

```pkl
run {
// restrictToHTTPMethods specifies the HTTP methods required for the request.
// If none are specified, all HTTP methods are permitted. This restriction is only
// in effect when APIServerMode is enabled. If the request method is not in this list,
// the action will be skipped.
restrictToHTTPMethods {
"GET"
}

// restrictToRoutes specifies the URL paths required for the request.
// If none are specified, all routes are permitted. This restriction is only
// in effect when APIServerMode is enabled. If the request path is not in this list,
// the action will be skipped.
restrictToRoutes {
"/api/v1/whois"
}

// allowedHeaders specifies the permitted HTTP headers for the request.
// If none are specified, all headers are allowed. This restriction is only
// in effect when APIServerMode is enabled. If a header used in the resource is not
// in this list, the action will be skipped.
allowedHeaders {
"Content-Type"
// "X-API-KEY"
}

// allowedParams specifies the permitted query parameters for the request.
// If none are specified, all parameters are allowed. This restriction is only
// in effect when APIServerMode is enabled. If a parameter used in the resource is
// not in this list, the action will be skipped.
allowedParams {
"user_id"
"session_id"
}
}
```

### Validation Details

- **restrictToHTTPMethods**:
- Validates the request’s HTTP method (e.g., `GET`, `POST`) against the specified list.
- Example: If set to `["GET"]`, a `POST` request will cause the action to be skipped.
- Case-insensitive matching is used (e.g., `get` matches `GET`).

- **restrictToRoutes**:
- Validates the request’s URL path (e.g., `/api/v1/whois`) against the specified list.
- Example: If set to `["/api/v1/whois"]`, a request to `/api/v1/users` will cause the action to be skipped.
- Exact path matching is used; patterns or wildcards are not currently supported.

- **allowedHeaders**:
- Validates headers used in `request.header("header_id")` calls within the resource file against the specified list.
- Example: If set to `["Content-Type"]`, a `request.header("Authorization")` call will cause the action to be skipped.
- Case-insensitive matching is used.

- **allowedParams**:
- Validates query parameters used in `request.params("param_id")` calls within the resource file against the specified list.
- Example: If set to `["user_id"]`, a `request.params("token")` call will cause the action to be skipped.
- Case-insensitive matching is used.

### Behavior in APIServerMode

- **Enabled (`APIServerMode = true`)**:
- All validations are enforced.
- If any validation fails, the action is skipped, and a log message is recorded (e.g., "Skipping action due to method validation failure").
- The workflow continues processing the next resource in the dependency stack.

- **Disabled (`APIServerMode = false`)**:
- Validations are bypassed, and all HTTP methods, routes, headers, and parameters are permitted.
- Actions proceed without restriction, subject to other checks like `skipCondition` or `preflightCheck`.

### Example Workflow

Consider a resource with the above configuration and a request with:
- Method: `POST`
- Path: `/api/v1/users`
- Headers: `Content-Type`, `Authorization`
- Query Parameters: `user_id`, `token`

In `APIServerMode`:
- The `restrictToHTTPMethods` validation fails (`POST` is not in `["GET"]`), so the action is skipped.
- The `restrictToRoutes` validation would also fail (`/api/v1/users` is not in `["/api/v1/whois"]`).
- The `allowedHeaders` validation would fail if `request.header("Authorization")` is used, as it’s not in `["Content-Type"]`.
- The `allowedParams` validation would fail if `request.params("token")` is used, as it’s not in `["user_id", "session_id"]`.

The action is skipped at the first validation failure, and a log entry details the reason.

### Best Practices

- **Use Specific Restrictions:** Define only the necessary HTTP methods and routes to minimize skipping and ensure intended behavior.
- **Leverage Logging:** Review log messages for skipped actions to diagnose validation issues (e.g., incorrect method or path).
- **Test Configurations:** Validate resource configurations in a test environment to ensure the correct methods, routes, headers, and parameters are permitted.
- **Combine with Preflight Validations:** Use HTTP request validations alongside [Preflight Validations](#preflight-validations) for comprehensive checks, as they serve complementary purposes.

By incorporating HTTP request validations into your resources, you can enforce strict request compliance, enhance security, and streamline action execution in API-driven workflows.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ require (
github.com/google/uuid v1.6.0
github.com/joho/godotenv v1.5.1
github.com/kdeps/kartographer v0.0.0-20240808015651-b2afd5d97715
github.com/kdeps/schema v0.2.12
github.com/kdeps/schema v0.2.14
github.com/kr/pretty v0.3.1
github.com/spf13/afero v1.14.0
github.com/spf13/cobra v1.9.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kdeps/kartographer v0.0.0-20240808015651-b2afd5d97715 h1:CxUIVGV6VdgZo62Q84pOVJwUa0ONNqJIH3/rvWsAiUs=
github.com/kdeps/kartographer v0.0.0-20240808015651-b2afd5d97715/go.mod h1:DYSCAer2OsX5F3Jne82p4P1LCIu42DQFfL5ypZYcUbk=
github.com/kdeps/schema v0.2.12 h1:jilrgh+ZzcNMiEqVAPkIlgvGZkfMu3KvI8FHp9U66Eg=
github.com/kdeps/schema v0.2.12/go.mod h1:jcI+1Q8GAor+pW+RxPG9EJDM5Ji+GUORirTCSslfH0M=
github.com/kdeps/schema v0.2.14 h1:awZB3x9orhgAl898sGx8gjmLQJ8Enxz9A9aZG89yesA=
github.com/kdeps/schema v0.2.14/go.mod h1:jcI+1Q8GAor+pW+RxPG9EJDM5Ji+GUORirTCSslfH0M=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
Expand Down
2 changes: 1 addition & 1 deletion main.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func main() {
ctx = ktx.CreateContext(ctx, ktx.CtxKeyAgentDir, agentDir)

if env.DockerMode == "1" {
dr, err := resolver.NewGraphResolver(fs, ctx, env, logger.With("requestID", graphID))
dr, err := resolver.NewGraphResolver(fs, ctx, env, nil, logger.With("requestID", graphID))
if err != nil {
logger.Fatalf("failed to create graph resolver: %v", err)
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/docker/api_server.go
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ func APIServerHandler(ctx context.Context, route *apiserver.APIServerRoutes, bas

newCtx := ktx.UpdateContext(ctx, ktx.CtxKeyGraphID, graphID)

dr, err := resolver.NewGraphResolver(baseDr.Fs, newCtx, baseDr.Environment, logger)
dr, err := resolver.NewGraphResolver(baseDr.Fs, newCtx, baseDr.Environment, c, logger)
if err != nil {
resp := APIResponse{
Success: false,
Expand Down
2 changes: 1 addition & 1 deletion pkg/docker/docker_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -465,7 +465,7 @@ func itWillInstallTheModels(arg1 string) error {
}

func kdepsWillCheckThePresenceOfTheFile(arg1 string) error {
dr, err := resolver.NewGraphResolver(testFs, ctx, environ, logger)
dr, err := resolver.NewGraphResolver(testFs, ctx, environ, nil, logger)
if err != nil {
return err
}
Expand Down
101 changes: 100 additions & 1 deletion pkg/resolver/resolver.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ import (
"errors"
"fmt"
"path/filepath"
"regexp"
"runtime"
"time"

"github.com/apple/pkl-go/pkl"
"github.com/gin-gonic/gin"
"github.com/kdeps/kartographer/graph"
"github.com/kdeps/kdeps/pkg/environment"
"github.com/kdeps/kdeps/pkg/ktx"
Expand All @@ -30,6 +32,7 @@ type DependencyResolver struct {
Graph *graph.DependencyGraph
Environment *environment.Environment
Workflow pklWf.Workflow
Request *gin.Context
RequestID string
RequestPklFile string
ResponsePklFile string
Expand All @@ -49,7 +52,7 @@ type ResourceNodeEntry struct {
File string `pkl:"file"`
}

func NewGraphResolver(fs afero.Fs, ctx context.Context, env *environment.Environment, logger *logging.Logger) (*DependencyResolver, error) {
func NewGraphResolver(fs afero.Fs, ctx context.Context, env *environment.Environment, req *gin.Context, logger *logging.Logger) (*DependencyResolver, error) {
var agentDir, graphID, actionDir string

contextKeys := map[*string]ktx.ContextKey{
Expand Down Expand Up @@ -119,6 +122,7 @@ func NewGraphResolver(fs afero.Fs, ctx context.Context, env *environment.Environ
ResponsePklFile: responsePklFile,
ResponseTargetFile: responseTargetFile,
ProjectDir: projectDir,
Request: req,
}

workflowConfiguration, err := pklWf.LoadFromPath(ctx, pklWfFile)
Expand All @@ -128,6 +132,7 @@ func NewGraphResolver(fs afero.Fs, ctx context.Context, env *environment.Environ
dependencyResolver.Workflow = workflowConfiguration
if workflowConfiguration.GetSettings() != nil {
dependencyResolver.APIServerMode = workflowConfiguration.GetSettings().APIServerMode

agentSettings := workflowConfiguration.GetSettings().AgentSettings
dependencyResolver.AnacondaInstalled = agentSettings.InstallAnaconda
}
Expand Down Expand Up @@ -164,6 +169,68 @@ func (dr *DependencyResolver) processResourceStep(resourceID, step string, timeo
return nil
}

// validateRequestParams checks if params in request.params("header_id") are in AllowedParams.
func (dr *DependencyResolver) validateRequestParams(file string, allowedParams []string) error {
if len(allowedParams) == 0 {
return nil // Allow all if empty
}

re := regexp.MustCompile(`request\.params\("([^"]+)"\)`)
matches := re.FindAllStringSubmatch(file, -1)

for _, match := range matches {
param := match[1]
if !utils.ContainsStringInsensitive(allowedParams, param) {
return fmt.Errorf("param %s not in allowed params: %v", param, allowedParams)
}
}
return nil
}

// validateRequestHeaders checks if headers in request.header("header_id") are in AllowedHeaders.
func (dr *DependencyResolver) validateRequestHeaders(file string, allowedHeaders []string) error {
if len(allowedHeaders) == 0 {
return nil // Allow all if empty
}

re := regexp.MustCompile(`request\.header\("([^"]+)"\)`)
matches := re.FindAllStringSubmatch(file, -1)

for _, match := range matches {
header := match[1]
if !utils.ContainsStringInsensitive(allowedHeaders, header) {
return fmt.Errorf("header %s not in allowed headers: %v", header, allowedHeaders)
}
}
return nil
}

// validateRequestPath checks if the actual request path is in AllowedRoutes.
func (dr *DependencyResolver) validateRequestPath(req *gin.Context, allowedRoutes []string) error {
if len(allowedRoutes) == 0 {
return nil // Allow all if empty
}

actualPath := req.Request.URL.Path
if !utils.ContainsStringInsensitive(allowedRoutes, actualPath) {
return fmt.Errorf("path %s not in allowed routes: %v", actualPath, allowedRoutes)
}
return nil
}

// validateRequestMethod checks if the actual request method is in AllowedHTTPMethods.
func (dr *DependencyResolver) validateRequestMethod(req *gin.Context, allowedMethods []string) error {
if len(allowedMethods) == 0 {
return nil // Allow all if empty
}

actualMethod := req.Request.Method
if !utils.ContainsStringInsensitive(allowedMethods, actualMethod) {
return fmt.Errorf("method %s not in allowed HTTP methods: %v", actualMethod, allowedMethods)
}
return nil
}

// HandleRunAction is the main entry point to process resource run blocks.
func (dr *DependencyResolver) HandleRunAction() (bool, error) {
// Recover from panics in this function.
Expand Down Expand Up @@ -206,6 +273,38 @@ func (dr *DependencyResolver) HandleRunAction() (bool, error) {
continue
}

if dr.APIServerMode {
// Read the resource file content for validation
fileContent, err := afero.ReadFile(dr.Fs, res.File)
if err != nil {
return dr.HandleAPIErrorResponse(500, fmt.Sprintf("failed to read resource file %s: %v", res.File, err), true)
}

// Validate request.params
if err := dr.validateRequestParams(string(fileContent), *runBlock.AllowedParams); err != nil {
dr.Logger.Error("request params validation failed", "actionID", res.ActionID, "error", err)
return dr.HandleAPIErrorResponse(400, fmt.Sprintf("Request params validation failed for resource %s: %v", res.ActionID, err), false)
}

// Validate request.header
if err := dr.validateRequestHeaders(string(fileContent), *runBlock.AllowedHeaders); err != nil {
dr.Logger.Error("request headers validation failed", "actionID", res.ActionID, "error", err)
return dr.HandleAPIErrorResponse(400, fmt.Sprintf("Request headers validation failed for resource %s: %v", res.ActionID, err), false)
}

// Validate request.path
if err := dr.validateRequestPath(dr.Request, *runBlock.RestrictToRoutes); err != nil {
dr.Logger.Info("skipping due to request path validation not allowed", "actionID", res.ActionID, "error", err)
continue
}

// Validate request.method
if err := dr.validateRequestMethod(dr.Request, *runBlock.RestrictToHTTPMethods); err != nil {
dr.Logger.Info("skipping due to request method validation not allowed", "actionID", res.ActionID, "error", err)
continue
}
}

// Skip condition
if runBlock.SkipCondition != nil && utils.ShouldSkip(runBlock.SkipCondition) {
dr.Logger.Infof("skip condition met, skipping: %s", res.ActionID)
Expand Down
2 changes: 1 addition & 1 deletion pkg/resolver/resolver_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ func iLoadTheWorkflowResources() error {
logger := logging.GetLogger()
ctx = context.Background()

dr, err := resolver.NewGraphResolver(testFs, ctx, environ, logger)
dr, err := resolver.NewGraphResolver(testFs, ctx, environ, nil, logger)
if err != nil {
log.Fatal(err)
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/schema/schema.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (
var (
cachedVersion string
once sync.Once
specifiedVersion string = "0.2.12" // Default specified version
specifiedVersion string = "0.2.14" // Default specified version
UseLatest bool = false
)

Expand Down
4 changes: 2 additions & 2 deletions pkg/schema/schema_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ func TestSchemaVersion(t *testing.T) {
t.Parallel()
ctx := context.Background()

const mockLockedVersion = "0.2.12" // Define the version once and reuse it
const mockVersion = "0.2.12" // Define the version once and reuse it
const mockLockedVersion = "0.2.14" // Define the version once and reuse it
const mockVersion = "0.2.14" // Define the version once and reuse it

// Save the original value of UseLatest to avoid test interference
originalUseLatest := UseLatest
Expand Down
Loading
Loading