/area networking

What version of Knative?

Knative Serving v1.17, v1.18, v1.19, v1.20.0

Also confirmed on main branch: knative-v1.20.0-52-gabbe514be

Expected Behavior

When creating a Knative Service with a private Docker Hub image using a tag (e.g., docker.io/username/private-image:latest) and properly configured imagePullSecrets in the Service spec with valid credentials in a kubernetes.io/dockerconfigjson secret, the Revision controller should:

• Use the imagePullSecrets to authenticate during tag-to-digest resolution
• Successfully resolve the tag to a digest
• Create the pod and pull the image

Actual Behavior

The Revision fails with a 401 Unauthorized error during tag-to-digest resolution:

Unable to fetch image "docker.io/username/private-image:v0.1": 
failed to resolve image to digest: 
HEAD https://index.docker.io/v2/username/private-image/manifests/v0.1: 
unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

The imagePullSecrets are not being used for authentication during the HTTP HEAD request. The pod is never created because digest resolution fails before pod creation.

Key observations:

• The secret is correctly formatted (kubernetes.io/dockerconfigjson)
• The secret is properly referenced in spec.template.spec.imagePullSecrets
• The same credentials work for standard Kubernetes Pods (Deployments, StatefulSets)
• Using image digest format directly (image@sha256:...) works as a workaround

Attempted workarounds:

1. Adding docker.io to registries-skipping-tag-resolving in config-deployment ConfigMap - still fails with 401
2. Using image digests directly - ✅ works but breaks CI/CD pipelines

Steps to Reproduce the Problem

1. Create a Docker Hub private registry secret:

kubectl create secret docker-registry my-docker-secret \
  --docker-server=docker.io \
  --docker-username=<username> \
  --docker-password=<password> \
  -n testing-zone

2. Create a Knative Service with imagePullSecrets:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: test-private-image
  namespace: testing-zone
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
        autoscaling.knative.dev/max-scale: "1"
    spec:
      imagePullSecrets:
      - name: my-docker-secret
      containers:
      - image: docker.io/username/private-repo:latest
        ports:
        - containerPort: 8080

3. Observe the failure:

kubectl get revision -n testing-zone
# Output: READY=False, REASON=ContainerMissing

kubectl get revision <revision-name> -n testing-zone -o jsonpath='{.status.conditions[?(@.type=="ContainerHealthy")].message}'
# Output: 401 Unauthorized error

4. Verify credentials work with standard Kubernetes Pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: testing-zone
spec:
  imagePullSecrets:
  - name: my-docker-secret
  containers:
  - name: test
    image: docker.io/username/private-repo:latest

Result: The standard Pod successfully pulls the image using the same credentials.

Root Cause

The issue appears to be in pkg/reconciler/revision/resolve.go:116 where remote.Head() is called with remote.WithAuthFromKeychain(kc). Despite imagePullSecrets being passed to k8schain.Options in revision.go:90-99, the authentication fails during the HTTP HEAD request to Docker Hub.

The registries-skipping-tag-resolving workaround (originally added in PR #1390 for local dev registries) doesn't help because it returns an empty digest at resolve.go:112-114, preventing pod creation entirely.

Impact

This affects all users deploying private images from Docker Hub or other authenticated registries. The only workaround (using image digests) breaks CI/CD pipelines that rely on tags.