Cannot mirror images using oc-mirror: Unauthorized errors from registry.redhat.io

[bmanoj27]

Version

$ oc-mirror version
Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.14.0-202410182001.p0.ga0733c1.assembly.stream.el8-a0733c1", GitCommit:"a0733c17322f96d14f9158604fe74f1bfecc4a53", GitTreeState:"clean", BuildDate:"2024-10-18T22:52:57Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

What happened?

I attempted to use the oc-mirror command to mirror the images specified in the image-set.yaml configuration file to a local registry. However, the process fails with 401 Unauthorized errors when trying to pull from registry.redhat.io. Below is the relevant portion of the log:

The initial part of the logs give the 200 response code like this:

level=debug msg=fetch response received digest=sha256:05ede1ab60b724c1ac46464faf1ce1e1d20d690008e641779a775b1263e7477f mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.access-control-allow-methods=GET, POST, PUT, DELETE, OPTIONS response.header.access-control-allow-origin=* response.header.cache-control=max-age=1500 response.header.cf-cache-status=HIT response.header.cf-ray=8ebb0808bfb1470d-BOM response.header.connection=keep-alive response.header.content-length=29634 response.header.content-type=application/octet-stream response.header.date=Mon, 02 Dec 2024 11:38:35 GMT response.header.etag="7aed8d660cd4e9fced355100de8426a3" response.header.expires=Mon, 02 Dec 2024 15:38:35 GMT response.header.last-modified=Mon, 02 Dec 2024 10:52:08 GMT response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2=3lx91Q7t4BGXtGP9UKj+jfE2YCAax3+mXv3ZTcbo6sHEVJbdKNeP/LHdvZm7inCg74o0UBzBJqM= response.header.x-amz-replication-status=COMPLETED response.header.x-amz-request-id=KNNJ21TQN0ADJZ95 response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=Ilcyx6d.RFcGvesRQtG_winW_fHKJ1Ug response.status=200 OK size=29634 url=https://registry.redhat.io/v2/redhat/redhat-operator-index/blobs/sha256:05ede1ab60b724c1ac46464faf1ce1e1d20d690008e641779a775b1263e7477f

but right before it fails:
level=debug msg=fetch response received host=registry.redhat.io response.header.cache-control=max-age=0, no-cache, no-store response.header.connection=keep-alive response.header.content-length=99 response.header.content-type=application/json response.header.date=Mon, 02 Dec 2024 09:25:10 GMT response.header.docker-distribution-api-version=registry/2.0 response.header.expires=Mon, 02 Dec 2024 09:25:10 GMT response.header.pragma=no-cache response.header.www-authenticate=Bearer realm="https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth",service="docker-registry",scope="repository:redhat/certified-operator-index:pull" response.status=401 Unauthorized url=https://registry.redhat.io/v2/redhat/certified-operator-index/manifests/v4.14

error: error rendering new refs: render reference "registry.redhat.io/redhat/redhat-operator-index:v4.14": error resolving name for image ref registry.redhat.io/redhat/redhat-operator-index:v4.14: pulling from host registry.redhat.io failed with status code [manifests v4.14]: 401 Unauthorized

What did you expect to happen?

I have correctly logged in using docker to all the required registries, including registry.redhat.io and stored the file in ~/.docker/config.json and am able to pull certified/all images from redhat registry manually.

The oc-mirror command should authenticate correctly with registry.redhat.io using valid credentials and successfully pull the required images for mirroring.

How to reproduce it (as minimally and precisely as possible)?

1. image-set.yaml as follows:

apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
storageConfig:
 registry:
   imageURL: utility.techlife.local:5000/mirror/metadata
   skipTLS: true
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
        - name: elasticsearch-operator      
          channels:                    
            - name: stable-5.8             
            - name: stable         
              minVersion: '5.8.15'      
              maxVersion: '5.8.15'

        - name: jaeger-product        
          channels:                    
            - name: stable             
            - name: stable         
              minVersion: '1.57.0-10'      
              maxVersion: '1.57.0-10'

        - name: kiali-ossm        
          channels:                    
            - name: stable             
            - name: stable         
              minVersion: '1.89.7'
              maxVersion: '1.89.7'
        - name: servicemeshoperator
          channels:
            - name: stable
            - name: stable
              minVersion: '2.6.3-0'
              maxVersion: '2.6.3-0'

2. run the command: oc-mirror --config=image-set.yaml docker://utility.techlife.local:5000 --dest-skip-tls

I have actually mirrored elastic-search & jaeger-product operator a day before and it worked, and just after few hours facing this issue.

Appreciate the help, Thanks!
Manoj.

[soukron]

Same issue here no matter how many times we try.

Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.14.0-202405222237.p0.ga0733c1.assembly.stream.el8-a0733c1", GitCommit:"a0733c17322f96d14f9158604fe74f1bfecc4a53", GitTreeState:"clean", BuildDate:"2024-05-23T02:32:24Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

It's interesting to know that the image can not only be manually pulled but also be properly mirrored by the plugin if we include it as part of additionalImages but as soon as it's pulled/processed as part of a operator mirror, it fails consistently for us.

Also, there's no difference in mirroring to local nor to another registry.

To be honest, I think it's a misleading message and not related with network nor credentials.

[gammohamed]

@soukron @bmanoj27 , Any updates about the issue ?

[bmanoj27]

Hi @gammohamed

No but I have just tried by removing all the packages from the YAML, and by adding one by one.
Like starting with one package, and then adding one after the first one completes. Just make sure you don't remove the any package after the mirroring completes, just keep on adding...

[soukron]

No updaets from my side neither. I just moved to a newer version of the operator so I don't need to mirror that one anymore.

[bmanoj27]

No updaets from my side neither. I just moved to a newer version of the operator so I don't need to mirror that one anymore.

Hi @soukron
oc-mirror --v2

Is this how you are using the new version?

Thanks.

[soukron]

No updaets from my side neither. I just moved to a newer version of the operator so I don't need to mirror that one anymore.

Hi @soukron oc-mirror --v2

Is this how you are using the new version?

Thanks.

No, I meant I moved to a newer version of the operator that I was trying to mirror. Interestingly it stopped failing...

[aguidirh]

Hi @bmanoj27,

Are you still facing this issue?

If yes, could you try oc-mirror --v2 ?

[bmanoj27]

Hi @bmanoj27,

Are you still facing this issue?

If yes, could you try oc-mirror --v2 ?

Hello @aguidirh

Yes I have migrated to --v2. I will close this case.

Thanks!