[CRI] unable to pull dockerhub library image, reference for unknown type: application/octet-stream #4756

This seems to be cased by the containerd client's Pull API logic rather than CRI.
redis:2.8.23 has a legacy application/octet-stream config blob (unsupported by Docker/OCI specs; see also #2456 (comment)), which isn't recognized by the Pull API logic so layer won't be pulled.

$ curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" ...omit... https://registry-1.docker.io/v2/library/redis/manifests/2.8.23 | jq '.config'
{
  "mediaType": "application/octet-stream",
  "size": 5946,
  "digest": "sha256:481995377a044d40ca3358e4203fe95eca1d58b98a1d4c2d9cec51c0c4569613"
}

The solution will be to just add application/octet-stream MediaType here. But I'm not sure we should support this MediaType.

What's especially odd here is how it succeeds under manual pull with ctr but not when leaving the pull up to CRI.

ctr uses different API so it doesn't see this.
We can repro this manually with crictl.

# crictl pull docker.io/library/redis:2.8.23
...
FATA[0010] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/redis:2.8.23": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:4dcab49015d47e8f300ec33400a02cebc7b54cadd09c37e49eccbc655279da90: failed to get reader from content store: content digest sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b: not found 

Thanks for filing & investigating @BenTheElder & @ktock !

I agree we should update the redis image and I'll work on that - the whole example is very out of date. I don't know whether this is could be a problem for containerd/crictl (i.e. whether similar problems exist in other images); we couldn't find an existing bug for this problem.

/cc @tianon

There was fairly complete discussion on this in #2456. I think the net of it is:

• images built prior to Docker 1.11/mid-2016 could have the wrong media type encoded in the config; the bug was fixed in April 2016; released in Docker 1.11.
• While I understand it might be possible to depend on an image that is now over four years old, that flies pretty hard in the face of "cloud native" best practices in 2020 :)

The open question in my mind (possibly directed at @tianon 😇 ): I assume these very old images are truly not "official images" (in the sense of updates/support) anymore--has there ever been any thought to actually cull them from DockerHub?

Those old images were official images, but you are correct that they are no longer supported -- here's a standard reply we give to maintainers that I think might be helpful context:

Removing tags here will remove them from the "Supported" section on the Hub readme (and will prevent us from spending cycles rebuilding them on the official build servers), but the tags will still be available to users who want them. (See https://github.com/docker-library/official-images#library-definition-files for more detail on this.)

(And just to be explicitly clear, official images have never been built any more esoterically than docker build + docker push at the lowest levels, so this being due to bugs in Docker 1.11 seems likely 😞)

This issue cropped up because Kubernetes examples repo has very old sample deployments etc. that otherwise still function for demo purposes, so the images are not actively upgraded because nobody truly runs these. Of course we want them to work on containerd though.

Bumping seems reasonable, though I wonder what the impact of compat for this would be.

To clarify:

• I think we clearly need someone in the Kubernetes space to patch up examples to supported images (as we've done for the particular image from this issue).
• I wonder if more users will be surprised when migrating dockershim => containerd breaks running some image. I think a surprising amount of ""cloud native"" users are not necessarily keeping everything on the bleeding edge 🙃 .. If the compatibility patch would be small (I suspect it is, but I'm not sure?), it might be worth reconsidering?

Yeah, it's a tough spot, but what makes me hope to hold the line here is:

• images with this "bug" should be extremely rare. If we were talking this is something that happened to a lot of images from 6 months or even a year ago, I get it. Given we are approaching year 5 (in a few months) since this was fixed, I'm thinking we can figure out how to do some better image hygiene, even in examples. I know it's a pain--I have lots of out of date code in my GitHub examples, too!
• we really should be in an environment in 2020 where, as much as possible, compliance to image spec should be held as a gold standard by all build tools and runtimes. Containerd has some experience with this re: registry/distribution implementations that depend on lax Docker behaviors, but in the end, we've convinced several registries to fix inconsistencies rather than dot the code with a lot of "oh, and we have to allow this because.." clauses. It's much harder to ever remove those, as you might imagine.

I hope this particular issue with dockershim -> containerd migration experience is extremely rare, and hopefully would not impact anyone who is building and deploying code today as they will never have images built with Docker 1.10, I would hope! I'm going to ping @containerd/containerd-maintainers as others might have a different view.

I generally agree; I don't see a direct need for containerd to carry backward compatibility code for images created before containerd 1.0 even existed.

That said; I asked in our internal slack if we have data on manifests still being pushed to Docker Hub with this mediatype in the last year or so

That said; I asked in our internal slack if we have data on manifests still being pushed to Docker Hub with this mediatype in the last year or so

IMO the more interesting additional metric would be the frequency of users pulling them 😇

Resolving? containerd won't support container images with application/octet-stream.

This seems to be cased by the containerd client's Pull API logic rather than CRI.
redis:2.8.23 has a legacy application/octet-stream config blob (unsupported by Docker/OCI specs; see also #2456 (comment)), which isn't recognized by the Pull API logic so layer won't be pulled.

$ curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" ...omit... https://registry-1.docker.io/v2/library/redis/manifests/2.8.23 | jq '.config'
{
  "mediaType": "application/octet-stream",
  "size": 5946,
  "digest": "sha256:481995377a044d40ca3358e4203fe95eca1d58b98a1d4c2d9cec51c0c4569613"
}

The solution will be to just add application/octet-stream MediaType here. But I'm not sure we should support this MediaType.

containerd/unpacker.go

Line 319 in af963cc

case images.MediaTypeDockerSchema2Config, ocispec.MediaTypeImageConfig:

What's especially odd here is how it succeeds under manual pull with ctr but not when leaving the pull up to CRI.

ctr uses different API so it doesn't see this.
We can repro this manually with crictl.

# crictl pull docker.io/library/redis:2.8.23
...
FATA[0010] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/redis:2.8.23": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:4dcab49015d47e8f300ec33400a02cebc7b54cadd09c37e49eccbc655279da90: failed to get reader from content store: content digest sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b: not found 

Pr(#2814) haved supported application/octet-stream MediaType, And pr(#3870) haved missed this. we should add application/octet-stream MediaType back.

Additionally,
in containerd v1.3.4, crictl pull redis:2 is ok.
but in containerd v1.4.3, crictl pull redis:2 is error.

closing this issue with the guidance provided here:
#5497 (comment)