Skip to content

πŸ› Bug Report β€” httpx via Cloudflare Python Worker drops User-Agent headersΒ #7242

Open
hoodmane/httpx2-jsfetch
#4

Description

@jGowgiel

After some debugging, I believe I have found some kind of bug or issue with how User-Agent headers are handled by httpx as provided within the Python Worker runtime.

The following snippet sets up a simple worker that makes a few requests, and outputs the result.

class Default(WorkerEntrypoint):
    async def fetch(self, request):
        # A manually constructed request for reference
        req = Request.new("https://httpbin.org/headers")
        req.headers.set("User-Agent", "some custom user-agent", "Custom-Header")
        req.headers.set("Custom-Header", "another value")
        result = await fetch(req)
        print(await result.json())

        # An httpx driven request (which does not function as expected)
        print(
            (
                await httpx.AsyncClient().get(
                    "https://httpbin.org/headers",
                    headers={
                        "User-Agent": "some custom user-agent",
                        "Custom-Header": "another value",
                    },
                )
            ).json()
        )

The response for the first request is:

{'headers': {'Cf-Worker': 'worker.example.com', 'Custom-Header': 'another value', 'Host': 'httpbin.org', 'User-Agent': 'some custom user-agent', 'X-Amzn-Trace-Id': 'Root=1-6a9a557a-2bfffbe61affbf6b149be877'}}

For the second request, however, the response is:

{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Cf-Worker': 'worker.example.com', 'Custom-Header': 'another value', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6a9a557a-37c5a13b304346096bb8e308'}}

Aside from the few extra headers, the request through httpx does not have the custom User-Agent. It appears to have been uniquely dropped at some point, since the other Custom-Header survives.

I have exhausted most other debugging options, and am left thinking that this must be either an intentional exemption in the runtime (preventing User-Agent from being overridden, but only sometimes?) or a bug internally with how httpx is bridged to Python Workers.

When htps is run in a local script outside of the Python Worker runtime, the correct User-Agent header is attached.

Environment info:

  • Pywrangler: 1.16.7
  • Node: v26.7.0
  • httpx: 0.28.1
  • Compatibility date: 2026-08-20
  • Using the python_workers compatibility flag

Activity

  1. changed the title [-]πŸ› Bug Report β€” httpx via Cloudflare Python Worker refuses User-Agent headers[/-] [+]πŸ› Bug Report β€” httpx via Cloudflare Python Worker drops User-Agent headers[/+] on Sep 4, 2026
  2. ryanking13 commented on Sep 7, 2026

    @ryanking13
    Contributor

    Thanks for the report. We can patch httpx to fix that, but httpx is currently not maintained and we recommend replacing it with httpx2, which is better maintained by Pydantic team.

    I wonder if you are using httpx directly, or using it as a dependency of another package. If former is the case, could you try to replace it with httpx2?

  3. jGowgiel commented on Sep 7, 2026

    @jGowgiel
    Author

    I am using httpx directly, so I could swap to httpx2 - however, the same issue occurs with httpx2 as well.

    Whichever is easiest to fix upstream, I'm happy to use, but it does seem like both versions have this same problem.

    I will say that 90% of the Python Workers docs I have seen refer to httpx if they refer to any third party HTTP client, so if httpx2 is recommended, it's probably worth auditing and updating all of that documentation to match.

  4. ryanking13 commented on Sep 7, 2026

    @ryanking13
    Contributor

    Whichever is easiest to fix upstream, I'm happy to use, but it does seem like both versions have this same problem.

    I see. Let us see if we can fix it in upstream httpx2.

    I will say that 90% of the Python Workers docs I have seen refer to httpx if they refer to any third party HTTP client, so if httpx2 is recommended, it's probably worth auditing and updating all of that documentation to match.

    Yeah, we need to update them all... thanks. There was some drama in httpx/httpx2/httpxyz lately.

  5. hoodmane commented on Sep 8, 2026

    @hoodmane
    Contributor

    This was added intentionally at some point because attempting to set a user agent header in a browser causes a warning saying it's not allowed. But we shouldn't do it in node or workerd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions