Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Randomly receiving proxy IP as X-Forwarded-For header using Traefik or Caddy #3436

Open
1 of 2 tasks
toverux opened this issue Sep 14, 2024 · 10 comments
Open
1 of 2 tasks
Labels
💎 Bounty Issues with a monetary reward for completion.

Comments

@toverux
Copy link

toverux commented Sep 14, 2024

Description

Edit: The issue is caused by Coolify not using an IPv6 network, so you get proxy IP for IPv6 users. You can solve this manually: #3436 (comment)
I'm leaving the issue open to signify that the issue is not solved at Coolify's level, but feel free to close when a more focused issue is created.

Hello!

I am having an issue with Traefik and Caddy proxies where they frequently (and kinda randomly) send me a forwarded-for IP address header that is actually a proxy local IP.
Ex.

  • When it works I'm reading this in my debug logs as the IPs received by my app: ["172.18.0.2", "xxx.xxx.xxx.xxx"] with the first IP being a proxy IP, second is the end user IP, and I have the right X-Forwarded-For header.
  • When it doesn't work I get this: ["172.18.0.2", "172.18.0.1"] with X-Forwarded-For being the second and "wrong" IP.

I tried adding traefik.entryPoints.http.forwardedHeaders.insecure=true to my project's Traefik config to see if it would just forward headers as-is, but it didn't change a thing.

I use pretty much all the defaults coming with a proxy install, didn't change anything. This is the project's config:

traefik.enable=true
traefik.http.middlewares.gzip.compress=true
traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.entryPoints=http
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.middlewares=redirect-to-https
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.rule=Host(`example.com`) && PathPrefix(`/`)
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.service=http-0-bk4sk40wgcwc4k0gc8skwogw
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.entryPoints=https
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.middlewares=gzip
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.rule=Host(`example.com`) && PathPrefix(`/`)
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.service=https-0-bk4sk40wgcwc4k0gc8skwogw
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.tls.certresolver=letsencrypt
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.tls=true
traefik.http.services.http-0-bk4sk40wgcwc4k0gc8skwogw.loadbalancer.server.port=4000
traefik.http.services.https-0-bk4sk40wgcwc4k0gc8skwogw.loadbalancer.server.port=4000
caddy_0.encode=zstd gzip
caddy_0.handle_path.0_reverse_proxy={{upstreams 4000}}
caddy_0.handle_path=/*
caddy_0.header=-Server
caddy_0.try_files={path} /index.html /index.php
caddy_0=https://example.com
caddy_ingress_network=coolify

I have no load balancing between server, both the proxy and the projects are on the same machine, one running instance only.

I'm completely unfamiliar with the stack (Coolify, Docker, Traefik, Caddy, etc) so I don't know where to look. I posted in the Discord, but no one was able to help.

I thought this is a serious enough bug though (and blocking, for me), since it makes us unable to track users' IPs (I need this for rate limiting and IP-banning), so I thought I could post an issue here.

Minimal Reproduction (if possible, example repository)

console.log(req.ip) with proxy headers trusting enabled in your application's HTTP router.

Exception or Error

No response

Version

4.0.0-beta.332

Cloud?

  • Yes
  • No
Copy link

algora-pbc bot commented Sep 14, 2024

💎 $50 bounty • algora0

Steps to solve:

  1. Start working: Comment /attempt #3436 with your implementation plan
  2. Submit work: Create a pull request including /claim #3436 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to coollabsio/coolify!

Add a bountyShare on socials

@algora-pbc algora-pbc bot added the 💎 Bounty Issues with a monetary reward for completion. label Sep 14, 2024
@peaklabs-dev
Copy link
Member

Are you sure the request where you do not seem to get the right IP are not just internal request in coolify? For example the healthcheck of your container?

@toverux
Copy link
Author

toverux commented Sep 14, 2024

@peaklabs-dev Yes, it has been verified that those are attached to users.

Edit: removed a comment that could me misleading, check out fa-sharp answer for the solution.

@toverux
Copy link
Author

toverux commented Sep 15, 2024

This was too much of a pressing issue for me, so in the meantime I've disabled the Coolify-managed proxy and apt-installed Nginx that I configured manually. Sadly, I loose load balancing and rolling releases, so a fix would still be much appreciated.

@enkiark
Copy link

enkiark commented Sep 16, 2024

same issue

@toverux
Copy link
Author

toverux commented Sep 26, 2024

I've disabled the Coolify-managed proxy and apt-installed Nginx that I configured manually.

So this was on the host machine and it worked well. Then I migrated Nginx to a container again and had the exact same issue.

So I guess this is not an issue with Coolify's handling of proxies specifically, but something about Coolify's Docker network or Docker in general that I don't understand.

@fa-sharp
Copy link

@toverux Don't know if you've resolved the issue or not, but I was running into a similar issue and I realized it was related to IPv6. Any client connecting with an IPv6 address was not getting their IP address correctly through and showed up with an internal/Docker IP address.

There are various threads here and there on the issues with Docker and IPv6, but I was able to use the official docs to get this resolved. Basically there are 2 choices: remove and recreate the Coolify network with IPv6 enabled, or edit the /etc/docker/daemon.json file to enable IPv6 and restart the Docker daemon. Once I did that, IPv6 addresses started coming through correctly.

(Note I'm not using Cloudflare tunnels or anything like that)

@toverux
Copy link
Author

toverux commented Jan 17, 2025

@fa-sharp No, I was not able to solve it and it's still an issue for me, so, many thanks for you investigation!

@toverux
Copy link
Author

toverux commented Jan 25, 2025

It works! Changing the daemon settings did not work for me even after a hard reboot, so I had to remove each container attached to the coolify network, then delete it, and recreate it with IPv6 enabled. Then I had to redeploy manually each Coolify-managed app so they'd be attached to the new network.

So, it would be nice if Coolify created its network with IPv6 enabled!

@fa-sharp
Copy link

Glad to hear that worked! I think there might be internet providers and devices that don't support IPv6, which is probably why it isn't the default. I just wish Docker made it a lot more straightforward to toggle it.

I'm hoping that the container/network management mentioned in #2495 would have a setting for IPv6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💎 Bounty Issues with a monetary reward for completion.
Projects
None yet
Development

No branches or pull requests

4 participants