Tracked separately in T377805

In T376762#10246366, @CDanis wrote:

Should we attempt this on a cluster or two? One of the stagings, and then perhaps aux?

Good question. Let me add some data points. We currently use:

To debug apparmor profiles, an easy way is to add the complain flag. So, flags=(attach_disconnected) becomes flags=(complain) and the actions will be allowed and logged. That should some pretty specific hints as to what exactly would break in the "normal" (not fallbacks implemented in code to workaround failures) mode.

In T357950#10236864, @santhosh wrote:

@akosiaris We deployed this code in staging. Only issue we observe is ECS logging is not parsed by logstash.

the msg field has values like 2024-10-16T07:19:25.484793362Z stdout F {"@timestamp":"2024-10-16T07:19:25.484Z","ecs.version":"8.10.0","log.level":"info","message":"GET /_info 200"}

Example: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2024.10.16?id=E9I2lJIB-wz6FOsqP_-w
(please ignore the info level log, we changed the log level after this)

The application prints logs to stdout as json lines
{"@timestamp":"2024-10-16T07:19:25.484Z","ecs.version":"8.10.0","log.level":"info","message":"GET /_info 200"}

Do you know why this parsing failed?

I 've evaluated the change today

Can you re-run these with strace so that we can figure out whether it open /dev/null for read or write? I 99% expect read, but wanna be sure. I think allow reads on /dev/null (if we don't already) is going to be fine.

For what is worth, I also update the dashboard at https://grafana-rw.wikimedia.org/d/U4TuF-lMk/proton?orgId=1 to allow querying both DCs at once as well as selectively and fixed the per container saturation panels that were broken. I also removed the nodejs garbage collection panels as those metrics aren't being emitted anymore (support has been dropped at the service runner level IIRC)

In T349118#10237211, @Mvolz wrote:

Tilerator exists no more in the WMF environment. I 'll close this av invalid, feel free to reopen.

Mistakenly removed @dcausse, re-adding.

I suggest we let the change ride next week's train and activate the rerouting on or after Oct. 28th

In T359820#10222688, @JJMC89 wrote:

In T359820#10209474, @SLyngshede-WMF wrote:

@taavi I just added you to the list of of people who can block users. You should have a "Block/unblock accounts" in the menu now.

What about anyone else? Please document a process for getting the necessary access.

I 'll resolve this. All hosts that could be renumbered have been renumbered. 6 hosts have been decomed instead. And then there is a number of jobrunners that will hopefully soon be reimaged to wikikube-worker nodes.

In T374888#10227426, @MarkAHershberger wrote:

Probably happened once more on ganeti1034 today. node is still bullseye fwiw.

@HCoplin-WMF The internal mappings on rest-gateway work, all that's left is merging https://gerrit.wikimedia.org/r/c/1080232 and external traffic will move from being routed to restbase to being routed, via rest-gateway, to /w/script.php. It looks like, something major happening aside, we will be able after all to meet the Oct 17th date. Is there some communication or other action that you 'd like to take before the change is merged and deployed? Or should we just deploy it on that date?

After discussing with @hnowlan, I think I was wrong. We already have some infrastructure (in the form of lua scripts for ATS) that was built with Traffic to allow easier configuration of the routing and mappings of /api/rest_v1/ that I wasn't aware of. I 'll untag Traffic, I don't think they are needed for this one. I 've already pushed and merged a patch to configure the rest-gateway to do the mapping.

Adding content transform too.

I just had enough time to review this. This can't be implemented in the infrastructure that serviceops maintains but rather the CDN/Edge. Adding Traffic. I 'll also try to post a patch, but I 'll be needing traffic's OK. I 'll also post a patch but I think that implementing such a logic at the CDN/Edge layer is exposing too much information to the it. If done, it must be temporary and reverted at a very clearly agreed date and not left lingering.

statsd-exporter deployment merged and deployed in both eqiad and codfw. It is addressable via the standard mechanisms that the other deployment share. General docs are at https://wikitech.wikimedia.org/wiki/Kubernetes/Metrics and https://wikitech.wikimedia.org/wiki/Prometheus/statsd_k8s, more specific to this implementation T365265: Create a per-release deployment of statsd-exporter for mw-on-k8s (and we need to update docs for this one once we resolve that task). I 'll resolve this for now, in the interest of not letting it linger. Feel free to reopen!

The host has some history of failure per T370633: hw troubleshooting: CPU 2 machine check error detected for rdb1014.eqiad.wmnet

It won't require full rebuilds indeed. But it will require restarting all containers so that they pick up the new domain name. Which for most should be a noop cause, on purpose (due to discovery), we don't rely on these DNS records for most workloads. The hard part is probably figure out what (if anything) will break across all clusters and amend proactively.

@hashar Patch merged today. Starting next week, there should be some speed improvements (roughly cutting down the time in half). The unfortunate consequence is that someone using WikimediaDebug extension during that stage of the deployment might see errors. As I point out in the task, whether that's going to be acceptable is something to balance against the speedier deployments. Target group is rather small, so if they are made aware, it might just be worth it.

@ssastry, @Arlolra, fyi scandium is no more, may it RIP.

machine powered off manually

We 've already discussed this in a 1on1 and just for transparency's sake, this finds me in agreement. The outage we had this week carried a signal that we don't want to lose, that is that memory usage exploded over the course of a few hours, which in itself is a signal that something else was amiss (which is what T376795: mwscript-k8s creates too many resources is about). At the same time, an outage is the worst possible messenger. So finding some other way to keep the signal, like the alert pointed out above SGTM.

I 'll resolve this one. Things overall are OK deployment times wise. In fact, there is a downward trend in the following graph

In T371130#10214102, @Trizek-WMF wrote:

The reasons you give make a lot of sense. I'll document our process so that the person in charge books the time needed to change local times in the different messages, if not done by a volunteer first.

Using a bot would be nice, but we don't have the skills within the team to have one running. :)

+1 on an upgrade regardless of the rest.

In T371130#10213880, @Trizek-WMF wrote:

The retro is a bit late, sorry.

Noting that in Q3 FY24-25, that is the quarter starting on January 2025, we 'll be refreshing mw[2291-2376], which includes wikikube-ctrl2001 and wikikube-ctrl2001

@Pginer-WMF @santhosh, I 've tried below to summarize our conversation in P+T meeting. Please do point out mistakes and I 'll correct them, I am mostly reconstructing from memory. I 've also added a suggested path forward for next actions, let me know what you think.

In an ideal world, this process would inform the upper and lower bounds of an HPA and we wouldn't need to come up with exact numbers, but rather rely on some metric (e.g. PHP idle workers) and let the system balance itself. And still we would need to run the process to update those bounds every now and then, because the world (and our traffic) isn't set in stone. So automating it a bit would be worth it.

/etc/envoy/envoy.yaml was empty on the new host. Deleting it and running puppet fixed it, and it seems fine now

We 'll be tracking decom of scandium in T376632: decommission scandium, I 'll resolve this, feel free to reopen if something weird comes up with parsoidtest1001.

Nice job!

@wiki_willy wikikube-ctrl1001 is in the batch that is being decommissioned in this task. It was procured in 2019. Given the work done in T353464 where we had to somehow get hold of a 10G nic card and reimage the node and all, I am wondering whether we can push refresh back for like 1 year or so and put it in the next fy budget. It feels like wasted work on all sides to find some other node and redo this. Does that sound ok?

3 years later, we no longer have appservers, this is probably best closed as invalid

This is done

In T374683#10176123, @HCoplin-WMF wrote:

Just want to update that we should be good to go on this by Monday. The final changes to support the compatibility layers are currently riding the train.

@akosiaris -- can we plan to take action on this on Monday from the ServiceOps side? Is there anything else y'all need beforehand?

In T375845#10182322, @JMeybohm wrote:

Note that per past experience, changing the ippool is a arduous and dangerous process. We probably don't need that though and can live without aggregating on the configuration level the 2 /18s to a /17. On the BGP level, it's /26s anyway that get announced.

It indeed is, at least when changing the ip block size T345823: Wikikube staging clusters are out of IPv4 Pod IP's. It might be possible though to migrate to a new IPPool that contains the old one. We might as well aggregate the two pools into one when recreating the cluster for T341984: Update Kubernetes clusters to >1.25.

But we need to keep in mind that the cluster CIDR (pod IP range) is configured in multiple places in hiera (at least) and it needs to be configured in kube-proxy as well - where only one CIDR per stack (IPv4/IPv6) is supported.

@ssastry , @ihurbain parsoidtest1001 is up and running and should be a full replacement of scandium. Ι 've already updated https://wikitech.wikimedia.org/w/index.php?title=Parsoid&diff=prev&oldid=2230761 to reference the new hosts, if you know of other places, please update them. Basic alerts and checking does work but I have have missed something.

And we 've just seen this on parsoidtest1001 which is bullseye. Old host, scandium is on buster.

In T354169#10087192, @ayounsi wrote:

Upgrade is done, I had a bit more time to look into that.

Taking this list as baseline https://netbox.wikimedia.org/ipam/prefixes/?q=kubernetes (all the prefixes that mention "kubernetes" in their description.
We can see that there is already a drift between the tagged prefixes, some (like 2620:0:861:301::/116 are missing the k8s tag.

To my understanding, it would be the same functionally to replace the usage of the k8s tag with a Kubernetes "Prefix role" https://netbox.wikimedia.org/ipam/roles/ same kind of filtering, etc.
However getting in a smaller granularity (for example having a wikikube device role) seems overkill as it risks creating too many roles, but that's more a feeling than an absolute truth :)
Roles also don't support nesting, so we can't define a potential wikikube role as a kubernetes one.

I also suggest that we cleanup the current descriptions as the data they contain is redundant with the various prefixes attributes.
For example, taking some random prefixes from that list:

• Kubernetes IP spaces (eqiad) -> (no description), kubernetes role, keep the current site and status attributes
• kubernetes service IPs reservation (eqiad) -> Description: "WikiKube service", kubernetes role, keep the current site and status attributes
• kubernetes pod IPs staging (eqiad) -> Description: "WikiKube staging pod", kubernetes role, keep the current site and status attributes
• ML/DE Team Kubernetes IP spaces - ml-serve service IPs (eqiad) -> Description: "ml-serve service", kubernetes role, keep the current site and status attributes

Than way the descriptions are easier to read, filtering works more efficiently, we can decom the tags, and the usecase from the task description is taken care of.

(43) mw[1352-1357,1360-1363,1367-1371,1374-1397,1399,1405,1408-1409].eqiad.wmnet are wikikube nodes (albeit not renamed and can be depooled and decomed ASAP. And they should suffice for the complication 3 from above. Proceeding

1240 1241 1242 1243 1244 1246 1249 1250 1252 1253 1254 1257 1258 1259 1260 1262 1264 1265 1270 1271 1273 1274 1278 1279 1281 1282 1284 1286 1289 1290 1294 1296 1301 1303 1304 have been pooled and are service.

In T330768#10173616, @Jdforrester-WMF wrote:

Whilst poking around open alerts, I noticed that both citoid and zotero have a few active alerts about "Container …-tls-proxy is consistently using [>95]% of its memory limit"; is this expected? Other TLS proxies for prod services seem to be using much less (and even the ones not alerting for citoid/zotero seem to be using more than average). Is it just an artefact of making many outbound requests?

I 'll mark this as resolved. The failures were specific to the swagger probe and were due to lacking egress network policy rules for text-lb eqiad and codfw.

In T370755#10179050, @Pginer-WMF wrote:

In T370755#10178842, @jijiki wrote:

@santhosh given the low traffic and the low storage needs, we could start off by adding memcached pods in translation service's namespace, and start from there. If we find that resource wise this is not enough, we re-iterate. What do you think?

For context, regarding the "low traffic" observation, it is worth noting that we had to disable a feature that exposed MinT machine translation to Wikipedia readers on 23 wikis because the number of requests were exceeding the server capacity. More details in T363338#10038113 and in this report.

Exclusion merged, doesn't look like there is a need for another followup I think I 'll close this as resolved.

In T374888#10171478, @Jdforrester-WMF wrote:

In T374888#10170930, @akosiaris wrote:

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Collection/+/1075157 is up, how do I get anyone to review it?

C+2'ed.

We can probably shave off ~2 minutes for the mw-debug environment following the same approach as in T375477. I 'll let the approach sit for 1 week or so and if no issues arise I 'll apply it to mw-debug too.

For what is worth, if scap was the only way helmfile commands would ever be run, the above would be totally ok. However, we can not rule out that someone will run a helmfile command and apply the newer version. This could lead to an incident if the newer version was broken in some way (which is probably the case after a rollback). Also, it's a surprise waiting to happen. It's not the end of the world as a bug, but has the potential to lead to an incident at some point.

Adding 1 more data point. In the previous deployment I see also

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Collection/+/1075157 is up, how do I get anyone to review it?

In T375477#10170585, @jnuche wrote:

It seems timeouts are currently hardcoded: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/mw-api-int/helmfile.yaml#6

Maybe this could be parametrized and the values increased for the canaries? At e.g. https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/mw-api-int/values-canary.yaml

This could make sense if the expectation is there could be bumps in deployment times for the canaries in the future.

This probably was to embed graphite powered graphs in MediaWiki regarding extension download numbers. And the URL hasn't worked in quite a while anyway.

https://logstash.wikimedia.org/goto/69fa724990f8f554ac97601360675c79 points out the the slowest image pull was at 2m27s, most were way faster. webserver ones at a few seconds, but also multiversion ones tend to be ~1m 30s (the 2m27s is for the mwdebug ones plus a couple from other deployments).

I see some evictions happening during the deployment that could explain this, trying to correlate.

This increase in deployment times coincides with the deployment to production of the following scap change: https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/447. The corresponding task being T366778

Hmm, I 've dug into Turnilo too and I fear my change isn't going to have any impact. Per https://w.wiki/BHjy, the top 5 user agents request wise, all showing Chrome version that are at least ~2 years old (a sign of bot activity), are without a Referer and from singaporean IPs

Diving a bit deep in the rabbithole.

I 've dug a bit into this. And it's a mess of historical reasons.

Drive by comment to say that pediapress book printing has probably been broken for a very long time and no-one noticed. See T374888.

Nicely done! Thanks for the detailed writeup!

@matmarex , I am pretty confident I 've found the reason and have a patch at https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1074365. I 'd appreciate a review, mostly for avoiding side effects I am not aware of.

In T374907#10153807, @akosiaris wrote:

In T374907#10153052, @hashar wrote:

For the OCI image being pulled, scap has a step described as docker pull on k8s nodes which happens before the sync-testservers. I can't tell whether it was fast this morning but surely I can look it up in the scap logs.

That specific step is being re-evaluated in T366778 and unless something changes dramatically, it will soon be no more. Don't spend cycles on it.

Thanks for fixing it @JMeybohm. For posterity's sake, the "fooing" part was related to T374366 and trying to figure out the race condition(s).

In T374907#10153052, @hashar wrote:

Looking back in time a bit this isn't unheard of, so it doesn't look like some recent regression.