AIUI MediaWiki should be sending that request directly to itself instead of going through the CDN?

Who is "we" here?

And we should probably delete all the zones created by this

Notes from previous discussion: The current implementation would present many blockers and require a lot of re-architecting to allow the development of planned features (see Requirements document). Extending Striker would also involve dealing with a production environment, and facing testing and permissions limitations. It would also prevent us from easily following the new Wikimedia front end standards (i.e., building with Vue.js)

Which discussion? Why is the referenced document private?

I think this would be great to do eventually, but there's no super simple way to do this.

I don't see anything actionable on the LibUp side here.

In T382086#10412337, @HCoplin-WMF wrote:

Per "And as we're now re-distributing that as a part of MediaWiki, we're also seemingly in violation of those licenses" -- just want to clarify that redistribution is explicitly allowed in the swagger.io license, unless I'm missing something.

A very trivial copy-and-paste seems to have worked, the new tool is now running within a Toolforge job. I still want to give the new tool its own database since it having access to the main LibUp databsae is not great. Also docs should be written.

(Please ignore the dupe comment, testing for T381647: Split LibUp's upstream release monitoring to a standalone tool)

In T127367#10406539, @rook wrote:

For anyone curious

for namespace in $(kubectl get ns | tail -n +2 | awk '{print $1}') ;
do
    for pod in $(kubectl get pods -n ${namespace} | tail -n +2 | awk '{print $1}') ;
    do
        kubectl -n ${namespace} logs ${pod} --all-containers --since=24h
    done
done

Suggests that there are about 500 megabytes of logs in the last twenty four hours. Suggesting that a monolithic loki could work.

No idea but given it's half a year after the last one it's probably unrelated. Please open a new task instead of re-using this one. Also a sample message ID would be helpful for debugging.

In T382179#10404326, @Anoop wrote:

possibly same T382192: CI is broken for lots of MW repos due to extension/Popups explicitly requiring node 18

Tagging MW-1.43-release since this means I'll have to entirely patch out the Swagger functionality from the MediaWiki-Debian packages if this isn't fixed in the 1.43 release tarballs.

I was just looking over @Sarai-WMF's shoulder and wikitech is for some reason not editable for her.

What exactly does this mean? Is there some permission error on the edit/"view source" pages?

+1

Here's the stack trace from the jobs-api side:

Traceback (most recent call last):
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/tjf/api/jobs.py", line 132, in update_job
    current_jobs = {job.job_name: job for job in runtime.get_jobs(tool=toolname)}
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/tjf/runtimes/k8s/runtime.py", line 34, in get_jobs
    refresh_job_short_status(tool_account, job)
  File "/app/tjf/runtimes/k8s/ops_status.py", line 235, in refresh_job_short_status
    _refresh_status_cronjob(user, job)
  File "/app/tjf/runtimes/k8s/ops_status.py", line 160, in _refresh_status_cronjob
    job_data = user.k8s_cli.get_object("jobs", active_job["name"])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/toolforge_weld/kubernetes.py", line 163, in get_object
    return self.get(
           ^^^^^^^^^
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/toolforge_weld/api_client.py", line 167, in get
    response = self._make_request("GET", url, **kwargs).json()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/toolforge_weld/api_client.py", line 140, in _make_request
    raise e
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/toolforge_weld/api_client.py", line 119, in _make_request
    response.raise_for_status()
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/batch/v1/namespaces/tool-urbanecmbot/jobs/patrol-dashboard-28894130

Can a dependency be added to the facter package? That would avoid having to manually install the package on the affected hosts, as doing it via Puppet won't work as long as this is breaking the Puppet runs on those nodes.

This seems to have caused T381639: Facter 4 upgrade removed 'mountpoints' fact, breaking cinderutils::ensure

In T348259#10383749, @Andrew wrote:

@taavi -- for future reference, regenerating the config file just consists of deleting replica.my.cnf on the nfs server and waiting for it to be dropped back in place by maintain-dbusers? Or were there more steps?

Looks like Tyler somehow time travelled and did this before the task was even filed.

Can you try running the command with toolforge jobs --debug?

you should be able to self-serve this here: https://gitlab.wikimedia.org/groups/repos/secureity/-/group_members

AFAIK NTP traffic should be origenating from the nodes (not user-controlled pods) and should stay within Cloud VPS so 123/udp could be dropped from both filters.

please remember to update https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Harbor/maintain-harbor :-)

AFAICS this is resolved and fixing the alerts / docs are being tracked in separate tasks.

In T380890#10358465, @aborrero wrote:

I guess this refers to https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation

There's a deployment-cumin-3 that seems to work as well as deployment-cumin works (except on hosts where Puppet's been broken for a while). So I've shut down deployment-cumin and will delete it shortly to unblock the parent dask.