I did almost nothing (2 edits) for 19 minutes. Started VFC again for some forgotten edits. 5 edits and BOOM, throttled. 3 more and BOOM, throttled.

Maybe VFC is performing null edits and the limit on that is lower or something? But that's rubbish, if I try to edit (not null-edit) right after VFC pauses, the action gets throttled.

Hi @AlexisJazz, thanks for taking the time to report this. I don't know what VFC is and we don't know which specific steps you performed. Please see https://www.mediawiki.org/wiki/How_to_report_a_bug and provide clear steps to reproduce. Thanks.

In T245214#5883646, @Aklapper wrote:

Hi @AlexisJazz, thanks for taking the time to report this. I don't know what VFC is and we don't know which specific steps you performed. Please see https://www.mediawiki.org/wiki/How_to_report_a_bug and provide clear steps to reproduce. Thanks.

Sorry, VFC is so well known to power users on Commons that I forgot. It's https://commons.wikimedia.org/wiki/Help:VisualFileChange.js.

I had https://commons.wikimedia.org/wiki/Category:Media_from_Beeld_en_Geluid_Wiki_with_unchecked_source opened in VFC and replaced:

• "http://www.beeldengeluidwiki.nl/index.php" with "https://wiki.beeldengeluid.nl/index.php"
• "https://www.beeldengeluidwiki.nl/index.php" with "https://wiki.beeldengeluid.nl/index.php" (I know, I'm lazy)
• "{{Beeld en Geluid Wiki|source=}}" with "{{Beeld en Geluid Wiki|source={{subst:User:Alexis Reggae/BenGwiki license|1=File:{{subst:PAGENAME}}}}}}"

Less than 1000 files were selected and 76 edits ultimately resulted. I started the job at 22:07 and it finished at 23:39. Two edits failed because it took so damn long I got edit conflicts for https://commons.wikimedia.org/wiki/File:Slade_-_TopPop_1973_12.png and https://commons.wikimedia.org/wiki/File:StephenStills1972.jpg.

So the Commons edit ratelimit configuration is

"ip" => [ 120, 900, 8, 60 ],
"newbie" => [ 120, 300, 8, 60 ],
"user" => [ 900, 180, 90, 60 ],
"image-reviewer" => [ 10500, 180 ],
"patroller" => [ 10500, 180 ],
"autopatrolled" => [ 10500, 180 ],

(tries / seconds - no idea what are the extra two values in some of them).

If this is something similar to T234743: User rights validation is sometimes malfunctioning (with FlaggedRevs only?), then maybe you are getting the user limits (on some edits only) instead of the patroller limits. If you have more tasks lined up, can you hit the limit and then manually test a few times whether you are being reliably throttled?

I guess throttling is owned by Security?

Is the extra values maybe the array keys not merged properly? I think 8, 60 is default for ip.

In T245214#5883532, @AlexisJazz wrote:

I did almost nothing (2 edits) for 19 minutes. Started VFC again for some forgotten edits. 5 edits and BOOM, throttled. 3 more and BOOM, throttled.

Maybe VFC is performing null edits and the limit on that is lower or something? But that's rubbish, if I try to edit (not null-edit) right after VFC pauses, the action gets throttled.

This sounds like the rate limit key doesnt have an expiry. I think that happened once before in the past with abusefilter and nobody figured out why so we just manually deleted they key

In T245214#5883727, @Tgr wrote:

So the Commons edit ratelimit configuration is

"ip" => [ 120, 900, 8, 60 ],
"newbie" => [ 120, 300, 8, 60 ],
"user" => [ 900, 180, 90, 60 ],
"image-reviewer" => [ 10500, 180 ],
"patroller" => [ 10500, 180 ],
"autopatrolled" => [ 10500, 180 ],

(tries / seconds - no idea what are the extra two values in some of them).

If this is something similar to T234743: User rights validation is sometimes malfunctioning (with FlaggedRevs only?), then maybe you are getting the user limits (on some edits only) instead of the patroller limits. If you have more tasks lined up, can you hit the limit and then manually test a few times whether you are being reliably throttled?

Okay, I selected 924 files in VFC. None of these would result in actual edits, but possibly null edits as VFC didn't color them yellow which I'm guessing is the result of my substitution. (pre-substitution wikitext differs from the source, so VFC will submit it, but post-substitution is identical so no actual edit is made)

With 889 edits remaining (924-889=35..?) VFC paused. I tried editing 6 different pages by hand (not null edits) and all 6 edits were rejected. I kept trying and when I was able submit edits again (made two edits) I looked at VFC: 120-something seconds remaining. So about a minute had passed and I was able to edit again.

In T245214#5883800, @Bawolff wrote:

Is the extra values maybe the array keys not merged properly? I think 8, 60 is default for ip.

Yeah, that must be it. I guess not much harm in it, User::pingLimiter() just takes the first two items.

In T245214#5883804, @Bawolff wrote:

This sounds like the rate limit key doesnt have an expiry. I think that happened once before in the past with abusefilter and nobody figured out why so we just manually deleted they key

Shouldn't the user/IP get locked out permanently in that case?

Shouldn't the user/IP get locked out permanently in that case?

Good point

I tried it again, selected 500 files (same stuff, just null edits) and with 467 files remaining I got throttled.

I'm now going to request Account Creator (noratelimit) because I simply can't work like this. (I can do more testing with a sock if needed)

Edit:
A note: "Well, this was introduced because of vandals editing with speed over 30000 edits per hour (this is too high). The reference is T192668 but the task is restricted because of security." (T194864, see also here)

This kind of vandalism has (to my knowledge) never been a thing on Commons to begin with. And it CERTAINLY has never been a thing with users who are autopatrolled and up. Those are NOT vandals.

• Translation administrators had problems with the ratelimit: T155162
• When the ratelimit for edits was first installed, Commons just broke and the limit had to be raised significantly.
• Not all was solved, eventually people spoke up because moving pages was a problem. So we raised that limit too: T232657

And I am under the impression (but can't provide hard data to back this up) that ever since ratelimit was introduced, bulk editing became significantly slower. By which I mean without actually getting throttled. I remember how VFC would just swoosh through the files, it was beautiful. Never seen that again after May 2018.

Ratelimit is a solution in search of a problem for autopatrolled and up. It's done literally no good, only give people headaches. Just something for you to tinker about.

191 files selected in VFC. With 40 remaining (191-40=151) I got throttled. 120 actual edits were made and I guess 31 null edits.

Before this I made 190 or so edits (no null edits) without being throttled.

So I suspect it's really the null edits that get throttled at roughly 30/minute, and for some reason when you get throttled for null edits, you also get throttled for regular edits. Where's that image of Darth Vader with a jug in the ocean?

The purge rate limit is 30/60 so apparently that is being triggered for null edits.

Change 572339 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Increase Commons purge rate limit for patrollers

https://gerrit.wikimedia.org/r/572339

Patch is probably not ideal, as real (recursive) purges are expensive. Null edits probably should not count as purges (they are cheaper than real edits). But it works as a quick fix.

Thanks for figuring out what's going on @AlexisJazz.

In T245214#5886042, @Tgr wrote:

Patch is probably not ideal, as real (recursive) purges are expensive. Null edits probably should not count as purges (they are cheaper than real edits). But it works as a quick fix.

Thanks for figuring out what's going on @AlexisJazz.

Thanks! I know these null edits waste some resources, but excluding those pages from my selection is very difficult. It would mean either categorizing files before I run VFC (that would be an additional edit, that's more expensive! I do this only when there's no other way) or spending like an hour or two figuring out a way every time I make a run. Between a human wasting 2 hours of their life or a CPU wasting 2 minutes of its life, I prefer putting the burden on the CPU. ;-)

With 1000/minute I can continue my work without fear of getting throttled. :-)

Btw, I guess this does mean that if you exceed any ratelimit, your entire account gets locked out, not just whatever action you were performing?

In T245214#5886175, @AlexisJazz wrote:

Btw, I guess this does mean that if you exceed any ratelimit, your entire account gets locked out, not just whatever action you were performing?

Oh, right, you were blocked from normal editing as well, not just null edits. I probably misidentified the problem then.
(Also, looking at the code, I see no obvious way how the purge throttle would be involved, although I might be missing some non-obvious way.)

At the very least, I think the purge rate limit should be higher than the edit rate limit.

I guess it would be possible to confirm this theory by next time Alexis is rate limited, have someone with shell access look at all the various memcache keys to see which one's are too high. Weirdly we don't seem to log throttle events (unless its login related).

(Also, looking at the code, I see no obvious way how the purge throttle would be involved, although I might be missing some non-obvious way.)

I think linkpurge throttle is more likely than purge ( https://github.com/wikimedia/mediawiki/blob/master/includes/EditPage.php#L2112 ) . linkpurge will be checked before any edit (but not incremented unless your edit is a null edit) so if you hit it, you get locked out of editing until it is reset. It is also currently set to 30 per minute.

The other question is why is this happening just now? linkpurge limits have been in place since at least 2013-ish

Ideally of course, VFC would just see that its edit is a null edit and not do it. Null edits are kind of expensive for the servers to do (Because they trigger refreshing all db tables), so avoiding doing them unintentionally would probably be good.

In T245214#5886329, @Bawolff wrote:

At the very least, I think the purge rate limit should be higher than the edit rate limit.

I guess it would be possible to confirm this theory by next time Alexis is rate limited, have someone with shell access look at all the various memcache keys to see which one's are too high. Weirdly we don't seem to log throttle events (unless its login related).

(Also, looking at the code, I see no obvious way how the purge throttle would be involved, although I might be missing some non-obvious way.)

I think linkpurge throttle is more likely than purge ( https://github.com/wikimedia/mediawiki/blob/master/includes/EditPage.php#L2112 ) . linkpurge will be checked before any edit (but not incremented unless your edit is a null edit) so if you hit it, you get locked out of editing until it is reset. It is also currently set to 30 per minute.

The other question is why is this happening just now? linkpurge limits have been in place since at least 2013-ish

I guess few people do what I do. (and not everyone reports issues) Though I have done this before, and probably made mass null edits before without issue. But, probably. Maybe I didn't. Can't say for sure.

In T245214#5886373, @Bawolff wrote:

Ideally of course, VFC would just see that its edit is a null edit and not do it. Null edits are kind of expensive for the servers to do (Because they trigger refreshing all db tables), so avoiding doing them unintentionally would probably be good.

It does, but I'm using a substitution to insert data I gathered locally and dumped into my userspace. VFC has no way to see whether or not that substitution will result in an edit or not.

Easy reproduction steps:

• Load any category (or uploads from a user, or search, whatever) with VFC. Make sure enough (say, 100) files are loaded and select all.
• Select "custom replace"
• Replace "a" with "a{{subst:void}}" (without the quotation marks)
• Execute

This results in nothing but null edits and quickly hitting the ratelimit.

In T245214#5886373, @Bawolff wrote:

Ideally of course, VFC would just see that its edit is a null edit and not do it. Null edits are kind of expensive for the servers to do (Because they trigger refreshing all db tables), so avoiding doing them unintentionally would probably be good.

Ideally, we'd have an API option to say "if this is a null edit, just discard it silently". Or maybe just never trigger the null edit purge logic on API edits. They were always a UI hack to provide editors with an intuitive way to refresh the page.

Given the state of EditPage code, probably better to wait until it gets refactored for MCR.

In T245214#5886683, @AlexisJazz wrote:

Easy reproduction steps:

• Load any category (or uploads from a user, or search, whatever) with VFC. Make sure enough (say, 100) files are loaded and select all.
• Select "custom replace"
• Replace "a" with "a{{subst:void}}" (without the quotation marks)
• Execute

This results in nothing but null edits and quickly hitting the ratelimit.

Thanks. I saved the profile in VFC. Will use it for testing.

The next deploy window is Tuesday 12:00 UTC, can one of you be around to test it?

In T245214#5887035, @Tgr wrote:

The next deploy window is Tuesday 12:00 UTC, can one of you be around to test it?

If Masumrezarock100 isn't around for some reason and if I forget, just ping me on-wiki.

The patch only affects patrollers. I've just been granted account creator (noratelimit) and I have no alternate accounts with patroller right. (I have an alternate account with autopatroller right..)

@Masumrezarock100, can you test it after deployment?

If nobody is around for testing at that moment, maybe ping me (@ IRC!) and I might be able to grant patrollers for few hours.

In T245214#5887452, @AlexisJazz wrote:

In T245214#5887035, @Tgr wrote:

The next deploy window is Tuesday 12:00 UTC, can one of you be around to test it?

If Masumrezarock100 isn't around for some reason and if I forget, just ping me on-wiki.

The patch only affects patrollers. I've just been granted account creator (noratelimit) and I have no alternate accounts with patroller right. (I have an alternate account with autopatroller right..)

@Masumrezarock100, can you test it after deployment?

Sure.

In T245214#5888211, @Masumrezarock100 wrote:

Sure.

Thanks! I added you for SWAT.

Change 572339 merged by jenkins-bot:
[operations/mediawiki-config@master] Increase Commons linkpurge rate limit for patrollers

https://gerrit.wikimedia.org/r/572339

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:06:20Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 4b193dd: Increase Commons linkpurge rate limit for patrollers (T245214) (duration: 01m 31s)

I've deployed the fix @Tgr uploaded. Wasn't tested yet, but the purge limit is indeed increased.

>>> $wgRateLimits['linkpurge']
=> [
     "patroller" => [
       3000,
       180,
     ],
     "ip" => [
       30,
       60,
     ],
     "user" => [
       30,
       60,
     ],
   ]
>>>

I have tested the patch per reproduction steps in T245214#5886683. My rate limit indeed got increased. The API response from https://commons.wikimedia.org/w/api.php?action=query&format=json&meta=userinfo&formatversion=2&uiprop=ratelimits is as follows.

{
  "batchcomplete": true,
  "query": {
    "userinfo": {
      "id": 7299736,
      "name": "Masumrezarock100",
      "ratelimits": {
        "move": {
          "user": {
            "hits": 8,
            "seconds": 60
          },
          "patroller": {
            "hits": 32,
            "seconds": 60
          }
        },
        "edit": {
          "user": {
            "hits": 900,
            "seconds": 180
          },
          "patroller": {
            "hits": 10500,
            "seconds": 180
          }
        },
        "upload": {
          "user": {
            "hits": 380,
            "seconds": 4320
          },
          "patroller": {
            "hits": 999,
            "seconds": 1
          }
        },
        "linkpurge": {
          "user": {
            "hits": 30,
            "seconds": 60
          },
          "patroller": {
            "hits": 3000,
            "seconds": 180
          }
        },
        "badcaptcha": {
          "user": {
            "hits": 30,
            "seconds": 60
          }
        },
        "emailuser": {
          "user": {
            "hits": 20,
            "seconds": 86400
          }
        },
        "changeemail": {
          "user": {
            "hits": 4,
            "seconds": 86400
          }
        },
        "rollback": {
          "user": {
            "hits": 100,
            "seconds": 60
          }
        },
        "purge": {
          "user": {
            "hits": 30,
            "seconds": 60
          }
        },
        "renderfile": {
          "user": {
            "hits": 700,
            "seconds": 30
          }
        },
        "renderfile-nonstandard": {
          "user": {
            "hits": 70,
            "seconds": 30
          }
        },
        "cxsave": {
          "user": {
            "hits": 10,
            "seconds": 30
          }
        },
        "urlshortcode": {
          "user": {
            "hits": 50,
            "seconds": 120
          }
        },
        "thanks-notification": {
          "user": {
            "hits": 10,
            "seconds": 60
          }
        },
        "badoath": {
          "user": {
            "hits": 10,
            "seconds": 60
          }
        }
      }
    }
  }
}

I opened VFC and selected 250 files (means 250 null edits) https://commons.wikimedia.org/wiki/Category:Featured_pictures_on_Wikimedia_Commons to test the limit. I didn't ran into any rate-limits during the run and I was able to complete the job without any interruption.
I tested it again, now with 1000 files (means 1000 null edits) but I didn't run into any rate limit.

Should we keep this task private for a couple of days just in case there are any unintended consequences and we have to revert?

Related bug : #lingua_libre > T260649 .

In T245214#5883727, @Tgr wrote:

So the Commons edit ratelimit configuration is

"ip" => [ 120, 900, 8, 60 ],
"newbie" => [ 120, 300, 8, 60 ],
"user" => [ 900, 180, 90, 60 ],
"image-reviewer" => [ 10500, 180 ],
"patroller" => [ 10500, 180 ],
"autopatrolled" => [ 10500, 180 ],

(tries / seconds - no idea what are the extra two values in some of them).

If this is something similar to T234743: User rights validation is sometimes malfunctioning (with FlaggedRevs only?), then maybe you are getting the user limits (on some edits only) instead of the patroller limits. If you have more tasks lined up, can you hit the limit and then manually test a few times whether you are being reliably throttled?

@Tgr : To solve our upload ratelimit issue on LinguaLibre -> Commons uploads, I'am looking for the source of your statement : what are the ratelimite configurations of a given wiki, by example Commons.

I found this : https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/InitialiseSettings.php#9983

Please correct me if I'am wrong.

	'+commonswiki' => [ // T132930
		'move' => [ // T232657
			'autopatrolled' => [ 32, 60 ],
			'patroller' => [ 32, 60 ],
			'image-reviewer' => [ 32, 60 ],
		],
		'edit' => [
			'ip' => [ 8 * 15, 60 * 15 ], // T225148
			'newbie' => [ 8 * 15, 60 * 5 ], // T231463
			'user' => [ 900, 60 * 3 ], // T194864
			// Higher rate limit for trusted users
			'image-reviewer' => [ 10500, 60 * 3 ],
			'patroller' => [ 10500, 60 * 3 ],
			'autopatrolled' => [ 10500, 60 * 3 ],
		],
		'upload' => [
			// 380 uploads per 72 minutes
			'user' => [ 380, 4320 ],
			// Effectively no upload rate limit for members of these groups
			'image-reviewer' => [ 999, 1 ],
			'patroller' => [ 999, 1 ],
			'autopatrolled' => [ 999, 1 ],
		],
		'linkpurge' => [
			'patroller' => [ 3000, 60 * 3 ], // T245214
		],
	],