Skip to content

[HttpClient] CurlHttpClient not closing file descriptors #60513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
digilist opened this issue May 22, 2025 · 7 comments
Open

[HttpClient] CurlHttpClient not closing file descriptors #60513

digilist opened this issue May 22, 2025 · 7 comments
Labels
Bug HttpClient Status: Needs Review

Comments

@digilist
Copy link
Contributor

Symfony version(s) affected

at least 6.4.x, 7.2.x

Description

When sending requests with the CurlHttpClient against different host names, the file descriptors are not garbage collected and eventually further requests might be blocked by hitting the open file descriptors limit (ulimit -n).

Consider this example:

<?php

use Symfony\Component\HttpClient\CurlHttpClient;

require 'vendor/autoload.php';

$requests = [
    'https://symfony.com/',
    'https://github.com/',
    'https://google.com/',
    'https://stackoverflow.com/',
    'https://spotify.com/',
];

$client = new CurlHttpClient();
foreach ($requests as $url) {
    $response = $client->request('GET', $url);
    dump($url . ' ' . $response->getStatusCode());

    $fdCount = count(scandir('/proc/self/fd'));
    dump("Open file descriptors: {$fdCount}");
}

Executing this script gives me the following output:

"https://symfony.com/ 200"
"Open file descriptors: 12"
"https://github.com/ 200"
"Open file descriptors: 13"
"https://google.com/ 200"
"Open file descriptors: 15"
"https://stackoverflow.com/ 200"
"Open file descriptors: 16"
"https://spotify.com/ 200"
"Open file descriptors: 19"

As you can see the number of open file descriptors is increasing, even though the response objects can be garbage collected.

Extending the requests array with further URLs on the same hosts will not increase the file descriptors, but adding further hosts will do so.

Once the number of file descriptors is reached, it is not possible to send further requests and I get the following error:

PHP Fatal error: Uncaught Symfony\Component\HttpClient\Exception\TransportException: Could not resolve host: {hostname}

It took me a while to pinpoint the DNS resolution error to the file descriptors and this is a very subtle bug.

A workaround that solved the issue for me was creating a new HTTP client every x requests to stay below my file descriptor limit, but that's not really a good solution and needs awareness of this issue.

How to reproduce

You can change the file descriptor limit for the current shell with e.g. ulimit -n 15. Afterwards, when you run the script above you should see the described error.

Instead of reducing the limit, you could also increase the number of hosts, but this might require a larger list depending on the current limit.

Possible Solution

No response

Additional Context

No response
@digilist digilist added the Bug label May 22, 2025
@joelwurtz
Copy link
Contributor

Did you try to add Connection: close header to your request ?

Having a file descriptor per host is normal IMO, as connection should be kept alive if there is other request so you don't have to reconnect each time you do a request on the same host.

@digilist
Copy link
Contributor Author

Did you try to add Connection: close header to your request ?

Yes, I tried that, but I did not notice any difference.

Having a file descriptor per host is normal IMO, as connection should be kept alive if there is other request so you don't have to reconnect each time you do a request on the same host.

In general I agree on that, but after some time idle connections should be closed, or at least there should be a (configurable) maximum number of idle connections. This is especially a problem in long running processes.

(Or maybe there is a configuration for that, but I haven't seen anything about this yet?)

@joelwurtz
Copy link
Contributor

Yes, I tried that, but I did not notice any difference.

Maybe it's because the curl handle is shared and never close, did you try to set CURLOPT_FORBID_REUSE to true (their is an option to set curl options) ?

@digilist
Copy link
Contributor Author

Thank you, that's a good hint and actually solves the issue:

    $response = $client->request('GET', $url, [
        'extra' => [
            'curl' => [
                CURLOPT_FORBID_REUSE => true,
            ],
        ],
    ]);

I also did some more digging and noticed that the behavior changed with v6.4.12, and I guess that's because of PR #58278. Due to the changes in that PR, CURLMOPT_MAXCONNECTS is set to 4294967295 if there is any $maxHostConnections parameter (and it is 6 by default). That effectively means, there is more or less no limit on the number of connections, doesn't it? 🤔

This is the code in 6.4.12:

symfony/src/Symfony/Component/HttpClient/Internal/CurlClientState.php

Lines 52 to 57 in 5532c34

if (\defined('CURLMOPT_MAX_HOST_CONNECTIONS') && 0 < $maxHostConnections) {
$maxHostConnections = curl_multi_setopt($this->handle, \CURLMOPT_MAX_HOST_CONNECTIONS, $maxHostConnections) ? 4294967295 : $maxHostConnections;
}
if (\defined('CURLMOPT_MAXCONNECTS') && 0 < $maxHostConnections) {
curl_multi_setopt($this->handle, \CURLMOPT_MAXCONNECTS, $maxHostConnections);
}

Before #58278 CURLMOPT_MAXCONNECTS was never set when CURLMOPT_MAX_HOST_CONNECTIONS was set succesffully. This is 6.4.11:

symfony/src/Symfony/Component/HttpClient/Internal/CurlClientState.php

Lines 52 to 57 in 658b70d

if (\defined('CURLMOPT_MAX_HOST_CONNECTIONS')) {
$maxHostConnections = curl_multi_setopt($this->handle, \CURLMOPT_MAX_HOST_CONNECTIONS, 0 < $maxHostConnections ? $maxHostConnections : \PHP_INT_MAX) ? 0 : $maxHostConnections;
}
if (\defined('CURLMOPT_MAXCONNECTS') && 0 < $maxHostConnections) {
curl_multi_setopt($this->handle, \CURLMOPT_MAXCONNECTS, $maxHostConnections);
}

Removing this line after 6.4.12 solves the issue for me and there is not an unlimited number of open connections anymore:

symfony/src/Symfony/Component/HttpClient/Internal/CurlClientState.php

Line 56 in cb08480

curl_multi_setopt($this->handle, \CURLMOPT_MAXCONNECTS, $maxHostConnections);

While it's okay to increase the max number of connections, I do not think that quasi unlimited is a good default, since the errors when hitting the file descriptor limit is really unhelpful.

@joelwurtz
Copy link
Contributor

If CURLMOPT_MAXCONNECTS is not set it means no limit, so old behavior should also have your current problem.

However i do believe it should be 2 separate options for the state instead of having the first option having an effect on the second one (i don't really undersand why also : you may want 1 connection per host max, but 100 max across all differents hosts ?)

@nicolas-grekas do you know why having the CURLMOPT_MAXCONNECTS set should have an effect on the CURLMOPT_MAXCONNECTS option ?

@nicolas-grekas
Copy link
Member

One setting to rule them all I guess.
If the number of connections per host is limited, I'd expect the number of different hosts to be naturally bounded, which means the max number of open connections should be hosts x max/host
Looks like I missed something.

@digilist
Copy link
Contributor Author

If CURLMOPT_MAXCONNECTS is not set it means no limit, so old behavior should also have your current problem.

That's not the case: Previously there was a limit, because if it's not set, curl is using a default value of 4 times the number of handlers. See https://curl.se/libcurl/c/CURLMOPT_MAXCONNECTS.html:

By default libcurl enlarges the size for each added easy handle to make it fit 4 times the number of added easy handles.

I am not sure what the number of handlers (or a handler) actually is, but I think it's a lower number. curl_multi_init() returns a CurlMultiHandle and the options are set for that handle object. So maybe that's the handle the documentation is talking about? In that case the number would always be 1, which would set the default max connections to 4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug HttpClient Status: Needs Review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@joelwurtz @nicolas-grekas @digilist @carsonbot
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy