Get Started With Python’s Requests Library

Even though the Requests library is a common staple for many Python developers, it’s not included in Python’s standard library. That way, the library can continue to evolve more freely as a self-standing project.

Since Requests is a third-party library, you need to install it before using it in your code. As a good practice, you should install external packages into a virtual environment, but you may choose to install requests into your global environment if you plan to use it across multiple projects.

Whether you’re working in a virtual environment or not, you’ll need to install requests:

$ python -m pip install requests

Once pip finishes installing requests, you can use it in your application. Importing requests looks like this:

import requests

Now that you’re all set up, it’s time to begin your journey with Requests. Your first goal will be to make a GET request.

Make a GET Request

HTTP methods, such as GET and POST, specify the action you want to perform when making an HTTP request. In addition to GET and POST, there are several other common methods that you’ll use later in this tutorial.

One of the most commonly used HTTP methods is GET, which retrieves data from a specified resource. To make a GET request using Requests, you can invoke requests.get().

To try this out, you can make a GET request to GitHub’s REST API by calling get() with the following URL:

>>> import requests
>>> requests.get("https://api.github.com")
<Response [200]>

Congratulations! You’ve made your first request. Now you’ll dive a little deeper into the response of that request.

Inspect the Response

A Response is the object that contains the results of your request. Try making that same request again, but this time store the return value in a variable so you can get a closer look at its attributes and behaviors:

>>> import requests
>>> response = requests.get("https://api.github.com")

In this example, you’ve captured the return value of requests.get(). It’s an instance of Response, and you stored it in a variable called response. You can now use response to see a lot of information about the results of your GET request.

>>> response.status_code
200

if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Not Found.")

if response:
    print("Success!")
else:
    raise Exception(f"Non-success status code: {response.status_code}")

import requests
from requests.exceptions import HTTPError

URLS = ["https://api.github.com", "https://api.github.com/invalid"]

for url in URLS:
    try:
        response = requests.get(url)
        response.raise_for_status()
    except HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
    except Exception as err:
        print(f"Other error occurred: {err}")
    else:
        print("Success!")

>>> import requests

>>> response = requests.get("https://api.github.com")
>>> response.content
b'{"current_user_url":"https://api.github.com/user", ...}'

>>> type(response.content)
<class 'bytes'>

>>> response.text
'{"current_user_url":"https://api.github.com/user", ...}'

>>> type(response.text)
<class 'str'>

>>> response.encoding = "utf-8"  # Optional: Requests infers this.
>>> response.text
'{"current_user_url":"https://api.github.com/user", ...}'

>>> response.json()
{'current_user_url': 'https://api.github.com/user', ...}

>>> type(response.json())
<class 'dict'>

>>> response_dict = response.json()
>>> response_dict["emojis_url"]
'https://api.github.com/emojis'

>>> import requests

>>> response = requests.get("https://api.github.com")
>>> response.headers
{'Server': 'github.com',
...
'X-GitHub-Request-Id': 'AE83:3F40:2151C46:438A840:65C38178'}

>>> response.headers["Content-Type"]
'application/json; charset=utf-8'

>>> response.headers["content-type"]
'application/json; charset=utf-8'

Add Query String Parameters

One common way to customize a GET request is to pass values through query string parameters in the URL. To do this using get(), you pass data to params. For example, you can use GitHub’s repository search API to look for popular Python repositories:

import requests

response = requests.get(
    "https://api.github.com/search/repositories",
    params={"q": "language:python", "sort": "stars", "order": "desc"},
)

json_response = response.json()
popular_repositories = json_response["items"]
for repo in popular_repositories[:3]:
    print(f"Name: {repo['name']}")
    print(f"Description: {repo['description']}")
    print(f"Stars: {repo['stargazers_count']}\n")

By passing a dictionary to the params parameter of get(), you’re able to modify the results that come back from the search API.

You can pass params to get() either as a dictionary, as you’ve just done, or as a list of tuples:

>>> import requests

>>> requests.get(
...     "https://api.github.com/search/repositories",
...     [("q", "language:python"), ("sort", "stars"), ("order", "desc")],
... )
<Response [200]>

You can even pass the values as bytes:

>>> requests.get(
...     "https://api.github.com/search/repositories",
...     params=b"q=language:python&sort=stars&order=desc",
... )
<Response [200]>

Query strings are useful for parameterizing GET requests. Another way to customize your requests is by adding or modifying the headers that you send.

Customize Request Headers

To customize headers, you pass a dictionary of HTTP headers to get() using the headers parameter. For example, you can change your previous search request to highlight matching search terms in the results by specifying the text-match media type in the Accept header:

import requests

response = requests.get(
    "https://api.github.com/search/repositories",
    params={"q": '"real python"'},
    headers={"Accept": "application/vnd.github.text-match+json"},
)

json_response = response.json()
first_repository = json_response["items"][0]
print(first_repository["text_matches"][0]["matches"])

The Accept header tells the server what content types your application can handle. In this case, since you’re expecting the matching search terms to be highlighted, you’re using the header value application/vnd.github.text-match+json. This is a proprietary GitHub Accept header where the content is a special JSON format.

When you run this Python script, you’ll get a result similar to the one shown below:

$ python text_matches.py
[{'text': 'Real Python', 'indices': [23, 34]}]

Before you learn more ways to customize requests, you’ll broaden your horizons by exploring other HTTP methods.

Use Other HTTP Methods

Aside from GET, other popular HTTP methods include POST, PUT, DELETE, HEAD, PATCH, and OPTIONS. For each of these HTTP methods, Requests provides a function with a similar signature to get().

You’ll notice that Requests provides an intuitive interface for all the common HTTP methods:

>>> import requests

>>> requests.get("https://httpbin.org/get")
<Response [200]>
>>> requests.post("https://httpbin.org/post", data={"key": "value"})
<Response [200]>
>>> requests.put("https://httpbin.org/put", data={"key": "value"})
<Response [200]>
>>> requests.delete("https://httpbin.org/delete")
<Response [200]>
>>> requests.head("https://httpbin.org/get")
<Response [200]>
>>> requests.patch("https://httpbin.org/patch", data={"key": "value"})
<Response [200]>
>>> requests.options("https://httpbin.org/get")
<Response [200]>

In the example above, you called each function to make a request to the httpbin service using the corresponding HTTP method.

All of these functions are high-level shortcuts to requests.request(), which takes the method name as its first argument:

>>> requests.request("GET", "https://httpbin.org/get")
<Response [200]>

You could use the equivalent lower-level function call, but the power of Python’s Requests library is in its human-friendly, high-level interface. You can inspect the responses just like you did before:

>>> response = requests.head("https://httpbin.org/get")
>>> response.headers["Content-Type"]
'application/json'

>>> response = requests.delete("https://httpbin.org/delete")
>>> json_response = response.json()
>>> json_response["args"]
{}

No matter which method you use, you get a Response object that provides access to headers, response bodies, status codes, and more.

Next, you’ll take a closer look at the POST, PUT, and PATCH methods and learn how they differ from the other request types.

Send Request Data

According to the HTTP specification, POST, PUT, and the less common PATCH requests pass their data through the message body rather than through parameters in the query string. With Requests, you pass this payload to the corresponding function’s data parameter.

The data parameter takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data you send in the body of your request to the specific needs of the service you’re interacting with.

For example, if your request’s content type is application/x-www-form-urlencoded, then you can send the form data as a dictionary:

>>> import requests

>>> requests.post("https://httpbin.org/post", data={"key": "value"})
<Response [200]>

You can also send that same data as a list of tuples:

>>> requests.post("https://httpbin.org/post", data=[("key", "value")])
<Response [200]>

If you need to send JSON data, then you can use the json parameter. When you pass JSON data via json, Requests will serialize your data and add the correct Content-Type header for you.

As you learned earlier, the httpbin service accepts test requests and responds with data about the requests. For example, you can use it to inspect a basic POST request:

>>> response = requests.post("https://httpbin.org/post", json={"key": "value"})
>>> json_response = response.json()
>>> json_response["data"]
'{"key": "value"}'
>>> json_response["headers"]["Content-Type"]
'application/json'

You can see from the response that the server received your request data and headers just as you sent them. Requests also provides this information to you in the form of a PreparedRequest, which you’ll look at more closely in the next section.

Inspect the Prepared Request

When you make a request, the Requests library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.

You can view the PreparedRequest object by accessing .request on a Response object:

>>> import requests

>>> response = requests.post("https://httpbin.org/post", json={"key":"value"})

>>> response.request
<PreparedRequest [POST]>

>>> response.request.headers["Content-Type"]
'application/json'

>>> response.request.url
'https://httpbin.org/post'

>>> response.request.body
b'{"key": "value"}'

Inspecting PreparedRequest gives you access to all kinds of information about the request being made, such as payload, URL, headers, authentication, and more.

So far, you’ve made a lot of different kinds of requests, but they’ve all had one thing in common: they were unauthenticated requests to public APIs. Many services you’ll come across will want you to authenticate in some way.

Use Authentication

Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the Authorization header or a custom header defined by the service. All the functions in Requests that you’ve seen to this point provide a parameter called auth, which allows you to pass your credentials directly:

>>> import requests

>>> response = requests.get(
...     "https://httpbin.org/basic-auth/user/passwd",
...     auth=("user", "passwd")
... )

>>> response.status_code
200
>>> response.request.headers["Authorization"]
'Basic dXNlcjpwYXNzd2Q='

The request succeeds if the credentials that you pass in the tuple to auth are valid.

When you pass your credentials in a tuple to the auth parameter, Requests applies the credentials using HTTP’s basic access authentication scheme under the hood.

You could make the same request by passing explicit basic authentication credentials using HTTPBasicAuth:

>>> from requests.auth import HTTPBasicAuth
>>> requests.get(
...     "https://httpbin.org/basic-auth/user/passwd",
...     auth=HTTPBasicAuth("user", "passwd")
... )
<Response [200]>

Although you don’t need to be explicit for basic authentication, you might want to authenticate using another method. Requests provides other methods of authentication out of the box, such as HTTPDigestAuth and HTTPProxyAuth.

A real-world example of an API that requires authentication is GitHub’s authenticated user API. This endpoint provides information about the authenticated user’s profile.

If you try to make a request without credentials, then you’ll see that the status code is 401 Unauthorized:

>>> requests.get("https://api.github.com/user")
<Response [401]>

If you don’t provide authentication credentials when accessing a service that requires them, then you’ll get an HTTP error code as a response.

To make a request to GitHub’s authenticated user API, you first need to generate a personal access token with the read:user scope. Then, you can pass this token as the second element in a tuple to get():

>>> import requests

>>> token = "<YOUR_GITHUB_PA_TOKEN>"
>>> response = requests.get(
...     "https://api.github.com/user",
...     auth=("", token)
... )
>>> response.status_code
200

As you learned earlier, this approach passes the credentials to HTTPBasicAuth, which expects a username and a password, and sends the credentials as a Base64-encoded string with the prefix "Basic ":

>>> response.request.headers["Authorization"]
'Basic OmdocF92dkd...WpremM0SGRuUGY='

This method works, but it’s not the right way to authenticate with a Bearer token—and using an empty string input for the superfluous username is awkward.

With Requests, you can supply your own authentication mechanism to fix that. To try this out, create a subclass of AuthBase and implement .__call__():

from requests.auth import AuthBase

class TokenAuth(AuthBase):
    """Implements a token authentication scheme."""

    def __init__(self, token):
        self.token = token

    def __call__(self, request):
        """Attach an API token to the Authorization header."""
        request.headers["Authorization"] = f"Bearer {self.token}"
        return request

Here, your custom TokenAuth mechanism receives a token, then includes that token in the Authorization header of your request, also setting the recommended "Bearer " prefix to the string.

You can now use this custom token authentication to make your call to GitHub’s authenticated user API:

>>> import requests
>>> from custom_token_auth import TokenAuth

>>> token = "<YOUR_GITHUB_PA_TOKEN>"
>>> response = requests.get(
...     "https://api.github.com/user",
...     auth=TokenAuth(token)
... )

>>> response.status_code
200
>>> response.request.headers["Authorization"]
'Bearer ghp_b...Tx'

Your custom TokenAuth created a well-formatted string for the Authorization header. This gives you a more intuitive and reusable way to work with token-based authentication schemes, like those required by parts of GitHub’s API.

Poor authentication mechanisms can lead to security vulnerabilities. Unless a service requires a custom authentication mechanism for some reason, it’s best to stick with a tried-and-true method, like built-in basic authentication or OAuth—for example, via Requests-OAuthlib.

While you’re thinking about security, consider dealing with TLS/SSL certificates with Requests.

Communicate Securely With Servers

Whenever the data you’re trying to send or receive is sensitive, security becomes essential. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using Transport Layer Security (TLS). TLS is the successor to Secure Sockets Layer (SSL), offering enhanced security and efficiency in secure communications. Yet, it’s still common for programmers to use the term SSL instead of TLS.

Requests verifies the digital certificate of servers for you by default, and you rarely need to make adjustments to this behavior. However, there are some cases where you might need to customize the verification process.

For example, when you’re working in a corporate environment with custom certificate authorities, you may need to provide your own certificate bundle:

>>> import requests

>>> requests.get(
...     "https://internal-api.company.com",
...     verify="/path/to/company-ca.pem"
... )
<Response [200]>

You can also provide a directory containing certificate files if you need to trust multiple custom authorities.

If you’re debugging certificate issues in development, then you might be tempted to disable verification entirely. While this works, it’s a significant security risk and you should never use this approach in production:

>>> requests.get("https://api.github.com", verify=False)
InsecureRequestWarning: Unverified HTTPS request is being made to host
⮑ 'api.github.com'. Adding certificate verification is strongly advised.
⮑ See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
⮑  warnings.warn(
<Response [200]>

Requests warns you about this dangerous practice because disabling verification makes you vulnerable to man-in-the-middle attacks.

Now that you know how to handle certificate verification securely, you might be wondering how to keep your program running as quickly as possible.

Improve Performance

When using Requests, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.

Even in synchronous code, there are several important techniques you can use to optimize your HTTP requests and prevent performance bottlenecks.

>>> requests.get("https://api.github.com", timeout=1)
<Response [200]>

>>> requests.get("https://api.github.com", timeout=0.01)
Traceback (most recent call last):
  ...
requests.exceptions.ConnectTimeout:
⮑ HTTPSConnectionPool(host='api.github.com', port=443):
⮑ Max retries exceeded with url: / (Caused by ConnectTimeoutError(...))

>>> requests.get("https://api.github.com", timeout=(3.05, 5))
<Response [200]>

import requests
from requests.exceptions import Timeout

try:
    response = requests.get("https://api.github.com", timeout=(3.05, 5))
except Timeout:
    print("The request timed out")
else:
    print("The request did not time out")

 1import requests
 2from custom_token_auth import TokenAuth
 3
 4TOKEN = "<YOUR_GITHUB_PA_TOKEN>"
 5
 6with requests.Session() as session:
 7    session.auth = TokenAuth(TOKEN)
 8
 9    first_response = session.get("https://api.github.com/user")
10    second_response = session.get("https://api.github.com/user")
11
12print(first_response.headers)
13print(second_response.json())

import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import RetryError
from urllib3.util.retry import Retry

retry_strategy = Retry(
    total=2,
    status_forcelist=[429, 500, 502, 503, 504]
)
github_adapter = HTTPAdapter(max_retries=retry_strategy)

with requests.Session() as session:
    session.mount("https://api.github.com", github_adapter)
    try:
        response = session.get("https://api.github.com/")
    except RetryError as err:
        print(f"Error: {err}")

Conclusion

Nice work! You’ve made it to the end of the tutorial and come a long way in expanding your knowledge of Python’s powerful Requests library.

In this tutorial, you’ve learned how to:

• Make requests using a variety of HTTP methods such as GET, POST, and PUT
• Customize your requests by modifying headers, authentication, query strings, and message bodies
• Inspect the data you send to the server and the data the server sends back to you
• Work with TLS/SSL certificate verification
• Use Requests effectively with max_retries, timeout, sessions, and transport adapters

Because you learned how to use Requests, you’re equipped to explore the wide world of web services—and build awesome applications using the fascinating data they provide.

Frequently Asked Questions

Now that you have some experience with Python’s Requests library, you can use the questions and answers below to check your understanding and recap what you’ve learned.

These FAQs are related to the most important concepts you’ve covered in this tutorial. Click the Show/Hide toggle beside each question to reveal the answer.

Take the Quiz: Test your knowledge with our interactive “Python's Requests Library” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Python's Requests Library

Test your understanding of the Python Requests library for making HTTP requests and interacting with web services.