requests#

What it is#

requests is a synchronous HTTP client library created by Kenneth Reitz in 2011 and now maintained under the Python Software Foundation umbrella. It wraps urllib3 with a high-level, human-friendly API and has consistently ranked as the single most-downloaded package on PyPI for over a decade.

Reach for requests when you need a stable, blocking HTTP client with broad ecosystem support — almost every Python SDK, scraper, and integration tutorial assumes it. Reach for httpx instead when you need async/await or HTTP/2.

Install#

pip install requests

Output: (none — exits 0 on success)

uv add requests

Output: dependency resolved + added to pyproject.toml

poetry add requests

Output: updated lockfile + virtualenv install

pip install "requests[security]"     # legacy extra — see notes

Output: installs requests plus optional security backports (no-op on modern Python)

Versioning & Python support#

• Current stable line is the 2.x series (as of mid-2026). The project has stayed on 2.x since 2014 — major-version bumps are rare and treated as breaking.
• Supports Python 3.8+ on recent releases. Older releases support 3.7 and earlier; check the changelog for the exact floor.
• Loose semver — patch releases are bug fixes, minor releases may deprecate but rarely break import-level API.
• Python 2.7 support was dropped in 2.28 (2022).

Package metadata#

• Maintainer: Python Software Foundation / psf org on GitHub
• Project home: github.com/psf/requests
• Docs: requests.readthedocs.io
• PyPI: pypi.org/project/requests
• License: Apache-2.0
• Governance: PSF-stewarded, community-maintained
• First released: 2011
• Downloads: in the hundreds of millions per month — consistently #1 on PyPI

Optional dependencies & extras#

• requests[security] — historically pulled in pyOpenSSL, cryptography, idna. Now mostly a no-op on Python 3.x because the stdlib ssl module is sufficient. Kept for backward compatibility.
• requests[socks] — installs PySocks for SOCKS4/SOCKS5 proxy support (socks5://user:pass@host:port).
• requests[use_chardet_on_py3] — installs chardet for response-encoding detection instead of the default charset-normalizer.

Core dependencies pulled in automatically:

• urllib3 — connection pooling, retries, TLS
• certifi — bundled CA certificate store
• charset-normalizer — response encoding detection
• idna — internationalized domain name handling

Alternatives#

Common gotchas#

1. Synchronous only. No async/await support — calls block the event loop. In FastAPI / asyncio contexts, use httpx.AsyncClient instead.
2. certifi CA bundle, not system trust store. TLS verification uses certifi’s bundled CAs. If you need OS-managed certs (corporate CA, mitmproxy), install pip-system-certs or set REQUESTS_CA_BUNDLE.
3. Vendored urllib3 history. Old releases vendored urllib3 under requests.packages.urllib3; modern releases import the top-level one. Avoid the legacy import path — it’s a compatibility shim, not the real module.
4. Connection-pool default is 10 per host. For high-concurrency workers, mount a custom HTTPAdapter(pool_connections=N, pool_maxsize=N) on a Session.
5. Session is not safe to share across threads for all operations. Reading is generally fine, but mount(), header mutation, and cookie jar updates during requests can race. Prefer one session per thread.
6. No retry by default. Failed requests raise immediately. Wire urllib3.util.Retry into an HTTPAdapter to get exponential backoff.
7. Implicit JSON encoding charset is UTF-8. Posting bytes that aren’t UTF-8 via json= will silently re-encode — pass data= with an explicit Content-Type header instead.

Real-world recipes#

Concrete, copy-paste solutions to problems that come up the first week any team puts requests into production. Each focuses on package-level concerns (pool tuning, adapter mounts, transport-level configuration) rather than the day-to-day API patterns covered in the companion article.

Recipe 1 — Production-grade session with pool sizing, retries, and timeouts wired together.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

def make_session(pool_size: int = 50) -> requests.Session:
    s = requests.Session()
    retry = Retry(
        total=5,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset(["GET", "HEAD", "PUT", "DELETE", "OPTIONS", "TRACE", "POST"]),
        respect_retry_after_header=True,
    )
    adapter = HTTPAdapter(
        pool_connections=pool_size,
        pool_maxsize=pool_size,
        max_retries=retry,
        pool_block=False,
    )
    s.mount("http://", adapter)
    s.mount("https://", adapter)
    s.headers.update({"User-Agent": "myapp/1.0 (+contact: ops@example.com)"})
    return s

Output: session ready for use; per-request timeout=(connect, read) still required.

Recipe 2 — Paginated cursor-based API client iterator.

def iter_pages(session, url, params=None):
    params = dict(params or {})
    while url:
        r = session.get(url, params=params, timeout=(5, 30))
        r.raise_for_status()
        body = r.json()
        yield from body["items"]
        url = body.get("next_url")
        params = None  # cursor embedded in next_url

Output: lazy generator; one HTTP request per page, no extra memory beyond a page.

Recipe 3 — File upload with progress reporting via MultipartEncoderMonitor.

# pip install requests-toolbelt
from requests_toolbelt.multipart.encoder import MultipartEncoder, MultipartEncoderMonitor

def upload(session, url, path):
    encoder = MultipartEncoder(fields={"file": (path, open(path, "rb"), "application/octet-stream")})
    def callback(monitor):
        pct = monitor.bytes_read * 100 // monitor.len
        print(f"\r{pct}%", end="", flush=True)
    monitor = MultipartEncoderMonitor(encoder, callback)
    r = session.post(url, data=monitor, headers={"Content-Type": monitor.content_type}, timeout=(5, 600))
    r.raise_for_status()
    return r.json()

Output: progress percentages stream to stdout; final JSON response returned.

Recipe 4 — Mounting a per-host adapter for one slow upstream.

s = make_session()
slow_adapter = HTTPAdapter(
    pool_connections=10, pool_maxsize=10,
    max_retries=Retry(total=2, backoff_factor=2.0),
)
s.mount("https://slow-api.example.com/", slow_adapter)

Output: other hosts keep the fast default; only slow-api.example.com uses the patient adapter.

Security considerations#

requests ships secure defaults, but each is overridable — and the override often ends up in production by accident. Treat every TLS, redirect, and authentication setting as load-bearing.

• Never set verify=False in production. It silently disables certificate validation and prints a InsecureRequestWarning that gets filtered out of structured logs. If you need a private CA, set verify="/path/to/ca-bundle.pem" or the REQUESTS_CA_BUNDLE env var.
• certifi is the cert store, not the OS trust store. Corporate MITM proxies and self-signed certs require either bundling them into a custom PEM or installing pip-system-certs to redirect lookups to the OS keychain.
• Redirects follow by default and forward auth headers to same-host targets only. A redirect to a different host strips Authorization and Cookie headers since 2.x, but cross-protocol downgrades (HTTPS → HTTP) still happen unless you wire a redirect_hook or set allow_redirects=False.
• CRLF injection through header values. Pre-2.32 releases were vulnerable to header smuggling via newlines in user-supplied header values; modern releases reject them. Pin to a recent patch level — see GitHub security advisories.
• Proxy URL leaks credentials in logs. If your proxy URL embeds user:pass@host, the password ends up in req.proxies and any logger that pickles the request prints it. Strip to a ProxyManager or load via env var.
• charset-normalizer runs untrusted input through a heuristic. For very large responses, prefer setting response.encoding explicitly rather than letting it sniff.
• Pin urllib3. requests does not cap upstream urllib3 tightly; a major bump in urllib3 (the 1.x → 2.x switch in 2023 broke many downstream users) can change retry semantics and proxy handling.

Testing & CI integration#

The library is sync, so test doubles are simple — the choice is whether to mock at the Session level, the HTTP-call level, or via a recorded VCR cassette.

# pytest + responses (recommended for unit tests)
# pip install responses
import responses, requests

@responses.activate
def test_fetch_user():
    responses.add(responses.GET, "https://api.example.com/users/42",
                  json={"id": 42, "name": "Alice Dev"}, status=200)
    r = requests.get("https://api.example.com/users/42", timeout=5)
    assert r.json()["name"] == "Alice Dev"
    assert len(responses.calls) == 1

Output: test passes; HTTP traffic intercepted, no socket opened.

# requests-mock — pytest fixture style
# pip install requests-mock
def test_with_fixture(requests_mock):
    requests_mock.get("https://api.example.com/health", text="ok")
    assert requests.get("https://api.example.com/health", timeout=5).text == "ok"

Output: the requests_mock fixture is provided by the package — no decorator needed.

# vcrpy — record real traffic once, replay forever
# pip install vcrpy
import vcr

@vcr.use_cassette("cassettes/users.yaml")
def test_real_user():
    r = requests.get("https://api.example.com/users/42", timeout=5)
    assert r.status_code == 200

Output: first run records to YAML; subsequent runs replay offline. Useful for end-to-end style integration tests.

CI guidance: pin urllib3 and certifi in requirements.txt (not just requests) — a fresh resolver run can pull in a new urllib3 major that breaks tests with subtle retry-behaviour differences.

Troubleshooting common errors#

Performance tuning#

requests is sync, so its perf knobs are about avoiding waste — pool reuse, connection keep-alive, response streaming. The library can’t compete with httpx (HTTP/2) or aiohttp (async) on pure throughput; tune for predictable latency instead.

• One Session per process, mounted with a sized adapter. A new Session() per call pays full TLS handshake (~50-300 ms on first request) and skips keep-alive.
• pool_connections × pool_maxsize = total open sockets. Both should match your worker concurrency. The defaults (10 / 10) bottleneck multi-threaded workers fast.
• stream=True for responses > a few MB. Default reads the entire body into memory; iter_content(chunk_size=8192) lets you process incrementally.
• Connection: keep-alive is implicit. As long as you reuse a Session, subsequent requests to the same host reuse the TCP+TLS state. The host must send Connection: keep-alive (or omit Connection: close) for this to work.
• requests-cache — drop-in middleware that caches responses to SQLite / Redis. Massive win for read-heavy clients hitting idempotent endpoints.
• niquests — a drop-in fork with HTTP/2, HTTP/3, and async. Use when you’ve outgrown requests perf but don’t want to rewrite to httpx.
• Compression. Accept-Encoding: gzip, deflate is auto-added; requests decompresses transparently. Add brotli (via pip install brotli) for br-aware servers.
• Avoid json.loads(response.text). Use response.json() which decodes from bytes — one fewer copy and uses the cached charset.

Version migration guide#

requests has been on 2.x since 2014. The library is stable enough that “migration” usually means a transitive-dep bump (especially urllib3) rather than requests itself. The list below is the gotcha list for users moving across recent minor releases.

• 2.28 — dropped Python 2.7. Code targeting Python 2 needs a hard fork or replacement.
• 2.30 — switched the default character-detection backend from chardet to charset-normalizer. The [use_chardet_on_py3] extra restores the old behavior. Most code is unaffected.
• 2.31 — security fix for Proxy-Authorization header forwarding on redirects. Pin to >=2.31 if you proxy auth.
• 2.32 — security fix for CRLF injection in user-supplied headers. The library now rejects newlines in header values; code that built headers with f"value\r\n..." (always a bug) now raises.
• urllib3 1.x → 2.x (2023) — requests works with both, but the urllib3 major bump changed retry-after-header parsing, default certificate validation, and the pool_connections semantics slightly. Re-run integration tests when crossing this boundary.

The often-discussed requests 3.0 plan has been on the roadmap for years; treat the 2.x series as the production line indefinitely.

# Defensive pinning for production
pip install 'requests>=2.32,<3' 'urllib3>=2.0,<3' 'certifi>=2024.2'

Output: ranges that pick up security patches without surprise major bumps.

Ecosystem integrations#

The requests ecosystem is so broad that almost any Python SDK either uses it or copies its API shape. Below are the integrations most teams reach for:

• requests-toolbelt — official extension pack: streaming uploads, multipart progress, source-address binding, advanced sessions.
• requests-cache — transparent caching to SQLite, Redis, MongoDB; massive win for read-heavy clients.
• requests-mock / responses — test doubles; responses integrates with pytest, requests-mock provides a fixture.
• vcrpy — record real HTTP traffic once, replay forever; good for integration-style tests.
• requests-oauthlib — OAuth1 and OAuth2 client flows.
• requests-html — adds HTML/CSS-selector parsing on top of requests (long-tail maintained).
• niquests — drop-in API-compatible fork with HTTP/2, HTTP/3, and async.
• pip-system-certs — redirects requests/urllib3 to the OS trust store (useful behind corporate proxies).
• opentelemetry-instrumentation-requests — auto-tracing for OTel exporters.
• sentry-sdk — automatic request breadcrumbs and error capture.

When NOT to use this#

requests is the right default, but a handful of situations call for something else:

• Inside async def handlers (FastAPI, Litestar, asyncio jobs) — requests blocks the event loop. Use httpx.AsyncClient instead.
• HTTP/2 or HTTP/3 required — requests is HTTP/1.1 only. Use httpx[http2] or niquests.
• Tens of thousands of concurrent outbound calls — sync model can’t keep up. Switch to aiohttp or httpx.
• WebSockets — requests has no WS support. Use websockets or websocket-client.
• Streaming uploads with progress — usable, but requests-toolbelt’s MultipartEncoder is the practical path (recipe above).
• Tight memory or zero-dep constraints — urllib (stdlib) is uglier but ships with Python.

Compatibility matrix#

Critical dependency pairs:

• urllib3 1.x works with all requests 2.x; some retry semantics differ at the boundary.
• urllib3 2.x requires requests >= 2.30 and OpenSSL 1.1.1+ at runtime.
• certifi is unversioned in requests deps — a stale certifi cache silently misses new CA roots. Refresh annually.
• charset-normalizer >= 3 shipped a major rewrite with subtle detection differences; pin minimums if you depend on exact charset-detection results.

See also#

• Python: requests — API tutorial, examples, recipes
• Concept: HTTP — protocol fundamentals
• Packages: pip-httpx — the async-capable alternative
• Linux: curl — the upstream CLI inspiration