tqdm — Progress Bars#

What it is#

tqdm wraps any Python iterable and displays an auto-updating progress bar with ETA, rate, and elapsed time in the terminal or Jupyter notebook. It adds less than 60 ns of overhead per iteration and requires no configuration beyond installation — just wrap an iterable and the bar appears. It is the de-facto standard for loop progress in Python data pipelines.

Install#

pip install tqdm

Output: (none — exits 0 on success)

Quick example#

from tqdm import tqdm
import time

for _ in tqdm(range(100)):
    time.sleep(0.02)

Output:

100%|████████████████████████████| 100/100 [00:02<00:00, 49.8it/s]

When / why to use it#

• Long-running loops over files, API pages, dataset rows, or model batches where ETA matters.
• CLI tools where a bar communicates liveness to the user.
• Data pipelines with pandas (.progress_apply) or PyTorch data loaders.
• Notebooks where it renders as an HTML widget.
• Shell pipelines where it acts as a byte-throughput meter (like pv).

Common pitfalls#

[!WARNING] Wrapping a generator with unknown length — if total= is omitted and tqdm cannot infer the length from len(), the ETA shows ? and the percentage is hidden. Always pass total=len(items) when the iterable is a generator.

[!WARNING] Nested bars without position= — each inner bar overwrites the outer bar on the same line. Use position=0 on the outer bar and position=1, leave=False on the inner bar.

[!WARNING] print() inside a tqdm loop — ordinary print interleaves with bar output and creates visual noise. Replace with tqdm.write("msg"), which prints above the bar without disturbing it.

[!TIP] tqdm(iterable, desc="Loading") is the most readable one-liner. The desc= prefix appears left of the bar and doubles as a log label when redirected.

[!TIP] from tqdm.auto import tqdm auto-selects the notebook HTML widget when running in Jupyter and falls back to the terminal bar everywhere else — the cleanest single import for code that runs in both environments.

Richer example — file hashing pipeline#

from tqdm import tqdm
import pathlib, hashlib

files = list(pathlib.Path(".").glob("**/*.py"))

results = {}
with tqdm(files, desc="Hashing", unit="file", colour="green") as pbar:
    for path in pbar:
        pbar.set_postfix(file=path.name, refresh=False)
        results[str(path)] = hashlib.md5(path.read_bytes()).hexdigest()

print(f"Hashed {len(results)} files")

Output:

Hashing: 100%|██████████| 42/42 [00:00<00:00, 312.4file/s, file=utils.py]
Hashed 42 files

trange — range shorthand#

trange(n) is identical to tqdm(range(n)) and is the idiomatic way to wrap a counter loop. It accepts all the same keyword arguments as tqdm.

from tqdm import trange
import time

total = 0
for i in trange(50, desc="Summing", unit="step"):
    total += i
    time.sleep(0.01)

print(f"Total: {total}")

Output:

Summing: 100%|████████████████████| 50/50 [00:00<00:00, 87.3step/s]
Total: 1225

Manual updates with update() and set_postfix()#

When the loop body controls progress (streaming downloads, chunked reads, batch training), open tqdm as a context manager with total= and call pbar.update(n) to advance by n units. set_postfix attaches live key-value metadata to the right of the bar without interrupting the display.

from tqdm import tqdm
import time

with tqdm(total=1000, desc="Download", unit="KB") as pbar:
    downloaded = 0
    while downloaded < 1000:
        chunk = 64
        time.sleep(0.005)
        downloaded += chunk
        pbar.update(chunk)
        pbar.set_postfix(speed="12800 KB/s", refresh=False)

Output:

Download: 100%|█████████| 1000/1000 [00:00<00:00, 12.8KB/s, speed=12800 KB/s]

pandas integration — progress_apply#

tqdm patches pandas Series, DataFrame, and GroupBy objects with a progress_apply method. Call tqdm.pandas() once at module level to activate the patch; then replace .apply() with .progress_apply() throughout.

import pandas as pd
from tqdm import tqdm

tqdm.pandas(desc="Transforming")

df = pd.DataFrame({"value": range(200)})
df["doubled"] = df["value"].progress_apply(lambda x: x * 2)
print(df.tail(3))

Output:

Transforming: 100%|████████| 200/200 [00:00<00:00, 4123.7it/s]
   value  doubled
197   197      394
198   198      396
199   199      398

For groupby, use progress_apply on the grouped object the same way:

df = pd.DataFrame({"group": ["a", "b"] * 100, "val": range(200)})
result = df.groupby("group")["val"].progress_apply(sum)
print(result)

Output:

Transforming: 100%|████| 2/2 [00:00<00:00, 312.4it/s]
group
a    9900
b    10100
Name: val, dtype: int64

Nested bars#

Use position= (0-indexed from the bottom of the terminal block) and leave=False on inner bars so they erase themselves when complete. The outermost bar uses position=0 (default) and leave=True (default).

from tqdm import tqdm
import time

epochs = 3
batches = 5

for epoch in tqdm(range(epochs), desc="Epoch", position=0):
    for batch in tqdm(range(batches), desc="  Batch", position=1, leave=False):
        time.sleep(0.04)

Output (mid-run):

Epoch:  67%|██████████████       | 2/3 [00:01<00:00,  1.39it/s]
  Batch:  60%|████████████       | 3/5 [00:00<00:00,  9.87it/s]

Async — asyncio and concurrent.futures#

For asyncio, use tqdm.asyncio.tqdm which provides gather() and as_completed() drop-ins that track coroutine completion. For concurrent.futures, wrap the iterator returned by pool.map with a standard tqdm.

import asyncio
from tqdm.asyncio import tqdm as atqdm

async def fetch(i):
    await asyncio.sleep(0.05)
    return i * i

async def main():
    tasks = [fetch(i) for i in range(20)]
    results = await atqdm.gather(*tasks, desc="Fetching")
    print(results[:5])

asyncio.run(main())

Output:

Fetching: 100%|████████████████████| 20/20 [00:05<00:00,  3.94it/s]
[0, 1, 4, 9, 16]

from concurrent.futures import ThreadPoolExecutor
from tqdm import tqdm
import time

def work(x):
    time.sleep(0.02)
    return x ** 2

with ThreadPoolExecutor(max_workers=4) as pool:
    results = list(tqdm(pool.map(work, range(20)), total=20, desc="Threads"))
print(results[:5])

Output:

Threads: 100%|████████████████████| 20/20 [00:00<00:00, 52.3it/s]
[0, 1, 4, 9, 16]

Jupyter / notebook mode#

In Jupyter, from tqdm.notebook import tqdm renders an HTML progress widget with colour gradients and smooth updates. The API is identical to the terminal version. from tqdm.auto import tqdm is the recommended import — it picks notebook mode automatically.

from tqdm.auto import tqdm   # widget in Jupyter, terminal bar elsewhere
import time

for _ in tqdm(range(50), desc="Training epoch"):
    time.sleep(0.02)

Custom format string and bar characters#

bar_format= controls every token in the rendered string. ascii= replaces the default Unicode block characters with a custom set of ASCII fill characters (lowest to highest density).

from tqdm import tqdm
import time

fmt = "{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}]"

for _ in tqdm(range(60), bar_format=fmt, ascii="░▒█", desc="Custom"):
    time.sleep(0.02)

Output:

Custom:  85%|░░░░░░░░░░░░░░░░░░░░░░░░░▒█   | 51/60 [00:01<00:00, 42.1it/s]

Available format tokens: {l_bar}, {bar}, {r_bar}, {n}, {n_fmt}, {total}, {total_fmt}, {percentage}, {elapsed}, {elapsed_s}, {remaining}, {remaining_s}, {rate}, {rate_fmt}, {rate_noinv}, {rate_noinv_fmt}, {postfix}, {desc}, {unit}.

Dynamic description and colour#

from tqdm import tqdm
import time

stages = ["Loading", "Preprocessing", "Fitting", "Evaluating"]
with tqdm(total=len(stages), colour="cyan") as pbar:
    for stage in stages:
        pbar.set_description(stage)
        time.sleep(0.3)
        pbar.update(1)

Output:

Evaluating: 100%|███████████████████████| 4/4 [00:01<00:00,  3.33it/s]

CLI piping — byte-count filter#

Invoked as python -m tqdm, tqdm reads stdin, forwards it to stdout, and displays throughput. It is a portable drop-in for pv on systems where pv is unavailable.

cat large_file.bin | python -m tqdm --bytes > /dev/null

Output:

 512MB [00:04, 121MB/s]

Count lines instead of bytes:

cat records.jsonl | python -m tqdm --unit=line --unit-scale > output.jsonl

Output: (none — exits 0 on success)

Disabling bars for non-interactive contexts#

import sys
from tqdm import tqdm

verbose = True  # set via CLI arg or environment variable

# Explicit disable flag
for item in tqdm(range(100), disable=not verbose):
    pass

# Auto-detect TTY — suppresses bar in CI, cron, piped output
for item in tqdm(range(100), disable=not sys.stdout.isatty()):
    pass

Real-world recipes#

A set of self-contained snippets that show how tqdm composes with the most common parallelism, networking, and ML primitives.

1. Batched API scraper with retries#

import httpx
from tqdm import tqdm

def fetch(url: str) -> dict:
    for attempt in range(3):
        try:
            return httpx.get(url, timeout=5).json()
        except httpx.RequestError:
            if attempt == 2:
                raise

urls = [f"https://api.example.com/items/{i}" for i in range(200)]
results = {}
with tqdm(urls, desc="Scraping", unit="page", smoothing=0.3) as bar:
    for url in bar:
        bar.set_postfix(url=url.split("/")[-1], refresh=False)
        try:
            results[url] = fetch(url)
        except Exception as e:
            tqdm.write(f"FAIL {url}: {e}")
print(f"Got {len(results)} / {len(urls)} pages")

Output:

Scraping: 100%|██████████| 200/200 [00:14<00:00, 13.7page/s, url=199]
Got 197 / 200 pages

2. ML training loop with nested bars and live metrics#

import random, time
from tqdm import tqdm

epochs = 4
batches = 12

with tqdm(total=epochs, desc="Epoch", position=0) as ep_bar:
    for epoch in range(epochs):
        loss, acc = 1.0, 0.0
        with tqdm(total=batches, desc="  Batch", position=1, leave=False) as bt_bar:
            for batch in range(batches):
                time.sleep(0.04)
                loss = loss * (0.95 + random.random() * 0.03)
                acc = min(1.0, acc + 0.04 + random.random() * 0.02)
                bt_bar.set_postfix(loss=f"{loss:.3f}", acc=f"{acc:.2%}", refresh=False)
                bt_bar.update(1)
        ep_bar.set_postfix(loss=f"{loss:.3f}", acc=f"{acc:.2%}")
        ep_bar.update(1)

Output (mid-run):

Epoch:  50%|██████      | 2/4 [00:01<00:01, loss=0.612, acc=68.40%]
  Batch:  67%|████      | 8/12 [00:00<00:00, loss=0.617, acc=64.20%]

3. Multiprocessing pool with shared progress#

from multiprocessing import Pool
from tqdm import tqdm
import time

def work(n: int) -> int:
    time.sleep(0.05)
    return n * n

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        # imap_unordered yields as workers finish — bar advances ASAP
        results = list(tqdm(
            pool.imap_unordered(work, range(80)),
            total=80,
            desc="Pool",
            unit="task",
        ))
    print(sum(results))

Output:

Pool: 100%|██████████| 80/80 [00:01<00:00, 65.4task/s]
167960

4. asyncio concurrency-bounded fetch#

import asyncio
from tqdm.asyncio import tqdm as atqdm

semaphore = asyncio.Semaphore(8)

async def fetch(i: int) -> int:
    async with semaphore:
        await asyncio.sleep(0.1)
        return i * 2

async def main():
    coros = [fetch(i) for i in range(40)]
    out = await atqdm.gather(*coros, desc="Fetching", unit="req")
    print(sum(out))

asyncio.run(main())

Output:

Fetching: 100%|██████████| 40/40 [00:00<00:00, 78.1req/s]
1560

5. Streaming download with byte progress#

import httpx, pathlib
from tqdm import tqdm

url = "https://example.com/big.iso"
dst = pathlib.Path("big.iso")

with httpx.stream("GET", url) as r:
    total = int(r.headers.get("Content-Length", 0))
    with open(dst, "wb") as f, tqdm(
        total=total, unit="B", unit_scale=True, unit_divisor=1024, desc=dst.name
    ) as bar:
        for chunk in r.iter_bytes(chunk_size=1024 * 64):
            f.write(chunk)
            bar.update(len(chunk))

Output:

big.iso: 100%|██████████| 512M/512M [00:18<00:00, 28.4MB/s]

6. joblib + tqdm — patched parallel callback#

from joblib import Parallel, delayed
from tqdm import tqdm
import time

def work(x):
    time.sleep(0.02)
    return x * x

# Wrap the input iterator; tqdm advances as each task is dispatched
inputs = list(range(60))
results = Parallel(n_jobs=4)(delayed(work)(i) for i in tqdm(inputs, desc="joblib"))
print(sum(results))

Output:

joblib: 100%|██████████| 60/60 [00:00<00:00, 1842it/s]
70210

7. logging interleave — keep the bar at the bottom#

import logging
from tqdm import tqdm
from tqdm.contrib.logging import logging_redirect_tqdm

log = logging.getLogger("etl")
logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")

with logging_redirect_tqdm():
    for i in tqdm(range(50), desc="Loading"):
        if i % 10 == 0:
            log.info(f"checkpoint at {i}")

Output:

INFO | checkpoint at 0
INFO | checkpoint at 10
Loading:  40%|████      | 20/50 [00:00<00:00, 50.2it/s]

Performance tuning#

tqdm adds ~60 ns per iteration in the fast path, but unnecessary refreshes can dominate a tight loop. The right knobs depend on iteration speed.

from tqdm import tqdm
import time

# Slow loop (>1 it/s) — defaults are fine
for _ in tqdm(range(50), desc="Default"):
    time.sleep(0.05)

# Fast loop (>10k it/s) — raise miniters to skip per-tick refreshes
for _ in tqdm(range(10_000_000), desc="Fast", miniters=10_000, mininterval=0.2):
    pass

# Streaming-ish loop — smoothing controls ETA averaging window
for _ in tqdm(range(200), desc="Variable", smoothing=0.05):
    time.sleep(0.01)

# Adaptive terminal width — set if columns change mid-run (e.g. in tmux)
for _ in tqdm(range(50), desc="Resize", dynamic_ncols=True):
    time.sleep(0.05)

Rules of thumb:

• For ML training (1–100 it/s): defaults are fine.
• For tight CPU loops (>10k it/s): miniters=1000+, mininterval=0.5.
• For variable-rate downloads: smoothing=0.1 (lower = more responsive ETA).
• For nested terminal pagers (tmux, screen): dynamic_ncols=True.

Troubleshooting common errors#

Integration patterns#

Beyond the recipes above, a few cross-cutting patterns repeatedly come up.

Periodic checkpoint with set_postfix:

from tqdm import tqdm

best = float("inf")
with tqdm(range(100), desc="Search") as bar:
    for i in bar:
        score = (i - 73) ** 2 + (i % 7)
        if score < best:
            best = score
        bar.set_postfix(best=best, refresh=(i % 10 == 0))

Output:

Search: 100%|██████████| 100/100 [00:00<00:00, 12.4kit/s, best=0]

dask.diagnostics progress bar interop:

from dask.diagnostics import ProgressBar
import dask.dataframe as dd

# Dask ships its own ProgressBar; tqdm.dask provides an adapter
from tqdm.dask import TqdmCallback

with TqdmCallback(desc="Dask compute"):
    ddf = dd.read_parquet("data/*.parquet")
    result = ddf.groupby("k")["v"].mean().compute()

Output:

Dask compute: 100%|██████████| 12/12 [00:03<00:00,  3.42partition/s]

tqdm.contrib.concurrent — one-liner thread/process map:

from tqdm.contrib.concurrent import thread_map, process_map
import time

def slow(x):
    time.sleep(0.05)
    return x * 2

print(thread_map(slow, range(40), max_workers=8, desc="Threads")[:5])
print(process_map(slow, range(40), max_workers=4, desc="Procs")[:5])

Output:

Threads: 100%|██████████| 40/40 [00:00<00:00, 158.3it/s]
[0, 2, 4, 6, 8]
Procs:   100%|██████████| 40/40 [00:00<00:00,  75.1it/s]
[0, 2, 4, 6, 8]

Testing patterns#

In tests, you usually want bars suppressed — they pollute pytest output and interfere with CI log parsers. Disable globally via env var or a fixture.

# conftest.py
import os
import pytest
from tqdm import tqdm

@pytest.fixture(autouse=True)
def silence_tqdm(monkeypatch):
    monkeypatch.setenv("TQDM_DISABLE", "1")

# Or explicitly via the API
@pytest.fixture
def quiet_tqdm(monkeypatch):
    monkeypatch.setattr("tqdm.tqdm.__init__",
                        lambda self, *a, **kw: super(tqdm, self).__init__(*a, **{**kw, "disable": True}))

# Assert bar emits expected postfix values
def test_postfix_updates(capsys):
    from tqdm import tqdm
    with tqdm(range(3), file=sys.stdout) as bar:
        for i in bar:
            bar.set_postfix(step=i)
    out = capsys.readouterr().err  # tqdm writes to stderr by default
    assert "step=2" in out

Output: (none — exits 0 on success)

[!TIP] TQDM_DISABLE=1 in CI environment variables is the cleanest way to silence every bar without touching application code.

When NOT to use this#

tqdm is almost-always the right choice for interactive loops, but a handful of contexts are better served by alternatives.

• Long-running batch jobs in CI — bars produce ANSI control sequences and \r carriage returns that flood the log. Use a periodic log.info(f"{i}/{total}") every N seconds instead.
• Web servers / request handlers — never inside a request handler; the user can’t see stderr. Emit progress via WebSocket or SSE.
• Structured-logging environments (Datadog, Splunk) — the carriage-return output line breaks log ingestion. Disable with TQDM_DISABLE=1.
• Loops with side effects faster than display refresh — for >100k it/s loops, the bar update overhead can be visible; tune miniters or skip the bar entirely.
• When you want rich tables, panels, or live metrics, prefer rich.progress — it supports multi-column live displays.

Quick reference#