Python urllib HTTP Error 403: Forbidden (User-Agent, tested)

urllib.error.HTTPError: HTTP Error 403: Forbidden on a page that opens fine in your browser usually means the server rejected your script's User-Agent. We captured what Python sends: urlopen() identifies itself as Python-urllib/3.13 and requests as python-requests/2.32.3 -- strings many CDNs and firewalls refuse. A specific, honest User-Agent fixed it in our test; if it doesn't, the 403 body tells you what else is wrong.

1. What urllib and requests send, and a server that filters it

The server below stands in for the many CDNs and web application firewalls that refuse default library User-Agents. It records the request headers and returns 403 for Python-urllib and python-requests:

import socket, threading, time, urllib.request, urllib.error, requests

PORT = 8963
seen = []
def serve():
    srv = socket.socket()
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", PORT))
    srv.listen()
    while True:
        conn, _ = srv.accept()
        req = conn.recv(65536).decode()
        headers = dict(l.split(": ", 1) for l in req.split("\r\n")[1:] if ": " in l)
        ua = headers.get("User-Agent", "")
        seen.append(headers)
        # like many CDNs/WAFs: refuse the default library User-Agents
        if ua.startswith(("Python-urllib", "python-requests")):
            conn.sendall(b"HTTP/1.1 403 Forbidden\r\nContent-Length: 9\r\n\r\nForbidden")
        else:
            conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
        conn.close()
threading.Thread(target=serve, daemon=True).start()
time.sleep(0.2)
url = f"http://127.0.0.1:{PORT}/page"

try:
    urllib.request.urlopen(url)
except urllib.error.HTTPError as e:
    print("urlopen          ->", e, "| sent User-Agent:", seen[-1].get("User-Agent"))
try:
    requests.get(url).raise_for_status()
except requests.HTTPError as e:
    print("requests.get     ->", e.response.status_code, "| sent User-Agent:", seen[-1].get("User-Agent"))

UA = "my-report-script/1.0 (+https://example.com/contact)"
req = urllib.request.Request(url, headers={"User-Agent": UA})
print("urlopen + UA     ->", urllib.request.urlopen(req).status)
print("requests + UA    ->", requests.get(url, headers={"User-Agent": UA}).status_code)
print("all headers urlopen sent by default:", {k: v for k, v in seen[0].items()})

urlopen          -> HTTP Error 403: Forbidden | sent User-Agent: Python-urllib/3.13
requests.get     -> 403 | sent User-Agent: python-requests/2.32.3
urlopen + UA     -> 200
requests + UA    -> 200
all headers urlopen sent by default: {'Accept-Encoding': 'identity', 'Host': '127.0.0.1:8963', 'User-Agent': 'Python-urllib/3.13', 'Connection': 'close'}

Measured: Python 3.13.5's urlopen() sends User-Agent: Python-urllib/3.13 and requests 2.32.3 sends python-requests/2.32.3. Those strings are easy to filter, and urllib's whole default request is just four headers (no Accept, no Accept-Language). Sending a real, descriptive User-Agent changed the same request from 403 to 200 for both libraries.

2. The fix: send an honest, specific User-Agent

import urllib.request

UA = "my-report-script/1.0 (+https://example.com/contact)"
req = urllib.request.Request(url, headers={"User-Agent": UA})
with urllib.request.urlopen(req) as resp:
    data = resp.read()

# requests: once per session
import requests
s = requests.Session()
s.headers["User-Agent"] = UA

Name your script and give a contact URL or email -- many sites' policies (Wikipedia's, for example) ask for exactly that, and it lets an operator reach you instead of blocking you. Pretending to be a browser is not a fix: it can break the site's terms of service, and bot protection that checks more than the User-Agent will block it anyway.

3. Still 403 with a proper User-Agent?

• Read the body: e.read() on the HTTPError (or r.text) often says why -- missing API key, region block, rate limit, or a bot-challenge page.
• Authentication: 403 can mean "you are logged in but not allowed"; see 401 vs 403.
• A JavaScript challenge page (Cloudflare and similar) means the site does not want automated access to that page. Use the site's official API or feed, or ask the operator; don't try to get around it.
• Check robots.txt and the terms before automating requests to a site you don't own.

Written with AI assistance. Measured on 2026-10-09 with Python 3.13.5's urllib.request and requests 2.32.3 against a local socket server that records request headers and refuses the default library User-Agents. PlainHeaders is our extension; Python and requests are free.