How to extract all links from a web page

Five ways to pull every link out of a page, from a one-line console snippet to crawling a whole site.

1. Browser console (no install, one page)

Open DevTools (F12) → Console, paste this and press Enter:

copy([...new Set([...document.links].map(a => a.href.split('#')[0]))].join('\n'))

Every unique link on the page, already made absolute, is now on your clipboard, one per line. copy() is a DevTools helper available in Chrome, Edge and Firefox. Chrome may ask you to type allow pasting the first time you paste into the console. That warning is there because pasted code runs with full access to the page, so only paste code you understand.

2. A link-extractor extension (filters and Excel export)

PlainLinks (ours, free) lists every link on the page or in your selection, removes duplicates and lets you filter by text, domain, file type (PDFs, documents, images) or a regular expression. Copy the URLs or a Markdown list, or export CSV / Excel with the anchor text, rel (nofollow) and internal/external columns. It reads the page only when you click its button.

3. Online, from HTML you already have

Paste page source or an HTML snippet into the free link extractor. Add the page's URL as the base so relative links like /about become absolute. It runs in your browser.

4. Python: requests + BeautifulSoup

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/Markdown"
html = requests.get(url, headers={"User-Agent": "link-script/1.0 (you@example.com)"}, timeout=30).text
links = {urljoin(url, a["href"]).split("#")[0] for a in BeautifulSoup(html, "html.parser").find_all("a", href=True)}
print("\n".join(sorted(links)))

urljoin turns relative links into absolute ones, and the set removes duplicates. Set a descriptive User-Agent: some sites, Wikipedia included, reject the default one. This only sees links in the HTML the server sends. Links added by JavaScript need a real browser (methods 1 and 2).

5. A quick shell one-liner (rough)

curl -s https://example.com | grep -oE 'href="[^"]+"' | sed 's/^href="//; s/"$//' | sort -u

Good for a quick look, but it also catches stylesheets and icons, leaves relative links relative, and misses single-quoted or unquoted attributes.

All links on a whole website

A page only shows its own links. For every page and every link of a site (and which ones are broken), you need a crawler. Our website SEO audit on Apify crawls a site and reports broken links and on-page issues, billed per page. If the site has a sitemap, the sitemap tools and the sitemap URL checker cover its pages without crawling.

Written with AI assistance. The console snippet, the Python script and the shell one-liner were run before publishing. PlainLinks is our extension; the other methods are free.