scrapyd RunnerError: No module named 'pkg_resources' (fix)

Q: Why does scrapyd start but deploy fails?

The web server does not import scrapyd.runner. Only the subprocess that lists spiders and runs jobs does, and that is where pkg_resources is imported.

Q: Where do I install the setuptools pin?

In the Python environment that runs the scrapyd server, not on the machine where you run scrapyd-deploy.

On setuptools 82 or newer, scrapyd 1.6.0 (the latest) starts, but scrapyd-deploy and addversion.json fail with RunnerError: ... ModuleNotFoundError: No module named 'pkg_resources', from scrapyd/runner.py line 6. Tested fix: pip install "setuptools<81" where scrapyd runs.

1. The error: scrapyd 1.6.0 with setuptools 84

pip install scrapyd==1.6.0 (the latest release, 2025-07-22) into a Python 3.11.17 venv with setuptools 84.2.1. The scrapyd server starts and daemonstatus.json answers "status": "ok". Uploading a project egg is what fails (that is what scrapyd-deploy sends):

$ curl http://127.0.0.1:6800/addversion.json -F project=demo -F version=1.0 -F egg=@demo-1.0-py3.11.egg
{"status": "error", "message": "RunnerError: ..."}

The message field, unescaped:

RunnerError: RunnerError: Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "scrapyd/runner.py", line 6, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

To list the spiders in your egg, and to run every scheduled job, scrapyd starts a subprocess with python -m scrapyd.runner (runner = scrapyd.runner in its default config). Line 6 of scrapyd/runner.py is a plain import pkg_resources, and setuptools 82.0.0 removed that module. You can reproduce it without a server:

$ python -m scrapyd.runner list
...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "scrapyd/runner.py", line 6, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

2. Fix: pin setuptools below 81 in the scrapyd environment (tested)

$ pip install "setuptools<81"    # gave 80.10.2
# restart scrapyd, then:
$ curl http://127.0.0.1:6800/addversion.json -F project=demo -F version=1.0 -F egg=@demo-1.0-py3.11.egg
{"project": "demo", "version": "1.0", "spiders": 1, "status": "ok", "node_name": "myhost"}
$ curl http://127.0.0.1:6800/schedule.json -d project=demo -d spider=q
{"jobid": "ce3f73d6cf7111f18ae895f0214921f0", "status": "ok", "node_name": "myhost"}

The job finished and its log showed 'item_scraped_count': 1. Install the pin into the Python that runs scrapyd (its venv or Docker image), not the machine you deploy from. There is no newer scrapyd release to upgrade to (1.6.0 is the latest on PyPI), so keep setuptools<81 in that image's requirements. With the pin, scrapyd's own log (not the job log) showed the harmless pkg_resources is deprecated warning. To hide it, set PYTHONWARNINGS="ignore:pkg_resources is deprecated as an API" where scrapyd starts.

Test setup: a scrapy startproject project with one spider on a data: URL, packed with python setup.py bdist_egg, and scrapyd on 127.0.0.1. We used another port; the commands above show the default 6800.

Frequently asked questions

Why does scrapyd start but deploy fails?

The web server does not import scrapyd.runner. Only the subprocess that lists spiders and runs jobs does, and that is where pkg_resources is imported.

Where do I install the setuptools pin?

In the Python environment that runs the scrapyd server, not on the machine where you run scrapyd-deploy.

Written with AI assistance. Measured on 2026-10-24 on Linux with scrapyd 1.6.0 and the Scrapy it installed, CPython 3.11.17, setuptools 80.10.2 and 84.2.1. Output pasted unedited except shortened paths and host name. PlainTables is our extension; scrapyd is free.