How to block or download a pdf

[danielvanmilgx]

Hello,

Are there working examples of blocking or downloading a pdf with Playwright crawler?

I keep getting errors like:

Traceback (most recent call last):
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\crawlee\crawlers\_basic\_context_pipeline.py", line 100, in __call__
    result = await middleware_instance.action()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\crawlee\crawlers\_basic\_context_pipeline.py", line 40, in action
    self.output_context = await self.generator.__anext__()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\crawlee\crawlers\_playwright\_playwright_crawler.py", line 320, in _navigate
    response = await context.page.goto(
               ^^^^^^^^^^^^^^^^^^^^^^^^
        context.request.url, timeout=remaining_timeout.total_seconds() * 1000, **context.goto_options
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\playwright\async_api\_generated.py", line 9045, in goto
    await self._impl_obj.goto(
        url=url, timeout=timeout, waitUntil=wait_until, referer=referer
    )
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\playwright\_impl\_page.py", line 552, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\playwright\_impl\_frame.py", line 153, in goto
    await self._channel.send(
        "goto", self._navigation_timeout, locals_to_params(locals())
    )
  File "c:\danielm\git\gitlab\services\subsidiegezocht\.venv\Lib\site-packages\playwright\_impl\_connection.py", line 69, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^

as for JS there is: https://crawlee.dev/js/docs/examples/file-download

[danielvanmilgx]

as for JS there is: https://crawlee.dev/js/docs/examples/file-download

[janbuchar]

Hi @danielvanmilgx, could you please provide a minimal code snippet that shows the error and/or desired functionality?

[danielvanmilgx]

For example the code here:

https://crawlee.dev/python/docs/introduction/crawling

[janbuchar]

So you run the exact code there against that Shopify site and you get the error above?

[UnknownHawkins]

I need a minimal example to demonstrate the issue. The error likely stems from missing dependencies or incorrect use of Playwright in the crawler setup. Let me walk through it.

[vdusek]

Navigation to a PDF fails because headless browser can't render it, so page.goto() aborts. PDFs shouldn't go through the browser render at all, they should be downloaded with a plain HTTP request instead.

The good news is that this is currently being added in #2043: FileDownloadCrawler, the Python counterpart of the JS file-download example you linked. It can fetch any content type without parsing the body and can stream large files to disk.

Once merged, it could look something like this:

from crawlee.crawlers import FileDownloadCrawler, FileDownloadCrawlingContext

crawler = FileDownloadCrawler()

@crawler.router.default_handler
async def handler(context: FileDownloadCrawlingContext) -> None:
    content = await context.http_response.read()
    kvs = await context.get_key_value_store()
    await kvs.set_value('sample.pdf', content, 'application/pdf')

await crawler.run(['https://example.com/sample.pdf'])

If you need this today, load the website with PlaywrightCrawler, collect the .pdf links, and fetch each one with context.send_request() and store them where you want to. Don't enqueue_links() them, otherwise they'll hit page.goto() again.

If you only want to skip PDFs, use the exclude argument and filter them out like this:

await context.enqueue_links(exclude=[Glob('**/*.pdf')])

[vdusek]

cc @Mantisus

[Mantisus]

For small files, this is already possible, you need to use HttpCrawler. The main advantage of FileDownloadCrawler is that it reads the response in a stream, that is important for large files.

from crawlee.crawlers import HttpCrawler, HttpCrawlingContext

crawler = HttpCrawler()

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
    content = await context.http_response.read()
    kvs = await context.get_key_value_store()
    await kvs.set_value('sample.pdf', content, 'application/pdf')

await crawler.run(['https://example.com/sample.pdf'])