restart after crash

[mikadev]

crawlee

Issue description

1. run crawlee using CRAWLEE_PURGE_ON_START=0 or purgeOnStart: false
2. kill
3. restart (\storage\request_queues\default are full)
4. nothing happen

Package version

crawlee@3.15.3

Node.js version

v22.20.0

Operating system

win 11

[barjin]

Hello and thank you for your interest in Crawlee.

The following snippet simulates a crashing process with a crawler that is run with CRAWLEE_PURGE_ON_START set to '0'. Running this script multiple times scrapes different parts of the website (subsequent invocations "resume" the scrape).

import { CheerioCrawler } from '@crawlee/cheerio';

async function main() {
    const crawler = new CheerioCrawler({
        requestHandler: async ({ request, log, enqueueLinks }) => {
            log.info(`Processing: ${request.url}`);
            await enqueueLinks();
        },
    });

    await crawler.run(['https://docs.apify.com/']);
}

process.env.CRAWLEE_PURGE_ON_START = '0';
setTimeout(() => process.exit(129), 2e3);
main();

Could you please verify if you encounter any issues with this script? If so, can you describe the issue in bigger detail - or update the repro script so it matches your scenario?
Thank you!

[mikadev]

@barjin thanks you for Crawlee is so great :) is now working i spend almost 2 days on this not understand...

Thank you !