Web crawler based on Puppeteer

npm install @opd/crawler

import Crawler from '@opd/crawler'
// or commonjs
const Crawler = require('@opd/crawler').default

const crawler = new Crawler(options)

create crawler instance

options: crawler instance config

• parallel: maximum number of crawlers, default is 5
• pageEvaluate: evaluate function on current page, see Puppeteer, cannot support extra args now

launch browser use puppeteer.launch

add urls to crawler queue

Note: check url strictly, means url must start with https?

start crawl page, if urls is presented, will call crawler.queue firstly.

const result = await crawler.start()
console.log(result)

// [
//   {
//     url, // page url
//     result // crawled result
//   }
// ]

Note: if you call start before launch, browser will also be launched, but with no extra launch options