I support a less example code to use context.Context with colly,

which not change anything in colly.

// contextTransport wrapper a context.Context for cancel requests
type contextTransport struct {
	ctx   context.Context
	trans *http.Transport
}

func (t *contextTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	req = req.WithContext(t.ctx)
	return t.trans.RoundTrip(req)
}

func collectorWithContext(c *colly.Collector, ctx context.Context) {
    // We can stop all requests at `OnRequest` callback 
    // before send request to HTTP client.
    c.OnRequest(func(req *colly.Request) {
		select {
		case <-ctx.Done():
			req.Abort()
		default:
		}
	})

    // Use custome Transport to cancel all pending requests at HTTP client,
    // which not have chance to stop at OnRequest callback.
    trans := &contextTransport{
		ctx:   ctx,
		trans: &http.Transport{},
	}
    c.WithTransport(trans)
}

This will work when cancel the context.Context,

ctx.Done() will return, and all pending request will immediately be canceled.

Hey! Any reason why this became stuck?

Is it because this PR changes lots of function signatures to have context argument, which is a pretty radical change?

If so, what about making it much less drastic by only adding Collector.WithContext method, which will be used for the http.Request? This way API compatibility would be kept, and it still would be possible to cancel HTTP requests in a clean way.

And since the common way to use colly is to have OnHTML etc. handlers be closures, you can simply use your own context from the closure. E.g.

func crawl(ctx context.Context, url string) {
	c := colly.NewCollector(colly.WithContext(ctx))
	c.OnResponse(func(res *colly.Response) {
		storeIntoSomeDatabase(ctx, res)
	})
	// skip
}

If so, what about making it much less drastic by only adding Collector.WithContext

I like this approach a lot! @WGH- could you work on this?

@asciimoo yes

FWIW, I said it in other issue, but the solution outlined by @fooofei is not perfect since it loses the original request's context which might already have deadline set. See net/http/client.go, where setRequestCancel(req, rt, deadline) is called just before RoundTrip.

FWIW, I said it in other issue, but the solution outlined by @fooofei is not perfect since it loses the original request's context which might already have deadline set

I think it isn't a deal breaker. We can document this behavior or perhaps we can throw a warning if the user sets a custom timeout and also uses custom ctx. What do you think?

FWIW, I said it in other issue, but the solution outlined by @fooofei is not perfect since it loses the original request's context which might already have deadline set

I think it isn't a deal breaker. We can document this behavior or perhaps we can throw a warning if the user sets a custom timeout and also uses custom ctx. What do you think?

What I said earlier applies only to the hack with setting context inside RoundTrip. Setting context before passing the http.Request to Client.Do does not have this problem.