curl --request POST \
--url https://api.context.dev/v1/batch/submit \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"input": {
"type": "scrape",
"data": {
"type": "markdown",
"urls": [
{
"url": "https://example.com/products/anvil",
"itemId": "sku-1",
"meta": {
"category": "tools"
}
},
{
"url": "https://example.com/products/hammer",
"itemId": "sku-2"
}
],
"options": {
"useMainContentOnly": true
}
}
}
}
'{
"id": "batch_9f2c8a",
"status": "queued",
"mode": "scrape",
"format": "markdown",
"tags": [
"docs"
],
"crawl": {
"source": {
"type": "start_url",
"url": "https://example.com/docs"
},
"max_pages": 500,
"max_depth": 0,
"follow_subdomains": true,
"url_pattern": "^https://example\\.com/docs/"
},
"input": {
"reserved": 24817,
"reserved_is_ceiling": false,
"submitted": 25000,
"duplicates": 183,
"invalid": 0
},
"credits": {
"reserved": 24817
},
"created_at": "<string>",
"invalid_urls": [
{
"url": "<string>",
"reason": "Must be a public http:// or https:// URL"
}
],
"cache_metadata": {
"status": "hit",
"age_ms": 1
},
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"webhook_secret": "<string>",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}Submit a Batch
Scrape 25K URLs or crawl large websites asynchronously.
curl --request POST \
--url https://api.context.dev/v1/batch/submit \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"input": {
"type": "scrape",
"data": {
"type": "markdown",
"urls": [
{
"url": "https://example.com/products/anvil",
"itemId": "sku-1",
"meta": {
"category": "tools"
}
},
{
"url": "https://example.com/products/hammer",
"itemId": "sku-2"
}
],
"options": {
"useMainContentOnly": true
}
}
}
}
'{
"id": "batch_9f2c8a",
"status": "queued",
"mode": "scrape",
"format": "markdown",
"tags": [
"docs"
],
"crawl": {
"source": {
"type": "start_url",
"url": "https://example.com/docs"
},
"max_pages": 500,
"max_depth": 0,
"follow_subdomains": true,
"url_pattern": "^https://example\\.com/docs/"
},
"input": {
"reserved": 24817,
"reserved_is_ceiling": false,
"submitted": 25000,
"duplicates": 183,
"invalid": 0
},
"credits": {
"reserved": 24817
},
"created_at": "<string>",
"invalid_urls": [
{
"url": "<string>",
"reason": "Must be a public http:// or https:// URL"
}
],
"cache_metadata": {
"status": "hit",
"age_ms": 1
},
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"webhook_secret": "<string>",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}webhook or webhookUrl to receive batch.completed, batch.failed, and batch.cancelled events.
Set webhook.retry to {} to use the default automatic retry schedule, then use List Webhook Deliveries to inspect delivery status.Authorizations
Bearer authentication header of the form Bearer <API_KEY>, where <API_KEY> is your api key.
Headers
Any string unique to this submission. Retries with the same key return the original batch.
200Body
Create an asynchronous scraping job.
Scrape up to 25K URLs in one batch.
- Scrape
- Crawl
Show child attributes
Show child attributes
Legacy URL notified when the batch finishes. Preserves one best-effort attempt. Cannot be combined with webhook.
Completion webhook settings. Cannot be combined with webhookUrl. Omitting retry preserves legacy delivery; retry: {} opts into durable retries.
Show child attributes
Show child attributes
Tags stored on the batch. Filter the batch list by them later.
201 - 50["docs", "competitor"]
Response
Batch accepted. Read progress and results from GET /batch/{batch_id}.
Batch ID. Poll GET /batch/{batch_id} with it.
"batch_9f2c8a"
Always queued. An accepted batch has not started yet.
queued How pages will be selected.
scrape, crawl What each page will be returned as.
markdown, html Tags stored on the batch.
["docs"]
How the crawl was configured. Null for scrape batches.
Show child attributes
Show child attributes
What submission took in, and what it charged for.
Show child attributes
Show child attributes
What accepting this batch cost.
Show child attributes
Show child attributes
When the batch was created.
Rejected URLs, up to 100. These are not charged.
Show child attributes
Show child attributes
Cache outcome for this response. Composite responses are hits only when every cache-controlled fetch contributing to the output was a hit; age_ms is the oldest contributing hit.
Show child attributes
Show child attributes
Unique id of this API call, also sent in the X-Request-Id response header. Quote it when contacting support about a failed request.
"3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91"
Signing secret for the completion webhook, returned only here and never again. Store it now; it is not repeated by GET /batch/{batch_id}.
API key usage for this request.
Show child attributes
Show child attributes
Was this page helpful?