Codestin Search App

Context.dev supports gzip response compression. It is most useful for large HTML, Markdown, crawl, and batch-result payloads; small JSON responses may not benefit enough to be compressed.

Enable compression

Most modern HTTP clients request compression and decode it automatically. Your application should continue to parse ordinary JSON. The SDK examples use their HTTP transport’s defaults without overriding Accept-Encoding.

curl --compressed --get https://api.context.dev/v1/web/scrape/markdown \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "useMainContentOnly=true"

import ContextDev from "context.dev";

const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

const response = await client.web.webScrapeMd({
  url: "https://example.com",
  useMainContentOnly: true,
});
console.log(response.markdown);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

response = client.web.web_scrape_md(
    url="https://example.com",
    use_main_content_only=True,
)
print(response.markdown)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

response = client.web.web_scrape_md(
  url: "https://example.com",
  use_main_content_only: true,
)
puts response.markdown

package main

import (
	"context"
	"fmt"
	"os"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))
	response, err := client.Web.WebScrapeMd(context.Background(), contextdev.WebWebScrapeMdParams{
		URL: "https://example.com",
		UseMainContentOnly: contextdev.Bool(true),
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(response.Markdown)
}

<?php
require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$response = $client->web->webScrapeMd(
    url: 'https://example.com',
    useMainContentOnly: true,
);
echo $response->markdown, PHP_EOL;

Do not add manual decompression when your client already handles Content-Encoding. In Go’s net/http and Ruby’s Net::HTTP, setting Accept-Encoding yourself can change automatic decoding behavior; prefer the client’s defaults unless you are measuring the wire response.

Check the response headers

This cURL-only diagnostic inspects the wire response. Request gzip explicitly and inspect the headers without printing the body; SDK transports may remove compression headers after decoding.

cURL

curl --silent --output /dev/null --dump-header - \
  --get https://api.context.dev/v1/web/scrape/markdown \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Accept-Encoding: gzip" \
  --data-urlencode "url=https://example.com"

For a compressible response, look for:

content-encoding: gzip

An absent header is not necessarily an error. The response may be too small to compress, an intermediary may have changed negotiation, or your application client may expose headers after decoding.

Request less content

Compression should not replace endpoint-level controls:

Request Markdown instead of HTML when text is enough.
Use useMainContentOnly=true to omit repeated page chrome.
Use includeSelectors and excludeSelectors to keep only relevant subtrees.
Keep includeImages=false unless image references are required.
Leave shortenBase64Images=true when Markdown images are enabled.
Use cursor pagination or gzipped NDJSON files for large batch results.
Bound crawl page counts instead of transferring content you will discard.

The best optimization is returning fewer unnecessary bytes.

Measure the effect

Compression trades CPU work for fewer transferred bytes. Its latency effect depends on payload size, client location, network, runtime, and intermediaries. Compare representative requests with Accept-Encoding: identity and Accept-Encoding: gzip. Record:

compressed and uncompressed bytes transferred;
time to first byte and total response time;
client CPU and memory during decode;
behavior through your production proxy or gateway;
results for small, median, and large payloads.

Do not publish a universal improvement percentage from one page or one network path.

Reuse connections

Keep a long-lived HTTP client or connection pool per process. Compression reduces response bytes, while connection reuse avoids repeating DNS, TCP, and TLS setup. Set explicit request timeouts and cap concurrency as described in Best practices.

Next steps

Scrape webpages as Markdown

Retrieve page content with the output options your application needs.

Crawl a website

Collect linked pages with explicit depth and page limits.

Rate limits

Bound concurrency and retry rate-limited requests.

Get started

Web data

Brand data

Automation

Production

Response compression

Enable compression

Check the response headers

Request less content

Measure the effect

Reuse connections

Next steps

Scrape webpages as Markdown

Crawl a website

Rate limits

​Enable compression

​Check the response headers

​Request less content

​Measure the effect

​Reuse connections

​Next steps

Scrape webpages as Markdown

Crawl a website

Rate limits

Enable compression

Check the response headers

Request less content

Measure the effect

Reuse connections

Next steps