> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Response compression

> Use gzip to reduce the size of large scraping and crawl responses.

Context.dev supports gzip response compression. It is most useful for large HTML, Markdown, crawl, and batch-result payloads; small JSON responses may not benefit enough to be compressed.

## Enable compression

Most modern HTTP clients request compression and decode it automatically. Your application should continue to parse ordinary JSON. The SDK examples use their HTTP transport's defaults without overriding `Accept-Encoding`.

<CodeGroup>
  ```bash cURL theme={null}
  curl --compressed --get https://api.context.dev/v1/web/scrape/markdown \
    --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --data-urlencode "url=https://example.com" \
    --data-urlencode "useMainContentOnly=true"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.web.webScrapeMd({
    url: "https://example.com",
    useMainContentOnly: true,
  });
  console.log(response.markdown);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.web.web_scrape_md(
      url="https://example.com",
      use_main_content_only=True,
  )
  print(response.markdown)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.web.web_scrape_md(
    url: "https://example.com",
    use_main_content_only: true,
  )
  puts response.markdown
  ```

  ```go Go theme={null}
  package main

  import (
  	"context"
  	"fmt"
  	"os"

  	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
  	"github.com/context-dot-dev/context-go-sdk/v2/option"
  )

  func main() {
  	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))
  	response, err := client.Web.WebScrapeMd(context.Background(), contextdev.WebWebScrapeMdParams{
  		URL: "https://example.com",
  		UseMainContentOnly: contextdev.Bool(true),
  	})
  	if err != nil {
  		panic(err)
  	}
  	fmt.Println(response.Markdown)
  }
  ```

  ```php PHP theme={null}
  <?php
  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->web->webScrapeMd(
      url: 'https://example.com',
      useMainContentOnly: true,
  );
  echo $response->markdown, PHP_EOL;
  ```
</CodeGroup>

Do not add manual decompression when your client already handles `Content-Encoding`. In Go's `net/http` and Ruby's `Net::HTTP`, setting `Accept-Encoding` yourself can change automatic decoding behavior; prefer the client's defaults unless you are measuring the wire response.

## Check the response headers

This cURL-only diagnostic inspects the wire response. Request gzip explicitly and inspect the headers without printing the body; SDK transports may remove compression headers after decoding.

```bash cURL theme={null}
curl --silent --output /dev/null --dump-header - \
  --get https://api.context.dev/v1/web/scrape/markdown \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Accept-Encoding: gzip" \
  --data-urlencode "url=https://example.com"
```

For a compressible response, look for:

```text theme={null}
content-encoding: gzip
```

An absent header is not necessarily an error. The response may be too small to compress, an intermediary may have changed negotiation, or your application client may expose headers after decoding.

## Request less content

Compression should not replace endpoint-level controls:

* Request [Markdown](/guides/scrape-websites-to-markdown) instead of HTML when text is enough.
* Use `useMainContentOnly=true` to omit repeated page chrome.
* Use `includeSelectors` and `excludeSelectors` to keep only relevant subtrees.
* Keep `includeImages=false` unless image references are required.
* Leave `shortenBase64Images=true` when Markdown images are enabled.
* Use cursor pagination or gzipped NDJSON files for [large batch results](/guides/scrape-websites-in-batches#read-result-records-as-json).
* Bound crawl page counts instead of transferring content you will discard.

The best optimization is returning fewer unnecessary bytes.

## Measure the effect

Compression trades CPU work for fewer transferred bytes. Its latency effect depends on payload size, client location, network, runtime, and intermediaries.

Compare representative requests with `Accept-Encoding: identity` and `Accept-Encoding: gzip`. Record:

* compressed and uncompressed bytes transferred;
* time to first byte and total response time;
* client CPU and memory during decode;
* behavior through your production proxy or gateway;
* results for small, median, and large payloads.

Do not publish a universal improvement percentage from one page or one network path.

## Reuse connections

Keep a long-lived HTTP client or connection pool per process. Compression reduces response bytes, while connection reuse avoids repeating DNS, TCP, and TLS setup. Set explicit request timeouts and cap concurrency as described in [Best practices](/optimization/best-practices).

## Next steps

<CardGroup cols={2}>
  <Card title="Scrape webpages as Markdown" icon="file-lines" href="/guides/scrape-websites-to-markdown">
    Retrieve page content with the output options your application needs.
  </Card>

  <Card title="Crawl a website" icon="sitemap" href="/guides/crawl-website">
    Collect linked pages with explicit depth and page limits.
  </Card>

  <Card title="Rate limits" icon="gauge-high" href="/optimization/rate-limits">
    Bound concurrency and retry rate-limited requests.
  </Card>
</CardGroup>