API Documentation

1. Authentication

All API requests require a valid Pro or Enterprise plan. Generate your API key from the Dashboard under the API Key section.

Pass the key via the Authorization header:

Authorization: Bearer sk-your-api-key-here

2. Convert Single URL

Extract clean plain text from a single webpage.

Endpoint: POST /api/scrape

curl Example

curl -X POST https://crawlyourweb.online/api/scrape \
  -H "Authorization: Bearer sk-your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article"}'

Python Example

import requests

API_KEY = "sk-your-api-key-here"

resp = requests.post(
    "https://crawlyourweb.online/api/scrape",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"url": "https://example.com/article"},
)
data = resp.json()
print(data["text"])      # clean extracted text
print(data["char_count"])  # character count

JavaScript Example

const API_KEY = "sk-your-api-key-here";

const resp = await fetch("https://crawlyourweb.online/api/scrape", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/article" }),
});
const data = await resp.json();
console.log(data.text);

Response (JSON)

{
  "text": "Cleaned content of the page goes here...",
  "word_count": 523,
  "char_count": 3142,
  "credits_used": 4,
  "monthly_credits_remaining": 4996,
  "addon_credits_remaining": 0,
  "total_credits_remaining": 4996
}

3. Bulk Extraction (Pro & Enterprise)

Scrape multiple URLs simultaneously. Returns a ZIP file containing the text and metadata for each processed URL.

Endpoint: POST /api/scrape/bulk

curl Example

curl -X POST https://crawlyourweb.online/api/scrape/bulk \
  -H "Authorization: Bearer sk-your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/page1", "https://example.com/page2"]}' \
  --output bulk_export.zip

Request Body (JSON)

{
  "urls": [
    "https://example.com/page1",
    "https://example.com/page2"
  ]
}

Response (application/zip)

A downloadable ZIP archive structured as follows:

bulk_export.zip
├── example-com__page1/
│   ├── text.txt
│   ├── raw.txt
│   ├── formatted.txt
│   └── meta.json
├── example-com__page2/
│   ├── text.txt
│   ├── raw.txt
│   ├── formatted.txt
│   └── meta.json
└── _index.json

4. Check Account Status & Credits

Get your current credit balance, plan, and API key status.

Endpoint: GET /api/auth/me

curl Example

curl -X GET https://crawlyourweb.online/api/auth/me \
  -H "Authorization: Bearer sk-your-api-key-here"

Response (JSON)

{
  "id": 1,
  "email": "[email protected]",
  "monthly_credits": 30,
  "addon_credits": 0,
  "total_credits": 30,
  "plan": "pro",
  "is_admin": false,
  "is_verified": true,
  "is_active": true,
  "has_api_key": true,
  "api_key_prefix": "sk-a1b2c3d4",
  "auth_provider": "local",
  "created_at": "2026-04-24T00:00:00"
}

5. Extract History

Retrieve your past extractions without consuming credits again.

Endpoint: GET /api/scrape/history

curl Example

curl -X GET "https://crawlyourweb.online/api/scrape/history?limit=10&offset=0" \
  -H "Authorization: Bearer sk-your-api-key-here"

6. Error Codes