How to Extract Brand Assets from 1,000 URLs in Bulk (Python)

The person with 1,000 URLs is never the person who wants to babysit 1,000 browser tabs. Bulk brand extraction in Python is an asyncio problem wearing a design-data hat: a semaphore for rate limits, retries for the 10% of URLs that misbehave, and checkpointing so a crash at URL 900 does not cost you the first 899. Here is the working script, then the failure math.

Why not 1,000 Playwright launches

The single-site Playwright approach does not scale linearly. A thousand headless Chromium sessions means gigabytes of RAM, cookie banners in six languages, and a long tail of sites that hang networkidle forever. It is buildable - a browser pool, per-site timeouts, crash recovery - but at that point you are maintaining scraping infrastructure as a hobby. For bulk, put the browser problem on someone else's side of the API.

The script

pip install httpx

import asyncio, csv, json
import httpx

API = "https://miromiro.app/api/v1/brand"
KEY = "mm_live_your_key"
CONCURRENCY = 5          # stay under your plan's req/min cap
RETRIES = 3

async def extract(client, sem, url):
    async with sem:
        for attempt in range(RETRIES):
            try:
                r = await client.get(
                    API,
                    params={"url": url, "fields": "logo,palette"},
                    headers={"Authorization": f"Bearer {KEY}"},
                    timeout=60,
                )
                if r.status_code == 429:            # rate limited: back off
                    await asyncio.sleep(2 ** attempt * 5)
                    continue
                if r.status_code == 200:
                    return {"url": url, "ok": True, "data": r.json()}
                return {"url": url, "ok": False, "error": f"HTTP {r.status_code}"}
            except httpx.HTTPError as e:
                if attempt == RETRIES - 1:
                    return {"url": url, "ok": False, "error": str(e)}
                await asyncio.sleep(2 ** attempt)

async def main(urls):
    sem = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*(extract(client, sem, u) for u in urls))
    with open("brands.jsonl", "w") as f:
        for row in results:
            f.write(json.dumps(row) + "\n")
    failed = [r for r in results if not r["ok"]]
    print(f"{len(results) - len(failed)} ok, {len(failed)} failed")

if __name__ == "__main__":
    urls = [row[0] for row in csv.reader(open("domains.csv"))]
    asyncio.run(main(urls))

Swap the endpoint for /v1/extract if you want full token sets instead of brand data, and adjust fields= to trim the response to what you store - it does not change the credit cost, but it makes the output files dramatically smaller.

The three things that make bulk runs survive

1. The semaphore is the contract. Your plan has a requests-per-minute cap (the ladder is in the rate limits docs). Five concurrent with backoff on 429 stays comfortably inside the entry tiers; raise it as your plan allows. Blasting unbounded gather at any API gets you rate-limited into taking longer than the polite version.

2. Expect 5-15% failures and capture them. On a real 1,000-domain list some are dead, parked, or behind bot walls. The script writes per-URL errors instead of dying, so the run always finishes and the failures are a re-runnable list, not a mystery.

3. Re-runs are cheap because of caching. Responses are cached for 24 hours and a cache hit costs 0 credits ("cached": true in the usage object). Re-running today's failed batch only pays for URLs that actually re-extract - so retry freely.

Budgeting a 1,000-URL run

The free 100 credits cover a 6-10 URL pilot - do that first, on your ugliest URLs, before committing a plan to the full list. Wall-clock: the rate limiter sets the pace, not the extraction; expect tens of minutes for 1,000 URLs, not hours.

Summary

Bulk extraction is a queue-discipline problem: bounded concurrency, backoff on 429, per-URL error capture, and a re-run list. The 24-hour cache turns retries from a cost into a free operation. Pilot on 10 URLs with a free key, then size the plan to the list.

Frequently asked questions

How do I scrape brand colors from a list of websites in Python?

For a few sites, Playwright and computed styles. For hundreds or thousands, the async httpx script above against an extraction API.

How long does 1,000 URLs take?

Tens of minutes - your requests-per-minute cap is the bottleneck, not the extraction itself.

What failure rate should I expect?

5-15% on real-world lists: dead domains, parked pages, bot walls. Capture and re-run rather than aiming for zero.

Do repeated extractions cost credits again?

Not within 24 hours - cache hits cost 0 credits and report cached: true.