Scraping API
Scraping API · on-demand jobsOn-demand scraping jobs — submit, poll, download.
The Scraping API runs scrapes on demand: you submit URLs or product IDs, we fetch them now, and you download the raw results. Use it for pages the universe doesn't track or when you need this hour's numbers. If you want cleaned, history-deep market data without running a scrape, that's the Data API.
Batch jobs (Variant Sold, Listing product, App Search) go through team approval — typically within working hours (Monday–Friday, 08:00–17:00 Asia/Jakarta / WIB). Single-page endpoints run immediately, no approval.
| Endpoint | Path | What it does | Approval |
|---|---|---|---|
| Variant Sold | /v1/shopee/variant_sold_v1/submit · retrieve/{job_id} | Batch variant-sold scrape, up to 50,000 items | Approval required |
| Shopee Product Page | /v1/shopee/get_pc/submit · retrieve/{task_id} | Single product-page fetch (get_pc) | No approval |
| Shopee Product Page (bulk) | /v1/shopee/get_pc_bulk/submit · tasks · retrieve/{task_id} | Submit CSV, recover task IDs, then poll and download CSV or JSONL results | No approval |
| Shopee PDP Bundling | /v1/shopee/pdp-bundling/submit · tasks · retrieve/{task_id} | Submit CSV, recover task IDs, then poll and download captured API bundles as CSV or JSONL | No approval |
| Shopee Product Ratings | /v1/shopee/product-ratings · retrieve/{job_id} | Retrieve one explicit ratings page per product, up to 50 products per job | No approval |
| Shopee Listing product | /v1/shopee/search-items · merchant-items | Search / category / merchant listing pages by URL | Approval required |
| Shopee App Search | /v1/shopee/app/search-items/submit · retrieve/{job_id} · jobs/{job_id}/pages/{item_id}/data | Mobile keyword results as JSON; up to 50 searches and 100 pages per search, billed per successful page | Approval required |
| Shopee App Ratings | /v1/shopee/app/ratings/submit · retrieve/{job_id} · jobs/{job_id}/data | One product, fixed 50-rating pages; billed per successful non-empty page | Approval required |
| Shopify Compatibility Probe | /v1/shopify/probe/submit · retrieve/{job_id} | Check whether a Shopify site is scrapeable before running a real job — same tool as the checker above | No approval |
| Tokopedia PDP Bulk | /v1/tokopedia/pdp/bulk/submit · retrieve/{job_id} | Bulk Tokopedia PDP scrape, up to 2,000 items | No approval |
| Tokopedia Search Bulk | /v1/tokopedia/search/bulk/submit · retrieve/{job_id} | Bulk Tokopedia search/category listing scrape | No approval |
| Tokopedia Merchant Bulk | /v1/tokopedia/merchant/bulk/submit · retrieve/{job_id} | Bulk Tokopedia merchant product listing scrape | No approval |
| Blibli PDP Bulk | /v1/blibli/pdp/bulk/submit · retrieve/{job_id} | Bulk Blibli PDP scrape, up to 2,000 items | No approval |
| Blibli Merchant Bulk | /v1/blibli/merchant/bulk/submit · retrieve/{job_id} | Bulk Blibli merchant product listing scrape | No approval |
| Mercado Libre PDP Bulk | /v1/mercadolibre/pdp/bulk/submit · retrieve/{job_id} | Up to 2,000 exact product URLs; one region per batch; 18 configured regions | No approval |
| Mercado Libre Search Bulk | /v1/mercadolibre/search/bulk/submit · retrieve/{job_id} | Up to 50 listing URLs; all pages by default; one region per batch; 18 configured regions | No approval |
| Amazon PDP Bulk | /v1/amazon/pdp/bulk/submit · retrieve/{job_id} | Bulk Amazon PDP scrape, up to 2,000 items (per-item country: sg, jp, us, uk, de, fr, au, it, es, ca) | No approval |
| Amazon Search Bulk | /v1/amazon/search/bulk/submit · retrieve/{job_id} | Bulk Amazon search/category listing scrape, up to 50 items (per-item country: sg, jp, us, uk, de, fr, au, it, es, ca) | No approval |
| Shopify Search Bulk | /v1/shopify/search/bulk/submit · retrieve/{job_id} | Bulk Shopify collection/search page scrape, up to 50 URLs | No approval |
| Shopify PDP Bulk | /v1/shopify/pdp/bulk/submit · retrieve/{job_id} | Bulk Shopify PDP scrape, up to 2,000 product URLs | No approval |
| TikTok PDP Bulk | /v1/tiktok/pdp/bulk/submit · retrieve/{job_id} | Bulk TikTok PDP scrape, up to 2,000 items (per-item country: id, my, sg, th, vn, ph) | No approval |
| Lazada | /v1/lazada/* | Product search / details / reviews, category tree, seller directory / details / promotions, keyword tracking | No approval |
Per-marketplace guides with request examples and costs: Shopee · Tokopedia · TikTok Shop · Lazada · Blibli · Amazon
Each endpoint bills its own credit rate (per request, URL or item) from the shared wallet — see the rate card.
Shopee App Search
Get raw mobile keyword search results as JSON, separately from the existing Shopee Listing product API. Submit up to 50 searches from one country and request 1–100 pages per search with max_pages. Set start_page for an absolute page offset (default 0). Identical search page ranges must not overlap; categories and URLs are not accepted. Team approval is required, with processing during working hours (Monday–Friday, 08:00–17:00 Asia/Jakarta / WIB).
Create an API key in Scraping jobs and fund your wallet before submitting. Set MAGPIE_API_KEY in your shell; keep it private. Track submitted jobs in the Shopee App Search tab.
| Input | Default / limit | Meaning |
|---|---|---|
| country | Required | id, sg, th, ph, my, vn, tw, or br; one country per job. |
| searches | 1–50 entries | Each entry requires a nonempty keyword. |
| start_page / max_pages | 0 / 1 (maximum 100) | Absolute zero-based start page and number of pages requested; 60 items per page. |
| sort_by / order | relevancy / desc | Sort: relevancy, sales, price, ctime. Order: asc or desc. |
| filters | Empty object | Optional nonnegative integer price_min and price_max in the marketplace currency; minimum must not exceed maximum. |
1. Submit
BASE="https://api-staging.magpieiq.com/scraping"
curl -X POST "$BASE/v1/shopee/app/search-items/submit" \
-H "Authorization: Bearer $MAGPIE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"country":"id","searches":[{"keyword":"lipstick","start_page":0,"max_pages":1,"sort_by":"relevancy","order":"desc","filters":{}}]}'
# Response: {"job_id":"...","status":"awaiting_approval","cost":10,"total_pages":1}
# cost is illustrative: check the current per-page rate card.2. Wait for approval, then poll
Keep the returned job ID. Approval changes the job from awaiting_approval to processing. Poll about once a minute; a terminal job reports completed, partial_complete, or failed.
JOB_ID="<job_id from submit>"
curl "$BASE/v1/shopee/app/search-items/retrieve/$JOB_ID" \
-H "Authorization: Bearer $MAGPIE_API_KEY"// Example retrieval response (one successful page)
{"job_id":"...","status":"completed","total_pages":1,"success_count":1,"failed_count":0,
"results":[{"search_index":0,"keyword":"lipstick","page":0,"status":"success",
"download_url":"/v1/shopee/app/search-items/jobs/.../pages/0:0/data"}]}3. Download successful pages
Each result identifies its search_index, keyword, and absolute page. Each successful entry includes a download_url path. Append it to BASE and use the same API key. The example below downloads the first search's page 0.
curl "$BASE/v1/shopee/app/search-items/jobs/$JOB_ID/pages/0:0/data" \
-H "Authorization: Bearer $MAGPIE_API_KEY"Credits are charged on approval at the current per-page rate. Unsuccessful pages and pages beyond a verified response.nomore=true boundary are refunded. The deadline stays fixed at 24 hours from the original publication attempt. Missing earlier pages can be retried once, five minutes after the last expected page is observed; priority does not guarantee completion time.
Status and troubleshooting
- awaiting_approval: no charge yet; wait for team approval. Do not resubmit to speed it up.
- processing: work is queued or results are being checked. An empty worker queue does not mean results are ready.
- partial_complete / failed: inspect individual page statuses. Unsuccessful or beyond-end pages are not billed; successful pages remain downloadable.
- 401: invalid or revoked API key. 403: key lacks endpoint access. 402: insufficient available credits, including pending-job reservations.
- 422: invalid input, unsupported fields, or overlapping identical searches. 503: endpoint unavailable; contact support rather than repeatedly submitting.
- Save the job_id. If a network error obscures the submit response, check your dashboard before retrying; another submission may create another paid job.