--- name: magpie-ecommerce-data description: Answer questions about Southeast Asian ecommerce with Magpie. Use for brands, market share, GMV, pricing, assortment or sellers on Shopee, Tokopedia, TikTok Shop, Lazada and Blibli (plus live Amazon pages), whether the question is historical ("who gained share last quarter", "what price bands exist") or live ("what does this product cost right now", "who is selling my brand"). Covers finished answers, SKU rows and bulk files from a maintained monthly dataset, and on-demand scraping of live marketplace pages, all through one account and wallet. --- # Magpie ecommerce data One account, one credit wallet, two kinds of data. **The dataset** is a maintained monthly panel, already scraped, cleaned, brand-labelled and deduplicated; agents read it as finished answers, SKU rows or bulk files. **Scraping** fetches live marketplace pages on request. ## Set up in one line Pick the line that fits where you are. None of them involves an API key in the conversation. - **Any agent:** paste `set up https://data-staging.magpieiq.com/setup.md`. The agent connects the Magpie MCP server, signs you in through the browser and saves the Magpie skill. - **Claude or ChatGPT (chat apps):** add the Magpie connector, URL `https://mcp-staging.magpieiq.com/v2/mcp`, and click Connect. Sign-in is OAuth in the browser; the app holds the token. - **Coding agents (Claude Code, Cursor, Codex, opencode):** run `npx @magpieiq/cli init`. It signs you in with a device code and registers the Magpie MCP server in the agents it finds. - **Terminal:** `magpie login` prints a short code and a URL; approve it at https://data-staging.magpieiq.com/device and the CLI stores its own key in `~/.magpie/credentials` (mode 0600). You can set a daily spend cap for that key on the approval page. No browser on this machine? `magpie login --email you@example.com` emails an Approve link instead (ask the user for the address, never for a key). - **Python:** `pip install magpie-data` (client + `magpie` CLI) is coming soon to PyPI. Until it ships, use the REST API or the MCP server. **Never ask the user to paste an API key into the chat**, and never print one back. If a call returns `login_required`, send the user through one of the lines above instead. Server keys for cron and CI are created in the dashboard by a developer and stored in a secret manager, then read from the environment (`MAGPIE_API_KEY`); they are not for conversations. You can start before signing in: discovery, the catalog, estimates and answer-tier calls on a few fixed sample scopes work without any key, limited to 10 calls a day per IP address. Everything else answers `login_required`. ### If you cannot make HTTP calls Some assistants cannot reach the network. Say so plainly rather than inventing results, and suggest the connector or a coding agent. If the user is happy to run commands, write the exact `curl` out for them, ask them to paste the response back (never a key), and carry on from there. **`data-staging.magpieiq.com` is the documentation and billing portal, not an API host.** The API host is `api-staging.magpieiq.com`. ## The method: discover, describe, estimate, run Agent API base: `https://api-staging.magpieiq.com/agent/v1`. Do not guess endpoint names, parameters or prices; the registry tells you, and it changes without a docs release. 1. **Discover** what exists: `GET /agent/v1/capabilities`, optionally filtered with `?q=brand share`, `?product=data|scrape` or `?marketplace=shopee`. Keyless. Each entry has a `name` such as `data/answer_brand_share`, a description, the price and unit, `mode` (sync or async), a latency hint and its `http_method` / `http_path`. Over MCP this is the `discover` tool. 2. **Describe** the one you picked: `GET /agent/v1/capabilities/{name}` returns its `input_schema` and `output_schema`. Use the parameter names and allowed values from the schema verbatim. Over MCP: `describe`. 3. **Estimate** before anything paid: send the same request with `estimate=true` (query string or JSON body). It returns the exact charge in `data.credits`, the units it counted, and `data.max_credits`. Nothing is charged. Tell the user the figure when it is more than a flat answer-tier call. 4. **Run** the real call with `max_credits` set to the estimate's `max_credits`. The server refuses to charge more than that (`estimate_exceeded`) instead of surprising the user. Over MCP: `run(name, params)`. ```bash A=https://api-staging.magpieiq.com/agent/v1 curl -s "$A/capabilities?q=brand+share" curl -s "$A/capabilities/data/answer_brand_share" # then the endpoint's own method + path, with estimate=true first, then max_credits ``` Over MCP the connector handles authentication. Over REST, calls that need an account send `Authorization: Bearer $MAGPIE_API_KEY`, where the key is a server key from the environment or the one `magpie login` stored in `~/.magpie/credentials`. Read it from there; never echo it or ask for it. Re-run discover when `meta.catalog_version` on any response differs from the one you saw at the start; `GET /agent/v1/changelog?since=` lists what changed. ### Pick the cheapest tier that answers the question 1. **Answer tier first** (`/agent/v1/answer/brand_share`, `category_trend`, `top_skus`, `seller_share`). A summary sentence plus a table of at most 50 rows, in about two seconds, at a flat price per call. Most questions end here: who leads, is it growing, what is the price band, which sellers matter. 2. **Rows** (`/agent/v1/rows`) only when the user asks for the SKUs behind an answer. Up to 1,000 rows per call (default 100), billed per row. Page with `meta.next_cursor` (pass it back as the cursor parameter named in describe). If the user needs more than a few thousand rows, the response suggests a bulk export with its estimate; take that route. 3. **Scrape** (`/agent/v1/scrape/{endpoint}`; MCP: `scrape_product` / `search`, or `run` with a `scrape/...` name from discover) for anything live: today's price, who is selling now, a shop's current catalogue, reviews, keyword rank. Check `describe` for the exact inputs first, and estimate before running. The call waits up to about 25 seconds and returns normalized rows inline (title, price, original price, stock, rating, reviews, seller, url, scraped_at); a longer job hands you a job handle instead — poll it with `get_job`. 4. **Bulk export** (`POST https://api-staging.magpieiq.com/data/v1/exports`) only when the user wants a file (Parquet or gzipped CSV) for a warehouse or a notebook. It is asynchronous and takes minutes. The dataset is monthly and historical; scraping is live. "How did share move" is the dataset. "What does it cost today" is scraping. Some questions need both: use an answer or rows call to pick the SKUs worth watching, then scrape those. ### The response envelope Every agent endpoint, data or scraping, answers in the same shape: ```json { "data": [ ... ], "summary": "One sentence you can repeat to the user.", "meta": { "credits_charged": 0, "balance": 0, "next_cursor": null, "as_of": "YYYY-MM", "refresh_id": "...", "catalog_version": "..." }, "job": null } ``` - `summary` is written to be quoted. Pair it with a small table from `data`. - `credits_charged` and `balance` are on every response; mention the charge when you report a result, and keep track of the balance across a session. - `as_of` is the data month an answer describes. Say it, so nobody mistakes a monthly figure for today's. - `meta.estimate: true` marks an estimate; `meta.deprecated` warns that the endpoint has a replacement (`use_instead`) and a `sunset` date. Switch to the replacement. - `job` is non-null when work continues in the background (below). ### Async jobs Scrapes and exports can outlive one call. Then `data` is empty and `job` is `{"id", "status", "eta", "next": "get_job"}`. - **Write the job id down immediately**, before polling. It is the only way to reach a result you have already paid for. Never resubmit a job that is still running: you pay twice. - Poll `GET /agent/v1/jobs/{id}` (MCP: `get_job`) with backoff, 10 s, 30 s, 60 s, on a wall-clock deadline you choose from the user's situation. When the deadline passes, stop, give the user the id and how to fetch it later, and move on. - **Shopee approval window.** Shopee search, shop listings and variant sales need a manual approval and are processed Monday-Friday, 08:00 to 17:00 Asia/Jakarta (WIB). Outside that window, or before approval, the job reads `status: "queued"` with `runs_after` set. That is not an error: tell the user when it will run and hand back the id. Approval times out after 24 hours. ### Errors and what to do Errors look like `{"error": {"code", "message", "next", ...}}`. `next` is the exact next step (a URL, a registry name, or an instruction). Follow it. | Code | HTTP | What to do | |---|---|---| | `login_required` | 401 | Not signed in, or the keyless allowance does not cover this call. Give the user the setup line for their client (connector, `npx @magpieiq/cli init`, `magpie login`). Never ask for a key in chat. | | `insufficient_credits` | 402 | Stop. Tell the user the balance is too low; if `next` has a top-up link, pass it on. Do not retry or split the call to squeeze under the balance. | | `spend_cap_reached` | 402 | This key hit the daily cap the user set. Tell them; they can raise it in the dashboard. Do not look for another key. | | `estimate_exceeded` | 409 | The price moved above your `max_credits`. Re-estimate, show the new figure, and run again only with the user's approval. | | `not_in_coverage` | 404 | The dataset has no data for that scope. Check the catalog for the nearest country, platform, category or month, or offer a live scrape instead. Never present it as zero sales. | | `rate_limited` | 429 | Wait (respect `Retry-After`) and retry later. On the keyless tier, signing in lifts the limit. | | `deprecated` | 410 | The endpoint is past its sunset. Call the one named in `use_instead` / `next`. | | `invalid_request` | 400 | A parameter is wrong. Re-read the endpoint's `input_schema` with describe and fix it; do not guess again. | | `unavailable` | 503 | Temporary. Retry in a minute; if it persists, say so rather than inventing data. | ## Currently available Generated from the live endpoint registry at 2026-10-31 16:38 UTC and refreshed about every ten minutes. `GET https://api-staging.magpieiq.com/agent/v1/capabilities` is authoritative; describe an entry before calling it. Prices are in credits; the rate card converts them: https://data-staging.magpieiq.com/pricing/api Dataset coverage right now: countries ID, MY, PH, SG, TH, VN; marketplaces Blibli, Lazada, Shopee, Tiktok, Tokopedia, Tokopedia | Shop; months 2025-01 to 2026-07 (ranges differ by market and category; the catalog has the exact range for each). ### Dataset: answers, rows, exports - `data/answer_brand_share` · Which brands lead a category: brand GMV share, GMV (USD + local), units, reviews and SKU count for a country, platform(s), category at any level and month or month range. · 100 credits per call · `POST /agent/v1/answer/brand_share` - `data/answer_category_trend` · Is a category growing: monthly GMV (USD + local), units, average price, SKU count and month-on-month / year-on-year growth (default: the trailing 12 months), or with by=subcategory the sub-categories… · 100 credits per call · `POST /agent/v1/answer/category_trend` - `data/answer_seller_share` · Which sellers (shops) lead a category: seller GMV share, GMV, units and an official-store flag, plus the official-store share of the whole scope. · 100 credits per call · `POST /agent/v1/answer/seller_share` - `data/answer_top_skus` · Best-selling SKUs of a category in ONE month: name, brand, seller, price, discount, rating, stock, units, GMV and share of the category's GMV. · 100 credits per call · `POST /agent/v1/answer/top_skus` ### Scraping: live pages #### Amazon - `scrape/amazon_pdp` · Bulk Amazon product-detail scrape (per item). · 5 credits per item · `POST /agent/v1/scrape/amazon_pdp` · async (returns a job) - `scrape/amazon_search` · Bulk Amazon search/category listing scrape (per item). · 5 credits per item · `POST /agent/v1/scrape/amazon_search` · async (returns a job) #### Blibli - `scrape/blibli_merchant` · Blibli merchant product listing scrape (per request). · 1 credit per request · `POST /agent/v1/scrape/blibli_merchant` · async (returns a job) - `scrape/blibli_pdp` · Blibli product-detail (per request). · 1 credit per request · `POST /agent/v1/scrape/blibli_pdp` · async (returns a job) #### Lazada - `scrape/lazada_category_tree` · Lazada category tree (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_category_tree` · async (returns a job) - `scrape/lazada_keyword_tracking` · Lazada keyword tracking logs (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_keyword_tracking` · async (returns a job) - `scrape/lazada_product_details` · Lazada product detail (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_product_details` · async (returns a job) - `scrape/lazada_product_reviews` · Lazada product reviews via the V1 API (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_product_reviews` · async (returns a job) - `scrape/lazada_product_reviews_v2` · Lazada product reviews via the H5 V2 API (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_product_reviews_v2` · async (returns a job) - `scrape/lazada_product_search` · Lazada product search by keyword/category/seller (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_product_search` · async (returns a job) - `scrape/lazada_seller_details` · Lazada seller details (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_seller_details` · async (returns a job) - `scrape/lazada_seller_directory` · Lazada seller directory (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_seller_directory` · async (returns a job) - `scrape/lazada_seller_promotions` · Lazada seller campaigns and promoted products (per request item). · 6 credits per request · `POST /agent/v1/scrape/lazada_seller_promotions` · async (returns a job) #### Mercado Libre - `scrape/mercadolibre_pdp` · Regional Mercado Libre product-detail scrape (per successful product). · 1 credit per item · `POST /agent/v1/scrape/mercadolibre_pdp` · async (returns a job) - `scrape/mercadolibre_search` · Regional listing scrape (per completed listing URL, including configured page/product limits). · 1 credit per item · `POST /agent/v1/scrape/mercadolibre_search` · async (returns a job) #### Shopee - `scrape/shopee_app_search_items` · Shopee search/category scrape (per requested page). · 10 credits per page · `POST /agent/v1/scrape/shopee_app_search_items` · async (returns a job) - `scrape/shopee_get_pc` · Shopee product-catalog fetch (flat per request). · 8 credits per request · `POST /agent/v1/scrape/shopee_get_pc` · async (returns a job) - `scrape/shopee_pgnt` · Shopee search/category/merchant listing scrape (per successful page). · 6 credits per page · `POST /agent/v1/scrape/shopee_pgnt` · async (returns a job) - `scrape/shopee_product_ratings` · Shopee product ratings (per successful page). · 6 credits per page · `POST /agent/v1/scrape/shopee_product_ratings` · async (returns a job) - `scrape/variant_sold` · Shopee variant-sold scrape (per item). · 10 credits per item · `POST /agent/v1/scrape/variant_sold` · async (returns a job) #### Shopify - `scrape/shopify_pdp` · Bulk Shopify product-detail scrape (per product URL). · 2 credits per item · `POST /agent/v1/scrape/shopify_pdp` · async (returns a job) - `scrape/shopify_search` · Bulk Shopify collection/search listing scrape (per input URL). · 2 credits per item · `POST /agent/v1/scrape/shopify_search` · async (returns a job) #### TikTok Shop - `scrape/tiktok_pdp` · Bulk TikTok product-detail scrape (per item). · 5 credits per item · `POST /agent/v1/scrape/tiktok_pdp` · async (returns a job) #### Tokopedia - `scrape/tokopedia_merchant` · Tokopedia merchant product listing scrape (per request). · 1 credit per request · `POST /agent/v1/scrape/tokopedia_merchant` · async (returns a job) - `scrape/tokopedia_pdp` · Tokopedia product-detail (per request). · 1 credit per request · `POST /agent/v1/scrape/tokopedia_pdp` · async (returns a job) - `scrape/tokopedia_search` · Tokopedia search/category listing scrape (per request). · 1 credit per request · `POST /agent/v1/scrape/tokopedia_search` · async (returns a job) --- ## House rules Every one of these exists because breaking it produced a wrong answer or an unnecessary charge. **1 · Never spend before costing.** Every paid endpoint accepts `estimate=true`, which returns the exact credits and charges nothing. Answer-tier calls are flat and small; for rows, scrapes and exports, show the estimate and wait for the user's approval (or a budget they gave you) before running. A full-category export and a top-N slice of the same scope routinely differ by a factor of a hundred, so show both when you quote an export. **2 · Ask the registry and the catalog what exists.** Use names, parameters and category strings verbatim from describe and the catalog. An invented category name errors; omitting a category on an export means every category in the market, which is expensive and rarely intended. **3 · Know what runs unattended.** Answer, rows and bulk export calls, and Shopee product pages, have no approval step and no working-hours window, so a schedule can rely on them. Shopee search, shop listings and variant sales are approval-gated and process only Monday-Friday, 08:00 to 17:00 Asia/Jakarta (WIB). Never build an unattended job on those three. **4 · Poll on a deadline and hand back what you cannot finish.** See *Async jobs*. Record every job id at submit time; never resubmit a running job. **5 · Persist raw results before analysing.** Write a downloaded file or a large result to disk first. Running it again costs credits again. **6 · Report what failed and what is missing.** Multi-URL scrapes sometimes fail the last few URLs in submission order, so put the URLs that matter most first, and never present a total without saying how many inputs are missing from it. A `not_in_coverage` is "no data", not zero. **7 · Say how old the data is.** Dataset answers carry `meta.as_of`; scraped rows carry `scraped_at`. Quote them. --- ## Scraping specifics The agent surface (`/agent/v1/scrape/{endpoint}`) wraps the Scraping API (`https://api-staging.magpieiq.com/scraping`, submit-then-poll, results as signed URLs to gzipped JSONL). Find the endpoint for a marketplace and action with `GET /agent/v1/capabilities?product=scrape&marketplace=`; the descriptions name the underlying operation (`get_pc`, `search-items`, `merchant-items`, `variant_sold_v1`, Tokopedia `search`, Lazada `product-reviews`, ...), which the recipes below refer to. A Shopee product URL ends in `-i.{shop_id}.{item_id}`; that is where those two values come from. Sorting and filtering on search and shop pages are expressed **in the URL**, exactly as on the site: `?page=0&sortBy=sales` for best-sellers, `sortBy=ctime` for newest. The market is set by the domain: shopee.co.id, shopee.com.my, shopee.co.th, shopee.ph, shopee.sg, shopee.vn. ### How long a job takes There is no published completion time and the spread is wide: the same Shopee product page has completed in under 7 minutes on one submission and stayed pending for more than 24 hours on another, and both finished in the end. A slow job is usually queued, not lost, and for the first several minutes you cannot tell the two apart by waiting longer. - **Pick the deadline from the user's situation, not from the API.** If someone is waiting in conversation, a few minutes is the whole budget. Say what you are waiting for before you start. - **When the deadline passes, hand the id back and stop.** "Job `37933300` is still running; fetch it later with get_job" is a useful answer. Silence while you poll is not. - **A job stuck for hours is not yours to retry.** Resubmitting pays twice for the same product. Keep the id and come back; escalate to the Magpie team if it is still pending the next day. - **For more than a handful of items, prefer bulk plus a webhook** so nothing depends on your session staying alive. Known failure that looks like slowness: Tokopedia, TikTok Shop and Blibli bulk jobs can reach `completed` with successes counted and still return no download URL and *"Results are being compiled into a JSONL file"*. The scrape worked and the packaging did not; report the id and stop, polling will not clear it. ### Webhooks One event exists, `get_pc_bulk.completed`, for Shopee bulk product pages only; everything else is polled. It is configured in the dashboard (Scraping jobs page: set one HTTPS endpoint, copy the signing secret, send a test delivery); an agent cannot set it up with an API key. Deliveries are signed with HMAC-SHA256 over `timestamp + "." + raw body`, arrive at least once (deduplicate on `id`), retry for five attempts over about an hour and a half, time out after 10 seconds and do not follow redirects. `download_url` is time-limited; fetch the job again for a fresh one. Full reference: ### Parsing Shopee responses Normalized rows cover title, price, stock, rating, reviews and seller. When you need fields beyond those, request the raw payload (see describe) or read the Scraping API's JSONL, and use these paths. They are **not in the OpenAPI spec**; they come from running the endpoints. Verify against an actual response, and say so if a path is missing rather than silently reporting zero. - Iterate `response['search_items']['items']` directly. Do not recurse looking for fields. - Product name is on the wrapper at `item_card_displayed_asset.name`, not the inner object. - Units sold is the numeric `item_card_display_sold_count.monthly_sold_count`. The `_text` version is a bucket like "10k+" and makes every listing look identical. - Price is `item_card_display_price.price`, **divided by 100000**. - Shop is `item_data.shop_data.shop_name`, id `item_data.shopid`, location `item_card_displayed_asset.shop_location`. - **Deduplicate by `itemid` before summing anything.** `merchant-items` behaves differently: its numeric sold field is `0`, so read `monthly_sold_count_text`, which is exact there. It also repeats the same item across a shop's category tabs, so deduplicate or you will overcount by roughly half. Lazada endpoints each validate their own URL pattern, and a URL accepted by one is often rejected by another. Read the error and try the pattern it names rather than assuming the endpoint is broken. --- ## Bulk export (files only) Use it when the user wants a file, not an answer. `POST https://api-staging.magpieiq.com/data/v1/exports` with `country`, `category_3` (array, catalog strings verbatim), `platforms`, `date_from` / `date_to` (`YYYY-MM`), optional `top` (top-N SKUs by GMV per category, platform and month), `format` (`parquet` or `csv`), `idempotency_key`, and `"estimate": true` for the free dry run. The estimate quotes both the full scope and the top-N slice; show both. The real call returns an export id; poll `GET /data/v1/exports/{id}` and download the signed URLs. Discover the scope first with `GET /data/v1/exports/catalog`. Two catalogue behaviours that trip people: Indonesia lists `Tokopedia | Shop` alongside `Tokopedia` (a surface inside Tokopedia, not another marketplace; use the string verbatim but do not report it as a sixth marketplace), and one Indonesian entry has empty category names (skip it when looping). --- ## Recipes ### 1.1 · Category market share and trend *Who leads a category, how concentrated it is, how that has moved.* Dataset, answer tier. Call `data/answer_brand_share` for the scope across the months asked (estimate first if the user has not agreed to spend), and `data/answer_category_trend` for the category's size and growth. Report the ten largest brands by GMV and their share in the most recent month; how each share moved since the first month; realised price per unit for each (GMV divided by units) against the category average as an index with the average at 100; and any brand whose share moved more than a third either way. Flag any month whose totals are sharply out of line with its neighbours as a likely collection artefact rather than a market move. ### 1.2 · Price architecture for a new entry *What price bands exist, and which are crowded.* Dataset, answer tier. Use `data/answer_brand_share` (and `data/answer_top_skus` for SKU-level prices) for the most recent full month. Compute realised price per unit per brand as GMV divided by units. Bucket brands into price deciles against the category average and report, per decile, how many brands sit there and what share of category GMV they hold together. Look for a band holding meaningful demand but few competing brands. **Say explicitly if there is no such gap rather than manufacturing one.** If the user wants every SKU, page through `data/rows` after an estimate. ### 1.3 · Is this category worth entering *Size, concentration, and whether the top is defended.* Dataset, answer tier. From `data/answer_brand_share` for the most recent month, report the combined share of the top ten brands, the number of brands needed to reach half of category GMV (say so if the returned rows do not reach half), and the realised price index of the top three against the category average. Add the category's growth from `data/answer_category_trend`. Then classify: concentrated and premium (leaders hold high share at high price), concentrated and cheap (leaders hold share by undercutting), or fragmented (no brand holds meaningful share). Each implies a different entry cost; give the reasoning, not just the label. ### 2.1 · Price match: is anyone undercutting us *Whether a published price is genuinely the lowest.* Scraping, **runs unattended**. For each product the user supplies with an offer price and a Shopee URL, take `shop_id` and `item_id` from the URL and scrape the product page (`get_pc`). Return a table of every product where the Shopee price is at or below theirs, with both prices and the gap. In the raw payload `is_lowest_price_at_shopee` is reliable only on each entry in `item.models`; the item-level value comes back `null`. Report the per-model value alongside each row. At scale, use the bulk product-page endpoint with a CSV, and its schedules to run it nightly. Estimate first: it is billed per product. ### 2.2 · Who is selling in this category right now *The shops that dominate a category today, official versus reseller.* Scraping, **approval-gated**. Scrape Shopee search pages (`search-items`) sorted by sales, several keywords covering the category, at most 50 URLs. Expect `queued` with `runs_after` outside the approval window. Parse per the rules above, compute GMV per listing as units × price, aggregate by shop, and report the twenty largest shops with their share of the total observed, listing count, average realised price and location. Separate official brand stores from resellers where the shop name makes that clear. Report how many submitted URLs returned data and how many failed. ### 2.3 · Unauthorised sellers and channel control *Who is selling your brand without permission, and at what price.* Scraping, **approval-gated**. Run 2.2 using the brand name and its common misspellings as keywords. Aggregate by shop. Given a list of authorised distributors, produce two tables: every shop selling the brand that is not on the list, with observed GMV, listing count and location, sorted by size; and any authorised shop whose realised price sits more than twenty per cent below the stated floor. **Do not accuse anyone of counterfeiting.** Report what the listing data shows: who is selling, at what price, from where. ### 2.4 · Enumerate a competitor's whole shop *A rival's full catalogue, pricing, and what actually sells.* Scraping, **approval-gated**. Scrape shop pages (`merchant-items`), at most 50 per request. The numeric sold field is 0 here, so read `monthly_sold_count_text`, and deduplicate by `itemid` first or the total roughly doubles. Report listing count, total observed monthly GMV, the twenty listings carrying most of it, and the price distribution. **Do not compare this shop's total against a market total built from keyword search.** The shop is completely enumerated and the market is not, so a share computed that way is a ceiling, not an estimate. If asked for a share, say so, or use the dataset's `data/answer_seller_share` instead. ### 2.5 · New product and assortment watch *What competitors have just launched.* Scraping, **approval-gated**. Run 2.2 with `sortBy=ctime` instead of `sortBy=sales`. Run weekly and keep the previous week's `itemid` list. Each week report listings new since the last run, grouped by shop, with price and any units already recorded. Highlight any new listing that took meaningful volume immediately; that usually means a launch with paid support behind it. ### 2.6 · SKU-level variant demand *Which size, shade or pack actually sells.* Scraping, **approval-gated**. Scrape variant sales (`variant_sold_v1`) with `items: [{item_id, shop_id, country}]`, one country per request, billed per item and charged only on approval (a rejection or the 24-hour timeout costs nothing). Report sold count per variant, ranked, with each variant's share of the product total, and say whether demand concentrates in one or two variants or spreads evenly; that determines whether a range needs pruning. ### 2.7 · Review harvesting for product feedback *What buyers complain about, in their own words.* Scraping, Lazada only. Neither Lazada review endpoint accepts a product page URL; both return `422` if you pass one. Take the numeric item ID from the product URL (`…-i5133872551.html` gives `5133872551`) and submit it in the form each endpoint expects. Prefer V1 (`product-reviews`) with `{"requests":[{"url":"https://my.lazada.co.id/pdp/review/getReviewList?itemId=5133872551"}]}` (`my.lazada.`, so `co.id` for Indonesia). The body carries `reviewTotal` and a `reviews` list; each review has `rating`, `reviewContent`, `reviewedAt`, `reviewerName`, `skuFields` and `isPurchased`. One call returns five reviews; add `&pageNo=2`, `&pageNo=3` and so on for later pages. Every page is billed, so agree a page budget with the user before reading a product with hundreds of reviews. V2 (`product-reviews-v2`) also works: `{"requests":[{"url":"https://acs-m.lazada.co.id/h5/mtop.lazada.review.item.getpcreviewlist/1.0/","method":"POST","payload":{"itemId":5133872551,"pageNo":1}}]}`. `itemId` must be an integer; as a string the job is accepted and then fails upstream. V2 puts reviews under `data.module.reviews` with the text in `reviewContentList`, tags under `data.module.impressionTags` and paging under `data.paging`. Group reviews by star rating, summarise recurring complaints in the one and two star reviews and recurring praise in the five star ones. Quote a few representative lines verbatim, and say how many reviews were read out of `reviewTotal`. ### 2.8 · Keyword rank tracking *Where your listings rank when shoppers search a keyword, and how that moves.* Scraping, **runs unattended**, Tokopedia. Scrape Tokopedia search (`search`) with `{"items":[{"list_type":"search","keyword":"gitar yamaha","product_limit":60}]}`, up to 50 keywords per request. Keyword search needs no approval, so it can run on a schedule. Download each result file promptly (the signed URL expires) and save it. The `products` array is in result order, so a listing's rank is its position, counting from 1. Match your own listings by `id` or `shop.id` and record rank per keyword per run. Each product also carries `price.number`, `rating`, `category.breadcrumb` and a sold label in `labelGroups` such as `"1rb+ terjual"` (a lifetime bucket, not monthly sales). `params.totalData` is the number of results. The request has no sort parameter (`params.ob` is `5`), so compare rank run over run in that order and do not present it as exactly the default shopper view. Treat a product with a filled `ads.id` as sponsored and keep it out of organic rank. To find keywords worth tracking, Lazada `keyword-tracking` with `{"requests":[{"url":"https://www.lazada.co.id/tag-order-last-30days-morethan0.xml?limit=20&offset=0"}]}` returns search terms that led to orders in the last 30 days: plain strings, no volumes, and very large. Save it to disk and filter it to your product words before using any of it as seeds. --- ## Choosing between them | The question | Use | Recipe | |---|---|---| | Brand shares and how they moved over months | Answer tier: brand share, category trend | 1.1 | | Price bands, where a category is crowded | Answer tier: brand share, top SKUs | 1.2, 1.3 | | The SKUs behind an answer | Rows | 1.2 | | Current price on specific listings | Scrape: Shopee product page | 2.1 | | Who is selling right now | Scrape: Shopee search | 2.2, 2.3 | | A competitor's full catalogue | Scrape: Shopee shop listing | 2.4 | | A market or category the dataset does not cover | Scrape | 2.2 | | Where your listings rank for a keyword, over time | Scrape: Tokopedia search | 2.8 | | A file for a warehouse or notebook | Bulk export | | | Anything that must run unattended on a schedule | Answer, rows, export, or Shopee product pages | 2.1, 2.8 | **The dataset is not universal.** Coverage is category by market, not country-wide, and a category present in one market is frequently absent in the next. Check the catalog before promising a scope. Where the dataset holds nothing, scraping still reaches the marketplace directly. --- Docs · rate card · short brief Use subject to the Magpie IQ terms of service.