Data dictionary
Data API · monthly dataset32 fields per row — what every export and metrics response is made of.
Row grain
One row per product × platform × country × month. Monthly metrics (gmv_monthly*, sold_monthly, review_monthly) are that month's figures; stock and rating are the month-end snapshot, while price is the lowest price recorded during the month.
Money fields are in the market's local currency unless the name says _usd. A full export delivers the 32 fields below as Parquet or CSV; rank is added only to Top-N slices — estimate a pull →
Download a real sample — 100 rows, exactly as a Top-N export ships:Parquet (38 KB)CSV (63 KB)ID · Shopee · Facial Serum · 2026-05 · top-100 by GMV — all 32 fields plus rank. A full export ships the same 32 without it.
Product identity
What the row is about — stable identifiers plus display fields.
| Field | Type | Description | Nullable | Origin |
|---|---|---|---|---|
| product_id | STRING | The listing's own ID on its platform, carried through unchanged from source. The primary key for joining every other field back to one specific listing. | No | scraped |
| sku_name | STRING | The listing's title exactly as scraped, with no cleanup — good for a human eyeballing the row, but wording and casing vary listing to listing, so use sku_type_complete instead when matching or grouping products. | No | scraped |
| sku_type_complete | STRING | Standardized product type, matched against a maintained reference dictionary rather than derived from the raw title — better for grouping identical products than sku_name. Shows a placeholder when no dictionary match is found. | No | labelled — Magpie's internal library |
| brand | STRING | Brand name, resolved by matching the listing's product/merchant identity against Magpie's brand reference data rather than read directly off the title. Shows the placeholder when no brand has been confirmed yet. | No | labelled — Magpie's internal library |
| image | STRING | Direct URL to the listing's main product photo, as hosted by the platform — carried straight through from the raw scrape. | Yes | scraped |
| url | STRING | Canonical listing URL on the platform. | No | scraped |
| created_date | DATE | Date the listing first appeared on the platform. (Shopee & Tokopedia only — not currently captured for Lazada, TikTok, or Blibli; empty for listings on those marketplaces.) | Yes | scraped |
| flag_GWP | BOOL | True when the listing is a gift-with-purchase / bundle freebie. | No | labelled |
Classification
Where the row sits — market, platform, category tree, month.
| Field | Type | Description | Nullable | Origin |
|---|---|---|---|---|
| country | STRING | Market code: ID, TH, VN, SG, PH, MY. Sets the local currency of non-USD money fields. | No | scraped |
| ecommerce_platform | STRING | Marketplace: Shopee, Lazada, Tiktok, Tokopedia, 'Tokopedia | Shop', Blibli. | No | scraped |
| category_1 | STRING | Top-level category (e.g. Beauty & Personal Care). | No | labelled — Magpie's internal category |
| category_2 | STRING | Second-level category (e.g. Skincare). | No | labelled — Magpie's internal category |
| category_3 | STRING | Leaf category (e.g. Facial Serum) — the export's atom grain. | No | labelled — Magpie's internal category |
| month | DATE | Calendar month representing the aggregation period. Identifies which month's daily/raw scrape records were rolled up to produce this row (e.g. sold_monthly = SUM of daily sold within the month). | No | system |
Pricing & discounts
Listed prices and realized (sold) averages. Local currency unless the field says USD.
| Field | Type | Description | Nullable | Origin |
|---|---|---|---|---|
| price | FLOAT64 | The lowest price recorded for the listing during the month, local currency — the cheapest price point observed, not a price taken specifically at month's end. Empty when the listing never displayed a price during the month (e.g. a pre-launch or 'coming soon' page); this shows up mostly on small, low-volume SKUs. | Yes | scraped |
| price_usd | FLOAT64 | Price converted to USD using a fixed exchange rate per country, not a live daily rate — treat cross-market USD comparisons as directionally useful rather than accounting-precise. Empty whenever price is, for the same reason. | Yes | scraped |
| before_discount_price | FLOAT64 | Pre-discount reference price, calculated from the price and discount columns rather than scraped from the listing's strike-through label. Equals price when no discount applies. | Yes | derived — from price and discount |
| discount | FLOAT64 | Listed discount as a fraction (0.70 = 70% off). | Yes | scraped |
| avg_price_sold | FLOAT64 | Average realized selling price across the month's sales, local currency. | No | derived — from gmv_monthly and sold_monthly |
Sales & performance
The analysis metrics — what actually moved this month.
| Field | Type | Description | Nullable | Origin |
|---|---|---|---|---|
| sold_monthly | INT64 | Units sold in this month. | No | scraped |
| gmv_monthly | FLOAT64 | GMV for this month, local currency (≈ units × realized price). Empty when the listing carried no price to value the units against (see price) — the same pre-launch / 'coming soon' small SKUs. An empty GMV means unknown, not zero: exclude those rows rather than summing them as 0. | Yes | scraped |
| gmv_monthly_usd | FLOAT64 | GMV for this month in USD — the cross-market analysis metric. Empty whenever gmv_monthly is, for the same reason. | Yes | scraped |
| review_monthly | INT64 | Reviews received in this month. | Yes | scraped |
| rating | FLOAT64 | Listing's rating (0–5) at month end. | Yes | scraped |
| stock | INT64 | Listing-level stock at month end. | Yes | scraped |
Merchant
Who sells it and from where — official-store detection and regional cuts.
| Field | Type | Description | Nullable | Origin |
|---|---|---|---|---|
| merchant_id | STRING | Platform's shop identifier. | No | scraped |
| merchant_name | STRING | Shop display name. | No | scraped |
| merchant_url | STRING | Shop URL on the platform. | Yes | scraped |
| merchant_badge | STRING | Platform badge, e.g. 'Shopee Mall', 'Star Seller' — official signal. | Yes | scraped |
| merchant_cities | STRING | Shop's city (as declared on the platform). | Yes | labelled — Magpie's internal library |
| merchant_province | STRING | Shop's province/state. | Yes | labelled — Magpie's internal library |
| merchant_region | STRING | Coarse region rollup (e.g. Jawa, Sumatera). | Yes | labelled — Magpie's internal library |
Top-N files only
Present only when you order a Top-N slice — a full export does not include this column.
| Field | Type | Description | Nullable | Origin |
|---|---|---|---|---|
| rank | INT64 | Position within this category × platform × country × month, 1 = highest GMV. Computed when the Top-N slice is built, ordering by gmv_monthly_usd descending (ties broken by product_id), so rank 1..N is exactly the set you paid for. Ranks are per file — they restart at 1 for every category and month, and are not comparable across them. | No | derived — from gmv_monthly_usd ordering |
Top-N export files additionally carry a rank column (1 = highest GMV in the atom). Schema questions? The live source of truth is the API reference.