# How to Monitor Competitor Prices With n8n (Free Template)
> Monitor competitor prices with n8n: find rival listings by model number, read prices from JSON-LD and get a weekly Slack report of undercuts.
*Last tested on n8n 2.45, November 2026.*
Checking competitor prices manually in a spreadsheet goes stale by Tuesday, while dedicated price-monitoring apps charge [US$99 to US$399 a month](https://prisync.com/pricing/) and still make you paste in every competitor URL yourself. Even most scraping templates get the economics backwards by routing every check through a paid headless browser API when a plain HTTP request can read the structured data on most stores for free.
By the end of this guide, you'll have a four-lane n8n workflow that searches Google for your product model numbers to discover competitor listings automatically, re-reads their prices politely on a schedule using JSON-LD structured data (with Firecrawl only as a fallback for blocked pages), and sends a weekly Slack report of every product where you're being undercut.
- Discovers competitor product URLs automatically by searching your MPNs via DataForSEO (including Google Shopping panel offers), or runs free from a list of URLs you paste in.
- Rejects lookalike model suffixes (`DS-2CD2366G2-IU` vs `DS-2CD2366G2-I`) and multi-packs (`2-Pack`), checks `robots.txt`, and reads prices from JSON-LD with zero AI token costs.
- Sends a weekly Slack report sorted by the biggest price undercuts, plus a health alert if sites start blocking your requests.
## What you'll build
Every Monday morning, the workflow posts a compact Slack digest listing the products where a verified competitor is cheaper than you beyond your threshold, showing your price, the competitor's price, the amount and percentage gap, and a direct link to their page.
Under the hood, the workflow separates discovery (finding which stores sell your exact model number) from recapture (re-reading known URLs). Searching once every two to four weeks and re-reading known URLs with a plain HTTP request is what keeps the monthly bill down to a few dollars.
## What you need
- n8n with Data Tables: the free self-hosted Community Edition (recommended) or n8n Cloud.
- A Shopify app with the `read_products` scope, made in the Shopify Dev Dashboard, plus its client ID and secret. Shopify stopped issuing permanent admin tokens for new custom apps in January 2026, so the workflow requests a fresh 24-hour token on each run.
- A DataForSEO account for automated Google SERP discovery ([US$0.002 per 10 results](https://dataforseo.com/pricing/serp/google-organic-serp-api), [US$50 minimum payment](https://dataforseo.com/help-center/minimum-payment)), or skip DataForSEO entirely and paste known competitor URLs into the table for the free path.
- Firecrawl (optional, [free tier includes 1,000 credits a month](https://www.firecrawl.dev/pricing)) as a fallback for pages behind bot walls, or Crawl4AI if you self-host.
- Slack (or Gmail) for the weekly undercut digest. No AI model is required.
## How it works
1. **Setup (Manual Trigger):** Creates `competitor_listings`, `competitor_prices`, and `competitor_searches` Data Tables.
2. **Discover (Schedule Trigger, daily batch):** Pulls active products and MPNs from Shopify, searches Google via DataForSEO every 14 to 30 days, filters for whole-token MPN matches (rejecting multi-packs and lookalike suffixes), and upserts verified listings.
3. **Recapture (Schedule Trigger, every 6 hours):** Checks `robots.txt`, re-reads known product URLs politely with a Wait node, parses JSON-LD `Product`/`Offer` prices, and uses Firecrawl only for blocked pages.
4. **Report (Schedule Trigger, weekly):** Compares verified competitor prices against yours and posts the top 20 undercuts to Slack.
## Step 1: Create the tables and settings
In the Setup lane, add a **Manual Trigger** connected to three **[Data table](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.datatable/tables/)** nodes set to **Table -> Create** (`Reuse Existing Tables` enabled). Remember that [n8n Data Tables share a 200 MiB default limit per instance](https://docs.n8n.io/build/work-with-data/data-tables/), so we only insert a row into `competitor_prices` when a price actually changes:
| Table | One row per | Columns |
| --- | --- | --- |
| `competitor_listings` | product per competitor page | **String:** `listing_key`, `product_key`, `sku`, `mpn`, `gtin`, `product_title`, `url`, `domain`, `seller`, `listing_title`, `matched_by`, `currency`, `status`, `method`
**Number:** `confidence`, `last_price`, `last_change`, `pending_price`
**Boolean:** `in_stock`
**Date:** `last_checked_at` |
| `competitor_prices` | price change (the price history) | **String:** `listing_key`, `product_key`, `domain`, `method`
**Number:** `price`
**Boolean:** `in_stock`
**Date:** `captured_at` |
| `competitor_searches` | product, with when it was last searched | **String:** `product_key`, `query`
**Number:** `listings_found`, `rejected`, `cost_usd`
**Date:** `last_searched_at` |
Next, add an **Edit Fields (Set)** node in JSON mode at the start of each operational lane, named `Discover settings`, `Recapture settings` and `Report settings`:
```json
{
"store_domain": "your-store.myshopify.com",
"own_domain": "yourstore.com",
"exclude_domains": [],
"currency": "USD",
"locale": "en-US",
"country": "US",
"search_location": "United States",
"language_code": "en",
"search_depth": 20,
"prices_include_tax": false,
"market_shows_tax": false,
"tax_rate": 0,
"tax_exclusive_domains": [],
"mpn_source": "metafield",
"mpn_metafield_namespace": "custom",
"mpn_metafield_key": "mpn",
"user_agent": "YourStorePriceCheck/1.0 (+https://yourstore.com/price-check)",
"robots_agent": "YourStorePriceCheck",
"discovery_batch": 100,
"research_with_listings_days": 30,
"research_without_listings_days": 14,
"recapture_every_hours": 120,
"recapture_batch": 300,
"min_delay_seconds": 10,
"max_price_move": 0.6,
"use_firecrawl": true,
"firecrawl_per_run": 8,
"min_confidence": 0.9,
"max_price_age_days": 7,
"min_price_ratio": 0.5,
"undercut_threshold": 0.02,
"report_limit": 20,
"slack_channel": "#pricing"
}
```
Set `market_shows_tax` to true where shoppers see tax-inclusive prices, and list any trade or B2B competitors whose JSON-LD prices exclude tax in `tax_exclusive_domains`.
## Step 2: Pull your products and their identifiers
Searching Google by your full product title brings back noisy results and blog posts, whereas searching by the manufacturer part number (MPN) or GTIN surfaces the exact product pages on rival stores. In the Discover lane, add a daily **Schedule Trigger** and `Discover settings`, then an **HTTP Request** node called `Get Shopify token` that swaps your client ID and secret for a 24-hour token through [the client credentials grant](https://shopify.dev/docs/apps/build/authentication-authorization/access-tokens/client-credentials-grant):
| Setting | Value |
| --- | --- |
| Method | POST |
| URL | `https://{{ $json.store_domain }}/admin/oauth/access_token` |
| Authentication | Generic Credential Type > Custom Auth: `{"body": {"client_id": "", "client_secret": ""}}` |
| Send Body | Form Urlencoded, `grant_type` = `client_credentials` |
| Settings > Retry On Fail | On |
Because n8n's built-in Shopify node doesn't return metafields, use a second **HTTP Request** node, `Get my products`, with that token calling Admin GraphQL API `2026-10` ([`productVariants` query](https://shopify.dev/docs/api/admin-graphql/2026-10/queries/productVariants)):
| Setting | Value |
| --- | --- |
| Method | POST |
| URL | `https://{{ $('Discover settings').first().json.store_domain }}/admin/api/2026-10/graphql.json` |
| Send Headers | `X-Shopify-Access-Token` = `{{ $('Get Shopify token').first().json.access_token }}` |
| Send Body | JSON (Using JSON) |
| Pagination > Mode | Update a Parameter in Each Request |
| Pagination > Type | Body |
| Pagination > Name | `variables` |
| Pagination > Value | `{{ { cursor: $response.body.data.productVariants.pageInfo.endCursor, ns: $('Discover settings').first().json.mpn_metafield_namespace, key: $('Discover settings').first().json.mpn_metafield_key } }}` |
| Pagination Complete When | Other |
| Complete Expression | `{{ !$response.body.data.productVariants.pageInfo.hasNextPage }}` |
Its JSON body carries the query and the `variables` the pagination updates:
```js
{{ JSON.stringify({
query: `query Variants($cursor: String, $ns: String, $key: String!) {
productVariants(first: 100, after: $cursor, query: "product_status:active") {
pageInfo { hasNextPage endCursor }
nodes {
id
sku
price
title
barcodes(first: 5) { nodes { value } }
mpn: metafield(namespace: $ns, key: $key) { value }
product {
title
vendor
mpn: metafield(namespace: $ns, key: $key) { value }
}
}
}
}`,
variables: { cursor: null, ns: $('Discover settings').first().json.mpn_metafield_namespace, key: $('Discover settings').first().json.mpn_metafield_key }
}) }}
```
Connect a **Data table** Get node for `competitor_searches` (`Get search log`, with Execute Once and Always Output Data on so the first, empty run doesn't stop the lane), then a **Code** node that keeps variants with an MPN or barcode, filters for items not searched within `research_with_listings_days` (`30` days, or `14` for products with no listings), prioritises never-searched items, and slices to `discovery_batch` (`100`):
```js
// One item per variant that has a model number or barcode and is due a search.
const s = $('Discover settings').first().json;
const day = 864e5;
const log = {};
for (const r of $('Get search log').all()) log[r.json.product_key] = r.json;
const due = [];
for (const page of $('Get my products').all()) {
for (const v of page.json.data.productVariants.nodes) {
const mpn = String((s.mpn_source === 'sku' ? v.sku : (v.mpn?.value || v.product.mpn?.value)) || '').trim();
const gtin = v.barcodes.nodes.map(b => String(b.value).replace(/\D/g, '')).find(b => b.length >= 8) || '';
if (!mpn && gtin.length < 8) continue; // nothing to match on: see "Make it your own"
const last = log[v.id];
const age = last ? (Date.now() - new Date(last.last_searched_at)) / day : 9999;
const every = last?.listings_found > 0 ? s.research_with_listings_days : s.research_without_listings_days;
if (age < every) continue;
const strong = /\d/.test(mpn) && !/\s/.test(mpn) && mpn.replace(/[^A-Za-z0-9]/g, '').length >= 6;
due.push({ json: { product_key: v.id, sku: v.sku, mpn, gtin, brand: v.product.vendor, age,
title: v.product.title + (v.title === 'Default Title' ? '' : ' ' + v.title),
query: mpn ? (strong ? mpn : `${v.product.vendor} ${mpn}`) : gtin } });
}
}
return due.sort((a, b) => b.json.age - a.json.age).slice(0, s.discovery_batch);
```
## Step 3: Search for each model number
To use the free path, paste competitor URLs into `competitor_listings` (with your variant's `gid://shopify/ProductVariant/…` ID as `product_key`, `status` `ok` and `confidence` 1) and skip to Step 5. Otherwise, call [DataForSEO's Google Organic Live Advanced endpoint](https://docs.dataforseo.com/v3/serp/google/organic/live/advanced/) (`POST https://api.dataforseo.com/v3/serp/google/organic/live/advanced`) in an **HTTP Request** node:
| Setting | Value |
| --- | --- |
| Method | POST |
| URL | `https://api.dataforseo.com/v3/serp/google/organic/live/advanced` |
| Authentication | Generic Credential Type > Basic Auth (your DataForSEO API login and password) |
| Send Body | JSON |
| Options > Batching > Items per Batch | 1 |
| Options > Response > Never Error | On |
| Options > Timeout | 60000 |
| Settings > On Error | Continue |
with this body:
```js
{{ JSON.stringify([{
keyword: $json.query,
location_name: $('Discover settings').first().json.search_location,
language_code: $('Discover settings').first().json.language_code,
depth: $('Discover settings').first().json.search_depth
}]) }}
```
DataForSEO returns both `organic` results (often carrying `price.current` and `price.currency`) and Google Shopping panel offers, giving many listings their first price reading at no extra cost. Never Error turns an empty balance's 402 into a message the next node reads, rather than a stopped run.
## Step 4: Keep only exact matches
Searching for a Hikvision `DS-2CD2366G2-I` also surfaces `DS-2CD2366G2-IU` (built-in mic, higher price), searching `RLC-810A` surfaces `RLC-810A 2-Pack` bundles, and wordy names like `Argus 4 Pro` need the brand (`Reolink`) in the title. Add a **Code** node named `Keep exact matches` to enforce whole-token MPN matching, brand presence on short or spaced MPNs, and pack rejection:
```js
// Turn each search into competitor listings, keeping only exact matches.
const s = $('Discover settings').first().json;
const sent = $('Pick products to search').all();
const results = $input.all();
const skip = [s.own_domain, ...s.exclude_domains].map(d => d.toLowerCase().replace(/^www\./, ''));
const host = u => (String(u).match(/^https?:\/\/([^/?#:]+)/i) || [])[1]?.toLowerCase().replace(/^www\./, '');
const DROP = /^(srsltid|utm_[a-z]+|gclid|variant|currency|country)$/i;
const cleanUrl = u => {
const [base, q = ''] = String(u).split('#')[0].split('?');
const kept = q.split('&').filter(kv => kv && !DROP.test(kv.split('=')[0]));
return base + (kept.length ? '?' + kept.join('&') : '');
};
// Model number as a whole token, any separator: DS-2CD2366G2-I matches "ds 2cd2366g2 i", not DS-2CD2366G2-IU.
const chars = m => String(m).toUpperCase().replace(/[^A-Z0-9]/g, '').split('');
const titleRe = m => new RegExp('(^|[^A-Z0-9])' + chars(m).join('[\\s\\-_/.]?') + '(?![A-Z0-9]|-[A-Z0-9])');
const slugRe = m => new RegExp('(^|[^A-Z0-9])' + chars(m).join('[\\-_.]?') + '(?![A-Z0-9])');
const PACK = /\b(\d+|two|three|four|twin)[\s-]*(pack|pk)\b|\bpack of \d+|\bbundle\b|\bkit\b/i;
function match(q, title, url) {
const T = String(title || '').toUpperCase(), U = String(url || '').split('?')[0].toUpperCase(); // path only: Google links carry the query
if (PACK.test(title) && !PACK.test(q.title)) return null; // a 2-pack isn't your single camera
if (q.gtin && (T + ' ' + U).includes(q.gtin)) return { by: 'gtin', confidence: 1 };
if (!q.mpn) return null;
const strong = /\d/.test(q.mpn) && !/\s/.test(q.mpn) && q.mpn.replace(/[^A-Za-z0-9]/g, '').length >= 6;
if (!strong && !T.includes(String(q.brand).toUpperCase())) return null; // short names need the brand
if (titleRe(q.mpn).test(T)) return { by: 'mpn', confidence: strong ? 1 : 0.9 };
if (slugRe(q.mpn).test(U)) return { by: 'mpn_url', confidence: strong ? 0.85 : 0.75 };
return null;
}
function offersFrom(items = []) {
const out = [];
for (const it of items) {
if (it.type === 'organic' && it.price?.current)
out.push({ seller: it.website_name || it.domain, url: it.url, title: it.title, price: it.price.current, currency: it.price.currency });
if (it.type === 'knowledge_graph')
for (const sub of it.items || []) for (const el of sub.items || [])
if (el.type === 'knowledge_graph_shopping_element' && el.price?.current)
out.push({ seller: el.source, url: el.url, title: el.title, price: el.price.current, currency: el.price.currency });
if ((it.type === 'popular_products' || it.type === 'shopping') && Array.isArray(it.items))
for (const el of it.items) if (el.price?.current)
out.push({ seller: el.seller || el.source, url: el.url, title: el.title, price: el.price.current, currency: el.price.currency });
}
return out;
}
const out = [];
const now = new Date().toISOString();
sent.forEach((p, i) => {
const q = p.json;
const task = results[i]?.json?.tasks?.[0];
if (task?.status_code !== 20000) {
out.push({ json: { kind: 'failed', query: q.query, error: task?.status_message || String(results[i]?.json?.error?.message || 'no response').slice(0, 200) } });
return;
}
let kept = 0, rejected = 0;
for (const o of offersFrom(task.result?.[0]?.items)) {
const d = host(o.url);
const m = d && o.currency === s.currency && !skip.some(x => d === x || d.endsWith('.' + x)) && match(q, o.title, o.url);
if (!m) { rejected++; continue; }
kept++;
const url = cleanUrl(o.url);
out.push({ json: { kind: 'listing', listing_key: q.product_key + '|' + url, product_key: q.product_key,
sku: q.sku, mpn: q.mpn, gtin: q.gtin, product_title: q.title, url, domain: d,
seller: String(o.seller || d).slice(0, 80), listing_title: o.title, matched_by: m.by, confidence: m.confidence,
last_price: Number(o.price), currency: o.currency, in_stock: null, status: 'ok', method: 'search',
last_checked_at: now, last_change: 0, pending_price: 0 } });
}
out.push({ json: { kind: 'search', product_key: q.product_key, query: q.query, last_searched_at: now,
listings_found: kept, rejected, cost_usd: Number(task.cost || 0) } });
});
return out;
```
The URL check reads only the path, because Google Shopping links carry your search in their query string. Connect a **Switch** node, `Route by kind`, on `{{ $json.kind }}`: `listing` goes to a **Data table** Upsert on `competitor_listings` (matching on `listing_key`), `search` to an Upsert on `competitor_searches` (matching on `product_key`), and `failed` to a **Slack** alert with Execute Once on.
## Step 5: Re-read each listing politely
In the Recapture lane, add a **Schedule Trigger** (every 6 hours) followed by a **Data table** Get node that pulls rows from `competitor_listings` where `status` is not `'robots_disallowed'` and `last_checked_at` is older than `recapture_every_hours` (`120` hours):
| Setting | Value |
| --- | --- |
| Must Match | All Conditions |
| Condition 1 | `last_checked_at` Less Than `{{ $now.minus({ hours: $json.recapture_every_hours }).toISO() }}` |
| Condition 2 | `status` Not Equals `robots_disallowed` |
| Order By | `last_checked_at`, Ascending |
| Limit | `{{ $json.recapture_batch }}` |
Before fetching product pages, check each domain's `robots.txt` once per run and space out requests to the same domain so you never hammer a small store's server. Following [RFC 9309 (the Robots Exclusion Protocol)](https://www.rfc-editor.org/rfc/rfc9309), a `4xx` response on `/robots.txt` means you may crawl, a `5xx` response means assume disallow, and the longest matching path wins. Add a **Code** node named `Check robots.txt`:
```js
// Fetch robots.txt once per site, drop what it disallows, and plan a polite gap per site.
// RFC 9309: a 4xx means no rules; a 5xx or no answer means treat the whole site as disallowed.
const s = $('Recapture settings').first().json;
const token = s.robots_agent.toLowerCase();
const originOf = u => (String(u).match(/^(https?:\/\/[^/?#]+)/i) || [])[1]?.toLowerCase();
const pathOf = u => String(u).replace(/^https?:\/\/[^/?#]+/i, '').split('#')[0] || '/';
function parseRobots(txt) {
const groups = [];
let cur = null, lastWasAgent = false;
for (const raw of String(txt).split(/\r?\n/)) {
const m = raw.replace(/#.*$/, '').trim().match(/^([A-Za-z-]+)\s*:\s*(.*)$/);
if (!m) continue;
const key = m[1].toLowerCase(), val = m[2].trim();
if (key === 'user-agent') {
if (!lastWasAgent) groups.push(cur = { agents: [], rules: [], delay: null });
cur.agents.push(val.toLowerCase());
lastWasAgent = true;
continue;
}
lastWasAgent = false;
if (!cur) continue;
if ((key === 'allow' || key === 'disallow') && val) cur.rules.push({ allow: key === 'allow', path: val });
if (key === 'crawl-delay' && !isNaN(Number(val))) cur.delay = Number(val);
}
const mine = groups.filter(g => g.agents.some(a => a !== '*' && token.includes(a)));
const chosen = mine.length ? mine : groups.filter(g => g.agents.includes('*'));
return { rules: chosen.flatMap(g => g.rules), delay: chosen.map(g => g.delay).find(d => d != null) ?? null };
}
const toRegex = p => {
const end = p.endsWith('$');
const body = (end ? p.slice(0, -1) : p).split('*').map(x => x.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*');
return new RegExp('^' + body + (end ? '$' : ''));
};
function allowed(rules, path) {
let best = null; // the longest matching rule wins; Allow wins a tie
for (const r of rules) if (toRegex(r.path).test(path) &&
(!best || r.path.length > best.path.length || (r.path.length === best.path.length && r.allow))) best = r;
return !best || best.allow;
}
const rows = $input.all().map(i => i.json);
const robots = {};
for (const origin of [...new Set(rows.map(r => originOf(r.url)))]) {
try {
const res = await this.helpers.httpRequest({ url: origin + '/robots.txt', returnFullResponse: true,
ignoreHttpStatusErrors: true, timeout: 15000, headers: { 'User-Agent': s.user_agent } });
robots[origin] = res.statusCode >= 500 ? { rules: [{ allow: false, path: '/' }] }
: res.statusCode >= 400 ? { rules: [] } : parseRobots(res.body);
} catch (e) {
robots[origin] = { rules: [{ allow: false, path: '/' }] };
}
}
// Interleave sites so one site is never hit twice inside its gap (min_delay_seconds or its Crawl-delay).
const queues = {};
const out = [];
for (const r of rows) {
const origin = originOf(r.url);
if (allowed(robots[origin].rules, pathOf(r.url))) (queues[origin] ||= []).push(r);
else out.push({ json: { ...r, fetch: false } });
}
let clock = 0;
const lastHit = {};
while (Object.values(queues).some(q => q.length)) {
for (const [origin, q] of Object.entries(queues)) {
if (!q.length) continue;
const gap = Math.max(s.min_delay_seconds, robots[origin].delay || 0);
const ready = origin in lastHit ? Math.max(clock, lastHit[origin] + gap) : clock;
out.push({ json: { ...q.shift(), fetch: true, wait_seconds: Math.ceil(ready - clock) } });
clock = lastHit[origin] = ready;
}
}
return out;
```
Route disallowed items with an **If** node (`Allowed by robots.txt?`) to a Data table Update that sets only their `status` to `'robots_disallowed'`, so a Google Shopping offer keeps its last price until it ages out. Pass allowed items into a **Loop Over Items** node (batch size `1`), a **Wait** node set to `{{ Math.max(1, $json.wait_seconds) }}` seconds, and an **HTTP Request** node called `Fetch page`:
| Setting | Value |
| --- | --- |
| Method | GET |
| URL | `{{ $json.url }}` |
| Headers | `User-Agent` = `{{ $('Recapture settings').first().json.user_agent }}` |
| Options > Response > Include Response Headers and Status | On |
| Options > Response > Never Error | On |
| Options > Response > Response Format | Text |
| Options > Redirects > Follow Redirects | Off |
| Options > Timeout | 30000 |
| Settings > On Error | Continue |
## Step 6: Read the price from the page
Most stores output [Google Merchant Listing JSON-LD](https://developers.google.com/search/docs/appearance/structured-data/merchant-listing) (`Product` and [Schema.org `Offer`](https://schema.org/Offer)) in the initial HTML. On a store I run, 26 of 29 prices on the first run came from JSON-LD.
Because [blocked requests often look like normal 200 pages](https://www.joinmassive.com/blog/competitor-price-monitoring), add a **Code** node called `Read price` that detects bot walls, reads JSON-LD (`@graph` and nested offers) then meta/`itemprop` tags, and holds moves over `max_price_move` (`60%`) as `'suspect'` until a second reading agrees:
```js
// Read the price from one page: JSON-LD Product offers first, then price meta tags, then itemprop.
// Reads a plain HTTP Request response or a Firecrawl /v2/scrape response.
const s = $('Recapture settings').first().json;
const src = $('Loop Over Items').first().json; // the listing being read
const r = $input.first().json;
const fc = $prevNode.name === 'Firecrawl scrape'; // which request fed this run
const via = fc ? 'firecrawl' : 'http';
const html = String((fc ? r.data?.rawHtml : (r.body ?? r.data)) || '');
const status = Number(fc ? r.data?.metadata?.statusCode : r.statusCode) || 0;
const errMsg = String(r.error?.message ?? r.error ?? '');
const num = v => { const n = Number(String(v ?? '').replace(/[^0-9.]/g, '')); return n > 0 ? n : null; };
const clean = v => String(v).toUpperCase().replace(/[^A-Z0-9]/g, '');
const ours = [src.mpn, src.gtin, src.sku].filter(Boolean).map(clean);
const idsOf = o => ['sku', 'mpn', 'gtin', 'gtin8', 'gtin12', 'gtin13', 'gtin14'].map(k => o?.[k]).filter(Boolean).map(clean);
function fromJsonLd(h) {
const re = /]*type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi;
const found = [];
let m;
while ((m = re.exec(h))) {
let data;
try { data = JSON.parse(m[1].trim()); } catch (e) { continue; }
for (const n of [].concat(data).flatMap(d => d?.['@graph'] || [d])) {
if (![].concat(n?.['@type'] || []).some(t => /Product/i.test(t))) continue;
for (const o of [].concat(n.offers || [])) {
for (const x of o?.offers ? [].concat(o.offers) : [o]) { // AggregateOffer holds nested offers
const price = num(x?.price ?? x?.lowPrice ?? x?.priceSpecification?.price);
if (!price) continue;
const avail = String(x?.availability || o?.availability || '');
found.push({ price, how: 'jsonld', ids: [...idsOf(x), ...idsOf(n)],
currency: x?.priceCurrency || o?.priceCurrency || x?.priceSpecification?.priceCurrency || null,
in_stock: avail ? /InStock|LimitedAvailability|PreOrder|BackOrder/i.test(avail) : null });
}
}
}
}
// On a page with several variants, prefer the offer that carries your model number or barcode.
return found.find(f => f.ids.some(id => ours.includes(id))) || found[0] || null;
}
function fromMeta(h) {
const p = h.match(/]+(?:property|name)=["'](?:product|og):price:amount["'][^>]*content=["']([^"']+)["']/i)
|| h.match(/]+content=["']([^"']+)["'][^>]*(?:property|name)=["'](?:product|og):price:amount["']/i);
if (p && num(p[1])) {
const c = h.match(/(?:product|og):price:currency["'][^>]*content=["']([^"']+)["']/i);
return { price: num(p[1]), currency: c ? c[1] : null, in_stock: null, how: 'meta' };
}
const ip = h.match(/itemprop=["']price["'][^>]*content=["']([^"']+)["']/i);
return ip && num(ip[1]) ? { price: num(ip[1]), currency: null, in_stock: null, how: 'itemprop' } : null;
}
const found = html ? (fromJsonLd(html) || fromMeta(html)) : null;
const head = html.slice(0, 6000).toLowerCase();
const walls = ['just a moment', 'verify you are human', 'checking your browser', 'access denied', 'captcha',
'cf-chl', 'attention required', 'request unsuccessful', 'are you a robot', 'pardon our interruption'];
const blocked = !found && (fc || [302, 303, 307, 401, 403, 503].includes(status) || /403|access denied|ECONNRESET/i.test(errMsg)
|| walls.some(w => head.includes(w)));
// Fetch page doesn't follow redirects. A permanent one (301, 308) saves the new address, which goes back through
// the robots check on the next run; a temporary one (302, 303, 307) is usually a bot check, so it counts as blocked.
const loc = !fc && [301, 308].includes(status) && (r.headers?.location || r.headers?.Location);
const moved = loc && (/^https?:/i.test(loc) ? loc : src.url.match(/^https?:\/\/[^/]+/i)[0] + (loc.startsWith('/') ? '' : '/') + loc);
// A 429 asks you to slow down: wait out Retry-After (seconds or a date; an hour if it's missing), no Firecrawl.
const ra = r.headers?.['retry-after'];
const raSec = ra ? (isNaN(Number(ra)) ? (Date.parse(ra) - Date.now()) / 1000 : Number(ra)) : NaN;
const retrySec = Number.isFinite(raSec) ? raSec : 3600;
// Start from what the table already holds, so a failed read never wipes the last good price.
const row = { status: 'ok', method: via, url: src.url, last_checked_at: new Date().toISOString(), last_price: src.last_price,
in_stock: src.in_stock, last_change: 0, pending_price: src.pending_price || 0 };
const last = Number(src.last_price) || 0;
if (moved) Object.assign(row, { status: 'moved', url: moved, last_checked_at: new Date(0).toISOString() });
else if (!fc && status === 429) Object.assign(row, { status: 'rate_limited',
last_checked_at: new Date(Date.now() + Math.max(retrySec, 60) * 1000 - s.recapture_every_hours * 36e5).toISOString() });
else if (blocked) row.status = 'blocked';
else if (!found) row.status = html ? 'no_price' : 'error';
else if (found.currency && found.currency !== s.currency) row.status = 'wrong_currency';
else {
row.method = via + ':' + found.how;
const jump = last && Math.abs(found.price - last) / last > s.max_price_move;
const confirmed = row.pending_price && Math.abs(found.price - row.pending_price) / row.pending_price < 0.02;
if (jump && !confirmed) Object.assign(row, { status: 'suspect', pending_price: found.price }); // wait for a second reading
else Object.assign(row, { last_price: found.price, in_stock: found.in_stock, pending_price: 0,
last_change: last ? Math.round((found.price - last) * 100) / 100 : 0 });
}
// Firecrawl only for blocked pages, and only up to firecrawl_per_run pages a run.
let try_firecrawl = false;
if (row.status === 'blocked' && !fc && s.use_firecrawl) {
const st = $getWorkflowStaticData('global');
if (st.run !== $execution.id) Object.assign(st, { run: $execution.id, firecrawl: 0 });
try_firecrawl = st.firecrawl++ < s.firecrawl_per_run;
}
return [{ json: { listing_key: src.listing_key, row, try_firecrawl } }];
```
A 429 marks the listing `rate_limited` and waits out the site's Retry-After time rather than going to Firecrawl. A permanent redirect (301, 308) saves the new address as `moved` so it passes the robots check before it's read, while a temporary one (302, 303, 307) counts as `blocked`, because it's usually a bot check.
## Step 7: Use Firecrawl for pages that block you
Add an **If** node after `Read price` (`Try Firecrawl?`) checking `{{ $json.try_firecrawl }}`, which is only true for blocked pages and only up to `firecrawl_per_run` (`8`) a run. The true branch goes to an **HTTP Request** node calling [Firecrawl's `/v2/scrape` endpoint](https://docs.firecrawl.dev/api-reference/endpoint/scrape) (`POST https://api.firecrawl.dev/v2/scrape`):
| Setting | Value |
| --- | --- |
| Method | POST |
| URL | `https://api.firecrawl.dev/v2/scrape` |
| Authentication | Generic Credential Type > Header Auth (Name `Authorization`, Value `Bearer `) |
| Options > Response > Never Error | On |
| Options > Timeout | 90000 |
| Settings > Always Output Data | On |
| Settings > On Error | Continue |
with this body:
```js
{{ JSON.stringify({
url: $('Loop Over Items').first().json.url,
formats: ['rawHtml'],
onlyMainContent: false,
location: { country: $('Recapture settings').first().json.country },
proxy: 'auto',
maxAge: 3600000
}) }}
```
Route Firecrawl's response back into the same parser. The false branch updates `competitor_listings` (a Data table Update on `listing_key`, mapping each column, including `url`, from `$json.row`, with Always Output Data on), and an **If** node, `Price changed?`, inserts into `competitor_prices` when `row.status` is `ok` and `row.last_change` isn't zero. Self-hosters can also run [Crawl4AI](https://docs.crawl4ai.com/core/self-hosting/) (`unclecode/crawl4ai` on port `11235`) for free JS rendering.
## Step 8: Find undercuts and send the report
In the Report lane, add a weekly **Schedule Trigger** (Mondays at 08:00) that reads current Shopify prices (copies of Step 2's two requests) and `competitor_listings`, then a **Code** node that keeps in-stock listings matched at `0.9` or higher and above `min_price_ratio`, evens out tax, calculates the gap, and flags health warnings if zero prices were read or more than half are blocked:
```js
// Cheapest trusted competitor per product, compared with your current price, plus a health check.
const s = $('Report settings').first().json;
const day = 864e5, now = Date.now();
const mine = {};
for (const page of $('Get my prices').all())
for (const v of page.json.data.productVariants.nodes) mine[v.id] = { price: Number(v.price), title: v.product.title };
// Compare like with like: tax-inclusive against tax-inclusive.
const comparable = p => s.market_shows_tax === s.prices_include_tax ? p
: s.market_shows_tax ? p * (1 + s.tax_rate) : p / (1 + s.tax_rate);
const money = n => new Intl.NumberFormat(s.locale, { style: 'currency', currency: s.currency }).format(n);
// Trade and B2B sites often put ex-tax prices in their JSON-LD even where shoppers see tax included.
const theirs = l => s.market_shows_tax && (s.tax_exclusive_domains || []).includes(l.domain)
? l.last_price * (1 + s.tax_rate) : l.last_price;
const listings = $input.all().map(i => i.json);
const best = {}, odd = [];
for (const l of listings) {
const me = mine[l.product_key];
// robots_disallowed keeps its last price (often a Google Shopping offer) until it's too old to trust.
if (!me || !['ok', 'robots_disallowed'].includes(l.status) || l.in_stock === false || l.confidence < s.min_confidence) continue;
if (now - new Date(l.last_checked_at) > s.max_price_age_days * day) continue;
const price = theirs(l);
if (price < comparable(me.price) * s.min_price_ratio) { odd.push(l); continue; } // too cheap to be the same thing
if (!best[l.product_key] || price < best[l.product_key].price) best[l.product_key] = { ...l, price };
}
const undercuts = Object.values(best).map(l => {
const my = comparable(mine[l.product_key].price);
return { ...l, title: mine[l.product_key].title, my, gap: my - l.price, pct: (my - l.price) / my };
}).filter(u => u.pct >= s.undercut_threshold).sort((a, b) => b.gap - a.gap);
const week = listings.filter(l => now - new Date(l.last_checked_at) < 7 * day);
const count = st => week.filter(l => l.status === st).length;
const unhealthy = count('ok') === 0 || count('blocked') > week.length / 2;
const blocks = [{ type: 'header', text: { type: 'plain_text', text: `${undercuts.length} products where a competitor is cheaper` } }];
if (unhealthy) blocks.push({ type: 'section', text: { type: 'mrkdwn',
text: `:warning: Only ${count('ok')} of ${week.length} pages read a price this week (${count('blocked')} blocked). Check the Recapture lane before trusting this report.` } });
for (const u of undercuts.slice(0, s.report_limit)) blocks.push({ type: 'section', text: { type: 'mrkdwn',
text: `*${u.title}*\n<${u.url}|${u.seller}> ${money(u.price)} vs yours ${money(u.my)}: ${money(u.gap)} (${Math.round(u.pct * 100)}%) cheaper` } });
blocks.push({ type: 'context', elements: [{ type: 'mrkdwn',
text: `${week.length} pages checked this week: ${count('ok')} ok, ${count('robots_disallowed')} search only, ${count('blocked')} blocked, ${count('no_price')} no price, ${count('suspect')} held back. ${odd.length} prices under ${s.min_price_ratio * 100}% of yours skipped as likely mismatches.` }] });
return [{ json: { blocks, text: `${undercuts.length} competitor undercuts`, undercuts: undercuts.length } }];
```
Send the output to Slack using a **Slack** node with Block Kit:
| Setting | Value |
| --- | --- |
| Resource | Message |
| Operation | Send |
| Send Message To | Channel, By Name: `{{ $('Report settings').first().json.slack_channel }}` |
| Message Type | Blocks |
| Blocks | `{{ JSON.stringify({ blocks: $json.blocks }) }}` |
| Notification Text | `{{ $json.text }}` |
| Options | Include Link to Workflow, Unfurl Links and Unfurl Media all off |
Test the Discover and Recapture lanes on 10 products first (`discovery_batch: 10`), including a lookalike suffix and a common 2-pack to confirm both are rejected, verify the matched URLs in `competitor_listings`, and then activate the schedule.
## What it costs
By searching for each model number once every two to four weeks and re-reading known URLs with plain HTTP requests, your costs stay tiny compared to SaaS price-tracking tools. All prices below were checked November 2026, and the DataForSEO line assumes about 1.5 searches per product a month:
| Component | Unit price | 500 products | 2,000 products | 10,000 products |
| --- | --- | --- | --- | --- |
| n8n | Free self-hosted, or [Starter €24 or Pro €60 a month](https://n8n.io/pricing/) (€20 and €50 billed annually) | 0 or €24 | 0 or €60 | 0 or €60 |
| DataForSEO SERP API | up to US$0.004 a 20-result search | US$3 | US$12 | US$60 |
| Plain HTTP recapture | Built-in n8n HTTP Request | 0 | 0 | 0 |
| Firecrawl fallback | Free up to 1,000 credits a month; Hobby US$19 for 5,000 | 0 | 0 | 0 to US$19 |
| Total, self-hosted | | US$3 | US$12 | US$60 to US$79 |
| Prisync, URL-based | [US$99 to US$399 a month](https://prisync.com/pricing/) | US$199 (up to 1,000) | US$399 (up to 5,000) | Quote |
[PageCrawl](https://pagecrawl.io/blog/competitor-price-monitoring-ecommerce-guide) charges per page, so four competitors on 500 products takes twice its US$99 top plan. I'd self-host n8n: the Community Edition is free and has no execution limits. On n8n Cloud, Starter stops a run after 5 minutes and Pro after 40 (Wait time doesn't count), so keep batches small. Two levers move your cost: increasing `research_with_listings_days` from 30 to 60 days, or pasting competitor URLs directly to skip DataForSEO altogether.
## Common mistakes
### Searching by title instead of model number
Searching a 12-word title brings back category pages and reviews. Search the MPN (`DS-2CD2366G2-I`) to surface exact product listings.
### Substring matching that catches lookalike suffixes
`title.includes("DS-2CD2366G2-I")` falsely matches `DS-2CD2366G2-IU`. Match the MPN as a whole token in Step 4.
### Matching bundles or multi-packs against single units
Competitor `RLC-810A 2-Pack` pages contain the single unit's MPN. Reject pack or bundle titles unless your own title is a pack.
### Mixing tax-inclusive and tax-exclusive prices or currencies
B2B sites often publish ex-tax prices in JSON-LD. Check `priceCurrency` and list known ex-tax domains in `tax_exclusive_domains` when `market_shows_tax` is on.
### Searching from the wrong country
Querying without a location returns foreign currencies from multi-region stores. Set `search_location` in DataForSEO and `country` for Firecrawl.
### Searching every product every day
The stores selling a product change slowly, so search every two to four weeks and spend the frequent runs re-reading known pages.
### Reacting to a single price jump
Pages occasionally render an accessory variant or glitch for one day. Hold moves over `max_price_move` (`60%`) as `'suspect'` until a second reading confirms them.
### Ignoring robots.txt and hammering one domain
Parallel requests trigger IP blocks quickly. Check `robots.txt` once per domain, honour Crawl-delay and Retry-After, and space requests with a Wait node.
### Paying a scraping API for every single page
Routing every check through a paid browser wastes credits on static HTML. Only call Firecrawl when Step 6 marks a page `'blocked'`.
### Blocked fetches failing quietly
Bot walls can make prices look unchanged for months. Always show the blocked count and the health alert in your weekly Slack report.
### Ignoring shipping costs
A US$5 cheaper listing with US$19 shipping isn't an undercut. Few stores put shipping in their JSON-LD, so check the competitor's delivery cost before you reprice, or raise `undercut_threshold`.
## Make it your own
### Matching products with no identifiers
For fashion or private-label items without MPNs, swap the Step 4 regex for a **Basic LLM Chain** node with a Structured Output Parser that compares title, brand and image and answers same, different or unsure, using an Ollama Chat Model on Ollama Cloud (base URL `https://ollama.com`) or any chat model. Save those matches at `0.8`, under the report's `0.9`, until you've checked a few dozen by hand.
### An instant alert for key products
Add an **If** node after `Read price` that posts to Slack straight away when a competitor drops one of your key products by more than 10%.
### Tracking marketplaces and acting on the report
Read Amazon and eBay through a marketplace data API (DataForSEO has one for Amazon) and write the results into `competitor_listings`, and add a repricing lane that reads the same table and adjusts your prices inside floors you set.
### When Data Tables aren't enough
Past the [200 MiB default limit](https://docs.n8n.io/build/work-with-data/data-tables/), data table inserts and updates fail. At four listings per product and one price change a week, 2,000 products add about 100 MB a year, and 10,000 products reach the limit in about five months. Self-hosters can raise it with `N8N_DATA_TABLES_MAX_SIZE_BYTES`; otherwise, swap the **Data table** nodes for n8n's **Postgres** node on [Supabase's free plan](https://supabase.com/pricing), which gives you a 500 MB database.
## Frequently asked questions
### Is it legal to scrape competitor prices?
The rules vary by country and by each site's terms, so treat this as general information rather than legal advice. Robots.txt states a site's preference rather than granting permission; this workflow uses a transparent `user_agent`, honours `robots.txt` per RFC 9309, never logs in, and spaces requests politely. Use what you collect for your own pricing, and don't republish a competitor's catalogue.
### How often should I check?
Re-reading known URLs every few days and running discovery every 14 to 30 days balances freshness and cost for most stores. During major sales, lower `recapture_every_hours` to `24` or `12`.
### Why not use AI to match products?
An exact model number does the job better and costs nothing, whereas a language model shown `DS-2CD2366G2-I` and `DS-2CD2366G2-IU` will often call them the same camera.
### Does it work without barcodes or part numbers?
Yes: either paste your known competitor URLs into `competitor_listings` manually, or use the optional AI attribute matcher in **Make it your own**.
### Will competitors see or block it?
They'll see your store's name in their server logs, because the `user_agent` names it on purpose. Most stores serve JSON-LD to a polite plain request, and for sites behind bot walls the Firecrawl fallback (or self-hosted Crawl4AI) can often read the page; set `use_firecrawl` to false if you'd rather not.
## Next steps
Import the template into n8n, run the **Setup** lane once, and test **Discover** and **Recapture** with `discovery_batch: 10` to check your first matched URLs.
- Complete four-lane workflow: Setup, SERP Discovery, Polite Recapture, and Weekly Slack Report.
- Includes whole-token MPN matching, pack filtering, `robots.txt` checks, and suspect price guards.
- Zero AI token cost; runs on self-hosted n8n or n8n Cloud.