How to Monitor Competitor Prices With n8n (Free Template)

Last tested on n8n 2.45, November 2026.

Checking competitor prices manually in a spreadsheet goes stale by Tuesday, while dedicated price-monitoring apps charge US$99 to US$399 a month and still make you paste in every competitor URL yourself. Even most scraping templates get the economics backwards by routing every check through a paid headless browser API when a plain HTTP request can read the structured data on most stores for free.

By the end of this guide, you’ll have a four-lane n8n workflow that searches Google for your product model numbers to discover competitor listings automatically, re-reads their prices politely on a schedule using JSON-LD structured data (with Firecrawl only as a fallback for blocked pages), and sends a weekly Slack report of every product where you’re being undercut.

What you’ll build

Every Monday morning, the workflow posts a compact Slack digest listing the products where a verified competitor is cheaper than you beyond your threshold, showing your price, the competitor’s price, the amount and percentage gap, and a direct link to their page.

Under the hood, the workflow separates discovery (finding which stores sell your exact model number) from recapture (re-reading known URLs). Searching once every two to four weeks and re-reading known URLs with a plain HTTP request is what keeps the monthly bill down to a few dollars.

What you need

• n8n with Data Tables: the free self-hosted Community Edition (recommended) or n8n Cloud.
• A Shopify app with the read_products scope, made in the Shopify Dev Dashboard, plus its client ID and secret. Shopify stopped issuing permanent admin tokens for new custom apps in January 2026, so the workflow requests a fresh 24-hour token on each run.
• A DataForSEO account for automated Google SERP discovery (US$0.002 per 10 results, US$50 minimum payment), or skip DataForSEO entirely and paste known competitor URLs into the table for the free path.
• Firecrawl (optional, free tier includes 1,000 credits a month) as a fallback for pages behind bot walls, or Crawl4AI if you self-host.
• Slack (or Gmail) for the weekly undercut digest. No AI model is required.

How it works

1. Setup (Manual Trigger): Creates competitor_listings, competitor_prices, and competitor_searches Data Tables.
2. Discover (Schedule Trigger, daily batch): Pulls active products and MPNs from Shopify, searches Google via DataForSEO every 14 to 30 days, filters for whole-token MPN matches (rejecting multi-packs and lookalike suffixes), and upserts verified listings.
3. Recapture (Schedule Trigger, every 6 hours): Checks robots.txt, re-reads known product URLs politely with a Wait node, parses JSON-LD Product/Offer prices, and uses Firecrawl only for blocked pages.
4. Report (Schedule Trigger, weekly): Compares verified competitor prices against yours and posts the top 20 undercuts to Slack.

Step 1: Create the tables and settings

In the Setup lane, add a Manual Trigger connected to three Data table nodes set to Table -> Create (Reuse Existing Tables enabled). Remember that n8n Data Tables share a 200 MiB default limit per instance, so we only insert a row into competitor_prices when a price actually changes:

Table	One row per	Columns
competitor_listings	product per competitor page	String: listing_key, product_key, sku, mpn, gtin, product_title, url, domain, seller, listing_title, matched_by, currency, status, method Number: confidence, last_price, last_change, pending_price Boolean: in_stock Date: last_checked_at
competitor_prices	price change (the price history)	String: listing_key, product_key, domain, method Number: price Boolean: in_stock Date: captured_at
competitor_searches	product, with when it was last searched	String: product_key, query Number: listings_found, rejected, cost_usd Date: last_searched_at

Next, add an Edit Fields (Set) node in JSON mode at the start of each operational lane, named Discover settings, Recapture settings and Report settings:

{
  "store_domain": "your-store.myshopify.com",
  "own_domain": "yourstore.com",
  "exclude_domains": [],
  "currency": "USD",
  "locale": "en-US",
  "country": "US",
  "search_location": "United States",
  "language_code": "en",
  "search_depth": 20,
  "prices_include_tax": false,
  "market_shows_tax": false,
  "tax_rate": 0,
  "tax_exclusive_domains": [],
  "mpn_source": "metafield",
  "mpn_metafield_namespace": "custom",
  "mpn_metafield_key": "mpn",
  "user_agent": "YourStorePriceCheck/1.0 (+https://yourstore.com/price-check)",
  "robots_agent": "YourStorePriceCheck",
  "discovery_batch": 100,
  "research_with_listings_days": 30,
  "research_without_listings_days": 14,
  "recapture_every_hours": 120,
  "recapture_batch": 300,
  "min_delay_seconds": 10,
  "max_price_move": 0.6,
  "use_firecrawl": true,
  "firecrawl_per_run": 8,
  "min_confidence": 0.9,
  "max_price_age_days": 7,
  "min_price_ratio": 0.5,
  "undercut_threshold": 0.02,
  "report_limit": 20,
  "slack_channel": "#pricing"
}

Set market_shows_tax to true where shoppers see tax-inclusive prices, and list any trade or B2B competitors whose JSON-LD prices exclude tax in tax_exclusive_domains.

Step 2: Pull your products and their identifiers

Searching Google by your full product title brings back noisy results and blog posts, whereas searching by the manufacturer part number (MPN) or GTIN surfaces the exact product pages on rival stores. In the Discover lane, add a daily Schedule Trigger and Discover settings, then an HTTP Request node called Get Shopify token that swaps your client ID and secret for a 24-hour token through the client credentials grant:

Setting	Value
Method	POST
URL	https://{{ $json.store_domain }}/admin/oauth/access_token
Authentication	Generic Credential Type > Custom Auth: {"body": {"client_id": "<your client ID>", "client_secret": "<your client secret>"}}
Send Body	Form Urlencoded, grant_type = client_credentials
Settings > Retry On Fail	On

Because n8n’s built-in Shopify node doesn’t return metafields, use a second HTTP Request node, Get my products, with that token calling Admin GraphQL API 2026-10 (productVariants query):

Setting	Value
Method	POST
URL	https://{{ $('Discover settings').first().json.store_domain }}/admin/api/2026-10/graphql.json
Send Headers	X-Shopify-Access-Token = {{ $('Get Shopify token').first().json.access_token }}
Send Body	JSON (Using JSON)
Pagination > Mode	Update a Parameter in Each Request
Pagination > Type	Body
Pagination > Name	variables
Pagination > Value	{{ { cursor: $response.body.data.productVariants.pageInfo.endCursor, ns: $('Discover settings').first().json.mpn_metafield_namespace, key: $('Discover settings').first().json.mpn_metafield_key } }}
Pagination Complete When	Other
Complete Expression	{{ !$response.body.data.productVariants.pageInfo.hasNextPage }}

Its JSON body carries the query and the variables the pagination updates:

{{ JSON.stringify({
  query: `query Variants($cursor: String, $ns: String, $key: String!) {
    productVariants(first: 100, after: $cursor, query: "product_status:active") {
      pageInfo { hasNextPage endCursor }
      nodes {
        id
        sku
        price
        title
        barcodes(first: 5) { nodes { value } }
        mpn: metafield(namespace: $ns, key: $key) { value }
        product {
          title
          vendor
          mpn: metafield(namespace: $ns, key: $key) { value }
        }
      }
    }
  }`,
  variables: { cursor: null, ns: $('Discover settings').first().json.mpn_metafield_namespace, key: $('Discover settings').first().json.mpn_metafield_key }
}) }}

Connect a Data table Get node for competitor_searches (Get search log, with Execute Once and Always Output Data on so the first, empty run doesn’t stop the lane), then a Code node that keeps variants with an MPN or barcode, filters for items not searched within research_with_listings_days (30 days, or 14 for products with no listings), prioritises never-searched items, and slices to discovery_batch (100):

// One item per variant that has a model number or barcode and is due a search.
const s = $('Discover settings').first().json;
const day = 864e5;
const log = {};
for (const r of $('Get search log').all()) log[r.json.product_key] = r.json;

const due = [];
for (const page of $('Get my products').all()) {
  for (const v of page.json.data.productVariants.nodes) {
    const mpn = String((s.mpn_source === 'sku' ? v.sku : (v.mpn?.value || v.product.mpn?.value)) || '').trim();
    const gtin = v.barcodes.nodes.map(b => String(b.value).replace(/\D/g, '')).find(b => b.length >= 8) || '';
    if (!mpn && gtin.length < 8) continue;                       // nothing to match on: see "Make it your own"
    const last = log[v.id];
    const age = last ? (Date.now() - new Date(last.last_searched_at)) / day : 9999;
    const every = last?.listings_found > 0 ? s.research_with_listings_days : s.research_without_listings_days;
    if (age < every) continue;
    const strong = /\d/.test(mpn) && !/\s/.test(mpn) && mpn.replace(/[^A-Za-z0-9]/g, '').length >= 6;
    due.push({ json: { product_key: v.id, sku: v.sku, mpn, gtin, brand: v.product.vendor, age,
      title: v.product.title + (v.title === 'Default Title' ? '' : ' ' + v.title),
      query: mpn ? (strong ? mpn : `${v.product.vendor} ${mpn}`) : gtin } });
  }
}
return due.sort((a, b) => b.json.age - a.json.age).slice(0, s.discovery_batch);

Step 3: Search for each model number

To use the free path, paste competitor URLs into competitor_listings (with your variant’s gid://shopify/ProductVariant/… ID as product_key, status ok and confidence 1) and skip to Step 5. Otherwise, call DataForSEO’s Google Organic Live Advanced endpoint (POST https://api.dataforseo.com/v3/serp/google/organic/live/advanced) in an HTTP Request node:

Setting	Value
Method	POST
URL	https://api.dataforseo.com/v3/serp/google/organic/live/advanced
Authentication	Generic Credential Type > Basic Auth (your DataForSEO API login and password)
Send Body	JSON
Options > Batching > Items per Batch	1
Options > Response > Never Error	On
Options > Timeout	60000
Settings > On Error	Continue

with this body:

{{ JSON.stringify([{
  keyword: $json.query,
  location_name: $('Discover settings').first().json.search_location,
  language_code: $('Discover settings').first().json.language_code,
  depth: $('Discover settings').first().json.search_depth
}]) }}

DataForSEO returns both organic results (often carrying price.current and price.currency) and Google Shopping panel offers, giving many listings their first price reading at no extra cost. Never Error turns an empty balance’s 402 into a message the next node reads, rather than a stopped run.

Step 4: Keep only exact matches

Searching for a Hikvision DS-2CD2366G2-I also surfaces DS-2CD2366G2-IU (built-in mic, higher price), searching RLC-810A surfaces RLC-810A 2-Pack bundles, and wordy names like Argus 4 Pro need the brand (Reolink) in the title. Add a Code node named Keep exact matches to enforce whole-token MPN matching, brand presence on short or spaced MPNs, and pack rejection:

// Turn each search into competitor listings, keeping only exact matches.
const s = $('Discover settings').first().json;
const sent = $('Pick products to search').all();
const results = $input.all();
const skip = [s.own_domain, ...s.exclude_domains].map(d => d.toLowerCase().replace(/^www\./, ''));
const host = u => (String(u).match(/^https?:\/\/([^/?#:]+)/i) || [])[1]?.toLowerCase().replace(/^www\./, '');
const DROP = /^(srsltid|utm_[a-z]+|gclid|variant|currency|country)$/i;
const cleanUrl = u => {
  const [base, q = ''] = String(u).split('#')[0].split('?');
  const kept = q.split('&').filter(kv => kv && !DROP.test(kv.split('=')[0]));
  return base + (kept.length ? '?' + kept.join('&') : '');
};

// Model number as a whole token, any separator: DS-2CD2366G2-I matches "ds 2cd2366g2 i", not DS-2CD2366G2-IU.
const chars = m => String(m).toUpperCase().replace(/[^A-Z0-9]/g, '').split('');
const titleRe = m => new RegExp('(^|[^A-Z0-9])' + chars(m).join('[\\s\\-_/.]?') + '(?![A-Z0-9]|-[A-Z0-9])');
const slugRe = m => new RegExp('(^|[^A-Z0-9])' + chars(m).join('[\\-_.]?') + '(?![A-Z0-9])');
const PACK = /\b(\d+|two|three|four|twin)[\s-]*(pack|pk)\b|\bpack of \d+|\bbundle\b|\bkit\b/i;

function match(q, title, url) {
  const T = String(title || '').toUpperCase(), U = String(url || '').split('?')[0].toUpperCase(); // path only: Google links carry the query
  if (PACK.test(title) && !PACK.test(q.title)) return null;          // a 2-pack isn't your single camera
  if (q.gtin && (T + ' ' + U).includes(q.gtin)) return { by: 'gtin', confidence: 1 };
  if (!q.mpn) return null;
  const strong = /\d/.test(q.mpn) && !/\s/.test(q.mpn) && q.mpn.replace(/[^A-Za-z0-9]/g, '').length >= 6;
  if (!strong && !T.includes(String(q.brand).toUpperCase())) return null; // short names need the brand
  if (titleRe(q.mpn).test(T)) return { by: 'mpn', confidence: strong ? 1 : 0.9 };
  if (slugRe(q.mpn).test(U)) return { by: 'mpn_url', confidence: strong ? 0.85 : 0.75 };
  return null;
}

function offersFrom(items = []) {
  const out = [];
  for (const it of items) {
    if (it.type === 'organic' && it.price?.current)
      out.push({ seller: it.website_name || it.domain, url: it.url, title: it.title, price: it.price.current, currency: it.price.currency });
    if (it.type === 'knowledge_graph')
      for (const sub of it.items || []) for (const el of sub.items || [])
        if (el.type === 'knowledge_graph_shopping_element' && el.price?.current)
          out.push({ seller: el.source, url: el.url, title: el.title, price: el.price.current, currency: el.price.currency });
    if ((it.type === 'popular_products' || it.type === 'shopping') && Array.isArray(it.items))
      for (const el of it.items) if (el.price?.current)
        out.push({ seller: el.seller || el.source, url: el.url, title: el.title, price: el.price.current, currency: el.price.currency });
  }
  return out;
}

const out = [];
const now = new Date().toISOString();
sent.forEach((p, i) => {
  const q = p.json;
  const task = results[i]?.json?.tasks?.[0];
  if (task?.status_code !== 20000) {
    out.push({ json: { kind: 'failed', query: q.query, error: task?.status_message || String(results[i]?.json?.error?.message || 'no response').slice(0, 200) } });
    return;
  }
  let kept = 0, rejected = 0;
  for (const o of offersFrom(task.result?.[0]?.items)) {
    const d = host(o.url);
    const m = d && o.currency === s.currency && !skip.some(x => d === x || d.endsWith('.' + x)) && match(q, o.title, o.url);
    if (!m) { rejected++; continue; }
    kept++;
    const url = cleanUrl(o.url);
    out.push({ json: { kind: 'listing', listing_key: q.product_key + '|' + url, product_key: q.product_key,
      sku: q.sku, mpn: q.mpn, gtin: q.gtin, product_title: q.title, url, domain: d,
      seller: String(o.seller || d).slice(0, 80), listing_title: o.title, matched_by: m.by, confidence: m.confidence,
      last_price: Number(o.price), currency: o.currency, in_stock: null, status: 'ok', method: 'search',
      last_checked_at: now, last_change: 0, pending_price: 0 } });
  }
  out.push({ json: { kind: 'search', product_key: q.product_key, query: q.query, last_searched_at: now,
    listings_found: kept, rejected, cost_usd: Number(task.cost || 0) } });
});
return out;

The URL check reads only the path, because Google Shopping links carry your search in their query string. Connect a Switch node, Route by kind, on {{ $json.kind }}: listing goes to a Data table Upsert on competitor_listings (matching on listing_key), search to an Upsert on competitor_searches (matching on product_key), and failed to a Slack alert with Execute Once on.

Step 5: Re-read each listing politely

In the Recapture lane, add a Schedule Trigger (every 6 hours) followed by a Data table Get node that pulls rows from competitor_listings where status is not 'robots_disallowed' and last_checked_at is older than recapture_every_hours (120 hours):

Setting	Value
Must Match	All Conditions
Condition 1	last_checked_at Less Than {{ $now.minus({ hours: $json.recapture_every_hours }).toISO() }}
Condition 2	status Not Equals robots_disallowed
Order By	last_checked_at, Ascending
Limit	{{ $json.recapture_batch }}

Before fetching product pages, check each domain’s robots.txt once per run and space out requests to the same domain so you never hammer a small store’s server. Following RFC 9309 (the Robots Exclusion Protocol), a 4xx response on /robots.txt means you may crawl, a 5xx response means assume disallow, and the longest matching path wins. Add a Code node named Check robots.txt:

// Fetch robots.txt once per site, drop what it disallows, and plan a polite gap per site.
// RFC 9309: a 4xx means no rules; a 5xx or no answer means treat the whole site as disallowed.
const s = $('Recapture settings').first().json;
const token = s.robots_agent.toLowerCase();
const originOf = u => (String(u).match(/^(https?:\/\/[^/?#]+)/i) || [])[1]?.toLowerCase();
const pathOf = u => String(u).replace(/^https?:\/\/[^/?#]+/i, '').split('#')[0] || '/';

function parseRobots(txt) {
  const groups = [];
  let cur = null, lastWasAgent = false;
  for (const raw of String(txt).split(/\r?\n/)) {
    const m = raw.replace(/#.*$/, '').trim().match(/^([A-Za-z-]+)\s*:\s*(.*)$/);
    if (!m) continue;
    const key = m[1].toLowerCase(), val = m[2].trim();
    if (key === 'user-agent') {
      if (!lastWasAgent) groups.push(cur = { agents: [], rules: [], delay: null });
      cur.agents.push(val.toLowerCase());
      lastWasAgent = true;
      continue;
    }
    lastWasAgent = false;
    if (!cur) continue;
    if ((key === 'allow' || key === 'disallow') && val) cur.rules.push({ allow: key === 'allow', path: val });
    if (key === 'crawl-delay' && !isNaN(Number(val))) cur.delay = Number(val);
  }
  const mine = groups.filter(g => g.agents.some(a => a !== '*' && token.includes(a)));
  const chosen = mine.length ? mine : groups.filter(g => g.agents.includes('*'));
  return { rules: chosen.flatMap(g => g.rules), delay: chosen.map(g => g.delay).find(d => d != null) ?? null };
}
const toRegex = p => {
  const end = p.endsWith('$');
  const body = (end ? p.slice(0, -1) : p).split('*').map(x => x.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*');
  return new RegExp('^' + body + (end ? '$' : ''));
};
function allowed(rules, path) {
  let best = null;                       // the longest matching rule wins; Allow wins a tie
  for (const r of rules) if (toRegex(r.path).test(path) &&
      (!best || r.path.length > best.path.length || (r.path.length === best.path.length && r.allow))) best = r;
  return !best || best.allow;
}

const rows = $input.all().map(i => i.json);
const robots = {};
for (const origin of [...new Set(rows.map(r => originOf(r.url)))]) {
  try {
    const res = await this.helpers.httpRequest({ url: origin + '/robots.txt', returnFullResponse: true,
      ignoreHttpStatusErrors: true, timeout: 15000, headers: { 'User-Agent': s.user_agent } });
    robots[origin] = res.statusCode >= 500 ? { rules: [{ allow: false, path: '/' }] }
      : res.statusCode >= 400 ? { rules: [] } : parseRobots(res.body);
  } catch (e) {
    robots[origin] = { rules: [{ allow: false, path: '/' }] };
  }
}

// Interleave sites so one site is never hit twice inside its gap (min_delay_seconds or its Crawl-delay).
const queues = {};
const out = [];
for (const r of rows) {
  const origin = originOf(r.url);
  if (allowed(robots[origin].rules, pathOf(r.url))) (queues[origin] ||= []).push(r);
  else out.push({ json: { ...r, fetch: false } });
}
let clock = 0;
const lastHit = {};
while (Object.values(queues).some(q => q.length)) {
  for (const [origin, q] of Object.entries(queues)) {
    if (!q.length) continue;
    const gap = Math.max(s.min_delay_seconds, robots[origin].delay || 0);
    const ready = origin in lastHit ? Math.max(clock, lastHit[origin] + gap) : clock;
    out.push({ json: { ...q.shift(), fetch: true, wait_seconds: Math.ceil(ready - clock) } });
    clock = lastHit[origin] = ready;
  }
}
return out;

Route disallowed items with an If node (Allowed by robots.txt?) to a Data table Update that sets only their status to 'robots_disallowed', so a Google Shopping offer keeps its last price until it ages out. Pass allowed items into a Loop Over Items node (batch size 1), a Wait node set to {{ Math.max(1, $json.wait_seconds) }} seconds, and an HTTP Request node called Fetch page:

Setting	Value
Method	GET
URL	{{ $json.url }}
Headers	User-Agent = {{ $('Recapture settings').first().json.user_agent }}
Options > Response > Include Response Headers and Status	On
Options > Response > Never Error	On
Options > Response > Response Format	Text
Options > Redirects > Follow Redirects	Off
Options > Timeout	30000
Settings > On Error	Continue

Step 6: Read the price from the page

Most stores output Google Merchant Listing JSON-LD (Product and Schema.org Offer) in the initial HTML. On a store I run, 26 of 29 prices on the first run came from JSON-LD.

Because blocked requests often look like normal 200 pages, add a Code node called Read price that detects bot walls, reads JSON-LD (@graph and nested offers) then meta/itemprop tags, and holds moves over max_price_move (60%) as 'suspect' until a second reading agrees:

// Read the price from one page: JSON-LD Product offers first, then price meta tags, then itemprop.
// Reads a plain HTTP Request response or a Firecrawl /v2/scrape response.
const s = $('Recapture settings').first().json;
const src = $('Loop Over Items').first().json;           // the listing being read
const r = $input.first().json;
const fc = $prevNode.name === 'Firecrawl scrape';        // which request fed this run
const via = fc ? 'firecrawl' : 'http';
const html = String((fc ? r.data?.rawHtml : (r.body ?? r.data)) || '');
const status = Number(fc ? r.data?.metadata?.statusCode : r.statusCode) || 0;
const errMsg = String(r.error?.message ?? r.error ?? '');
const num = v => { const n = Number(String(v ?? '').replace(/[^0-9.]/g, '')); return n > 0 ? n : null; };
const clean = v => String(v).toUpperCase().replace(/[^A-Z0-9]/g, '');
const ours = [src.mpn, src.gtin, src.sku].filter(Boolean).map(clean);
const idsOf = o => ['sku', 'mpn', 'gtin', 'gtin8', 'gtin12', 'gtin13', 'gtin14'].map(k => o?.[k]).filter(Boolean).map(clean);

function fromJsonLd(h) {
  const re = /<script[^>]*type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi;
  const found = [];
  let m;
  while ((m = re.exec(h))) {
    let data;
    try { data = JSON.parse(m[1].trim()); } catch (e) { continue; }
    for (const n of [].concat(data).flatMap(d => d?.['@graph'] || [d])) {
      if (![].concat(n?.['@type'] || []).some(t => /Product/i.test(t))) continue;
      for (const o of [].concat(n.offers || [])) {
        for (const x of o?.offers ? [].concat(o.offers) : [o]) {      // AggregateOffer holds nested offers
          const price = num(x?.price ?? x?.lowPrice ?? x?.priceSpecification?.price);
          if (!price) continue;
          const avail = String(x?.availability || o?.availability || '');
          found.push({ price, how: 'jsonld', ids: [...idsOf(x), ...idsOf(n)],
            currency: x?.priceCurrency || o?.priceCurrency || x?.priceSpecification?.priceCurrency || null,
            in_stock: avail ? /InStock|LimitedAvailability|PreOrder|BackOrder/i.test(avail) : null });
        }
      }
    }
  }
  // On a page with several variants, prefer the offer that carries your model number or barcode.
  return found.find(f => f.ids.some(id => ours.includes(id))) || found[0] || null;
}
function fromMeta(h) {
  const p = h.match(/<meta[^>]+(?:property|name)=["'](?:product|og):price:amount["'][^>]*content=["']([^"']+)["']/i)
         || h.match(/<meta[^>]+content=["']([^"']+)["'][^>]*(?:property|name)=["'](?:product|og):price:amount["']/i);
  if (p && num(p[1])) {
    const c = h.match(/(?:product|og):price:currency["'][^>]*content=["']([^"']+)["']/i);
    return { price: num(p[1]), currency: c ? c[1] : null, in_stock: null, how: 'meta' };
  }
  const ip = h.match(/itemprop=["']price["'][^>]*content=["']([^"']+)["']/i);
  return ip && num(ip[1]) ? { price: num(ip[1]), currency: null, in_stock: null, how: 'itemprop' } : null;
}

const found = html ? (fromJsonLd(html) || fromMeta(html)) : null;
const head = html.slice(0, 6000).toLowerCase();
const walls = ['just a moment', 'verify you are human', 'checking your browser', 'access denied', 'captcha',
               'cf-chl', 'attention required', 'request unsuccessful', 'are you a robot', 'pardon our interruption'];
const blocked = !found && (fc || [302, 303, 307, 401, 403, 503].includes(status) || /403|access denied|ECONNRESET/i.test(errMsg)
                           || walls.some(w => head.includes(w)));

// Fetch page doesn't follow redirects. A permanent one (301, 308) saves the new address, which goes back through
// the robots check on the next run; a temporary one (302, 303, 307) is usually a bot check, so it counts as blocked.
const loc = !fc && [301, 308].includes(status) && (r.headers?.location || r.headers?.Location);
const moved = loc && (/^https?:/i.test(loc) ? loc : src.url.match(/^https?:\/\/[^/]+/i)[0] + (loc.startsWith('/') ? '' : '/') + loc);
// A 429 asks you to slow down: wait out Retry-After (seconds or a date; an hour if it's missing), no Firecrawl.
const ra = r.headers?.['retry-after'];
const raSec = ra ? (isNaN(Number(ra)) ? (Date.parse(ra) - Date.now()) / 1000 : Number(ra)) : NaN;
const retrySec = Number.isFinite(raSec) ? raSec : 3600;

// Start from what the table already holds, so a failed read never wipes the last good price.
const row = { status: 'ok', method: via, url: src.url, last_checked_at: new Date().toISOString(), last_price: src.last_price,
              in_stock: src.in_stock, last_change: 0, pending_price: src.pending_price || 0 };
const last = Number(src.last_price) || 0;
if (moved) Object.assign(row, { status: 'moved', url: moved, last_checked_at: new Date(0).toISOString() });
else if (!fc && status === 429) Object.assign(row, { status: 'rate_limited',
  last_checked_at: new Date(Date.now() + Math.max(retrySec, 60) * 1000 - s.recapture_every_hours * 36e5).toISOString() });
else if (blocked) row.status = 'blocked';
else if (!found) row.status = html ? 'no_price' : 'error';
else if (found.currency && found.currency !== s.currency) row.status = 'wrong_currency';
else {
  row.method = via + ':' + found.how;
  const jump = last && Math.abs(found.price - last) / last > s.max_price_move;
  const confirmed = row.pending_price && Math.abs(found.price - row.pending_price) / row.pending_price < 0.02;
  if (jump && !confirmed) Object.assign(row, { status: 'suspect', pending_price: found.price }); // wait for a second reading
  else Object.assign(row, { last_price: found.price, in_stock: found.in_stock, pending_price: 0,
                            last_change: last ? Math.round((found.price - last) * 100) / 100 : 0 });
}

// Firecrawl only for blocked pages, and only up to firecrawl_per_run pages a run.
let try_firecrawl = false;
if (row.status === 'blocked' && !fc && s.use_firecrawl) {
  const st = $getWorkflowStaticData('global');
  if (st.run !== $execution.id) Object.assign(st, { run: $execution.id, firecrawl: 0 });
  try_firecrawl = st.firecrawl++ < s.firecrawl_per_run;
}
return [{ json: { listing_key: src.listing_key, row, try_firecrawl } }];

A 429 marks the listing rate_limited and waits out the site’s Retry-After time rather than going to Firecrawl. A permanent redirect (301, 308) saves the new address as moved so it passes the robots check before it’s read, while a temporary one (302, 303, 307) counts as blocked, because it’s usually a bot check.

Step 7: Use Firecrawl for pages that block you

Add an If node after Read price (Try Firecrawl?) checking {{ $json.try_firecrawl }}, which is only true for blocked pages and only up to firecrawl_per_run (8) a run. The true branch goes to an HTTP Request node calling Firecrawl’s /v2/scrape endpoint (POST https://api.firecrawl.dev/v2/scrape):

Setting	Value
Method	POST
URL	https://api.firecrawl.dev/v2/scrape
Authentication	Generic Credential Type > Header Auth (Name Authorization, Value Bearer <your API key>)
Options > Response > Never Error	On
Options > Timeout	90000
Settings > Always Output Data	On
Settings > On Error	Continue

with this body:

{{ JSON.stringify({
  url: $('Loop Over Items').first().json.url,
  formats: ['rawHtml'],
  onlyMainContent: false,
  location: { country: $('Recapture settings').first().json.country },
  proxy: 'auto',
  maxAge: 3600000
}) }}

Route Firecrawl’s response back into the same parser. The false branch updates competitor_listings (a Data table Update on listing_key, mapping each column, including url, from $json.row, with Always Output Data on), and an If node, Price changed?, inserts into competitor_prices when row.status is ok and row.last_change isn’t zero. Self-hosters can also run Crawl4AI (unclecode/crawl4ai on port 11235) for free JS rendering.

Step 8: Find undercuts and send the report

In the Report lane, add a weekly Schedule Trigger (Mondays at 08:00) that reads current Shopify prices (copies of Step 2’s two requests) and competitor_listings, then a Code node that keeps in-stock listings matched at 0.9 or higher and above min_price_ratio, evens out tax, calculates the gap, and flags health warnings if zero prices were read or more than half are blocked:

// Cheapest trusted competitor per product, compared with your current price, plus a health check.
const s = $('Report settings').first().json;
const day = 864e5, now = Date.now();
const mine = {};
for (const page of $('Get my prices').all())
  for (const v of page.json.data.productVariants.nodes) mine[v.id] = { price: Number(v.price), title: v.product.title };
// Compare like with like: tax-inclusive against tax-inclusive.
const comparable = p => s.market_shows_tax === s.prices_include_tax ? p
  : s.market_shows_tax ? p * (1 + s.tax_rate) : p / (1 + s.tax_rate);
const money = n => new Intl.NumberFormat(s.locale, { style: 'currency', currency: s.currency }).format(n);
// Trade and B2B sites often put ex-tax prices in their JSON-LD even where shoppers see tax included.
const theirs = l => s.market_shows_tax && (s.tax_exclusive_domains || []).includes(l.domain)
  ? l.last_price * (1 + s.tax_rate) : l.last_price;

const listings = $input.all().map(i => i.json);
const best = {}, odd = [];
for (const l of listings) {
  const me = mine[l.product_key];
  // robots_disallowed keeps its last price (often a Google Shopping offer) until it's too old to trust.
  if (!me || !['ok', 'robots_disallowed'].includes(l.status) || l.in_stock === false || l.confidence < s.min_confidence) continue;
  if (now - new Date(l.last_checked_at) > s.max_price_age_days * day) continue;
  const price = theirs(l);
  if (price < comparable(me.price) * s.min_price_ratio) { odd.push(l); continue; } // too cheap to be the same thing
  if (!best[l.product_key] || price < best[l.product_key].price) best[l.product_key] = { ...l, price };
}
const undercuts = Object.values(best).map(l => {
  const my = comparable(mine[l.product_key].price);
  return { ...l, title: mine[l.product_key].title, my, gap: my - l.price, pct: (my - l.price) / my };
}).filter(u => u.pct >= s.undercut_threshold).sort((a, b) => b.gap - a.gap);

const week = listings.filter(l => now - new Date(l.last_checked_at) < 7 * day);
const count = st => week.filter(l => l.status === st).length;
const unhealthy = count('ok') === 0 || count('blocked') > week.length / 2;

const blocks = [{ type: 'header', text: { type: 'plain_text', text: `${undercuts.length} products where a competitor is cheaper` } }];
if (unhealthy) blocks.push({ type: 'section', text: { type: 'mrkdwn',
  text: `:warning: Only ${count('ok')} of ${week.length} pages read a price this week (${count('blocked')} blocked). Check the Recapture lane before trusting this report.` } });
for (const u of undercuts.slice(0, s.report_limit)) blocks.push({ type: 'section', text: { type: 'mrkdwn',
  text: `*${u.title}*\n<${u.url}|${u.seller}> ${money(u.price)} vs yours ${money(u.my)}: ${money(u.gap)} (${Math.round(u.pct * 100)}%) cheaper` } });
blocks.push({ type: 'context', elements: [{ type: 'mrkdwn',
  text: `${week.length} pages checked this week: ${count('ok')} ok, ${count('robots_disallowed')} search only, ${count('blocked')} blocked, ${count('no_price')} no price, ${count('suspect')} held back. ${odd.length} prices under ${s.min_price_ratio * 100}% of yours skipped as likely mismatches.` }] });
return [{ json: { blocks, text: `${undercuts.length} competitor undercuts`, undercuts: undercuts.length } }];

Send the output to Slack using a Slack node with Block Kit:

Setting	Value
Resource	Message
Operation	Send
Send Message To	Channel, By Name: {{ $('Report settings').first().json.slack_channel }}
Message Type	Blocks
Blocks	{{ JSON.stringify({ blocks: $json.blocks }) }}
Notification Text	{{ $json.text }}
Options	Include Link to Workflow, Unfurl Links and Unfurl Media all off

Test the Discover and Recapture lanes on 10 products first (discovery_batch: 10), including a lookalike suffix and a common 2-pack to confirm both are rejected, verify the matched URLs in competitor_listings, and then activate the schedule.

What it costs

By searching for each model number once every two to four weeks and re-reading known URLs with plain HTTP requests, your costs stay tiny compared to SaaS price-tracking tools. All prices below were checked November 2026, and the DataForSEO line assumes about 1.5 searches per product a month:

Component	Unit price	500 products	2,000 products	10,000 products
n8n	Free self-hosted, or Starter €24 or Pro €60 a month (€20 and €50 billed annually)	0 or €24	0 or €60	0 or €60
DataForSEO SERP API	up to US$0.004 a 20-result search	US$3	US$12	US$60
Plain HTTP recapture	Built-in n8n HTTP Request	0	0	0
Firecrawl fallback	Free up to 1,000 credits a month; Hobby US$19 for 5,000	0	0	0 to US$19
Total, self-hosted		US$3	US$12	US$60 to US$79
Prisync, URL-based	US$99 to US$399 a month	US$199 (up to 1,000)	US$399 (up to 5,000)	Quote

PageCrawl charges per page, so four competitors on 500 products takes twice its US$99 top plan. I’d self-host n8n: the Community Edition is free and has no execution limits. On n8n Cloud, Starter stops a run after 5 minutes and Pro after 40 (Wait time doesn’t count), so keep batches small. Two levers move your cost: increasing research_with_listings_days from 30 to 60 days, or pasting competitor URLs directly to skip DataForSEO altogether.

Common mistakes

Searching by title instead of model number

Searching a 12-word title brings back category pages and reviews. Search the MPN (DS-2CD2366G2-I) to surface exact product listings.

Substring matching that catches lookalike suffixes

title.includes("DS-2CD2366G2-I") falsely matches DS-2CD2366G2-IU. Match the MPN as a whole token in Step 4.

Matching bundles or multi-packs against single units

Competitor RLC-810A 2-Pack pages contain the single unit’s MPN. Reject pack or bundle titles unless your own title is a pack.

Mixing tax-inclusive and tax-exclusive prices or currencies

B2B sites often publish ex-tax prices in JSON-LD. Check priceCurrency and list known ex-tax domains in tax_exclusive_domains when market_shows_tax is on.

Searching from the wrong country

Querying without a location returns foreign currencies from multi-region stores. Set search_location in DataForSEO and country for Firecrawl.

Searching every product every day

The stores selling a product change slowly, so search every two to four weeks and spend the frequent runs re-reading known pages.

Reacting to a single price jump

Pages occasionally render an accessory variant or glitch for one day. Hold moves over max_price_move (60%) as 'suspect' until a second reading confirms them.

Ignoring robots.txt and hammering one domain

Parallel requests trigger IP blocks quickly. Check robots.txt once per domain, honour Crawl-delay and Retry-After, and space requests with a Wait node.

Paying a scraping API for every single page

Routing every check through a paid browser wastes credits on static HTML. Only call Firecrawl when Step 6 marks a page 'blocked'.

Blocked fetches failing quietly

Bot walls can make prices look unchanged for months. Always show the blocked count and the health alert in your weekly Slack report.

Ignoring shipping costs

A US$5 cheaper listing with US$19 shipping isn’t an undercut. Few stores put shipping in their JSON-LD, so check the competitor’s delivery cost before you reprice, or raise undercut_threshold.

Make it your own

Matching products with no identifiers

For fashion or private-label items without MPNs, swap the Step 4 regex for a Basic LLM Chain node with a Structured Output Parser that compares title, brand and image and answers same, different or unsure, using an Ollama Chat Model on Ollama Cloud (base URL https://ollama.com) or any chat model. Save those matches at 0.8, under the report’s 0.9, until you’ve checked a few dozen by hand.

An instant alert for key products

Add an If node after Read price that posts to Slack straight away when a competitor drops one of your key products by more than 10%.

Tracking marketplaces and acting on the report

Read Amazon and eBay through a marketplace data API (DataForSEO has one for Amazon) and write the results into competitor_listings, and add a repricing lane that reads the same table and adjusts your prices inside floors you set.

When Data Tables aren’t enough

Past the 200 MiB default limit, data table inserts and updates fail. At four listings per product and one price change a week, 2,000 products add about 100 MB a year, and 10,000 products reach the limit in about five months. Self-hosters can raise it with N8N_DATA_TABLES_MAX_SIZE_BYTES; otherwise, swap the Data table nodes for n8n’s Postgres node on Supabase’s free plan, which gives you a 500 MB database.

Frequently asked questions

Is it legal to scrape competitor prices?

The rules vary by country and by each site’s terms, so treat this as general information rather than legal advice. Robots.txt states a site’s preference rather than granting permission; this workflow uses a transparent user_agent, honours robots.txt per RFC 9309, never logs in, and spaces requests politely. Use what you collect for your own pricing, and don’t republish a competitor’s catalogue.

How often should I check?

Re-reading known URLs every few days and running discovery every 14 to 30 days balances freshness and cost for most stores. During major sales, lower recapture_every_hours to 24 or 12.

Why not use AI to match products?

An exact model number does the job better and costs nothing, whereas a language model shown DS-2CD2366G2-I and DS-2CD2366G2-IU will often call them the same camera.

Does it work without barcodes or part numbers?

Yes: either paste your known competitor URLs into competitor_listings manually, or use the optional AI attribute matcher in Make it your own.

Will competitors see or block it?

They’ll see your store’s name in their server logs, because the user_agent names it on purpose. Most stores serve JSON-LD to a polite plain request, and for sites behind bot walls the Firecrawl fallback (or self-hosted Crawl4AI) can often read the page; set use_firecrawl to false if you’d rather not.

Next steps

Import the template into n8n, run the Setup lane once, and test Discover and Recapture with discovery_batch: 10 to check your first matched URLs.