The same Swingline stapler cost some Staples.com shoppers $15.79 and others $14.29. The only difference was where the website thought the visitor was. Wall Street Journal reporters checked the site from more than 42,000 US ZIP codes and found that people living within about 20 miles of a competitor’s store got the lower price (Source: The Wall Street Journal, 2012). Nobody typed in a ZIP code. The site worked it out from their IP address.
The same Swingline stapler cost some Staples.com shoppers $15.79 and others $14.29. The only difference was where the website thought the visitor was. Wall Street Journal reporters checked the site from more than 42,000 US ZIP codes and found that people living within about 20 miles of a competitor’s store got the lower price (Source: The Wall Street Journal, 2012). Nobody typed in a ZIP code. The site worked it out from their IP address.
The Signals a Website Uses to Decide Which Price You See
The IP address is the best-known signal, but it isn’t the only one and often isn’t the main one. Large retailers look at several at once and pick one by their own logic when they conflict.
Signal | What it changes | How to control it when monitoring |
|---|---|---|
IP geolocation | Default region, currency, price, availability | Use an IP from the target city, not just the country |
Region or store cookie | Keeps the first detected region for weeks | Start a clean session for each region |
ZIP code or delivery address | Final price, delivery time and cost, seller | Set the ZIP code explicitly if the site asks for one |
Account and history | Personal discounts, “member prices” | Monitor logged out, or use separate test accounts |
Device and browser | Result order, sometimes the assortment | Use the same user agent for every region |
IP type (datacenter, VPN, residential) | Block, CAPTCHA, or a silent default price | Use IPs that look like ordinary shoppers |
The last row matters more than it seems. Researchers at Northeastern University tested 16 major e-commerce sites and found personalization on nine of them. Home Depot, for example, showed smartphone users products averaging $230, while desktop users saw products averaging $120 (Source: Hannak et al., Northeastern University, 2014). The price you see depends not only on where you are but also on who the site thinks you are.
Five Ways to Get a Manipulated Price Without Noticing
Most bad data doesn’t come from being blocked. A block is obvious right away. These five mistakes are not:
- Your office IP. Every check runs from one city, so your “regional” data is really the same price collected many times.
- A sticky cookie. Many sites store your region in a cookie on the first visit and read only that from then on. You switch your VPN to another city, and the price stays the same.
- A detected VPN or datacenter IP. Large stores check IPs against MaxMind databases or through Cloudflare’s bot protection. Known VPN and hosting ranges get either a CAPTCHA or the default price, with no warning.
- CDN caching. The catalog page is served from cache, and the regional price loads in a separate request after a store is selected. A scraper that reads only the HTML picks up the cached number.
- Collecting at different times. You check New York at 9:00 a.m. and Dallas at 3:00 p.m., and the price changes in between. You end up reading a time difference as a regional pricing policy.
Warning: The most dangerous case is a site that doesn’t block you but quietly shows the national default price. Monitoring runs, the logs show no errors, yet every region in your spreadsheet has the same number. If your data says a competitor “doesn’t do regional pricing,” check that this is actually true first.
What to Collect: A Price Is More Than One Number
Shoppers decide on the total cost and availability, not the number on the product card. If that number is all you collect, you can miss a competitor who keeps the same price nationwide but offers free shipping only in some regions.
The minimum set of fields for every check:
- base price and sale price, recorded separately;
- delivery cost and delivery time to a specific ZIP code;
- availability (“in stock,” “backorder,” “out of stock”), or better, the stock count if the site shows it;
- the seller, if it’s a marketplace: different sellers often win the buy box in different regions;
- any active promotion or coupon on the page;
- metadata: collection time, IP city, the ZIP code you set, and the session ID.
Metadata seems like the boring part, but without it you can’t investigate a single anomaly. When the price in one city starts jumping around a month from now, the metadata is the only way to tell whether the market changed or your proxy did.
Which IP to Use for Checking Regional Prices
This choice decides whether a site sees you as a shopper in Denver or a bot on a hosting server. Here’s how the options compare:
Connection type | How it looks to the site | City accuracy | Risk of a block or default price | When it fits |
|---|---|---|---|---|
Office IP | A regular user from one city | Your city only | Low, while request volume is small | Manual checks of 5–10 products |
Consumer VPN | Often a known VPN range | Usually country or major city | Medium to high | One-off manual checks |
Datacenter proxy | Hosting, not a home user | Low, tied to the data center | High on major retailers | Sites without bot protection |
Residential proxy | Home internet from a regular ISP | City, sometimes ASN | Low | Ongoing regional monitoring |
Mobile proxy | A mobile carrier subscriber | Carrier’s region | Very low | Mobile apps, the most heavily protected sites |
For ongoing city-level monitoring, most teams use a residential proxy. The IP belongs to a regular home ISP in the target city, so the site treats the request as a visit from a local shopper and returns the same price, delivery terms and availability. When choosing a provider, don’t look at the advertised pool size. Look at three things: whether you can target a specific city, whether it offers sticky sessions (one IP held for a set time), and how it bills failed requests.
Pro Tip: Before you launch automated collection, open the competitor’s site in a regular browser through the same proxy and the same city. If the browser price matches what your script collected, the setup is correct. If it doesn’t, the problem is in the cookies, the ZIP code, or how your parser finds the price on the page.
How to Set Up Regional Monitoring, Step by Step
- Pick your products. You don’t need to monitor the whole catalog. Start with 50–200 items that drive most of your revenue or where you actually compete on price.
- Map your regions. Focus on places where the competitor has warehouses or stores and where they compete with someone else. The Staples case shows that price depends on local competition, not on city size.
- Set the region explicitly. Each city needs an IP from that city, a clean session with no old cookies, and, if the site asks, a delivery ZIP code that matches the IP’s city.
- Keep one IP for the whole pass. One region gets one sticky session while you check all its products. Switching IPs mid-session looks suspicious and can reset the selected region.
- Collect all regions in one window. A pass through every city should finish within 30–60 minutes, or dynamic pricing will leak into your data.
- Repeat every measurement. Check each product in each region at least 2–3 times in separate sessions.
- Store metadata with the price (time, IP city, ZIP code, session, HTTP status), not in a separate log.
Set the frequency by category. Once a day is enough for assortment analytics. Electronics and other categories with heavy price competition need several passes a day.
How to Tell Real Regional Pricing From Noise
A price gap between cities doesn’t by itself mean a regional strategy. There are at least three other explanations, and you should rule them out before drawing conclusions.
Dynamic pricing. According to Profitero, Amazon was already changing prices more than 2.5 million times a day back in 2013, sometimes more than 10 times in 24 hours for a single item (Source: Profitero, 2013–2015). If your regions are collected hours apart, you are comparing different moments, not different places.
A/B tests. The same Northeastern University study found that Expedia and Hotels.com randomly split users into groups and showed them options at different prices, which Expedia confirmed (Source: Hannak et al., 2014). That gap is random and has nothing to do with geography.
Session-based personalization. Being logged in, your browsing history, and an abandoned cart can all change the price too.
A practical rule: count a difference as regional only if it shows up in every check for that region and doesn’t appear in control checks from another city. If the price differs in one check out of three, it’s noise. If it differs in all three, it’s policy.
Note: Set up a control group: two or three IPs from the same city checking the same products at the same time. The variation within the control group shows the site’s background noise level. Anything below that level can’t be counted as a regional difference.
How Much Bandwidth Monitoring Uses and How to Estimate It
Residential proxies are usually billed per gigabyte, so your budget depends on page weight. Here is a sample calculation (industry estimate):
- a full product page with images, scripts and fonts weighs 1.5–3 MB;
- blocking images, fonts and media brings that down to 200–500 KB;
- if the site returns the price in a separate JSON request, each product takes 10–50 KB.
Take 500 products, 10 regions, 2 passes a day, and 3 repeats for control. That’s 30,000 requests a day. At 400 KB per request, it comes to about 12 GB a day, or roughly 360 GB a month. If you can pull the price from JSON, the volume drops 10–20 times. So the first optimization isn’t a discount from your provider. It’s finding out where the page actually gets its price from.
Is Collecting Competitor Prices Legal?
A price on a public page is public information, and collecting it is generally not prohibited by itself. There are caveats, though. Many sites’ terms of service forbid automated collection, and overly frequent requests can create a load that becomes a problem of its own. A sensible baseline: don’t bypass logins, don’t collect personal data, keep the request rate moderate, and respect robots.txt.
The EU is a separate case. Since December 3, 2018, Regulation 2018/302 on geo-blocking has been in force there. It prohibits unjustifiably restricting access for shoppers from other EU countries and automatically redirecting them to a “local” version of a site (Source: Regulation (EU) 2018/302). Different prices in different countries are not banned, as long as a shopper can buy on the same terms as a local customer. For monitoring, this means price differences between EU countries are legal, but the site must show them to any visitor.
This isn’t legal advice. If data collection is a significant part of your business, check the details with a lawyer in your jurisdiction.
Frequently Asked Questions
Can online store prices differ by location?
Yes, and it’s fairly common. The WSJ found regional differences at Staples, Home Depot and Lowe’s, and marketplaces change the price and the seller based on the delivery address. In the US this is usually legal, while the EU has restrictions under its geo-blocking regulation.
Why do I see a different price than my colleague on the same site?
There are several possible reasons: a different city based on IP, a region saved in a cookie, being logged in, different devices, or landing in different A/B test groups. To find out, open the site in an incognito window from the same IP and compare again.
What type of proxy works best for monitoring competitor prices?
For regular city-level collection, residential proxies with city targeting and sticky sessions. Datacenter proxies are cheaper, but major retailers often detect them and return a CAPTCHA or the default price. Mobile proxies are mainly needed for mobile apps and the most heavily protected sites.
How often should you check competitor prices?
It depends on the category. Once a day is enough for assortment analytics, while electronics and FMCG during promotions need several checks a day. More important than frequency is collecting all regions at the same time.
Can I use a VPN instead of a proxy?
For a one-off manual check of a few products, yes. For ongoing monitoring, no: consumer VPNs usually offer only a country or a major city, and their IP ranges are well known to bot protection systems.
The Bottom Line: A Regional Price Is a Measurement, Not a Screenshot
A single price snapshot from one city tells you little about a competitor’s strategy. Reliable data comes from setting the region explicitly (a city IP, a clean session, a delivery ZIP code), collecting all cities in the same time window, repeating measurements, and running a control group that shows the noise level.
Start small: 50 key products, the 5 regions where the competitor is strongest, and a manual browser check before you switch on automation. Within a week you’ll know whether the competitor has a regional pricing policy, or whether you’ve been comparing the price from your own office with itself all along