The Price Data Two Tools Couldn't Reach

At the parts company where I work, I needed to know what a competitor was charging for about a thousand parts. The prices were sitting right there on the competitor's website, in plain sight. I still couldn't get them. It took building something most people would have given up on to pull the first one: a rack and pinion, $195.99, plus a $56 core charge, off a page that was designed to keep me out.

Here's the catch. A price that's easy for one person to look up by hand turns into a different kind of problem when you need a thousand of them. Nobody is going to sit and copy a thousand prices off a website one at a time, and even if they did, those prices change, so you would be doing it again next month. The sane move is to have something read the page for you. The competitor knew that too, and their site was built to stop it.

The moment anything that looked like an automated tool tried to search the site, it slammed the door. The page came back with two words: "Access Denied." Not the price. Not a slow load. A flat refusal. A regular person opening the site in a normal web browser got the price every time. Anything faster or more automatic than a person hit the wall.

So I tried the two tools people normally reach for when they need to pull information off a website at any kind of scale. These are the standard, well-regarded options, the ones you would find if you searched for how to do this. Both of them hit the exact same wall. Same denial, every time, no matter how I set them up. That is usually the point where a project gets written off as impossible and quietly dropped.

The thing that finally worked was to stop trying to break through the wall and instead walk through the front door like a customer. Instead of a stripped-down tool that screams "automated," I ran a real web browser, the same kind you have open right now, and let it warm up on the site first the way an actual shopper would. It landed on the homepage, looked around, behaved like a person browsing, and only then went looking for a price. To the site, there was nothing to block. It was just another visitor. Then it read the number off the page.

I proved the whole thing end to end on a real part, which is the rack and pinion I mentioned: $195.99, plus the $56 core charge, pulled straight off the page the other two tools couldn't open. I also set up a second, independent method to fall back on, because sites like this one tighten their defenses over time and I did not want the whole thing to depend on a single trick.

Let me be straight about exactly where this stands, because it is easy to oversell. The method works and it is proven. The full run across all thousand-odd parts is built and ready, and it is designed to pick up where it left off if it gets interrupted partway through. I have not turned it loose on the entire list yet. But that was never the hard part. The hard part was getting the first real price out of a site built specifically to stop me from doing it, and that part is solved.

The problems worth taking usually look like this one. The volume is the easy half. The real work is the single thing everyone before you decided couldn't be done.