Learn how to use Node Unblocker for web scraping, integrate it with Puppeteer, and bypass IP bans, captchas, and restrictions with rotating proxies.
Web scraping is a crucial technique for extracting data from websites, but many sites employ anti-scraping mechanisms such as IP blocking, captchas, rate limiting, and geo-restrictions. To overcome these barriers, scrapers often rely on proxies, headless browsers like Puppeteer, and proxy servers like Node Unblocker.
In this guide, we will cover:
What Node Unblocker is and how it works How to set up Node Unblocker for web scraping How to integrate Puppeteer with proxies for bypassing restrictions Rotating proxies to prevent bans Deploying Node Unblocker on cloud services for large-scale scraping
Node Unblocker is a Node.js-based web proxy that routes traffic through an intermediary server, allowing users to access blocked content. Unlike traditional proxies, it works by modifying web requests to bypass restrictions.
Related Article: Node Unblocker
When scraping websites, especially those with anti-bot protections, using a simple IP-based proxy (HTTP/SOCKS5) may not be enough. Websites often block traffic that comes from known data center IPs. Node Unblocker acts as an intermediate layer that disguises requests, making them appear more human-like.
Ensure you have Node.js installed on your machine. Check by running:
1Copy
2Edit
3node -v
4npm -v
5
6If not installed, download and install Node.js from here.
Now, install Node Unblocker:
1Copy
2Edit
3npm install unblocker --save
4
5Alternatively, install it globally:
1Copy
2Edit
3npm install -g unblocker
4
5Create a new file named server.js and add the following code:
1Copy
2Edit
3const Unblocker = require('unblocker');
4const express = require('express');
5
6const app = express();
7const unblocker = new Unblocker({
8 prefix: '/proxy/',
9 requestMiddleware: [
10 function (req, res, next) {
11 console.log(`Proxying: ${req.url}`);
12 next();
13 },
14 ],
15});
16
17app.use(unblocker);
18app.listen(8080, () => {
19 console.log('Node Unblocker running on http://localhost:8080');
20});
21
22Run the following command:
1Copy
2Edit
3node server.js
4
5Now, your Node Unblocker proxy is running at http://localhost:8080. You can access websites by appending the URL after /proxy/, e.g.:
1Copy
2Edit
3http://localhost:8080/proxy/https://www.example.com
4
5To scrape websites using Puppeteer while routing requests through Node Unblocker, modify your scraping script as follows:
Example: Using Puppeteer with Node Unblocker
1Copy
2Edit
3const puppeteer = require('puppeteer');
4
5(async () => {
6 const browser = await puppeteer.launch();
7 const page = await browser.newPage();
8
9 // Access website through Node Unblocker proxy
10 await page.goto('http://localhost:8080/proxy/https://example.com', { waitUntil: 'networkidle2' });
11
12 console.log(await page.title());
13
14 await browser.close();
15})();
16
17Example: Using Axios with Node Unblocker If you prefer to fetch data using Axios, route your requests through the Node Unblocker proxy:
1Copy
2Edit
3const axios = require('axios');
4
5(async () => {
6 const response = await axios.get('http://localhost:8080/proxy/https://example.com');
7 console.log(response.data);
8})();
9
10For large-scale web scraping, residential proxies (like MoMoProxy) offer better performance than free proxies or data center IPs.
Example: Using an HTTP Proxy
1Copy
2Edit
3const puppeteer = require('puppeteer');
4
5(async () => {
6 const proxyServer = 'http://username:password@proxy-ip:port';
7
8 const browser = await puppeteer.launch({
9 headless: true,
10 args: [`--proxy-server=${proxyServer}`],
11 });
12
13 const page = await browser.newPage();
14 await page.authenticate({ username: 'your-username', password: 'your-password' });
15
16 await page.goto('https://www.example.com');
17 console.log(await page.title());
18
19 await browser.close();
20})();
21
22Example: Using a SOCKS5 Proxy
1Copy
2Edit
3const puppeteer = require('puppeteer');
4
5(async () => {
6 const proxyServer = 'socks5://proxy-ip:port';
7
8 const browser = await puppeteer.launch({
9 headless: true,
10 args: [`--proxy-server=${proxyServer}`],
11 });
12
13 const page = await browser.newPage();
14 await page.goto('https://www.example.com');
15
16 console.log(await page.title());
17
18 await browser.close();
19})();
20
21To avoid bans and captchas, rotate proxies using proxy-chain.
1Copy
2Edit
3const puppeteer = require('puppeteer');
4const proxyChain = require('proxy-chain');
5
6(async () => {
7 const oldProxyUrl = 'http://username:password@proxy-ip:port';
8 const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);
9
10 const browser = await puppeteer.launch({
11 headless: true,
12 args: [`--proxy-server=${newProxyUrl}`],
13 });
14
15 const page = await browser.newPage();
16 await page.goto('https://www.example.com');
17
18 console.log(await page.title());
19
20 await browser.close();
21 await proxyChain.closeAnonymizedProxy(newProxyUrl);
22})();
23
24For continuous operation, deploy Node Unblocker on a cloud server. You can use:
Use PM2 to keep the proxy running:
1Copy
2Edit
3npm install -g pm2
4pm2 start server.js --name "node-unblocker"
5pm2 save
6
7❌ Puppeteer Not Loading the Page? ✅ Check if the proxy is working using curl:
1Copy
2Edit
3curl -x http://proxy-ip:port -L https://www.example.com
4
5✅ If the website detects bot traffic, use stealth mode:
1Copy
2Edit
3npm install puppeteer-extra puppeteer-extra-plugin-stealth
4
51Copy
2Edit
3const puppeteer = require('puppeteer-extra');
4const StealthPlugin = require('puppeteer-extra-plugin-stealth');
5
6puppeteer.use(StealthPlugin());
7
8(async () => {
9 const browser = await puppeteer.launch();
10 const page = await browser.newPage();
11 await page.goto('https://www.example.com');
12 console.log(await page.title());
13 await browser.close();
14})();
15
16Node Unblocker is a powerful tool for bypassing website restrictions when web scraping. When combined with Puppeteer and rotating proxies, it provides an efficient and undetectable scraping solution.
Looking for reliable residential proxies? Consider MoMoProxy for geo-unblocking and anti-detection scraping.
