In today's e-commerce landscape, data scraping has become a crucial method for businesses to gather market intelligence and understand competitors. As a major e-commerce platform, Shopee offers a wealth of product information, user reviews, and pricing data that are vital for business decision-making. However, to protect platform security, Shopee implements various anti-scraping measures that can pose challenges for data collection. In this article, we will explore how to effectively use residential proxy IPs to scrape data from Shopee, along with practical techniques and code examples.

Residential proxy IPs are powerful tools that can effectively hide scraping behavior. Since these IPs come from real users' internet service providers, they are less likely to be detected as bots.

• Rotating Residential Proxies: These proxies frequently change in a short period, making them suitable for scenarios that require a high volume of requests. You can set them to switch IPs periodically to avoid having a single IP banned.
• Static Residential Proxies: These are ideal for situations that require a consistent connection, such as sessions after logging in. They provide more stable access. Make sure the proxy pool is large enough, and that the IPs’ geographic locations align with the target market (like Southeast Asia) to further reduce the risk of detection.

1import requests
2
3# Example: Using a dynamic residential proxy
4proxies = {
5    "http": "http://your_dynamic_proxy:port",
6    "https": "http://your_dynamic_proxy:port",
7}
8
9response = requests.get("https://shopee.com/", proxies=proxies)
10print(response.text)
11
12

Properly controlling the frequency and concurrency of requests is crucial to avoiding being blocked by Shopee's anti-scraping system. Shopee monitors for high-frequency requests made in a short time.

• Concurrency Control: While using multiple proxy IPs can increase request concurrency, it's recommended to limit the number of requests to 2-5 per second. Additionally, ensure that each IP does not make too many requests to lower the risk of being banned.
• Randomize Request Intervals: Introduce random time intervals (e.g., between 1 and 10 seconds) between requests to simulate normal user behavior, reducing the chances of detection.

1import time
2import random
3
4for _ in range(10):  # Make 10 requests
5    response = requests.get("https://shopee.com/", proxies=proxies)
6    print(response.text)
7    time.sleep(random.uniform(1, 10))  # Random delay between 1 to 10 seconds
8
9

To make scraping behavior appear more natural, it is essential to set the HTTP request headers appropriately.

1headers = {
2    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
3    "Referer": "https://shopee.com/",
4}
5
6response = requests.get("https://shopee.com/", headers=headers, proxies=proxies)
7print(response.text)
8
9

To prevent automated scraping, Shopee often employs CAPTCHA and other human verification mechanisms. If you encounter these obstacles, consider the following methods:

• Use CAPTCHA Solving Services: Integrate third-party CAPTCHA-solving solutions (like 2Captcha or Anti-Captcha).
• Simulate Login: Use tools like Selenium to simulate the login process, obtain cookies after logging in, and use these cookies for subsequent requests to avoid frequent login attempts.

1from selenium import webdriver
2from selenium.webdriver.common.by import By
3
4# Example: Simulating login using Selenium
5driver = webdriver.Chrome()
6driver.get("https://shopee.com/user/login")
7
8# Assume there are input fields and a login button
9username_input = driver.find_element(By.NAME, "username")
10password_input = driver.find_element(By.NAME, "password")
11login_button = driver.find_element(By.XPATH, "//button[@type='submit']")
12
13username_input.send_keys("your_username")
14password_input.send_keys("your_password")
15login_button.click()
16
17# Obtain cookies after logging in
18cookies = driver.get_cookies()
19print(cookies)
20
21# Close the browser
22driver.quit()
23
24

To effectively deal with Shopee's IP blocking strategies, it is essential to use a proxy pool and rotate IPs regularly.

1import random
2
3proxy_list = [
4    "http://proxy1:port",
5    "http://proxy2:port",
6    "http://proxy3:port",
7]
8
9# Randomly select a proxy IP
10selected_proxy = random.choice(proxy_list)
11proxies = {
12    "http": selected_proxy,
13    "https": selected_proxy,
14}
15
16response = requests.get("https://shopee.com/", proxies=proxies)
17print(response.text)
18
19

Selecting the right scraping tools can significantly improve data collection efficiency, especially for websites that load content dynamically.

• Selenium or Puppeteer: These tools can simulate user browser actions and handle JavaScript dynamically loaded content.
• Scrapy: Ideal for static pages, Scrapy supports multi-threaded scraping and proxy middleware.

1# Example: Code snippet using the Scrapy framework
2import scrapy
3
4class ShopeeSpider(scrapy.Spider):
5    name = "shopee"
6    start_urls = ["https://shopee.com/"]
7
8    def parse(self, response):
9        # Parse page data
10        products = response.css('.product-name::text').getall()
11        for product in products:
12            yield {'product_name': product}
13
14

7. Bypass Dynamic Content and API Scraping Some data on Shopee is dynamically loaded via JavaScript, making it impossible to retrieve complete information by simply scraping HTML. To address this issue, consider the following strategies:

1# Example: Using Selenium to fetch dynamic content
2driver = webdriver.Chrome()
3driver.get("https://shopee.com/")
4
5# Wait for the page to load completely
6driver.implicitly_wait(10)
7
8# Retrieve dynamically loaded product information
9products = driver.find_elements(By.CLASS_NAME, "product-name")
10for product in products:
11    print(product.text)
12
13# Close the browser
14driver.quit()
15
16

While it is technically possible to circumvent Shopee's anti-scraping measures, it is crucial to adhere to the guidelines outlined in their robots.txt file and relevant legal regulations.

• Check robots.txt: Shopee may restrict scraping of certain pages through its robots.txt file. It's advisable to check this file and follow its directives.
• Legal Compliance: Ensure that your scraping activities comply with local laws and regulations and adhere to Shopee's terms of service. Unauthorized data scraping can lead to legal risks.

By effectively utilizing residential proxy IPs, controlling request frequency, setting appropriate HTTP request headers, and managing CAPTCHA and dynamic content, you can successfully navigate Shopee's anti-scraping mechanisms and scrape data seamlessly. At the same time, it is essential to remain mindful of the legality and ethical standards of data scraping to ensure that your actions do not infringe upon others' rights. With the strategies and code examples provided above, you will be able to more efficiently gather the data you need from Shopee, providing robust support for your business decisions.