1
2 Comments

[Poll] What is your biggest challenge related to web scraping?

by Robert Balazsi

I'd be curious to find out what your biggest challenge is related to web scraping.

What is your biggest challenge related to web scraping?
  1. Coming up with robust element selectors
  2. Scraping AJAX pages
  3. Website's structure updated quite often
  4. Bypassing anti-scraping measures
  5. Getting past geo-blocking
  6. Something else (please comment)
Vote
posted to
Webscraping
 on February 8, 2021
Say something nice to robert_balazsi…
Post Comment
  1. 2

    Selenium does almost everything “well” BUT scraping sites with dynamic content is challenging. Don’t you think?

    xavierlesa
    ·
    3 years ago
    ·
    Reply
    1. 2

      It is indeed, especially for scalability reasons. A headless browser is much slower and more resource-intensive as a HTML parser, so a good practice would be to use headless browsers only if absolutely necessary.

      robert_balazsi
      ·
      3 years ago
      ·
      Reply
Trending on Indie Hackers
Passed $7k 💵 in a month with my boring directory of job boards 35 comments Reaching $100k MRR Organically in 12 months 32 comments 87.7% of entrepreneurs struggle with at least one mental health issue 14 comments How to Secure #1 on Product Hunt: DO’s and DON'Ts / Experience from PitchBob – AI Pitch Deck Generator & Founders Co-Pilot 11 comments Competing with a substitute? 📌 Here are 4 ad examples you can use [from TOP to BOTTOM of funnel] 10 comments Are you wondering how to gain subscribers to a founder's X account from scratch? 9 comments