What do you usually use for web scraping?
Code/No code, what tools do you use for web scraping? For me, it is Scrapy most of the time. Besides that, I also use Selenium, requests, beautifulsoup depending on the project.
What are your go-to tools for scraping? Do you use No Code tools? Would love to hear.
Trending on Indie Hackers
🔥 Roast My Landing Page
Scrapy for bigger/more complex sites. Python requests + lxml for simpler ones.
I've used Puppeteer in the past for this.
As a developer, I will instinctively reach for a code-based solution, but I'm not sure how well a no-code tool could work. In order to scrape effectively, you need to understand the technical structure of the page you are scraping and how it behaves, so I've always thought of it as an inherently technical task.
All of the scraping I've done in the past is to integrate with other systems, so using code sort of makes sense here anyway, as every other part of the integration is in code too.
eastbrightmarketing.gumroad.com/l/free-trial
webscraper.io is my fav by far :)
I also use Python (requests, BS4, selenium) But it always takes a lot longer to set up than webscraper
Out of curiosity, what do you use web scraping for? I generally have negative connotations towards it: it's brittle (i.e., there's no contract; the website you're scraping can change) and it could likely be against the terms of use for the site you're scraping.
There are a lot of use cases where you just can't ignore scraping. For e.g. I had a client who was tracking a lot of shipping containers from different vendor but non had an api. So we had to scrape those sites.
I don't know how to code, so I use a no-coding tool called Octoparse.
Just found their latest article and got to know more about web scraping:
https://www.octoparse.com/blog/what-is-web-scraping-basics-and-use-cases/?indie=