{ "version": "https://jsonfeed.org/version/1", "title": "Webscraping | Posts IndieHackers", "home_page_url": "https://feed.indiehackers.world", "description": "This is an unofficial feed of indiehackers.com", "items": [ { "content_html": "
I am looking to a build a quick proof of concept for idea validation.
\nThe product would be built on data scraped from public webpages. Instead of writing a scraper in the backend, I was hoping to make it completely client-side and instead use any webscraper cloud service.
\nDoes anyone have any recommendations for such a service?
\nAds here
", "url": "https://feed.indiehackers.world/post/ad964b0b45", "title": "Any recommendations for Web Scraper Cloud service?", "date_modified": "2022-12-19T07:17:05.577Z", "author": { "name": "vivekin" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/fd3a5072ba", "title": "Google Serp API", "date_modified": "2022-12-13T17:30:07.192Z", "author": { "name": "Laur" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/d24d6fb110", "title": "Scraping Google SERP with Geolocation", "date_modified": "2022-12-07T22:45:24.519Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/629ff2689b", "title": "Build Your Own Mobile Proxy for Web Scraping", "date_modified": "2022-10-29T13:47:49.325Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/0d95f2a3bb", "title": "100% Success Rate with This Proxy for Web Scrapers", "date_modified": "2022-09-22T18:21:59.815Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "I wrote a blog post on CSS selectors for web scraping.
\nIt covers the basics as well as some good practices for coming up with robust selectors to consistently scrape the correct data.
\nBe sure to check it out!
\nhttps://datagrab.io/blog/guide-to-css-selectors-for-web-scraping/
\nAds here
", "url": "https://feed.indiehackers.world/post/3aa25e542c", "title": "Blog Post: Guide to CSS selectors for web scraping", "date_modified": "2022-09-01T09:45:51.992Z", "author": { "name": "robert_balazsi" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/1a82573384", "title": "Meta Sues Web Scrapers", "date_modified": "2022-07-08T17:42:32.624Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/abe3b1f36a", "title": "Web scraping benchmark", "date_modified": "2022-06-28T07:24:16.445Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/28f629ab3c", "title": "Scraping Airbnb", "date_modified": "2022-05-26T08:48:15.238Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/6f2bdf4e93", "title": "Scraping Instagram", "date_modified": "2022-05-25T11:25:05.060Z", "author": { "name": "mateuszbuda" }, "tags": [ "Webscraping" ] }, { "content_html": "https://www.octoparse.com/blog/proxy-server-for-web-scraping/?indie=\nWeb scraping has become a popular way for collecting web data. However, some website owners are fighting back web scraping through limiting the access rate of any single IP. To reduce the chances of getting blocked, we should try to avoid scraping a website with a single IP address. In this article, we will introduce what is a proxy server and some popular web scrapers that have IP proxy features.
\nAds here
", "url": "https://feed.indiehackers.world/post/02480dbb0b", "title": "Use Proxy Server for Web Scraping", "date_modified": "2022-04-01T02:58:09.322Z", "author": { "name": "lizzhang" }, "tags": [ "Webscraping" ] }, { "content_html": "It seems that the content comes from the article: https://www.octoparse.com/blog/what-is-a-web-crawler-and-how-does-it-work-at-your-benefit/?indie=
\nAds here
", "url": "https://feed.indiehackers.world/post/6e2aab1060", "title": "Came across the infographic showing what is a web crawler and how does it work~", "date_modified": "2022-03-28T09:16:50.077Z", "author": { "name": "lizzhang" }, "tags": [ "Webscraping" ] }, { "content_html": "✨ Is web scraping legal?\n✨What kinds of data can be scraped? \n✨ What are common applications of web scraping?\nCheck out this video and find answers for all questions related to web scraping: https://youtu.be/WOuzDxHdz6I
\nAds here
", "url": "https://feed.indiehackers.world/post/94dfaecc49", "title": "Interesting Web Scraping Questions Answered", "date_modified": "2022-03-10T08:39:04.790Z", "author": { "name": "lizzhang" }, "tags": [ "Webscraping" ] }, { "content_html": "✨ What are the 3 methods of web scraping?\n✨What are the pros and cons of each web scraping way?\n✨ Which approach is your cup of tea?\nThis video got all the answers well covered: https://youtu.be/AeA-neSgON8
\nAds here
", "url": "https://feed.indiehackers.world/post/5261897ca2", "title": "How to start web scraping? Don't miss this one!", "date_modified": "2022-03-04T04:29:18.254Z", "author": { "name": "lizzhang" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/ff5e698732", "title": "Just found a new video introducing \"what is web scraping\". It's helpful for new users of web scrapers to understand how they work.", "date_modified": "2022-02-24T08:00:54.230Z", "author": { "name": "lizzhang" }, "tags": [ "Webscraping" ] }, { "content_html": "Hi all,\nI am learning web scraping these days, \nI want to work on some projects which will require data. But seems like my IPs will be blocked if i use it on large scale.\nCan someone please tell me which web scraping APIs are better so that i can scrape data smoothly? Thank you.
\nAds here
", "url": "https://feed.indiehackers.world/post/c787e7c47e", "title": "Which web scraping APIs are you using to scrape data?", "date_modified": "2022-02-23T19:06:18.805Z", "author": { "name": "Neerajk" }, "tags": [ "Webscraping" ] }, { "content_html": "Hi Folks,\nI am learning web scraping these days for gathering info about product prices listed on websites I want to use ready made solutions for web scraping...\nAlso, How many request do you make generally to scrape a particular website?\nIt would be more helpful with examples, \nYour time & suggestions will be highly appreciated, Thank you :)
\nAds here
", "url": "https://feed.indiehackers.world/post/86ede48f7a", "title": "What are the websites that you scrape for the lead data or price data?", "date_modified": "2022-02-22T15:35:09.475Z", "author": { "name": "Neerajk" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/7d1c2755df", "title": "Empowering Local-Based Web Scraping and More", "date_modified": "2022-02-22T08:46:14.777Z", "author": { "name": "lizzhang" }, "tags": [ "Webscraping" ] }, { "content_html": "I've recently started using Puppeteer and Node.js for web scraping and am finding the power and flexibility you get with it amazing. Of course you need to write a bit of code but its very simple and the browser emulation you get with it is great for javascript heavy websites. I'm currently using it on some more difficult to scrape sites like social media etc.
\nInterested to hear if anyone else has experience and what you've used it for
\nAds here
", "url": "https://feed.indiehackers.world/post/6f96c4b233", "title": "Anyone used Puppeteer for Web Scraping? Whats your coolest project?", "date_modified": "2021-09-09T09:58:32.335Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "I just made a new post where I curated the ultimate list of web automation and data scraping tools for technical and non-technical people who want to collect information from a website without hiring a developer or writing code.
\nCheck the full list here: https://automatio.co/blog/no-code-web-scrapers-ultimate-list/
\nHopefully, it will be of use to someone. Feel free to share in the comments what tool you already tried, which one you prefer, or suggest some that I didn't add to the list.
\nPeace!
\nAds here
", "url": "https://feed.indiehackers.world/post/751af2b059", "title": "No-code & Low-code web scrapers - the ultimate list", "date_modified": "2021-09-08T09:32:54.410Z", "author": { "name": "plavookac" }, "tags": [ "Webscraping" ] }, { "content_html": "Read More Article: https://www.3idatascraping.com/how-to-bypass-anti-scraping-tools-on-websites/
\nAds here
", "url": "https://feed.indiehackers.world/post/4122955343", "title": "Bypass Anti-Scraping Tools on Websites", "date_modified": "2021-08-12T11:13:27.944Z", "author": { "name": "3idatascraping" }, "tags": [ "Webscraping" ] }, { "content_html": "The Amazon Scraper from 3i Data Scraping is a tool for extracting PPC data. If you wish to track competitor’s PPC information from more e-commerce sites, or if you require data with more features and fields, 3i Data Scraping will provide a custom data solution for you.
\nOriginal Article: https://www.3idatascraping.com/how-to-monitor-competitor-ppc-data-on-amazon/
\nAds here
", "url": "https://feed.indiehackers.world/post/f38fb9776f", "title": "How to Monitor Competitor PPC Data on Amazon?", "date_modified": "2021-08-04T13:35:39.759Z", "author": { "name": "3idatascraping" }, "tags": [ "Webscraping" ] }, { "content_html": "Code/No code, what tools do you use for web scraping? For me, it is Scrapy most of the time. Besides that, I also use Selenium, requests, beautifulsoup depending on the project.
\nWhat are your go-to tools for scraping? Do you use No Code tools? Would love to hear.
\nAds here
", "url": "https://feed.indiehackers.world/post/fbab8372eb", "title": "What do you usually use for web scraping?", "date_modified": "2021-07-05T15:38:56.651Z", "author": { "name": "sagunsh" }, "tags": [ "Webscraping" ] }, { "content_html": "During my bachelor thesis I developed a list with more than 500 cookies with all their information inside. If it is interesting for you you can check it out here - https://github.com/akvaplus/json-cookieList \nIt is 100% free to use cookie library
\nAds here
", "url": "https://feed.indiehackers.world/post/943de017aa", "title": "Classified cookies long list", "date_modified": "2021-07-05T07:06:36.566Z", "author": { "name": "Akvaplus" }, "tags": [ "Webscraping" ] }, { "content_html": "Wie kann ich Online Geld verdienen? Was können Sie über dieses Thema sagen, das in letzter Zeit an Dynamik gewinnt?
\nAds here
", "url": "https://feed.indiehackers.world/post/3c047be52f", "title": "Wie kann ich Online Geld verdienen?", "date_modified": "2021-04-28T09:49:54.278Z", "author": { "name": "Venguer" }, "tags": [ "Webscraping" ] }, { "content_html": "I'd be curious to find out what your biggest challenge is related to web scraping.
\nAds here
", "url": "https://feed.indiehackers.world/post/c14a51933b", "title": "[Poll] What is your biggest challenge related to web scraping?", "date_modified": "2021-02-08T09:12:06.957Z", "author": { "name": "robert_balazsi" }, "tags": [ "Webscraping" ] }, { "content_html": "Folks, I wrote a guide on how to discover long-tail keywords using Node.js and Puppeteer. To do that, we'll scrape SERPs (Search Engine Results Pages). Check it out here: https://datagrab.io/blog/scraping-serps-to-find-long-tail-keywords/\nThis is my first ever blog post and I'm really excited about it! Let me know what you think. ;)
\nAds here
", "url": "https://feed.indiehackers.world/post/5340bd6eaf", "title": "How to research long-tail keywords using Puppeteer", "date_modified": "2021-02-02T10:54:27.439Z", "author": { "name": "robert_balazsi" }, "tags": [ "Webscraping" ] }, { "content_html": "I'd be curious about what delivery method do you usually prefer to get your scraped data.
\nAds here
", "url": "https://feed.indiehackers.world/post/3109920359", "title": "[Poll] How do you prefer to get your data delivered?", "date_modified": "2021-01-19T12:06:47.153Z", "author": { "name": "robert_balazsi" }, "tags": [ "Webscraping" ] }, { "content_html": "Question for the indiehacker community.
\nI'm targeting all sorts of agencies as listed on clutch.co
\nHow can I scrape the web for emails to get the emails of contacts and staff at these agencies?
\nIs LinkedIn worth a shot?
\nAny advice would help.
\nAds here
", "url": "https://feed.indiehackers.world/post/619910028e", "title": "How can I scrape web for emails?", "date_modified": "2020-12-08T21:15:56.146Z", "author": { "name": "ProcessLogicOmar" }, "tags": [ "Webscraping" ] }, { "content_html": "Thank you
\nAds here
", "url": "https://feed.indiehackers.world/post/70524e4b9d", "title": "The legal issues", "date_modified": "2020-12-05T22:15:34.611Z", "author": { "name": "NealCallaghan" }, "tags": [ "Webscraping" ] }, { "content_html": "We use selectors extensively at PixieBrix for web automation, scraping, and enhancement.
\nI've put together some tips we use to write more reliable selectors: https://medium.com/brick-by-brick/7-bite-sized-tips-for-reliable-web-automation-and-scraping-selectors-2612bc4de2a1
\nI'd love to hear what tips and techniques you have to selecting content
\nAds here
", "url": "https://feed.indiehackers.world/post/e2ef460e1c", "title": "Tips for writing reliable scraping selectors", "date_modified": "2020-12-03T14:26:07.583Z", "author": { "name": "tschiller" }, "tags": [ "Webscraping" ] }, { "content_html": "I am looking for examples , anything helps thanks!
\nAds here
", "url": "https://feed.indiehackers.world/post/d7e9419f10", "title": "Any examples of profitable websites made with web scraped data?", "date_modified": "2020-11-30T01:59:47.819Z", "author": { "name": "Joesdev" }, "tags": [ "Webscraping" ] }, { "content_html": "Thanks.
\nAds here
", "url": "https://feed.indiehackers.world/post/e9192a4632", "title": "Web automation question", "date_modified": "2020-11-19T06:51:37.819Z", "author": { "name": "RobertAndrews" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/51dc0e58d5", "title": "Notes on creating a scraper using nodejs", "date_modified": "2020-11-15T23:05:41.122Z", "author": { "name": "leima137" }, "tags": [ "Webscraping" ] }, { "content_html": "I've seen lots of use cases for lead generation, price comparison etc but I'm interested to see who is utilising scraped finance data, seems like there could be some really interesting use cases out there for that.
\nAds here
", "url": "https://feed.indiehackers.world/post/9c890731cb", "title": "Who Uses Web Scraping for Finance Projects?", "date_modified": "2020-11-12T09:29:08.833Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "Want to build a SaaS? Or find new leads? Or supercharge your marketing? IndieData gives you the benefits of scraping without the headache of scraping. Get a scraped database instantly 🚀. Build your next SaaS product with our SaaS (Scraping as a Service)
\nAds here
", "url": "https://feed.indiehackers.world/post/8d40125299", "title": "Get a scraped database instantly", "date_modified": "2020-11-08T04:34:09.166Z", "author": { "name": "karanbatra" }, "tags": [ "Webscraping" ] }, { "content_html": "Over at ScrapeDiary, we're building a community on top of slack for all things web scraping and its currently free to join.
\nIf you're working on a project and want to show it off, ask questions or get help, the community is the best place to do it.
\nTake a look around, we'd love to have you!
\nhttp://scrapediary.com/community
\nAds here
", "url": "https://feed.indiehackers.world/post/44b10dd8f0", "title": "Want to join a Web Scraping Community?", "date_modified": "2020-11-07T16:24:05.421Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "One of the main reason a lot of us turn to scraping websites is a lack of API, or limitations around an existing API that prohibit our use case. Given that, what websites don't currently, but really should have an API. Also, what API's exist that should be made much better to support your use case?
\nAds here
", "url": "https://feed.indiehackers.world/post/7807a5fe86", "title": "What websites do you wish had API's?", "date_modified": "2020-11-04T15:13:18.826Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/cef16d4177", "title": "How To: Scrape Facebook Business Pages 🤖", "date_modified": "2020-11-03T20:48:42.630Z", "author": { "name": "dekarboxylation" }, "tags": [ "Webscraping" ] }, { "content_html": "Any idea what blogs agency folks read, or what slack groups they're in?
\nAds here
", "url": "https://feed.indiehackers.world/post/b852234efa", "title": "Where do agency folks hang out?", "date_modified": "2020-11-02T21:53:25.716Z", "author": { "name": "OnboardNinja" }, "tags": [ "Webscraping" ] }, { "content_html": "And yes, the bot actually found me the apartment I now live in 🎉!
\nI hope this inspires some of you to give Webscraping a try.\nWhat annoying task would you like to outsource to a bot?
\nAds here
", "url": "https://feed.indiehackers.world/post/bf3a875cf7", "title": "How I built a bot to find an apartment 🤖", "date_modified": "2020-11-01T15:12:19.840Z", "author": { "name": "Trunksome" }, "tags": [ "Webscraping" ] }, { "content_html": "There are a growing number of no-code web scraping tools out there in market, both as browser extensions and cloud based services. I think this is a really good sign as it means we can introduce a whole group of people out there that aren't developers to web scraping. What are your favourite no-code or even low code web scraping tools? Why would you recommend them?
\nAds here
", "url": "https://feed.indiehackers.world/post/9ef154de29", "title": "What are your favourite No-Code Web Scraping Tools?", "date_modified": "2020-10-30T16:10:57.724Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "I have a SaaS product that makes web scraping super easy.
\nSo many times when I see other products, I can't help but think, damn if I can just make this one new feature by using my own scraping tool, this product could be even better!
\nUnfortunately, I'm having a hard time conveying that. I don't really have any real customer acquisition channels. Before I invest in content or paid marketing, I'd love to have that "it" factor in my offering, so that potential customers see in it, what I see in it :)
\nI need to make web scraping sexier. I want to remove people's hesitation (yes, in most cases it IS legal as evidenced by numerous supreme court rulings) and show them how easily they can use it to expand their own products.
\nAny ideas?
\nAds here
", "url": "https://feed.indiehackers.world/post/7390550f19", "title": "Need ideas: How do I make web scraping sexy?", "date_modified": "2020-10-29T16:59:40.135Z", "author": { "name": "JosephFritz" }, "tags": [ "Webscraping" ] }, { "content_html": "Ads here
", "url": "https://feed.indiehackers.world/post/9fdce03d6f", "title": "YCombinator - Extracting a list of all the latest companies attending in just 3 minutes.", "date_modified": "2020-10-29T10:11:26.981Z", "author": { "name": "maximg" }, "tags": [ "Webscraping" ] }, { "content_html": "Just thought I'd share.
\nFound an app online that I thought I could build a nice list from. GraphQL API was loading 10 prospects at a time. Tweak the request to 5000, 6 requests to get all of them, skip the rate limiting problem. Tried 10,000 but it crashed.
\nThx for the list!
\nAds here
", "url": "https://feed.indiehackers.world/post/e5ad302da5", "title": "Just scraped 30,000 prospects in 6 requests", "date_modified": "2020-10-28T04:30:05.000Z", "author": { "name": "connorbode" }, "tags": [ "Webscraping" ] }, { "content_html": "We see questions every day from beginners wondering how to get started web scraping. Often they're asking about the best language to learn or the best libraries or tools. If you had to give advice to a complete beginner, how would you tell them to start, what is really important to learn and what isn't?
\nAds here
", "url": "https://feed.indiehackers.world/post/8d4dd3add5", "title": "What Would You Teach a Web Scraping Beginner?", "date_modified": "2020-10-27T19:41:58.202Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "We've all tried to scrape a website and found ourselves being limited in some way, either through elements not loading, or being outright blocked. At a certain point, scraping the site becomes a matter of pride.
\nWhat sites have you struggled against and finally overcome? Which ones didn't you manage to overcome?
\nAds here
", "url": "https://feed.indiehackers.world/post/830cab3fc2", "title": "What are you best anti-web scraping stories?", "date_modified": "2020-10-22T16:35:48.802Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "I've seen lots of cool projects being build in IH that revolve around web scraping, a lot of them are aggregators of some sort. I've begun looking at building out a job board using web scraping to collect job postings from other sites and repackage them.
\nAnother great project I've seen is from a ScrapeDiary community member who built this Udemy course enroller bot;
\nhttps://github.com/aapatre/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE
\nWhat cool projects have you seen / working on at the moment?
\nAds here
", "url": "https://feed.indiehackers.world/post/36164bb827", "title": "What Project Are You Building With Web Scraping?", "date_modified": "2020-10-21T16:31:09.638Z", "author": { "name": "brycedavies" }, "tags": [ "Webscraping" ] }, { "content_html": "What else are you interested in extracting?
\nAds here
", "url": "https://feed.indiehackers.world/post/85f1b128cf", "title": "Mostly into email extraction?", "date_modified": "2020-10-20T03:22:31.257Z", "author": { "name": "Scott322" }, "tags": [ "Webscraping" ] } ] }