How I built a bot to find an apartment 🤖
I think most of you will agree that looking for an apartment can be quite annoying. Especially in popular cities with lots of competition. You have to regularly browse offerings and ideally be among the first to apply.
When I moved to Sweden 🇸🇪 this year, I had to do exactly that. And because I was looking for a specific move-in date on this Swedish Ebay-like site, I needed to click on every listing and look at the details to determine if it's a fit. Not how I like to spend my free time.
In an act of productive procrastination I decided to built my first WebScraper / Bot that would do the grunt work for me. And I wanted to try out some interesting tech along the way.
🛠 Building the bot
Its task: Scrape the apartment listings for offers that match my criteria. For all matches, send a personalised message to the landlord declaring interest. Do that every hour, from 5 in the morning until 11 at night.
This is the technology i used:
Node.js — Javascript for server-side apps. I used this framework for backends before, but this time I wanted to write it in Typescript.
Puppeteer: Run and control a chromium browser. I used the Stealth Plugin to avoid unnecessary attention. Important: make sure to create only one window and reuse it whenever possible. I only noticed my mistake when I my backend crashed 🔥 (out of memory).
cheerio: jQuery implementation for the server. I used this library to extract the relevant information out of the website HTML using CSS selectors. This part probably took me the longest time to get right.
MongoDB: A NoSQL database. The bot needed a way to remember which listings it already processed, and I always wanted to give MongoDB a try. I used their free Atlas cloud offering to set it up in minutes.
Docker: Package your app in a container to run it effortlessly in the cloud. Probably a bit unnecessary for this kind of project, but it's easy to do once you understand it.
Slack: I wanted to get updates on what my bot was doing. Instead of sending myself e-mails, I used Slack. More specifically, slack-notify.
💡 Takeaways
- A big part of webscraping is extracting information from HTML using CSS selectors. There are countless ways of building those queries, but at best you want them to stay valid even when the website changes. This is not an easy thing to get right.
- The best thing about Javascript is its ecosystem. There are a gazillion talented developers out there creating libraries, writing tutorials and solving problems. This really makes it easy to get started and build apps.
- Automation does not always save you time. In my case, I spent at least 10 hours automating something that would have probably ended up to be around 4 hours of work in total 😄. But I managed to turn something annoying into a fun learning experience.
And yes, the bot actually found me the apartment I now live in 🎉!
I hope this inspires some of you to give Webscraping a try.
What annoying task would you like to outsource to a bot?
Awesome post! I've built the same kind of bot 6-7 years ago that sent me a push notification every time a new apartment went up in Berlin. It helped me find an amazing apartment in the end, and it even lead to me dabbling in entrepreneurship and the startup founder world for the first time in my life :) Good old times!
When I moved to Sweden myself, it didn't help me much because of the housing queues. How did you find your apartment in the end? Blocket or Facebook groups?
Thank you! :) How did it lead you into the founder world - did you go further with that bot, as some kind of product?
Yeah true, here its much more regulated. I built the bot and found the apartment through blocket - I guess scraping facebook groups would be harder to do. I didn't even try, I figured they would ban me pretty quickly. Would be super valuable though, I find looking through facebook even more annoying than scanning the apartment sites 🙈
This is great @Trunksome I love hearing about new use cases for web scraping and this is a really handy one and solves a real problem, keep it up!!
Thanks Bryce! Glad to hear that :)
Wonder if you have a YouTube channel or so, because doing this kinda thing is always cool and fun, which is meant to be content (on YouTube or whatever platform).
This type of work is typically hard to create value for a lot of others as a product, but has considerable value as a piece of content.
I bet if you make a video doing this on YouTube, you can turn the current "4 hours" value into hundreds hours of viewing, if not thousnads in the future.
I agree, sharing the experience is a great way to make the time spent more impactful. :) And I want to try and come up with interesting posts in the future. IndieHackers exclusives though! :P
That's awesome! I'll follow your journey ;)
Hi Simon, I also like this product. One of my friends developed a similar scraper. In addition he developed a bot that creates appointments with government offices for him, because where he lives the places that do online appointments are usually packed.
When I was looking for an apartment recently, I had my personal assistant do the searching. I had her also call up the landlords to create appointments to view fitting apartments. The only thing I did was to drive to those appointments. Since I believe this topic is a pain in the ass for many, a full-service agency surrounding finding the perfect apartment would be quite nice.
A full-service agency would for sure be neat! Giving such a product directly to users is probably difficult to pull off, as those websites really don't want to be scraped. But using it as an internal tool to find apartment to clients sounds like it could work. You can start out doing things that don't scale, and step by step automating as much as possible.
Nice work! That's cool that it found a place that ended up working out. This reminds me of a similar project I saw a while back, but for searching Craigslist.
One thing I've done in the past is to write a bot to give me alerts when items I was interested in were posted on Craigslist.
Oh wow, cool stuff! Even with a distance-to-transit setting 😍Thanks for sharing!
I absolutely hate searching for an apartment. Been doing it for the last 3 months and have wanted a tool just like this. In Canada I need to check 4-5 websites and it's such a headache. I'd love to see the product in more detail.
Cheers,
Jer
Feel free to check out the source code on gitlab. Unfortunately every new website requires additional work. At least doing it "by hand" as I did.
I would be awesome to make a product out of something like this, but I guess the scraped pages would not appreciate 😅
Very cool @Trunksome 😃 How many different websites did you scrape? And how did that number affect your queries?
Glad you liked it! I actually only scraped one website, namely blocket.se. As the scraping code for this one site was probably 80% of the project, each additional one would have been quite a lot of added effort.
This comment was deleted 5 months ago.
Thanks! :) Just wanted to check out your app, but for some reason it gets stuck in "Loading your feed". In Brave browser.
This comment was deleted 5 months ago.
Ahh alright, its working now. Looks cool! Finding the interesting listings is the most time intensive task, sending an application was just the cherry on top. Hope you found a good apartment!
This comment was deleted 5 months ago.