Who Uses Web Scraping for Finance Projects?
I've seen lots of use cases for lead generation, price comparison etc but I'm interested to see who is utilising scraped finance data, seems like there could be some really interesting use cases out there for that.
Trending on Indie Hackers
🔥 Roast My Landing Page
Plaid does. I’m pretty sure almost all of Plaids infrastructure is based on web scraping.
This whole thread was very eye-opening, but how would scrapping potentially work with limitations like token generation to login into one's account?
I'm guessing lots of big banks use a similar security feature?
Which part of Plaid? Highly doubt a banking platform would use web scraping to log in to peoples bank accounts etc. They appear to use open banking APIs, unless there is something I'm missing?
I can confirm that the Canadian alternative to Plaid did it completely through scraping. (don't remember their name, but they pitched to us 3 years ago for a bank software project)
I remember asking them but what if the bank changes their interface and your automation breaks? They said they had people monitoring the automations 24h and they'd be able to resolve issues very quickly. That being said, you should expect a few hours of downtime for each bank every month.
You'd think that but it's (scarily) true - at least it was the last time I checked. There's this ongoing battle between the banks and aggregators like Plaid. Lots of banks have given in and have allowed Plaid to integrate with their APIs but lots haven't; that's why you see Plaid break a lot with Capital One and others. From a bank mindset, what benefit do they get from allowing themselves to be disintermediated by Plaid and Yodlee? Open Banking regulations like PSD2 in Europe basically tried to say that banks need to open up their APIs because it's better for consumers and unsafe to be building massive tools off scraping but even that hasn't been super successful.
Wow ok this is absolutely incredible to me, do we think this is because the API's are deficient in some way or these companies just don't really care about the risk here. I'm surprised they're not being blocked by more banks using 2FA now
Can confirm that this works. Currently working at Klarna, one of the biggest fintech's I Europe and sofort (subsidiary) is doing the same. They are quite clever around adding each banks support but cannot give more insight probably 😬
I can confirm as well that I personally was reached by big companies who utilize web automation/scraping to login to people bank accounts or insurance companies and get the data they need.