How Gab Deals With Scraping on the Open Web

Nearly everything that is publicly accessible on the Internet is being “scraped” or downloaded by various sources. In the beginning, it was primarily search engines doing this, but this has been forever expanding to include archive sites, researchers, advertising shops, government agencies, and many more organizations who are hungry for data to collect and analyze. It is a fact of life on the open web, and something we at Gab are very familiar with.

Twitter has recently implemented some harsh daily “reading” limits on normal user accounts, and even on paid verified accounts, complaining about the scraping they are being subjected to. We are skeptical if this is the true motivation behind the changes, because there are obviously better ways to combat excessive scraping than to implement blanket limitations on everyone. I use Twitter very lightly, quickly scanning it a few times a day, and I have already received the rate limiting error today. I find it very hard to believe that they cannot handle the traffic from the scraping, or that they cannot address the scraping in the same ways we have.

Through constant monitoring of the site and the sources of traffic we are able to identify specific IPs and networks where scraping is originating. We ban these sources, and continue to ban new ones as they appear. This is why some users on certain VPN locations will be unable to access the site, because that same location was being used to abuse the site. We do also have some rate-limits in place, although they are meant to be a second layer of defense and set at a level we believe is unlikely to be hit by a human user. Protecting the site while keeping everything open and fully accessible to our users and the general public, is our goal. On this, I think we do a great job and strike the right balance.

Twitter has seemingly left the open web, and joined Facebook in requiring everyone to be logged in to view content. This only makes Gab more important as a public source of news and free expression, accessible by anyone anywhere.

Take the ongoing situation in France, for example, and ask yourself, what site would you trust with your browsing and communication? Elon Musk from Twitter met with the French President and will certainly be censoring content on behalf of the government. Twitter will know French users and prevent French users from seeing things, even if they log in through an IP in another country. Gab CEO Andrew Torba tells the French government to get bent, and Gab will support free public discussion without censoring any of it. We seek to export the freedoms of the First Amendment to the World and keep the web free and open. So get on Gab.com and speak freely.

Fosco Marotto
CTO, Gab.com

Invest and become a shareholder in Gab! Gab is currently selling shares in its business in a Regulation Crowdfunding investment round conducted pursuant to Section 4(a)(6) of the Securities Act of 1933. Click here to be directed to the crowdfunding portal to learn more about the offering.