A "final solution" to the image hosting problem

14    08 Feb 2017 21:54 by u/user6675

Voat is really being crippled by the death of so many image links. In addition to the demise of sli.mg and the unreliability of other hosts, it feels like imgur is cracking down and removing all images that were submitted exclusively to voat.

The problem is that we just can't trust the next image host that pops up to be the one that'll actually last this time. We've been burned too many times for that. That means that no matter what else we do, some of us have to write scraping bots to make backups of all images linked on Voat. But how to redirect the users from the old image URLs to the new ones? I think there are 3 potential solutions:

  1. Make a change to Voat itself to where it automatically redirects an image host that's down to a fallback host (or shares the load between multiple fallback hosts).

  2. Create a special redirect plugin for the most popular browsers that will automatically redirect an image host that's been designated down to a fallback host.

  3. [Most insane solution] Create a peer to peer network that Voat users will be running that will keep some fraction of all image files on each user's computer, then have a browser extension automatically load the image corresponding to the URL from this p2p network if it's not accessible from the server.

I'm happy to start coding, but I want to hear from other people first. Do you think one of these ideas is best? Do you have your own? What do you think people would be most likely to actually use?

10 comments

3 u/Lord_of_the_rats 09 Feb 2017 01:49

im pretty sure that the scraping bot can be a shell script using curl/wget, sed, awk etc

does any1 want me to try doing dat or did some1 else already finnish it

1 u/user6675 [OP] 09 Feb 2017 03:09

If you're a regex wizard, that may be doable. Perhaps consider using Python instead? It'd probably involve less hair-pulling and be more maintainable by others.

In either case, please try it. Scraping the popular image hosts like Imgoat gallery individually would be one approach, or you can try to scrape Voat directly and back up every image that has at least one upgoat.

2 u/Reddit_traitor 08 Feb 2017 22:26

i think the p2p may be the best idea.

1 u/JustFeelsGoodMan 08 Feb 2017 22:26

Use .htaccess files server side to rewrite urls perhaps? It's what I would do but I'm a nerd like that

1 u/wesofx 10 Feb 2017 11:28

I think #1 is pretty smart. If you have Voat (or just a comment bot) automatically detect an image link and automatically upload the image to a list of alternate image hosts, there's copies on other hosts if one goes down. EDIT: with bots, it's not so long term because comments eventuall become outdate if all the hosts at the time the comment was made are down. Otherwise the bot has to update old comments. An offcial voat feature could be much more robust about staying updated.

1 u/mbenbernard 14 Feb 2017 14:41

The destiny of pretty much any service is to eventually disappear, so yes, it's a dead-end. However, solution #1 would allow us to avoid such problems in the future. It's a great solution, actually!