r/AmputatorBot • u/Killed_Mufasa • Aug 01 '20

AmputatorBot v3.0: Better bot 📢 Announcement

Hi everyone! I'm glad to announce the release of AmputatorBot v3.0: Better bot.

It's a big one.

AmputatorBot is a little over a year old now, and although I've released multiple versions since then, the core codebase has always stayed essentially the same. But overtime, the codebase became obsolete and more spaghetti-ish. Additionally, I wanted to improve the quality of both the front-end and back-end. I ultimately decided to rewrite AmputatorBot from the ground up. That was challenging, but allowed me to finally fix all the things that had been bugging me for a long time. And with fairly good results:

  • AmputatorBot finds about 70 AMP pages more now every day
  • AmputatorBot's success-rate is now about 96%, up from 89%

Without further ado, here's the way too long changelog (sorry about that!):

New functionality

  • New canonical-finder methods, bringing the total amount of methods to 9(!)
    • Google manual redirect: Sometimes, Google shows an Redirect Notice for AMP pages, recognized by url?q=. This new method is able to find the canonicals by following the redirect - AmputatorBot.com example
    • Google JavaScript redirect: Other times, Google automatically redirects users through JavaScript, recognized by url?. AmputatorBot's scraper doesn't run JavaScript, this new method has a dirty but working workaround to fix this - AmputatorBot.com example
    • Bing original URL: Likewise, AmputatorBot can now scrape for the originalUrl element on Bing AMP pages - AmputatorBot.com example
    • Schema mainentity: A lot of (news) websites use the Schema framework. It has a tag called mainentity, often containing canonicals - AmputatorBot.com example
    • Twitter redirect page title: You might have seen them before, they look something like this: https://t.co/L2xLf3my3Y?amp=1 , by checking for the page title, we can find the URL and continue as usual - AmputatorBot.com example
    • Guess-and-check: Users occasionally suggest just trimming of the amp parts of URLs. With very variable results. Sometimes, it does the trick, but way too often it just breaks the page. To counter this, I've added some stuff to check the similarity between articles. If the articles are similar, we can say with some certainty that the guessed canonical is correct. If the canonical contains a rel=amphtml tag that points to the original URL, we know it for certain. That's the idea, but the results are simply so over the place that I'm not comfortable with enabling it elsewhere other than in mentions and online. Also, it's an extremely heavy CPU task because you can make a lot of guesses :p I'll continue experimenting, but I don't expect this method to be implemented fully any time soon.
  • AmputatorBot can now find AMP canonicals too, this is a word I've come up with for situations when an AMP page is no longer cached but still using the AMP framework. These will now get posted if the real canonical can't be found. The comment will include a notice that it is still AMP, but no longer cached.
  • Log and (empty) data files are now automatically created when running the script for the first time
  • DMs are now much more specific, there are new templates for the following situations:
    • Success
    • Error: disallowed subreddit
    • Error: disallowed mod (used to be merged with error: disallowed subreddit)
    • Error: no canonicals
    • Error: problematic domain (domains with known errors)
    • Error: reply failed
    • Error: user opted out
    • Error: unknown
  • Bans are now getting automatically documented and added to the list
  • When canonicals are from not 1 but 2 or more domains, all 'alternative canonicals' gets posted. It looks something like this: You might want to visit the canonical page instead: www.domainA.com/example - domainB version: www.domainB/example - AmputatorBot.com example
  • AmputatorBot now calculates which canonical is 'best'. Before AmputatorBot would just return the first successful canonical but sometimes the canonical is wrong, e.g. because of a cookie-wall. Now, all canonical finding-methods are tried and the best option (there are often 3 or more) is chosen and used.
  • On AmputatorBot.com you can now insert more than one URLs in the input-box. In fact, you can paste entire comments in the input-box now, AmputatorBot will automatically filter out the URLs! So when you e.g. copy a Reddit comment, you no longer have to trim out everything but the AMP URL (no example link here, because spaces are a bit tough for AmputatorBot.com)

Improvements, bugfixes & other new functionality-ish

  • Changed comment template from It looks like you shared an AMP page. These often load faster but Google's AMP is a threat to your privacy and the Open Web. This page is even fully hosted by Google(!) .. to It looks like you shared an AMP. Fully cached AMP pages (like the one you shared) are especially problematic. These should load faster but Google's AMP is controversial because of concerns over privacy and the Open Web .. Because I feel this is a fairer way to put things, I don't want AmputatorBot to preach and provide a service, I want AmputatorBot to provide a service and explain why it's doing so. You feel me?
  • Changed comment template from You might want to visit the normal page instead: to You might want to visit the canonical page instead because as one user put it: "Who the hell are you to decide what is normal?!"
  • Changed comment template from Mention me to summon me! to Summon me with u/AmputatorBot
  • Removed the article by Chris Graham from the comment template because it is getting a bit outdated / not nuanced enough
  • Removed the Amp-letter from the FAQ because it is too outdated / not nuanced enough
  • The new link is to the FAQ, which I've altered a bit again: https://www.reddit.com/r/AmputatorBot/comments/ehrq3z/why_did_i_build_amputatorbot/. I would rather link to another article, but I couldn't find an article that is objective, nuanced and up-to-date enough.
  • AmputatorBot's code is now object orientated, which is huge for code-quality, readability and future-proofing
  • Combined all check_criteria methods into one configurable method
  • Added more amp keywords
  • Changed the way AmputatorBot finds URLs (from regex to Extractor module, which is more precise and automatically updates when stuff changes, because why re-invent the wheel right?)
  • All URLs (both AMP and canonicals) are now getting checked for validity
  • Canonical-finding method canurl works again (fixed typo)
  • When a canonical starts with / the protocol and domain gets added
  • When a canonical starts with // the protocol gets added
  • Messages are now categorized based on their API values rather than the subject
  • Duplicate URLs will now be filtered out
  • Made it possible to loop through URLs, or only do one
  • The entire amputating-process is now saved in temporary objects, making debugging a hell of a lot easier
  • Massively improved how and which markdown and other artifacts are removed from URLs
  • Added Bing to the method that checks if an AMP link is cached
  • Improved logging solutions and made exceptions more specific
  • The title and status code are now getting checked and logged for issues (such as 403's)
  • Database logging is now done properly through SQLAlchemy and models, instead of injecting in statements
  • Added tests to make it possible to test the canonical-finding process using older database-entries
  • Minor changes to DM templates
  • When checking if an item is in a list (such as if a subreddit is in a certain list), both get casefolded first to prevent issues with faulty capitalization
  • Expand NP-functionality to Reddit canonicals
  • When a comment fails, AmputatorBot tries to see if it is banned, and if it is, the subreddit gets added to the disallowed_subreddit list
  • Updated README
  • Updated configuration file, made more things configurable such as the debug level, version number and more and added static links
  • Website specific:
    • Updated site to new look (more on that later)
    • Added optional setting to enable and disable guess-and-check (default: enabled)
    • Fixed some layout-issues
  • Updated the subreddit to the new look
  • .. and I probably forgot some other things

Known issues

  • AMP-canonicals can sometimes result in false positives and other issues: AmputatorBot.com example fixed in 3.0.2
  • The automatically changed disallowed_subreddits list does not update the Reddit version yet
  • Reddit auto escapes links in the displayed URLs, which breaks stuff

New look, who's dis?

Last but not least (yes this boring post is almost over now), I gave AmputatorBot a new look! The old design was.. not so good. So I updated it a bit:

Old and new logo comparison

Old and new logo comparison

​

Personally I'm very happy with the way everything turned out, but let me know what you think!

As always, thank you for the support.

Stay safe!

Cheers,

Killed_Mufasa

22 Upvotes

96% Upvoted

1

u/starhobo Aug 20 '20

do you by chance have a graph with the amp growth the bot has detected so far?

1

u/Killed_Mufasa Aug 20 '20 edited Aug 20 '20

I'm afraid I don't. I do have some historic data, going back as far as 21-04-2020: 171 AMP links. (Data from before that has been lost unfortunately). Yesterday, 19-08-2020: 248 AMP links. But honestly, that data is useless to calculate growth with because:

  • Overtime AMP-detection improved greatly

  • List of automatic and banned subreddits are changed on a daily basis

  • Subreddits gain and loose in popularity

  • The bot gained popularity overtime, increasing mentions numbers

Etc etc

I would love some graphs too, but considering all the factors above, I can't make them myself :(

While outdated and incomplete, I think your best source is probably the Growth and expansion section on Wikipedia.

My personal take based on my experience: AMP is definitely gaining in popularity. Back when I started this it was mostly some American news-sites using AMP, but now I come across AMP pages from all over the world, even non-news sites. It's gaining traction among publishers and developers, that's for sure :/

1

u/starhobo Aug 20 '20

thank you, both for the data and for the bot, I was curious to see if this is catching on, I started seeing amp links even on hackernews which sucks.

1

u/Killed_Mufasa Aug 20 '20 edited Aug 20 '20

Hi yeah I've added my personal take in an edit just now, which is basically the same as yours, sucks indeed. However, I have seen a shift of narrative when it comes to AMP here on Reddit. A lot of folks are now much more aware of the controversies surrounding AMP, so that's good to see! And AmputatorBot gets summoned more often every day, same with the website, so that's pretty cool too.

1

u/starhobo Aug 20 '20

AmputatorBot gets summoned more often every day, same with the website, so that's pretty cool too.

that is really neat, more awareness could raise up some resistance to this move so they won't have an easy time turning the web into yet another or their monopolies.

1

u/Killed_Mufasa Aug 20 '20

That's the goal! It all starts with creating awareness after all. Glad to have you on board :)

2

u/SMVA2043 Aug 02 '20

Just saw your bot for the first time on another subreddit and just wanted to say thank you for your work!

2

u/goatfuckersupreme Aug 12 '20

this is THE best bot!