known-bots:validate's checks are now declared to any addon that runs a forum's self-checks over an API, so they can be read without a shell on the server. Only each check's outcome is returned, and the API token check, which reaches the KnownBots API, is listed as skipped
an API that is temporarily unavailable is no longer written to the server error log when sending user agents, checking the API token or running known-bots:fetch - it is logged as a warning and retried on the next run, as fetching bots already was. known-bots:validate now warns rather than fails when the API is unavailable, since nothing on the forum is wrong
bugfix: known-bots:import stopped reading at the first blank line in its input, so every user agent after it was silently skipped and the count it reported covered only the lines before the gap. Blank lines are now skipped and the whole file is read
bugfix: known-botsarse had the same fault - a blank line in a log file or on stdin ended the run early, and nothing in the output said so. It now reads to the end of the input, or until --limit is reached
bugfix: the fetch cron could exhaust PHP's memory limit on a forum with a large number of stored user agents, because reprocessing them after new bot definitions arrived loaded every one at once. They are now reprocessed in batches, and memory use no longer grows with the number stored. The send cron reads them more cheaply too
bugfix: when the API sent bot data that failed validation, the "update bots" admin tool and known-bots:fetch crashed with an error instead of reporting it. Both now say the data was refused and that the server error log has the details, and the command exits with a failure
bugfix: a user agent long enough to make a browser pattern fail to run was taken for a real browser, so it was never stored and reprocessing deleted it. It is now treated as unrecognised
bugfix: a user agent stored while the send cron was sending to the API was marked as sent without being included, so it was never sent. Only the user agents actually sent are marked now
bugfix: on a forum upgraded from XenForo 1 rather than freshly installed, the addon never loaded its bot definitions until the fetch cron next ran. Detection fell back to the much smaller list XenForo ships with, and because the browser and ignored rules were empty too, every ordinary visitor was stored as an unidentified user agent. The bundled definitions are now loaded on that upgrade path, as they always were on a fresh install
if your forum was upgraded from XenForo 1 while running 6.3.0 or earlier, the ordinary visitors stored as unidentified user agents are still in the database after this upgrade - run known-bots:reprocess from the command line once to clear them and identify any real bots among them. There is no admin control panel equivalent; left alone, the stored entries age out under the user agent retention period
the server error log now records when there is no bot list to load, instead of passing over it silently
the list of known bots is no longer included in the download as BOTS.md - use the "Show list of Known Bots" tool in the admin control panel, which lists the definitions currently on your forum rather than those current when the release was built
This update contains an important bugfix to handle malformed UTF-8 in user agent strings.
This update will simply ignore any user agents with malformed UTF-8, avoiding errors when trying to send updates via the API. These user agents are invalid and so there is no point undertaking any further analysis - thus they are silently discarded.
This new version also contains an additional CLI tool for importing user agents from a text file, for testing purposes.
v5.0.0 is a major rewrite of the core functionality of this addon aimed at improving processing speed, bot detection sophistication and greatly enhancing our ability to identify new bots.
Note that the options have changed - so please check the options after upgrading. More information about each option is provided on the main addon page.
major rewrite - no longer use "bot|spider|crawl" search strings and false-positive lists to identify possible bots, rely instead on search strings supplied by API to identify valid browsers and store them directly in the database rather than the SimpleCache, ready for emailing
more complete agent reprocessing - check for valid browsers and ignored agents
change the core userAgentMatchesRobot function to use strpos instead of preg_match, it's much faster and won't fall over with extremely high numbers of bot match strings
allow BotFetcher to be manually configured to bypass untrusted http agent - used for testing when API source is on a .local domain. Default action remains to use the untrusted http agent to allow for proxying outbound API calls.
change email cron to daily send
using new v2 API from KnownBots
replace generic bots with complex (regex) based searches
add "Fetch new bots" button to Known Bots List in admin UI
automatically reprocess user agents after loading new bot data
new Cli command for reprocessing user agents, including the option to force all user agents to be reprocessed
improvements to user agent test in admin ui to be more descriptive
bcc additional email address to keep them private
bugfix: don't linkify known bot list when no links supplied
This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.
If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.
After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root: