AEBC Manual

  1. Quick Start.
  2. System requirements.
  3. How it works.
  4. Usage patterns.
  5. Installation and configuration.
  6. NGINX Recommendations.
  7. Telegram setup.
  8. Testing.
  9. Cron setup.
  10. Configuration via an external online scheduler.
  11. config.php.
  12. FAQ and Solutions.

System requirements

Required:
  • Unix/Linux like OS only;
  • PHP 5.3 cli.
Optional:
  • Public http access (Apache/NGINX web-servers);
  • PHP mail() function for mailing;
  • Access to cron modifications;
  • ssh access or managed hosting.

How it works

The purpose and principle of work of Abusive Experience is described here.

AEBC by request in browser or with internal or external cron gets prepared, max full database of abused by Google sites. Scrips check domains list from file domains.txt. If at least one or more domains are found in it, AEBC show or send to email report.

The script caches the database, the search is optimized, which speeds up the script as much as possible and reduces the overhead of runs. The list of verified domains is limited only by reasonable limits. It is recommended not to load it with more than 10 thousand records (domains).

There are three main steps for each launch:

  1. Loading the database (if necessary);
  2. Search it for domains from the list;
  3. Sending an email or telegram message or displaying data in the browser.

You can get data in formatted format or as a dump:
reviewedSite | SourceDB | abusiveStatus/betterAdsStatus | filterStatus 

      reviewedSite:string
      The name/domain of the site reviewed.

      SourceDB:string
      Enum: AbusiveExperienceDb, ExperienceDb - Mobile, ExperienceDb - Desktop
      DB where domain was founded.

      abusiveStatus:string
      Enum: UNKNOWN, PASSING, FAILING
      The status of the site reviewed for the abusive experiences.

      betterAdsStatus:string
      Enum: UNKNOWN, PASSING, WARNING, FAILING
      The status of the site reviewed for the Better Ads Standards.

      filterStatus:string
      Enum: UNKNOWN, ON, OFF, PAUSED, PENDING
      The ad filtering status of the site.

      enforcementTime:string
      The date on which enforcement begins.

      lastChangeTime:string
      The last time that the site changed status.

The list of domains is simple. Take a list of your domains, for example, directly from the admin panel of the Registrar and in any text editor we bring to the view: 

domain1.com
domain2.com
#domain4.com  Dont disturb me anymore, skip
domain5.com
# Another list
domain6.com
domain7.com    # Not my domain
domain8.com

Where the lines after the character "#" are comments and will be ignored. Spaces and tabs are also ignored.

Usage patterns

By choosing a usage template, we will avoid unnecessary settings. If you already have a working php site on your server, then 99% of you already have everything you need for the highest or any another configuration.

  • Only via the web in a browser or another http client (without a schedule): The script is primarily intended for sending the results of work to the webmaster's email or telegram messenger. But if for some reason you only need results for the query, this usage pattern is also possible. For example, you can take raw data for http request for some other processing. In this case, you will need a server with public access. This can be either a site that is already running with its own domain, or just an IP sticking out. The crown is not needed in this case. You just fill the script on the server and access the URL to its front script, which gives you Results.
  • Scheduled Only (when http access is not available): This is a rare case if you still need to get reports by schedule an email or telegram if for some reason you do not have a web server installed, you can fill in the script to the server in such a way that it will not need to use external http access. For this method, access to the cron setup or a good admin is required.
  • Mixed mode (combining the previous two methods): The most common way to use the script, which includes the two methods described above, when you just fill scripts into the available software http directory (for example, a work site) and you can get reports on a schedule and in the web. AEBC does not interfere with the site, it works independently. You don't need to be able to configure crons, it can be used on remote free services. If you have web access, debugging the script is easier. Finally, web access can always be closed using the web server or a login form.

Installation and configuration

The installation process is briefly described on the quick start. Now let's look at it with a little more detail.

Even if you have never installed php scripts on the server or the word "cron" for you, "file permissions" are not quite clear for you, follow the instructions step by step, you should all earn. In extreme cases, give this instruction to the server admin.

  1. Download the archive with the script and unpack it. Open the file in a text editor
    aebc/back/config.php
    and replace text YOUR@EMAIL to your email (connected with licence key and to receive reports), and YOURKEY to your key, which you can receive here.
  2. Upload to server. For example, it can be web root: /home/web/yourdomain.com/ for domain yourdomain.com. Now we have: /home/web/yourdomain.com/aebc/
  3. Setup permissions 0777 (drwxrwxrwx) for dir back, for our example /home/web/yourdomain.com/aebc/back/. How to set permissions to file with FTP client ↬ or with SSH ↬.
  4. Add domains to file aebc/back/domains.txt one per line. (Or skip this step. You can do it via web GUI.)
  5. Testing of scripts.
  6. This could have been the end of installation, but we want to receive an email notification.
    We have 2 ways to do this:
    a) With cron (more reliable): if we familiar with ssh, cronjob or we have admin for it:
    0 1 * * * /PHP_BIN_PATH/php -q /FULL_PATH_TO_AEBC_DIR/back/email_cron.php
    b) Without using cron (simpler): we don't want to mess with the server settings and let's trust a third-party service that will pull our http file on our domain from the outside. External cron solution example are below.
  7. This could have been the end of installation, but we want to receive an telegram notification.
    We have 2 ways to do this:
    a) With cron (more reliable): if we familiar with ssh, cronjob or we have admin for it:
    0 1 * * * /PHP_BIN_PATH/php -q /FULL_PATH_TO_AEBC_DIR/back/telegram_cron.ph
    b) Without using cron (simpler): we don't want to mess with the server settings and let's trust a third-party service that will pull our http file on our domain from the outside. External cron solution example are below.

    * Telegram is a little more difficult than mail, only because you need to make some staps to create your own telegram bot. Nothing complicated, just a couple of clicks. But it will be your personal bot and all the data in it will pass only through you and the telegram service, without the participation of any other 3rd party. This is a digression about maintaining privacy in the case of telegrams as a means of getting reports. Setting up a telegram here, just below in a separate paragraph.
  8. Now completely done!

It is not recommended to run the crons more than once every 4 hours, but only once a day is optimal. Google itself does not update the database so often, and if you get at least one abusive domain in the database, your mail will inundated with emails with warnings. In the case of a remote scheduler, the script itself will cut off launches, which will be called more often every 3 hours by default.

NGINX or Apache+NGINX setup recommendations

Would like to note that the script is not sharpened for any particular server, it doesn't care Works with Apache+php without any problems. There are some features of working with NGINX. Directory aebc/back/ it may be open for reading. For the Apache server, this directory is closed by the back/.htaccess file, but the NGINX server without special add-ons ignored this file and it rules. NGINX keeps its settings in a special file, access to which usually be via ssh. You need to either independently or with the help of an admin to close access from the web to everything the contents of the aebc/back/ directory. If desired, you can set a password to access the aebc directory, but it still do not block access to aebc/back/ directory in case NGINX default settings.

Setting up work with the telegram bot

  1. There's a… bot for that. Just talk to BotFather (@botfather) ↬.
  2. Start it "/start".
  3. Then requeue new bot "/newbot". Then follow BotFather's simple steps.
    a) Create Bot Name.
    a) Create username.
    c) Once you've created a bot and received your authorization token, head down to the Bot API manual to see what you can teach your bot to do.
    Telgram Bot Setup Telgram Bot Setup 2
  4. Next go to your new bot by link from BotFather and start chat "/start".
  5. Back to BotFather, copy API Token and replace TELEGRAM_API_TOKEN in config.php by this token.
  6. Done! More info about telegram bots. ↬

Testing

  1. Make sure that in the list of file domains domains.txt there is at least one uncommented domain. There is nothing wrong with the list being empty (you can hold work of script with empty list). But for a complete test, you need at least one domain in the list for the test. For greater reliability, a banished domain that is already included in this database will be suitable. A list of banned domains can be found here.
  2. If you have web access, run http://yourdomain.com/aebc/ and make sure, that the page does not display error messages. Otherwise, the error will appear on the page as "[Error]: Error description".
  3. If you plan to use an external online scheduler, open this URL for the test in the browser: http://yourdomain.com/aebc/web_cron.php. Here you also can check that there are no error messages.
  4. Check access to any file in directory aebc/back/ from outside. For example, http://yourdomain.com/aebc/back/domains.txt in browser must NOT show file contents.
  5. Some unexpected errors may appear in the body of emails. But you need to check is antispam of your email service is block your server emails. You can check this by opening in your browser following URLs:
    http://yourdomain.com/aebc/email_test.php
    or in commandline:
    php email_test.php

Cron

Find out from the admin or using the command on the server which php or whereis php, on the right path to the server the php binary is located. In 90% of cases, it will be either /usr/bin/php or less frequently in /usr/local/bin/php.

Setup cronjob, so frequently as you want (but no often than once per 4 hour): 0 9 * * * /PHP_BIN_PATH/php -q /FULL_PATH_TO_AEBC_DIR/back/email_cron.php

It is important to use paths from the root of the file system that start with the symbol "/". For our example it must be:
0 11 * * * /usr/bin/php -q /home/web/yourdomain.com/aebc/back/email_cron.php
In this case cron run checking in 1:00AM by server time.

After that you can close optionally access to dir aebc/ with server tools, or using AEBC settings with password in aebc/back/config.php

Configuration via an external online scheduler

For case when you dont want use server cron, you can setup external service likke https://www.easycron.com/ ↬ or any similar.

The service must access the file aebc/web_cron.php via http. In fact, this is a remote browser that opens a our URL by a schedule. For our example, this must be the URL: http://yourdomain.com/aebc/web_cron.php. You can set the external scheduler interval so often as you want and set the minimum interval inside the file aebc/web_cron.php. In default case, this file is ready to work and you do not need to configure it.

config.php

If desired, you can set the date and time format in this file, set a password for the web face, and also choose how to get the result: formatted (by default) or raw dump (change 0 to 1 in the RESULT_AS_DUMP constant value).

Some User parameters of config.php:
  • define('EMAIL', 'YOUR@EMAIL'); -- email with a license and for mailing
  • define('KEY', 'YOURKEY'); -- licence key
  • define('TELEGRAM_API_TOKEN', '1111111111:AAbbCCddFFgg'); -- telegram token from BotFather (@botfather)
  • define('TELEGRAM_USERNAME', '@username'); -- can be empty, but recommend to fill with telegram username.
  • define('TELEGRAM_CHAT_ID', ''); -- can be empty, script try detect it automatically. Sometimes can be hardcoded for security reason. The chat_id number comes in message whe you run telegram_test.php.
  • define('RESULT_AS_DUMP', 0); -- if it is not 0, then reports will be displayed as raw dump in the web and in the emails
  • define('DATE_FORMAT', 'Y-m-d H:i:s'); -- date and time format
  • define('SECRET', ''); -- web access password
  • define('EMAIL_SUBJ_PREFIX', 'Alert '); -- sometimes you have several scripts, or for some other reason you want to add a distinctive prefix to the subject of emails with a report
  • define('SKIP_STATUSES', ''); -- don't alert domains with this statuses (comma separated). Example, we not want to see reports for domains with PAUSED and(or) PENDING statuses, then insert value 'PAUSED,PENDING'.

We don't recommend to modify other parameters.

FAQ and Solutions

  1. Problem: I don't get emails.
    Solution: When you first run the script, you need to add the domain \ ip of your server where script installed to the non-blocked. In the case of gmail usual all "nod gmail" emails will marked straight to spam. It is necessary to unmark them in the spam folder as NOT spam. To forcr email test oprn in browser:
    http://yourdomain.com/aebc/email_test.php
    or in commandline:
    php email_test.php
  2. Problem: How do I change the server where the script installed?
    Solution: To change server you must contact us. Script licence is connected to your server.