Page History

...

1. Prerequisites:
   
   1. mysql/mariadb
   2. PHP cli <= 5.6
   3. local web server
   4. ssh access to netbeans-vm.apache.org (to get logs)
2. Get the import & stats script sources from github GitHub apache/netbeans-tools/pp3/au
3. Create dedicated local database.
4. Bootstrap database with schema and basic data from sql SQL dump available in sources au/jchalupa.sql.gz
5. Note the DB will grow like hell, one day import brings in like 50k of records in some tables.
6. Download httpd logs for particular month from netbeans-vm:
   $ rsync -avz username@netbeans-vm.apache.org:/var/log/apache2/netbeans-vm.apache.org_access.log_yyyymm* ./
7. Put logs to location from where it can be served by local web server, keep them under /yyyy_mm/ folders for each month.
8. Scripts are written in super old PHP, run in cli and are compatible only with PHP up to 5.6. So you have to have php6.5 cli installed. Sorry about that.
9. Configure sctiptsscripts:
   
   1. au/config.php - for importing data; db connection, paths to logs are set there
   2.  au/db_connect.php.inc - for building stats; db connection
10. Make au/last-date-dlc file writable for user (if necessary).
11. Make sure au/last-date-dlc contains the date of last logfile you have before current import.
12. Note that to have correct monthly data, you have to have last 3 months already imported. Script looks 3month 3 months back to recognize AU user. So if you count for 2020-11 month, you have to import 2020-08, 09, 10 before that.
13. Import In order to import one month of new data run: in shell run
    $ for d in {1..31}; do php5.6 run-au-import.php -s dlc; done;
14. If there is a new release in the imported data, you have to register it in DB:
    1)
    1. get ID of new catalog: SELECT
    is
    1. id FROM `catalogs` WHERE `path` LIKE '%XX.Y%'; replace XX.Y with new version
    2)
    1. register it: INSERT INTO `releases` (`version`, `catalog_id`, `lang`, `stable`, `delay`, `product`) VALUES ('NB XX.Y', 'ID_FROM_STEP_ABOVE^^', 'en', 'Y', 'N', 'netbeans');
    1. Generate In order to generate stats run: in shell run
       $ php5.6 build-dashboard_v2.php -p netbeans -m 2020-09