Page History

...

1. Prerequisites:
   
   1. mysql/mariadb
   2. PHP cli <= 5.6
   3. local web server
   4. ssh access to netbeans-vm.apache.org (to get logs)
2. Create dedicated local database.
3. Bootstrap database with schema and basic data from jchalupa.sql.gz
4. Note the DB will grow like hell, one day import brings in like 50k of records in some tables.
5. Download httpd logs for particular month from netbeans-vm:
   $ rsync -avz username@netbeans-vm.apache.org:/var/log/apache2/netbeans-vm.apache.org_access.log_yyyymm* ./
6. Put logs to location from where it can be served by local web server, keep them under /yyyy_mm/ folders for each month
7. Get the import & stats script sources from github apache/netbeans-tools/pp3/au
8. Scripts are written in super old PHP, run in cli and are compatible only with PHP up to 5.6. So you have to have php6.5 cli installed. Sorry about that.
9. Configure sctipts:
   
   1. au/config.php - for importing data; db connection, paths to logs are set there
   2.  au/db_connect.php.inc - for building stats; db connection
10. Make au/last-date-dlc file writable for user (if necessary)
11. Make sure au/last-date-dlc contains the date of last logfile you have before current import
12. Note that to have correct monthly data, you have to have last 3 months already imported. Script looks 3month back to recognize AU user. So if you count for 2020-11 month, you have to import 2020-08, 09, 10 before that.
13. Import one month of new data: in shell run
    $ for d in {1..31}; do php5.6 run-au-import.php -s dlc;done;
14. If there is new release in the imported data, you have to register it in DB
    1) get ID of new catalog: SELECT is FROM `catalogs` WHERE `path` LIKE '%XX.Y%'; replace XX.Y with new version
    2) register it: INSERT INTO `releases` (`version`, `catalog_id`, `lang`, `stable`, `delay`, `product`) VALUES ('NB XX.Y', 'ID_FROM_STEP_ABOVE^^', 'en', 'Y', 'N', 'netbeans');
15. Generate stats: in shell run
    $ php5.6 build-dashboard_v2.php -p netbeans -m 2020-09

...