THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Prerequisites:
- mysql/mariadb
- PHP cli <= 5.6
- local web server
- ssh access to netbeans-vm.apache.org (to get logs)
- Get the import & stats script sources from GitHub apache/netbeans-tools/pp3/au
- Create dedicated local database.
- Bootstrap database with schema and basic data from SQL dump available in sources
au/jchalupa.sql.gz
- Note the DB will grow like hell, one day import brings in like 50k of records in some tables.
- Download httpd logs for particular month from netbeans-vm:
$ rsync -avz username@netbeans-vm.apache.org:/var/log/apache2/netbeans-vm.apache.org_access.log_yyyymm* ./
- Put logs to location from where it can be served by local web server, keep them under
/yyyy_mm/
folders for each monthGet the import & stats script sources from github apache/netbeans-tools/pp3/au. - Scripts are written in super old PHP, run in cli and are compatible only with PHP up to 5.6. So you have to have php6.5 cli installed. Sorry about that.
- Configure sctiptsscripts:
au/config.php
- for importing data; db connection, paths to logs are set there-
au/db_connect.php.inc
- for building stats; db connection
- Make
au/last-date-dlc
file writable for user (if necessary). - Make sure
au/last-date-dlc
contains the date of last logfile you have before current import. - Note that to have correct monthly data, you have to have last 3 months already imported. Script looks 3month 3 months back to recognize AU user. So if you count for 2020-11 month, you have to import 2020-08, 09, 10 before that.
- Import In order to import one month of new data run: in shell run
$ for d in {1..31}; do php5.6 run-au-import.php -s dlc; done;
- If there is a new release in the imported data, you have to register it in DB:
- get ID of new catalog:
SELECT
id FROM `catalogs` WHERE `path` LIKE '%XX.Y%';
replace XX.Y with new version
- register it:
INSERT INTO `releases` (`version`, `catalog_id`, `lang`, `stable`, `delay`, `product`) VALUES ('NB XX.Y', 'ID_FROM_STEP_ABOVE^^', 'en', 'Y', 'N', 'netbeans');
- get ID of new catalog:
- Generate In order to generate stats run: in shell run
$ php5.6 build-dashboard_v2.php -p netbeans -m 2020-09