...
The content of the update center (nbm files, etc.) is distributed through the https://www.apache.org/dyn/closer.cgi Apache Mirror System
These nbm files (NetBeans modules) are currently changed on each major Apache NetBeans release (or release candidate, etc.)
See
Jira | ||||||
---|---|---|---|---|---|---|
|
Analytics
The Apache NetBeans (incubating) project is interested in keeping track of how many modules are downloaded, so we can minimize the bandwidth consumed on each release, and understand which modules are most downloaded.
Since it's not currently possible to look at the log files of the mirror system Apache Mirror System, we've devised the following mechanism to keep track of this information:
- The Apache NetBeans IDE downloads the update center contents from https://netbeans.apache.org/nb/updates/9.0/
- This .htaccess file then redirects this request to the Apache NetBeans VM Box.
- The requested file is logged into the Apache NetBeans VM Box web server log.
- A second redirection (using the file at
/var/www/html/uc/9.0/.htaccess
in netbeans-vm.apache.org) sends the request to a closest Apache Mirror, building an url that uses the closer.lua mirror redirection scriptthe closer.lua Apache Mirror Redirection Script that returns the URL of the closest Apache Mirror for the binary.
NetBeans AU stats counting
This is how-to setup local infrastructure for counting NetBeans AU numbers from update center webserver logfiles.
- Prerequisites:
- mysql/mariadb
- PHP cli <= 5.6
- local web server
- ssh access to netbeans-vm.apache.org (to get logs)
- Get the import & stats script sources from GitHub apache/netbeans-tools/pp3/au
- Create dedicated local database.
- Bootstrap database with schema and basic data from SQL dump available in sources
au/jchalupa.sql.gz
- Note the DB will grow like hell, one day import brings in like 50k of records in some tables.
- Download httpd logs for particular month from netbeans-vm:
$ rsync -avz username@netbeans-vm.apache.org:/var/log/apache2/netbeans-vm.apache.org_access.log_yyyymm* ./
- Put logs to location from where it can be served by local web server, keep them under
/yyyy_mm/
folders for each month. - Scripts are written in super old PHP, run in cli and are compatible only with PHP up to 5.6. So you have to have php6.5 cli installed. Sorry about that.
- Configure scripts:
au/config.php
- for importing data; db connection, paths to logs are set thereau/db_connect.php.inc
- for building stats; db connection
- Make
au/last-date-dlc
file writable for user (if necessary). - Make sure
au/last-date-dlc
contains the date of last logfile you have before current import. - Note that to have correct monthly data, you have to have last 3 months already imported. Script looks 3 months back to recognize AU user. So if you count for 2020-11 month, you have to import 2020-08, 09, 10 before that.
- In order to import one month of new data run:
$ for d in {1..31}; do php5.6 run-au-import.php -s dlc; done;
- If there is a new release in the imported data, you have to register it in DB:
- get ID of new catalog:
SELECT id FROM `catalogs` WHERE `path` LIKE '%XX.Y%';
replace XX.Y with new version - register it:
INSERT INTO `releases` (`version`, `catalog_id`, `lang`, `stable`, `delay`, `product`) VALUES ('NB XX.Y', 'ID_FROM_STEP_ABOVE^^', 'en', 'Y', 'N', 'netbeans');
- get ID of new catalog:
- In order to generate stats run:
$ php5.6 build-dashboard_v2.php -p netbeans -m 2020-09