Page History

...

A major refactor is proposed, on the usage server to increase the usage server performance

...

Feature Specifications

This feature proposed a proposes a refactor on the usage server ensuring that the following requirements are met for administrators:

• Run the usage server in a multi-threaded mode (usage server is taking a lot of time, seems like single-threaded). Explore. The maximum number of threads is controlled by a new usage server property/configuration to the administrator. Explore/PoC if multiple nodes can be active at the same time.
• Usage server related APIs to work with specified domainid/accountid and specific usage types (example: regenerating data for a certain usage type on a time interval).
• In case of duplicates where healing is possible, the usage job must continue running and fix the issue. In other casecases, an email or alert must be sent to the administrator for repairing.
• Create an API to re-generate records between a start and end date (delete records for a slice of time/dates and insert new regenerated usage data records). Assumption: having accounts, events, and helper tables to regenerate usage data.
• Add a global setting to set the number of years (minimum 1 year as default for fresh installations, and default value = 0 upon -1 upon upgrade for backward compatibility) to keep the usage data and remove the older usage data.
• Refactor and extend the sanity check task to ensure the usage types are covered, report any errors in a separate sanity check logs and send an email to the administrator.

...

• A new API (regenerateUsageRecords) must be created to re-generate usage data for a start and end date interval.
• The API must accept an optional parameter for an account or a domain ID. If this parameter is not set, then all the accounts are processed. If the parameter is set, then only the account or the domain accounts specified are processed.
• The usage data generated for the selected date range is removed (if any) and replaced with the usage data re-generated by the API execution:
  • If the account ID (or domain ID) parameter is set, then only the usage records for the account(s) are removed from database and the re-generated values are inserted

  • If the account or domain ID are not set, then all the accounts’ usage data will be removed and re-generated

• The API execution will create and persist a new job with a type = ‘REGENERATE’ to differentiate from other usage jobs.

• The existing generateUsageRecords API must be fixed to allow re-generation if data from cloud_usage.cloud_usage is missing for the specified start/end date but as per the job metadata it is found to be available. The existing API does not regenerate the usage records properly, it only works if there is no usage data generated after the initial time set on the API 

Service Layer, Schema and Other Changes

...

• The helper tables parsing will become a parallel task which can be divided by the number of CPUs the usage server instance has. It is also proposed to expose a setting in the usage server’s configuration file for administrators configuration. As there is DB and possibly network I/O involved we can use more threads than CPUs available to optimise CPU usage.
• The parallel tasks will receive a portion of the total number of accounts and will process them in parallel, using a synchronized queue of account jobs
• The parallel tasks execution must synchronize on the database writing into the ‘cloud_usage’ table, making sure only one task at a time is able to write. As the writing to the database will be parallelised, then the order of the generated usage records is not kept as it currently is persisted on a per account basis. It is proposed to extend the list API for usage records to order the retrieved usage records on each query.
• In case of multiple usage server instances, the tasks can be parallel and concurrent across the different usage server instances. This is to be explored, it is proposed to explore the benefits of multi-threading vs multi-process
• The heartbeat task can be removed in favour of parallel coordination between tasks.
• In case of any failure, such as violating MySQL constraints by attempting to insert duplicated records, then the usage job must fail and log the issues found accordingly. All the usage data that is generated up to the moment of the failure must be persisted in database, and the administrator must be notified via alert or email about the issues on the usage jobs. The next daily job must not re-generate the same data again.

The usage job must be more robust on its execution. One known issue with the current usage server implementation comes from duplicate event records causing usage job to fail and never succeed unless the duplicates are removed. To increase robustness one additional step can be added after the helper records creation and before their parsing. The additional step must perform the following actions:

...

UI Changes

N/A

Marvin Tests

The following test must be included as a separate marvin class (e.g. have its own tag) to prevent it runs automatically on production environments as it will change configurations in the system, unless it is explicitly executed by an administrator. However, the test must be included on the Trillian automation

• Create a few resources on a certain account such as running VMs
• Set the global configurations for the aggregation range to a low value (around 5 minutes or less) and the usage job start time to a time close to the execution time so that a usage job can start almost immediately after the usage server is started

• Start the usage server

• Wait for 2 or 3 times the aggregation range minutes so that usage records can be generated.

• Stop the usage server

• List usage records for the account and ensure records have been generated for the resources created and a certain type, for example running time for VMs

  • Ensure the number of records match the expected number depending on the time spent generating records

• Wait for 2 or 3 times the aggregation range minutes, set the aggregation range to daily

• Start the usage server

• List the usage records for the account and check the number of records per a certain resource and usage type

• Regenerate usage records for account on the date

• List the usage records for the account and the same resource and usage type, and compare it to the number obtain above. Ensure the number of usage records is higher than the previous check.