Implement a unit testing framework and strategy to easily allow others to help in getting a solid set of tests built in a rapid and replicable manner
Address the challenges with our build processes such that units can more easily be included without adding more friction and build time to the contribution process
Addressing flakiness of existing tests
Ensure this happens smoothly in parallel to ongoing efforts led by Petri and Aleks around moving from raw strings to type-safe REST resource classes
Summary
Selection of Test Framework
Cucumber was chosen as it enables business-driven test design where semi and non-technical individuals with more domain knowledge can easily write test cases.
Cucumber is good because you can use keywords and prose for test case definiition and someone else can simply go in and fil in gaps with conditions to run test cases
Identify which unit test cases to focus on and ultimately have a wiki page with five categories, each category having 100 to 200 test cases and red or green tick mark to show if we have coverage.
Near-term step to having this page is for individuals like Nayan and Istvan based on the test cases their teams have implemented to input into a Google Sheet at least the categories and ideally the test cases that were implemented so these can be focused on.
Create a replicable cookbook for creation of test cases using DBUnit for setting up the data base and providing a few examples for others to follow. Test cases can be set up with simple CSV files.
In future can use generate Jacoco and other reports for test coverage.
Documentation In parallel we need to:
Update documentation on testing
Update our strategy and guidance on what tests (unit and/or integration) are required for PR to be contributed.
It is still outstanding if test should be part of the PR or separate
Don't want to add too much friction to developers by requiring too many test to be created
Could put skeleton of test in place in Cucumber, mark as ignore test and then set up Github Actions to create a todo list of ignored tests that other volunteers could work on.
Raw Notes
Discussion items
Time
Item
Who
Notes
Current State of Testing
Consensus on level of testing coverage at unit, integration, test level and objectives for what we should achieve
Aleks
Nayan: integration tests do cover most business scenarios but unit tests don't cover.
This isn't scalable - heavy dependency on developers to have to write this.
Domain experts can write far better test cases - need framework for semi-technical individuals to write test cases.
Aleks: a lot of manualy boilerplate code throughout application
Manual classes in REST API - impedes effort of getting tests out there.
Aleks: useful to use cucumber -
Disconnect between tickets, documentation and what is actually tested?
Difficult to understand test coverage
Only thing aleks can do is look at code and try to cross-read
With Cucumber cards can do annotation and any tags.
Can introduce standardized tags according to tags/modules
Tags could point to JIRA ticket and can create some stats
Also a good feature that converts reports into ASCII doc which will immediately.
Why our integration tests are peculiar
Testing business logic at rest API level
Better to do at unit test so isolate and not have to ramp up entire Fineract
Task: make integration tests faster and should also migrate logic that is there to proper unit tests
Can then run these during development and not have to wait like 5/6 minutes which is time it takes to run test cases currently
Should be as a button click in dev environment - simply click and run.
Metrics to Improve
Istvan: what overall do we want to improve?
Aleks: Recently got very frustrated by build process and tests = long time and too complex to understand
Raw numbers for time to wait - compile - gradle kicks in always - no usage of gradle cache whatsoever - effectively wait between 5 and 6 minutes until tests are ready
Even just selecting one integration test case, takes 10 minutes. Might only have one or two test functions but a lot of code - tests are easily 1000 lines of code.
Not easy to grok idea of what's being tested.
Overall an hour. to run in a consistent build
Added test containers but then took 2 hours without the optimizations
Ripped up build scripts - lot of manual task dependencies in place and removed all of that - some build steps might not be in right order but good
Dependencies on quality checkers but do you need these all the time.
Got build process down to under 20s with proper docker image.
Previously with docker compose file - docker image takes 8 to 10 minutes to finish - no caching used - replaced with google gradle plugin Jit?
Don't need tomcat plugin or additional mysql - can test easily with proper docker image.
Sneaky configurations with java parameters that was tricky to figure out.
Steps in build script to ensure not ignoring warnings/errors - settings were quite strict - array with parameters - build script different and then add more parameters - by splitting the paramters into two arrays - created a lot of waiting time.
Task: create pull request for improving build script.
Overall time - under 9 minutes for complete suite of tests and under 20s for build.
437 test cases but can't categorize where these are
Some areas where more things are tested.
INtegration tests aren't too bad but just very slow.
Nayan: reducing developer frustration with build time and test time
Instead of going with continuity analysis - not all test coverage is equally important
Features dealing with money - have most functional test coverage
Don't go just with thumb rule of what is my code coverage - what is critical business feature and how much coverage doing.
For banking project, typical code coverage for lending or deposit module have 10 to 20K test cases.
Reason for this low coverage is friction to contribute - only developers can write end to end test cases - tedious for developers
Don't have test data at the moment for writing test cases - i.e. to just test waive charge need to write dozen test cases to do so.
Task: Nayan - need available set of test data
Aleks: why a lot of time goes into integration tests - because test scenario setup is happening over API calls and even if local - this is not most efficient
Better approach is not have business cases tested in integration - test one by one in isolated unit tests - db unit or db writer
Can do data preparation of your database with CSV file - fill in few tables that need for busines case setup and then do business logic testing.
Nayan: agrees and thinks its obvious
Task: Test business cases in isolated unit tests - db unit or dbwriter, Test data with database
Aligning with ongoing community efforts
Old spring XML configuration still in place - when you try to integrate other stuff i.e. Apache Camel, couldn't do with simple Java Configuration - should just drop in Spring Boot application and run immediately
Petri migrated XML to more modern approach - javaConfig and also migrated outdated REST library Jersey 1.7 to most recent one - immediate performance gain.
Now REST API layer is in good shape technically but still hav emanual JSON mappings with GSON, ignores all infrastructure that get from SpringBoot
PIck up small portion of REST API layer and put in classic spring boot applicaiton with Jaxon and no mapping
A little bit of helper code with JSON parser. - some situations where can' seralize or de-serialize JSOn strucctures = not complex though and only for a handful of classes.
Outcome: REST API Layer that is type-safe
Instead of spring blob to transform - get proper java classes and let jaxon do stuff and remove all manual swagger maintenance code and proper swagger descriptors and in turn proper java API client, use that java client to do integration tests as well
Once REST API Layer is type-safe, if look at swagger descriptor, no categories so name clashes - name clashes sholud be resolved by code generator automatically. right now have mu;ltiple get-by-ids because in same namespace, can introduce different namespaces and have different functions
Customers look at this and get confused by couiple hundred rest endpoints
at Finflux doing similar effort, not sure if when running code generator multipel times.
Discuss optimal approach
Adam: Also in agreement with using Cucumber as well - other projects using it in-house for testing.
Previous project only wanted to see coverage through unit tests and not via integration tests.
Ensure that unit tests were there for business logic as well - both positive and negative scenarios.
Adam: take a hybrid path - unit tests for critical business logic - charge calculation, prepayment calculation, cover the logic by unit tests
Integration tests - most beneficial way is to use is for regression and to check that all is intact and as designed.
Quality of the tests was very important to them - usually not happy with coverage initially so were focused on if it was really tested. Regression costs are extremely high so want to avoid at all costs and catch as early as possible.
James' Summary
Cucumber Framework for business-driven test design
80%+ coverage
Metrics for reducing developer frustration
Execution Plan
WHat are sequencing of steps and dependencies
Create tickets for getting framework implemented for others to assist
Implement changes to build scripts that Aleks has done locally into upstream.
Are changes to REsT API Layer needed?
These changes can happen in paralle
Identify and separate out failing integration tests that should be in unit test cases
What is proces for getting cucumber into build - is quite easy to introduce that into build.
implementation of test containers for integration tests
Create test data set → have a plugin for test data.
Separate out busines logic
Write new unit test cases
Can maintain cucumber cards in JIRA - smartbear now owns and have plugin to syncrhonize cucumber card and Jira system
Not used that yet - in theory non-technical people with domain knowledge could edit requirements/specs and use cases in JIRA
Maintain structure format with keywords, etc - synchronize reports/specs with git repo.
Alternative approach is access to git repository - VS code online intetgrated with Git - plugin that works with cucumber to work with syntax highlighting - can just edit these text files -
Make a replicable cookbook for creating these test cases
DB Unit or DB writers - setting up use cases
Few examples that others can follow - for setup of test cases - don't need to be executed over internal API - easier done with simple CSV files - assume that's thee and not concern.
UPdate documentaation on testing
Update our straategy on what tests are required for a pr to be contributed.
If you update a feature or functionality , ensure that existing unit tests are all working
If add something new, add unit tests to test positive case and negative case for that.
James - functional are should be tested if
generate interest in overall framework rather than have developers do all of it
More imortant metric that for every 2 developers, one QA person writing tests
PRs need to reference at least 2 positive and negative tests.
Should tests be part of the PR or separate?
Can be frustrating to be asked for tests and provide positive/negtive for that
For cucumber this is really just text - put skelton in place, two placeholders cards and mark as ignore test
Put in github actions and see how many ignore tests have → becomes todo list for opeople to attack this and reduce amount of ignored tests.
Could be acceptable requirement for efforts
MVP
Implement cucumber - very simple
convert a few integration tests to unit tests as ecam
test containers - implement later
Timeline for implementation of frameworks
Near and long-term metrics.
if we implement this stratregy
Code to cover these tests.
Jacoco report is just done over java api layer and not rest API Layer
Will get more visiblit
Outlinig
Resourcing:
Nayan has domain experts
Istvan: Good to have framework but need to narrow down which unit test cases to focus on. - interest calculation, batch jobs, leap years
Aleks - have a blueprint for this but saw test scenarios that go. in this direction - difference now is setup and test execution is by REST API.
Blueprint is there
Read inside integration tests, repayment schedule
Aleks: can interpret what was aiming to do in REST as java rest apis
Istvan: shoud have HTML page with five caetegories
categories each have 100 to 200 test cases and a red or green tick to show if we have them.
Do we have these and a prioritization
Aleks: can take current number and have roughly
Wiki page - chaeecking/savings accounts - what are gourps to test, specific ocniditons to tests and batch jobs - high level would help
Google Sheet
Nayan: can take on have functional test caess to implement
Goals of Approach
Adopt framework such that semi-technical indivudals can write test cases as domain experts are far better
Aleks: Cucumber would be good - keywords and prose used for text case definition - someone can go in and fill in gaps with conditions to run test cases
useful to do parameterized tests - will reduce boilerplate code
Much of test code is setting up scenario - real testing only starts after set up the APIs
Can take shortcodes by describing in cucumber test card and it's also more readable.