The picture was taken for Zixia (Purple Light) Lake in famous Mt. Zijin of Nanjing (Nanking) city by my previous colleague. When we talk about migration to agile and automation test, it is rather to create a harmony with history assets and moderm methodology like to vitalize traditional building in morden metropolis with inner high-tech structural works, rather than a revolution on exteriority.
(\(\color{blue}{Lively\ Updating}\)) This post is based on the original Chinese blog articles and most parts are rewritten. I would keep on retrospectives and update it with new ideas periodically.
Brief
A better title could be "Myths in Test Automation -- a view from engineer". However, not to scare readers, we just keep it lively maintained as issue list with contourmeasures and thoughts.
4 versions of engineering team with/in testing
Version 1: Streamline Workers in Testing
Version 2: Agile Testing
Version 3: Role changed from tester to QA
Version 4: Full Stack Testing Team
Here Full Stack is a term to reflect an end-to-end view from QA perspective rather than just testing.
Where we are -- The shape of test automation
At first, this section describes from a high level view on how to reinforce test automation.
Test Pyramid -- Ideal Agile Testing
The chart of Pyramid model is from Martin Fowler. Test Pyramid is from Mkie Cohn. > The test pyramid is a concept developed by Mike Cohn, described in his book Succeeding with Agile. Its essential point is that you should have many more low-level unit tests than high level end-to-end tests running through a GUI. > 
Like bringing data as close to code as possible, the test shall be as close to code as possible. The distance of data and code increases the cost to warm data and rotate memory pages with cache line. The distance between test and code, in both space and time dimensions, could be regarded as a sign of cost of organization focus switch.
Alister Scott added manual test on top of the pyramid.
Fast iteration does not offer testing team a long time to perform more manual tests and the automated CI/CD becomes the fundamentals of agile to shorten the feedback cycle towards an autonomous learning organization. Even though, there are spaces for manual scenario based testing and exploring as an indispensable part of Agile Development cycle. It is true that the manual test sessions are a bit further to codes and extend a feedback cycle, but it is important to inspect from user scenario view and utilze human initiatives to qualify the software crafts and process.
Another take-away is the ability of test works to be reused. The lower it shows in above chart, the better adaptability it could have.
UT is closest to code and will be maintained with code changes together with CI protection. The gap is counted by minutes. With Lava Light and CI, UT is consistent with code within 20min for most cases.
API and component interfaces have a middle level re-useability. Most of the test works are stable by a well reganized design style. If there are issues that APIs change frequently according to busniess, it is a smell of Architecture Refactor. The original design does not expect the busniess evolution model.
For UI layer and manual scenario tests, it is costing to maintain through releases because there are presentation of business and expected to change accordingly. With technologies like Page Object and a good automation framework, the costs could be mitigated sometimes.
Icecream Cone Anti-Pattern -- The Reality
Alister Scott also pointed an "Icecream Cone Anti-Pattern" to describe the reality of testing industry.
Where the organizations have history assets or in an intermedium position to grow skilled engineers in testing, the most evident activities are still manual tests.
One possible reason is with culture building, as Conway's Law points out, the organization way of evaluating co-workers will come out with the output of software processes. If engineers work in an atmosphere that the bug numbers are a key to evaluate themselves, the manual test, with its time and resource consumption will be an asset, or, a debt, of the orgnazation.
Another possible reason is also on culture that developers don't have enough encourage or time to protect works with Unit Test. On the contrast, writing unit tests will influence developers to think about how peers are using that part of code and the structure will be designed to adapt a better decoupling philosophy and a better testability (the level of how the program is friendly to test, and then friendly to refactor).
Cupcake Anti-Pattern -- Reality of Agile Testing
Software Testing Cupcafe Anti-Pattern is a model by Fabio Pereira, Thoughworks to describe what agile turns out to be in most organizations.
The most siganificant take-away from this model is the isolation smell. It works partially as a mini-waterfall model that groups of team members make duplicated coverage. It usually happens when there is a separate automation testing team and an existing asset of "manual" test group to keep current business running.
Rugby Model -- Time to propose a theory to experiment
Focus on the middle layer instead of Unit Test. During the Agile Migration, pay most attention on API testing. For example, RESTful API testing can be extended to scenario test and create conditions and checkpoints for sceanrio level inspection. The RESTful API level automation testing can also provide a primary data analysis on which part is the most urgent section to protect with Unit Test.
It is not a fresh new concept. The term Subcutaneous Test covers the similar way on a pseudo end-to-end test but avoid the UI complexity (complexity is usually a singal of heavy design and indicator of high risk on test maintenance).
Thoughts on common practice issues in Automation Test
Tools matter -- Yes
Tools are not only tools in a transverse movement, but also a way of cultrual idea exchange and influence on way of works.
SW Dev Frameworks are tools and ROR almost defined the practice standard for a narrowed scope of RESTful. Without JUnit, XUnit, the basic Unit Test concepts would NOT become as popular as they are now. Without 120 fps technology, movie "Billy Lynn's Halftime Walk" could hardly bring audience the experience on what Ang Lee is exactly designed. A home-made toolkit or customized automation framework is a way to influence crew, too.
It is an effecient way to communicate with whole team on objectives, process, learning topics. As an old saying in Lucent, tools are ways to train people by supporting ourselves.
Mixed Coding and Test Skills -- Yes
This topic is still debating upon situations. This is the mostly heard challenge when quite a number of organizations do "SWOT" analysis towards test automation. However, the observation is that interruption matters more. Below is based on the close observation and discussions with test professionals.
When I am doing a manual test to GA a SW release sometimes, I don't expect an out-of-observation-scope interruption to system which drive me have to start over. It costs energy for human beings to concentrate for a while no matter what we are doing. Testing is somehow a lonely work without in-time feedbacks as in games. To predict the logic branches and weak points consume more than it looks like.
On the other side, coding skills are still inevitably critical to implement a stably reproducable test smoothly. Programming language expertise is vital not only on exploring and implementing tests, but also to form a predictable but effective ways to design the tests. Relationship between factors in space, time and logics are expected natrual to engineers in automaiton test role. Therefore, the tests, CI and CD can provide confidence to joint agile team. I keep an average of 1.5 issues discovered per working day in first two weeks of manual testing phase. The credit gives to development experiences. Which helps me to think in a systematic structure and perform test in well organized space as a journey with several dimensions.
This way, we say that testing is the way to use techniques to mitigate uncertaincy.
Tree vs. Forest -- Both
As a software engineer, I do have happy jouneys with occassional automation test in early years. Which are also called one time tools. The utilities were created to resolve a certain type of testings in such situations:
- Some tests are not able to complete manually. Constrains on space and time might be not easily to satisfy with manual actions.
One example is burst Wi-Fi connection and disconnection to a home gateway.
When I first realized an issue was with memory fragmentation, the test case in my mind is to call volunteers from my team together with all Android tablets, mobiles, laptops we can borrow and carry to lab and make a series of burst join/quit. It is true and I really did it once .I still keep the photo that a group of people are doing Wi-Fi connecting test together, for memory, for the sweats and cheers).
But after we reproduced the issue and successfully match the symptom as filed team suffered, it was not feasible to stay with the so called crowd action to qualify the patch-up component integration. Therefore an urgent request of tool to stably reproduce this issue is created. An one time tool could turn the unknown uncertaincy to known uncertaincy and release efforts.
- Some test case request special skill to reproduce.
- It is like that some piano songs can only play with computer because the pace is too quick human fingers can hardly move this way.
- An example is a race condition with home gateway GUI, TR69 management interface. It exhausts patience to reproduce it and record every execution sequences to table. Some of the pattern is sophisticated and ambiguous if we only tell from the text description.
- To explore more during the test.
- Tools like JBroFuzz provides precompiled fuzzers to offer randomization for different purpose. If the exploring is only on the space dimension of data feeding, as changing from a ascii-lower set to base64 set, or explore SQL-injection set for MySQL DB, it is time to introduce one time tools.
- Knowing there is an issue, but not sure how big the impacted scope is, or, what exact the trigger threshold is.
One-time automation test is like trees. This is an utilization of tools to achieve above expectations. It is happy and cheerful. However, this is not the same concept of test automation.
[Pub-2, 2019-05-19]: Update the image links]
[Pub-1, 2017-06-09]: Since most parts are rewritten or revised, the release date of this post is changed to recent day]
[draft-2, 2017-06-02]: Add agile test models, bring up Rugby model]
[draft-1, 2017-01-25]: Initial draft.
Some of the drafts are not on original Chinese version post, so a special date 12/31 will be used to track the global version.
[EOF]