06. 11. 2024
AI, Log-SIEM, Machine Learning, NetEye
The Research and Development team (short R&D) is the largest team in the Würth Phoenix System Integration business unit. Our responsibility is to develop, maintain and deliver high quality software to our customers. We also provide second level support in cooperation with our Service & Support team.
In past, the SI department was one of the fastest growing departments in the company. This rapid growth had also an impact on our team, forcing us to face new challenges in various areas. The most notable were workload distribution, issue prioritization, time efficiency, knowledge transfer and as a direct result continuously integrating new colleagues.
To keep delivery times low and product quality as well as the motivation of our members high, a decision had to be made, and the idea of the Agile Transformation was born.
First we restructured the organization of our team. Out of several tiny, product specific teams emerged one single R&D team. This “new” team could be more flexible, allocate new hires efficiently and distribute knowledge among all members. Finally, in spring of 2016, our team went from waterfall-ish to full agile development.
This agile transformation, lead by Professor Werner Wild, well known for his agile expertise in the IT world, had major effect on every aspect of how we approach improvements and how we handle requests and issues. To keep things simple and readable, every article of this series will treat one aspect of our development processes and organization.
In this first part of the series I want to provide a general overview over the restructuring of our team, our workflows, recent organizational changes and our goals.
Agile Methodologies & Task Management
The first step of the transformation was finding the root of our major shortcommings and understanding if we can apply existing solutions fitting them *and* also our team. We decided early on using the principles of Lean Software Development as a base with a Multi-Kanban setup for task management on top. (There will be an article about just this, later in the series). As digital aids we use the Atalassian Stack (BitBucket, JIRA, Confluence and HipChat for sourcecode-, issue- and knowledge-management.
In the real world we use three physical Kanban boards and two big monitors for our Dashboard/Andon which are used to keep track of what’s going on at any moment. We also reorganized our workday in four working time slots, 2 hours each, and enforced other teams to respect them in order to minimize interruptions.
With those new tools in place, we were finally able to get proof of what we already suspected during the analysis:
We prioritario too many tasks and issues at the same time, with the risk of a prioritization loop where sometimes issues would be re-prioritized repeatedly but hardly worked on.
Once we had the visual representation of the tasks written on Kanban cards and sticked to the the Kanban boards, it was a very satisfying experience for the whole team to see the queues shrink. From that point on we could stick to an agreement on the maximum concurrent workload (number of issues on Kanban).
To achieve a useful quantification of this concurrent workload we agreed on estimated boundaries for the size in time of each task. We agreed on one to six time slots which means two hours to two days. Some bigger tasks had to be split up in multiple sub-tasks and several minor tasks had to be organized into logical groups. In this way we gain multiple benefits and at the same time resolve several causes of our difficulties. Most notably are:
- Predictability: We know that in most of the cases if a task goes in the development queue, it will be in “Done Done”-State after at most 2 days. (=cycle time)
- Fast delivers: With this procedure and our new state of automation, clients can immediately get the additional value from new features in releases.
- Eliminate waste: An unfinished task does not add value to the product. As we are now continuously completing and deploying features for our products they benefit *directly* from our development.
- Quality: three of four steps on our main Kanban are directly or indirectly related to code quality (Development, Testing, Review)
Automation
Our new grade of automation on code deployment requires adequate and evenly automated quality control, however we are not only deploying and testing continuously but automated most of the previously time-consuming manual development and deployment steps. For each of those automation tasks we use Jenkins (previously Hudson), which will be the topic of a future article of the series. Now I’d like to just give you a short overview:
- Continuous Integration: Automated Builds directly from our development and maintenance branches
- Continuous Deployment: Automated deployment of new packages containing bugfixes and features from BitBucket to the correct repositories
- Continuous Testing: set of Integration- and UnitTests (+ 4 Selenium Instances) executed whenever a change is pushed to the repository in combination with our Dashboard/Andon to fulfill Lean Developments “Fail Fast” Principle.
Furthermore, we trained ourselves to recognize when we execute similar tasks repeatedly and to think about *if* and *how* they can be automated, such that the precious time of each team member can be allocated efficiently.
This will probably be an ongoing optimization process, and we still have a lot to do. However, the time we have to dedicate to those scripts will diminish and the saved time by automated solutions already now outnumbers the invested time by a few orders of magnitude. Practical examples from our everyday work are the full release processes, auto-generation of Kanban cards and time tracking.
Keeping an Overview
As previously mentioned we are strongly relying on automation, but also need immediate feedback on what is going on and in case, what is going wrong. To monitor this we are using the opensource solution Dashing an easy-to-setup and easy-to-extend solution to display various Dashboards in our office. We use them to display emergencies, build progress, last deployments, jenkins job status, issues, project progress and key metrics for our team. This topic will be covered in more detail by a separate article.
Daily standup meetings and weekly retrospectives were introduced as means for knowledge distribution and as self reflection tool. These help us to identify slowdowns and to keep up with ongoing events in the team, as well as giving each member the possibility to address important topics.
Before we started the transformation all knowledge was accumulated around the people who regularly developed on specific products or modules. New or less familiar team members had difficulties to aquire knowledge about a topic by reading. To counter this and actually share the available knowledge efficiently, we started to heavily document everything, not just in code, but also structured in Atlassian Confluence. This helped to encourage all members to work also on unfamiliar code. As a result every member is gaining experience with all projects.
Goals
As you can se we were starting from a situation where we were struggling when our environment and situation changed, but our organization didn’t. Together we managed to identify the causes and guided by Professor Wild we identified goals, which were defined such that reaching them, means resolving one or more of those key difficulties.
An example would be the limited task duration of 1-6 time slots. This effectively means a cycle time of less than 2 days. In this way we are able to predict the development duration and estimate a theoretical maximum effort. In combination with the Kanban boards this effectively limits the maximum concurrent workload in the team, reducing the “waste” of context switching, unfinished tasks and thus saving money and time.
Another major help for waste removal was investing time into automating tasks.
The time invested in the automation scripts already now paid back multiple times. Previously, deploying a package could take up to one hour (from the last commit to the deployment on RPM repository), whereas now it takes at most three minutes.
Automated tests, pair programming, code reviews and a dedicated step on the Kanban column for manual testing are having a significant impact on code quality.
Basically with this Agile Transformation we set up an Environment which allows us to respond quickly and efficiently to new and changing requirements not only for our software but also our teams. And I can honestly say that it was a total success.
Benjamin Gröber
R&D Software Architect at Wuerth Phoenix
Hi, my name is Benjamin, and I'm Software Architect in the Research & Development Team of the "IT System & Service Management Solutions" Business Unit of Würth Phoenix.
I discovered my passion for Computers and Technology when I was 7 and got my first PC. Just using computers and playing games was never enough for me, so just a few months later, started learning Visual Basic and entered the world of Software Development. Since then, my passion is keeping up with the short-lived, fast-paced, ever-evolving IT world and exploring new technologies, eventually trying to put them to good use. I'm a strong advocate for writing maintainable software, and lately I'm investing most of my free time in the exploration of the emerging Rust programming language.
Author
Latest posts by Benjamin Gröber
25. 11. 2022
Development
What is the perfect development team size?