This study used a true experimental design method with a pretest-posttest control group design approach that attempts a type of experimental design where the researcher randomly assigns test units and treatments (DevOps) to the experimental group (system analyst, programmer, developer, system administration and database administration), with the aim of systematically describing the facts and characteristics of the object under study precisely, using primary and secondary data from a previous ticketing system and implemented DevOps.
In elaboration of the methodology and to answering the research questions, the researchers studied the literature (Microsoft, 2021), seven processes are included: boards/backlog, apps source code web.config, repos, test plan, cicd pipeline, web apps, & review/insight as seen in Figure 1.
The data flows through new processes in Figure 1 as follows:
- Source code is revised or created by a programmer or developer.
- The source code repository in Repos is compiled with the application code (including the web.config file).
- Using Test Plans, continuous integration activates application builds and unit tests.
- Continuous deployment activates an automatic deployment of application objects with environment-specific configuration values within pipelines.
- App Service is used to deploy the objects.
- Review/perspectives based on wellness, efficiency, and consumption data that has been collected and analyzed.
- Developers keep track of and maintain data on health, performance, and use.
- Using Boards, backlog information is used to prioritize new functionality and bug fixes.
Analysis and Results
In this process, the researchers tried to revise the existing business process when deploying the application to maximize the delivery and timeline; initially the university had not adopted DevOps. The most common complaint that occurs when deploying applications is that the system administrator is not ready when the programmer has finished programming or coding. This often happens because the development of an application often changes according to business/stakeholder needs.
Shifting urgently from one project to another in business must be done by companies considering that those who cannot change or innovate will die; so, the final project often changes over time depending on the needs that occur. The impact of project changes and urgency on system analysts and programmers is that they must be able to keep up with the required changes with a short timeline and deployment times are quicker. Meanwhile, the number of system administrators is limited and when deployment is done manually takes time to serve a single deployment project so that there is often a queue for when deployments can be carried out.
Meanwhile, adding to the number of system administrators is also not effective when the deployment queue does not occur every day. They may only return at uncertain times. How DevOps can solve this problem could begin from good communication between the project manager/system analyst and the programmer(s) into the Boards.
Boards
The interactive lists and signboards below are provided by Boards. Each tool offers a filtered list of tasks. All of the tools allow you to display and define work objects.
- Work products include: Used to quickly identify assigned job objects. Work things that we follow, are listed in, or that we viewed or modified can be pivoted or filtered based on other criteria.
- Boards: Boards display work items as cards and allow for drag-and-drop status updates. Sticky notes on a physical whiteboard are similar to this feature. Used to imagine a team's work flow and incorporate Kanban activities.
- Backlogs: Backlogs are lists of job products. A product backlog is a repository for all of the information we need to track and share with our team, as well as our project plan. Portfolio backlogs allow us to categorize and organize our backlog. Work can be planned, prioritized, and organized using this tool.
- Sprints: Sprint backlogs and task boards offer a filtered view of work items allocated to a particular iteration course, or sprint, by a team. We can drag-and-drop tasks from our backlog onto an iteration course. The work can then be seen in its own sprint backlog. To put Scrum activities into action, use this tool.
- Queries: Queries are filtered lists of work items that you create using a query editor. The following tasks are supported by queries:
- Look for groups of work objects that have a similar theme.
- Create a list of job objects to share with others or update in bulk. To prioritize or delegate a collection of objects, triage them.
- Build status and trend maps, which we can then use to populate dashboards.
In Figure 2, project managers or system analysts and programmers, can see the progress of the status of each task or user story; they can find out which tasks are still not being done with a 'New' status. Meanwhile, the task currently still in progress by the team will be put into 'Active' and could be assigned to the designated programmer. When the task has been done by the programmer, the task is moved by the programmer to 'Resolved' to start testing and quality assurance checking by the system analyst. When the testing has been completed and the requirement has been fulfilled, the system analyst will move it into 'Closed', meaning it is ready for deployment or merging to a production branch, which is directly in line with the CI/CD of the production environment. With Boards, the monitoring of status of each task from backlogs or stories becomes easier to communicate and collaborate as a team.
In the Backlogs menu in Figure 3, all the tasks/backlogs/stories in the project can be seen, along with the points and efforts in their execution. Like Boards, Backlogs can also monitor any tasks or stories whose status is new, active, resolved or closed.
If the points get bigger, the effort in solving them will be even greater. In DevOps, the determination of these points and efforts is calculated by each team and how they determine it depends on the method used in making the project, whether waterfall, rapid, scrum, etc. In this study our point creation is based on man-days and the iteration per sprint is one week. Each project can be different in the length of the sprint depending on how big the scope of the project is. In Figure 4, we can see detailed tasks/stories per sprint, per iteration, and how much effort per iteration.
Repos
Repos are a series of version control tools for managing code. Using version control as soon as possible, whether a software project is big or small is a good idea. Version control systems are pieces of software that allow us to keep track of changes to our code over time. We instruct the version control system to take a snapshot of our files while we update them. The snapshot is stored permanently in the version control system so we can return to it later if appropriate. To save our work and organize code changes around our team, we use version control. Even if we only have one developer, version control keeps us coordinated while we fix bugs and add new features. Version control records our progress so that we can easily revisit and roll back to the previous version of our code. Version control is given by repos in two ways:
- Git is a distributed version control framework.
- Team Foundation Version Control (TFVC): version control that is centralized.
In Figure 5 above we can see the existing code file (source code) created and show the versioning every time there is a change/revision in the code, especially when it has been merged to the production branch.
If the file code has been changed to implement then we could monitor the history (Figure 6) so that if there are problems, we can find out when the application was changed, by whom, and what things were changed. The determination of naming when merged can be determined by each team.
In the branches menu in Figure 7, we can see what branches are available. In this research we only used one branch, the master branch, where if the code is merged to the master branch it will also be deployed to the production server.
Test plan
To track manual testing for sprints or milestones, we should build test plans and test suites. We will be able to see when the testing for a particular sprint or goal is finished this way. Test plans, test suites, and test cases are the three key forms of test management objects offered by DevOps Server's Test center. These components are saved as unique types of work objects in our work repository. We can export and distribute them with our team, and all our DevOps tasks are tightly integrated.
- Test plans: Individual test scenarios, group test sets, and test plans are all combined. Static test suites, requirements-based test suites, and query-based test suites are all examples of test plans.
- Test Suites: Within a single test strategy, we divide test cases into different testing scenarios. It is easier to see which situations are complete when test cases are grouped together.
- Test cases: Individual sections of our code or app deployment are validated. We should ensure that our code is error-free and meets both company and consumer requirements. If we do not want to build a test suite, we can add individual test cases to a test plan. A test case may be referenced by more than one test suite or test plan. Without having to copy or clone test cases for each suite or strategy, we can effectively reuse them.
In this research, we did not take advantage of the test plan feature of DevOps as in Figure 8, but instead took advantage of the test plan on .net where testing the behavior of code is done when the code has been compiled successfully before releasing or merging to the master branch.
Pipeline
The DevOps Starter simplifies the process of setting up a CI and CD pipeline. We can either carry our own code and Git repository or choose a sample program. The steps are as follows:
- We set up access to our GitHub repository and pick a system (we can use GitHub or an external Git code repository) as well as Branch.
- Build a new free organization in DevOps Organization or pick an existing organization from the drop-down menu to configure DevOps and a subscription. Then choose Subscription and either give the Web app a name or leave it blank, then choose a place. DevOps Starter creates a CI build and release trigger for you. Our code is held in your GitHub repository or another external repository, and a sample app is created in a DevOps Organization repository. DevOps Starter executes the build and deploys the app to the DevOps Starter environment.
- We are now able to work on our app with a team after committing improvements to GitHub and automatically deploying them to DevOps. The most recent work is automatically deployed to the website through the CI/CD process. Each update to the GitHub repo triggers a DevOps create, and a CD pipeline triggers a DevOps deployment.
- Examine the Pipelines CI/CD pipeline; DevOps Starter creates a CI/CD pipeline in Pipelines automatically. As required, explore and customize the pipeline.
- When App Service and the related services that we built are no longer needed, we can uninstall them.
In the pipeline, we create and activate a CICD for automatic deployment. After the code is merged to the master branch, it will be deployed automatically to the production server. Configuration to be automatic is created in the pipeline. In this study we made one example of Trial2-CI pipelines as in Figure 9 above.
In this Trial2-CI as in Figure 10, we created a task:
- Compile or build a solution or project.
- Perform testing on the code according to the test scenario that we created in the test plan.
- If there are no errors in the compilation of the test plan, the final code will be built.
- The compiled code is copied to the artifact and published into the production server.
Meanwhile, the configuration of deployment to the production server can be seen which is done from the branch master and the release menu.
In this research, testing of application behavior is carried out when compiling and merging to the master branch; in this example we create a calculator application and when compiled there will be addition, subtraction, multiplication and division tests as in Figure 12 below:
If the output did not match the test that was conducted, then the task will fail and the code cannot be merged to the branch master, so the programmer must determine the presence of bugs and these must be fixed first.
In the release pipelines menu that was created, it will automatically be deployed to the target server; the example in Figure 13 above is deployed to local drive C:. However, in overall research we deploy to a production server or cloud.
Artifacts
With artifacts, teams of any size can build and distribute Maven, npm, and NuGet package feeds from public and private sources. Our CI/CD pipelines will now provide fully integrated package management. Multiple feeds are introduced in Objects, which we can use to arrange and monitor access to our packages. If you are familiar with NuGet.org or npmjs, you can think of each of those places as a single feed. A word on terminology: whether you use npm or Maven, you might have seen feeds referred to as package repositories or package sources.
In this research, we have not taken advantage of the features of artifacts as in Figure 14, where artifacts basically can be used if we want to publish the results of our package code to several repositories such as Nuget, Git, etc. The compiled code is only copied to the artifact and published into the production server.
Review/Insights
In this stage we collected and analysed health, performance, and usage data as an insight/review for the deployment of new/revised source code. Some business processes/implemented flows have been changed after we applied the DevOps CICD. Initially, the university had not used DevOps but had used a ticketing system for tracking problems when there was a problem with applications. If there is a problem with the application, it could be tracked from the ticket by seeing when the application was last updated and what was updated in the application. It could then be traced if the new code caused problems. However, with this system there is no code repository. The code that is deployed to the production server is sent from the programmer.
In this research, we tried to change the deployment process by using DevOps, which contains a repository code and implements continuous integration/continuous development (CI / CD) in the pipeline, so it changes the new business processes. There is no more system administrator role; it is replaced by automation flow from DevOps application as in Figure 2.
The results of the analysis on this change shows that the application deployment process at university is better because it eliminates dependence on people, system administration, where application deployments were often improved on server code applications then updated to the database configuration. Using available primary and secondary data, the comparison of the time deployment before and after using DevOps can be seen in Table 1:
Table 1. Comparison per deployment before and after using DevOps.
Activities
|
Durations (minutes)
|
Before (O1)
|
After (O2)
|
The programmer fills in the form and creates a ticket for deployment
|
5
|
5
|
Approval by Technical Lead (Average length of time waiting for approval)
|
15
|
15
|
Approval by Database Admin (DBA) Section Head (Average length of time waiting for approval)
|
60
|
60
|
Done by the DBA team
|
60
|
60
|
Approval by the System Admin Section Head (Average length of time waiting for approval)
|
60
|
-
|
Deployment by Sys. Admin (Average)
|
277
|
-
|
Deployment using DevOps Sync
|
-
|
2
|
Closing Ticket
|
-
|
-
|
TOTAL
|
477
|
142
|
From the time difference figures above, we could see that deployment using DevOps became markedly faster. The impact resulting from using DevOps on each deployment activity of source code to the server application is:
O1 – O2 = 477 – 142 = 335 minutes
It means we will save as much as 335 minutes per deployment when using DevOps and it can increase the number of priority deployments that need to be performed. Then, if the savings are adjusted Indonesian Rupiah (IDR), if monthly wages are IDR 5 million and hourly wages are calculated by multiplying 1/173 times the monthly wage (Better-Work-Indonesia, 2012).
Table 2. Total savings per deployment in Rupiah.
Remarks
|
Rupiah (IDR)
|
Monthly wage
|
5,000,000
|
Hourly wage
|
28,902
|
Wage per minute
|
482
|
|
|
The amount of savings in 1x deployment (335 x Rp 482)
|
161,368
|
There were 647 deployment transactions in 2019, so the annual savings rate could reach almost 105 million Rupiah.
Table 2 below shows the comparison of the time before and after deployment using DevOps when rolling back the source code if there was an error:
Table 3. Comparison before and after using DevOps when rolling back the source code.
Activities
|
Durations (minutes)
|
Before
|
After
|
The programmer finds an error after deploying
|
-
|
-
|
Programmers inform Sys. Admin to rollback due to errors after deployment
|
5
|
5
|
Sys. Admin rolls back the deployment results file back to the previous version (average time)
|
90
|
-
|
Rolls back using DevOps
|
-
|
-
|
TOTAL
|
95
|
5
|
From the time difference in Table 3 the application server downtime when the system admin rolled back the source code application occured for 90 minutes before the application could be used again. Meanwhile, if the rollback was done through DevOps, there would be no downtime (Hodges, 2017). If it is calculated in terms of loss due to downtime, especially if the application is used by many people (tens of thousands of active students and thousands of lecturers), it will be very significant.