Introduction
DevOps and Agile, although very different concepts, share some common principles and adoption challenges. Moreover, many industry leading software professionals find themselves in the role of change agent in these exciting domains – challenging us to think deeper about Application Lifecycle Management (ALM).
Thus, the DevOps paradigm shift brings with it continuous improvement and creation of infrastructure, tools, and processes that take us far beyond the Agile Development paradigm. In this article, we will consider what is DevOps, why we need it, and several benefits and challenges experienced by organizations as they embark on the road to DevOps.
The information presented is based on personal experience, first hand work and conversations with ATG clients (anonymity maintained), and an interview with an expert DevOps practitioner who was also an early adopter within the Financial IT sector (Dr. Automation).
What is DevOps?
Beginning with investigation into the definition of DevOps, it is apparent that there is much confusion. Many definitions found on the internet focus primarily on the idea that development and operations must work together to more quickly deploy software. Some searches reveal the idea of “culture” as a central theme. Others find automation as the central idea of DevOps, and often reliability and security find their way into the definition. Based on what Advanced Technology Group (ATG) is experiencing with our clients, and some insights from Dr. Automation, I’m proposing the following definition:
“DevOps is a culture that fosters a way of unifying and facilitating: 1) the needs of creating and deploying Software and Infrastructure, and 2) how software development and information technology professionals, stake holders, and consumers work together.”
Within the definition, the needs of creating and deploying software and infrastructure are the things you might expect such as: time to market, security, reliability, and robustness. How people “work together” includes how the organization and systems support continuous interactions between DevOps teams, stake holders, and consumers. The Agile Manifesto states, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” DevOps as well embraces this principle, however, in addition to the software product, we include the very systems, infrastructure, and support needed to develop, deploy, host, and maintain the software. DevOps culture is about orchestrating people, process, technology and tools around creation and delivery of both software and infrastructure.
DevOps: People, Process, Technology and Tools
During an interview with Dr. Automation, he emphasized that successful DevOps teams are operating in the “center” of many software engineering and information technology competencies. This is revolutionary and where the traditional philosophies of Development, Operations, and QA are converging.
“Each team member bridges competencies; for example, a Unix Admin is expected to have solid Dev and QA skills. Team members are expected to have and/or to learn a broad set of competencies; however, automation is a must.” But a warning comes with this statement, and that is: DevOps “is not automation,” rather it “requires automation.” Actual DevOps failures have been attributed to this misconception. It requires more than hiring automation engineers to successfully move an organization from traditional operations into DevOps.
Overarching processes, technologies and tools found in modern Agile software development projects are also utilized in DevOps. For example, the Continuous Integration (CI) process applies directly to operational components such as building the VM where the software will execute. The difference comes with new technology and tools that will be added to the mix. Let’s say, for example, we have the following parts to our CI:
CI Tool such as Jenkins and an Integration Server
Single Source Repository such as GitHub
Source Control method such as GitFlow
Automated Build Triggered on Changes
Automated Unit Testing Triggered on Build
Automated Deploy to Staging or a Clone of Production Environment
Automated Functional Test Triggered on Deployment
With DevOps, we will now include managing, building, and deploying the environments as well. Some of the new tools and technologies include Vagrant, Docker, VMWare, Amazon AWS, Azure, Chef, Puppet, Ansible and so on. The integration, test, staging, and production environments are now under source control and changes to them will trigger the same types of testing and validation that happen on the software application. The builds can be managed to deploy in any manner and any combination of local device, on premise data center, remote data center, and any type of cloud, or even remote IoT devices. DevOps technology and tools facilitate the concepts of CI and CD (continuous deployment) by managing end-to-end all aspects of the ecosystem.
Why DevOps?
People working in traditional software development organizations, where development teams basically throw it over the wall to Operations, will tell you that it can take from weeks to months to deploy to production environments. Reasons for this phenomenon include trivial issues and important corporate standards around performance, security, patent and trademark, suitability for use, regulatory requirements and so on.
The key to DevOps is that the organization will foster a team that understands these things, and has the competencies to then bake them into process. Processes can enforce all kinds of automation, including: testing, notifications and processing related to compliance verification and validation, or whatever is needed to move work through the pipeline into production. Dr. Automation made the astounding remark that in the stringent area of Financial IT, by implementing DevOps, “We went from weeks or months before a deployment, to minutes!”
The statements above are intended to be very high level, and if you don’t have a good understanding of the challenges facing many organizations, the business novel The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (Gene Kim, 2013) can be eye-opening and well worth the read. Dr. Automation pointed out several things that are lacking in a pre-DevOps culture, and exemplify why DevOps is so important:
Environments are not available early, so days or weeks can be wasted during project startup
With DevOps, the team builds out the environments and understand it is critical for the code and environments to be tested together
Deployments are typically one-offs, and servers change over time so nobody knows what the software is running on in any given environment causing frustration and wasted development time and effort
Following DevOps principles, the deployment procedures are automated, and reusable scripts are managed in the single source repository.
Team members do not have ownership or responsibility for issues caused by their work outside of their domain, so issues can go on for days before the root cause is found and resolved
In a DevOps culture, you are responsible for your work and the consequences. When you change something, you are notified of any issues regardless of the point of failure. For example, if the Unix Admin applies a patch in Dev, then during the CI process some apparently unrelated software functional test breaks, everyone involved will be notified immediately with the current state, source of any changes, logs, etc. It must be fixed and restored to functional state STAT.
Challenges to DevOps Adoption
Clients all too often lament, “Why is the move to DevOps so challenging?”
Thought leaders in those organizations believe that, because they have undergone a successful and lasting Agile transformation, DevOps transformation is somehow an evolutionary guarantee. Agile thinking just does not lead to DevOps thinking. We must experience a true change in the way we think about both software development and operations – it is a true paradigm shift.
How we manage application bug fixes, versus an OS security patch, or whether we choose to store crypto keys and certificates in LastPass or a sophisticated HSM, all of it must be unified and facilitate improved velocity, quality, and cost effectiveness of the pipeline. While discussing this with Dr. Automation, we both concluded that DevOps is where “ALM morphs into Full-stack Lifecycle Management (FSLM).”
Another challenge is simply born out of resistance to change. This resistance is typically not an in-your-face bad attitude or even a hidden agenda, rather a resistance that comes from a mindset of the traditionally organized development and operations processes.
Essentially our brains have been trained contrary to the principles of DevOps, and this happens regardless of an organization’s size, or what development methodology is followed, or the talent level of our people. A great way to understand the challenge of successful DevOps transformation is The Backwards Brain Bicycle.
In this interesting YouTube video, Aerospace Engineer Destin Sandlin, founder of Smarter Every Day, demonstrates an interesting bicycle, and draws some noteworthy conclusions. The bicycle is essentially like any other two-wheel bike with one exception, the steering mechanism produces the opposite effect: left turn produces right turn, and visa-versa.
Unlike the many normal bikes one could attempt to ride impromptu, no one has been able to just hop on and ride the backwards brain bicycle. The application to DevOps is that to experience the paradigm shift will require daily effort over time, and like Sandlin’s experience, suddenly one day you will realize you really get it.
Oh, and another important observation from the video – it doesn’t take much effort to fall back to into the old practices. It is quite easy to go back to riding the old bicycle!
Conclusions
This article’s title, DevOps – the New Agile, emphasizes expanded roles of traditional Agile software development teams, and that a new way of thinking about development and operations is a critical factor for success. This is somewhat high level, however, we did cover a lot ground; including a definition of DevOps, some anti-patterns, tools and technologies, why we need it, and some benefits/challenges of adoption.
In addition, it is important to point out that DevOps is in the main stream. It is not only supported and fostered by noteworthy organizations like Google, Microsoft, Amazon, and Netflix, but also a large and growing community of DevOps professionals like the practitioners at ATG, who are ready to help you on your journey to achieving successful DevOps transformation.
Ed Harrold, MBA, BScEE, has over 20 years experience in information technology/software engineering and joined Advanced Technology Group as the Director of Architecture & Engineering in 2015.
