Master's Thesis

Cloud and CI/CD Challenges

I. INTRODUCTION

The introduction of the public cloud created numerous new information technology solutions and capabilities. The advanced method of deploying these solutions is known as the continuous integration / continuous delivery (CI/CD) pipeline and is used to streamline the process of implementing hardware and software solutions in the public cloud. The CI/CD pipeline is a framework that includes different stages to perform the continuous integration and continuous delivery tasks. Each stage operates by using different tools to create the aggregate pipeline. Continuous integration includes the ongoing coding, building and testing tasks. Continuous delivery is the process of releasing new changes and deploying to production. Cloud solutions that are implemented using the CI/CD pipeline are gaining more traction throughout the Information Technology (IT) community around the world. It’s a solution with unique options and various benefits. It does, however, include many complexities in its implementation, utilization, and ongoing maintenance. There are many challenges to consider in deciding if this solution is the right one to implement. Organizations need to research and answer a series of questions before deciding whether to proceed with this complicated framework. Those are questions are as follows: 1. What are the reasons an organization would want to deploy cloud solutions using CI/CD?

Who are the various cloud providers and what services do they offer?
What are the CI/CD stages and the corresponding products that should be researched and tested?
How does the public cloud and the CI/CD pipeline impact IT administrators?
What are the security challenges of running IT operations in the public cloud using the CI/CD pipeline?
What best practices, policies and compliance requirements need to be included?
When should the public cloud and CI/CD pipeline be used?
What can be learned from other organizations that made this change?

Following these deliberations, a strategy should be developed to keep the organization focused on its endeavor. All of these considerations will help contribute to the success of the organization’s IT tasks and goals. Throughout this research, each of these issues will be analyzed to determine whether the complexities associated with deploying solutions in the public cloud through the CI/CD pipeline are justified or if they are too complex to be successfully implemented.

II. CONVENTIONAL IT OPERATIONS

Understanding how traditional IT teams operate within organizations as it relates to existing solutions, processes, and procedures can illustrate why so many organizations are moving to the cloud. Many organizations operate by having separate teams manage engineering and operational tasks based on various technologies. For example, server teams provide physical or virtual capabilities with various operating systems like Unix or Windows. The network team offers a variety of solutions like switching, routing and firewall capabilities based on internal and external customer needs, as well as load balancing options and more. The storage team provides solutions like Storage Area Network (SAN), Network Attached Storage (NAS) and Object Unstructured Data Storage for different workloads and their corresponding performance needs. The security team provides services like Identity Access Management (IAM), as well as vulnerability scanning and patching. Development teams are also a huge cornerstone of IT departments that can build out company specific applications without purchasing Commercial-off-the-Shelf (COTS) software solutions. These are just a few of the teams that most robust IT departments are likely to maintain. Each of these teams work with their management and business partners to understand their business needs. Based on that information, they begin to formulate plans and strategies to build and implement solutions. Infrastructure engineering teams often start by researching products offered by several different vendors and compare and contrast the capabilities advertised. They then make a selection from one of those options and run that system through a multitude of tasks in a test environment. This would include building out test cases for hardware systems or software programs based on the requirements that were previously gathered. Typically, functional tests would be done to make sure the required features operate as they are advertised. In the event that they are not working as they should, or that the team runs into a bug, feedback is provided to vendors so improvements and enhancements can be made. Engineers also need to run a series of performance tests that validate the vendor claims on various speed related metrics like Input Output Operations (IOPs), throughput, latency, and central processing unit (CPU) utilization. Those are followed up by high availability tests to check the resiliency and scalability of the system to make sure it can meet the demands of the workload. Once these respective engineering teams complete these tests, they need to make a decision to move forward with an investment or look for another product that would be a better fit and start the process again. At this stage, it’s common for conversations between the infrastructure, vendor, and vendor management teams to take place regarding the cost of the products and services, including support and maintenance. The next step would be to begin the installation in pre-production and production environments by making a series of requests to other teams. A data center location needs to be provided, IP addresses and DNS names need to be reserved, and port assignments and configurations need to be made. The engineers need to create a detailed cabling spreadsheet for onsite resources to follow in order to accurately run cables. This is a particularly important step that could have major consequences when events like code upgrades are done or maintenance is performed. If power and switching are not drawn to two independent and redundant sides, the system will not be highly available resulting in an extended outage that can have impacts on productivity, revenue, as well as the company brand. The next step of the implementation process is to configure the systems to prepare them for the workloads that will be migrated onto them. These tasks require the collaboration of resources across multiple teams, as well as project managers to make sure that tasks are completed on schedule in order to meet project deadlines. This process can become very resource intensive. After the initial engineering implementation steps are complete, operations teams take over support of day-to-day tasks like provisioning requests, monitoring the health of the systems, responding to alerts, maintaining security, managing code upgrades and analyzing capacity. The operations resources often spend a good amount of time with developers understanding their requirements for changes to existing applications, as well as the new applications that need to be deployed in the environment. The operations engineers need to understand the workload the applications will generate to right size the infrastructure needed to support the applications and provision the solution within a reasonable amount of time. If the latter cannot be done in an appropriate amount of time, the operations team risks becoming a bottleneck for the organization - stunting potential growth. A developer’s tasks usually starts with the steps outlined in the Software Development Life Cycle (SDLC). They meet with their business partners to understand their application requirements and begin their project work with the design phase. In a pre-procured dedicated sandbox environment, they begin developing their code and creating test cases. Team members collaborate with one another throughout the development process to craft the best solution possible. Several levels of user acceptance testing (UAT) with various participants are completed to test the code for functionality. They then work with the operations team to deploy the first version of code into production. Feedback is provided by testers and business partners so that enhancements to the code and application can be made to offer additional capabilities to end users. This process can take weeks to months to complete. These are the typical teams, roles and responsibilities that make up major IT departments found in the largest companies in the world. While this may be how many of them operate now, it is debatable if this is the best way to run and function in the future.

III. CI/CD & CLOUD JUSTIFICATION

IT has always been in a state of change. Organizations have had to learn to adapt to changes and the teams that manage the solutions have also had to do the same. When the right technology and methodologies are put to use, new products and services can be delivered to the market on an expedited schedule. If companies don’t take the opportunity to innovate their solutions, processes, and procedures, they often end up running their IT departments in an unproductive manner. Employees end up spending excessive amounts of time working on repeatable tasks that can and should be automated. These organizations are also slow to embrace changes that can put them in direct contact with the most cutting-edge IT solutions like those that are available in the public cloud. Companies that avoid looking towards the future often get stuck in the here and now restricting their ability to grow stronger and more profitable businesses. Operating the IT environment in this way keeps companies on legacy systems that are difficult to maintain because of the reduced vendor support. Staffing becomes a problem too since there are less resources to manage these aging systems (Mitchell). In order for individual resources and IT departments to avoid becoming stagnant in their roles and offerings, they must be capable of change. If they fail to do so, they will start to see their profits drop, they will most likely lose a percentage of the market share and in extreme situations, the organization could go out of business. With the incredible number of products and services offered to IT professionals in the market, employees must challenge the status quo within their teams in order to bring real change and value to the organizations that they support. Implementing solutions in the public cloud through the CI/CD pipeline allows organizations to do things bigger, better and faster. It helps organizations stay current with emerging technology. There are plenty of solutions and options to choose from in the cloud as well as the CI/CD pipeline which can be configured into custom solutions to be used by any organization. The solutions offer many different opportunities to automate, expedite deployments and scale solutions to provide enhanced and new products and services. This helps keep the company relevant and profitable.

I. THE PUBLIC CLOUD

Twenty years ago, the notion of the public cloud was practically non-existent. CI/CD was not a commonly known acronym and the idea of Development Security Operations (DevSecOps) hadn’t been introduced yet. These functions, however, are part of mainstream IT departments now. They are useful tools to implement new capabilities that increase the IT footprint and bring increased value to companies. Before the public cloud came to the market, engineering and operations teams followed drawn out processes to run their environments. There was limited space in data centers which restricted the number of solutions that could be tested. Engineers and project managers were constantly worried about missing procurement, implementation and migration deadlines. They were also always on the lookout for cost savings and generally concerned that they either had too much or too little capacity to rely on. The public cloud had a substantial impact on companies that embraced the opportunities that were available. Cloud providers like Amazon Web Service (AWS), Google Cloud Platform (GCP) and Microsoft Azure invest billions of dollars developing new offerings that are available worldwide. Many of these solutions haven’t been available through more conventional IT providers. Because of these investments, these companies are able to offer a plethora of services for server, storage, network and security needs with customizable features. AWS created the S3 storage solution that is classified as object storage. They offer a similar solution known as Glacier that also provides object stores, however, it’s classified as cold storage and isn’t meant to be accessed frequently and is more affordable. These storage systems manage data in an unstructured format that organizes artifacts in buckets. One of the most important features offered by the cloud providers with regard to their storage solutions are automated snapshots and backups for restores and disaster recovery events. Database solutions are available in a variety of choices that include traditional relational databases, in-memory options that provide caching features for low response times and document based or non-relational databases (“AWS Cloud Databases”). AWS server services offer EC2 virtual machines (VM) instances with auto-scaling features as well as Lightsail VM’s that can be used for standing up and running websites (“Compute for Any Workload”). GCP offers similar solutions on their platform as well. Their network products include Cloud CDN that provides high performance and low latency response times that also has built-in features to protect operations from denial-of-service attacks (DOS) (“Leverage Google’s Decade of Experience Delivering Content”). Cloud DNS is a domain naming system (DNS) services that includes scaling and high availability features (“Cloud DNS”). Since security is a main concern of IT operations, core services like secret manager are available to store Application Programming Interface (API) tokens, keys and passwords (“Security and Identity"). Identity and Access Management services are included to accurately manage systems and permissions that users and groups are able to access through configurable roles (“Security and Identity"). Each of these providers offers many more cutting-edge technology solutions. These services help keep DevSecOps engineers focused on projects that can distinguish the organization’s products and services from their competitors, rather than being tied down by traditional IT tasks. The options provide teams with the opportunity to use agile methodologies and test different hardware and software solutions for their different workloads. Cloud solutions are offered worldwide to make high availability and scalability simpler. This means engineers don’t have to manage and facilitate complicated installations in foreign countries. They also don’t have to waste money over provisioning resources to make sure they don’t run out of capacity. Additionally, they don’t have to risk under provisioning which can lead to outages during traffic spikes. Events like this can have brand, profit and productivity impacts. Instead, they can build out automated procedures that pull from the capacity in the public cloud. Services can be configured to auto-scale based on current needs. Because there are different companies that offer services in the public cloud, IT teams can research the different products and services, costs, support models and availability of each, either through all the information provided on their website, through sales teams or by physically testing the options they are curious about. They can compare the results of each company to one another and make an informed decision as to which technologies/solutions in which they want to invest. Companies no longer need to make huge upfront investments in technology which can make budgeting difficult and usually takes away from things like new investments and training. The introduction of the cloud gave Chief Financial Officers (CFOs) a new and interesting theory to contemplate - capital expenditure versus operational expenditure. Cloud providers charge their customers for what they use. This is considerably different from the traditional IT model where capacity requirements had to be guessed well in advance of actual needs. IT departments would need to purchase massive quantities of hardware and software capabilities and pay vendors for them up front; these types of assets depreciate in value over several years. With the cloud model, capital expenditure is replaced by operational expenditure (“Why More CFOS are Shifting IT Investment from Capex to Opex”). The cloud lets companies consume IT resources as they need them. They pay for services as they use them, which makes the impossible task of capacity planning a little easier. The cloud charges customers for what they use on a monthly basis. That doesn’t mean that organizations don’t need to remain vigilant about the amount of money they spend. There are usually heavy fees for pulling data back from the cloud to on premise systems known as egress fees. Companies also need to make sure that systems aren’t being provisioned and left unused and idle in the environment (“The Essential Need to Understand Cloud Costs in 2023”). They may not be in use but are still incurring costs. The cloud gives companies flexibility with how they can create and manage their budgets; however, they must remain disciplined in order to be ultimately successful.

II. THE CI/CD PIPELINE

In the last couple of years, the term automation has gained a lot of popularity in the IT industry. Since IT is usually the team that is responsible for rolling out new products and services to the market, and everything has to be done bigger, better, and faster, automation has become a necessity. Before automation was such a big concept in IT, things had to be done in a slow monotonous manner. All tasks including testing, installation, provisioning and maintenance work had to be done manually. Running an IT department like this doesn’t give teams time to think about the future and be more innovative. It has them stuck in the past. The CI/CD pipeline is a big help to IT teams from multiple different angles. It automates jobs and workloads, facilitates the delivery of new capabilities, and has the ability to increase system resources and efficiency. To understand how the CI/CD pipeline achieves all these goals, it’s important to understand what technology is included. Code repositories like Github, Bitbucket and Gitlab are used to keep the process of developing code organized. They let developers check in and check out their code, create branches off the main repository to avoid making changes to and breaking production code. Repositories keep an auditable history of each revision that was made. This allows developers to roll back their code in case a bug or an error is encountered. Developers can pull down different code repositories to work on and because it is a centralized location it makes collaboration between developers simple since the code is viewable in the repository once it is committed. Repositories can also kick off a series of automated actions through yaml files that create containers, execute scripts, standup databases and provision infrastructure. While repositories can initiate automated actions, these steps can also be achieved by using dedicated orchestration applications as well. Containers are an important part of the CI/CD pipeline. One of the most recognizable brand names for containers is Docker and it is used by IT teams across the world. They are included in the CI/CD pipeline for a couple of key reasons. They stand up independent workspaces where everything that’s needed to run an application exists inside the container. That includes the code packages, libraries and dependencies that are requirements of the application. The underlying physical hardware has no impact on what runs inside the container and the resources the container uses are usually small in scope and have very little impact on the hardware. Containers are even capable of running multiple versions of the programming language in use so that developers can run the application using each version for even more robust testing. Volume mappings can be added to the container to save any data locally that is generated by the application. Port mappings can be included so that the container can make a connection to another container that may be running a database for example. Containers are lightweight and can easily be built and torn down to test code revisions and to help developers throughout the software development lifecycle. Container images can be created by developers to include environment-specific requirements needed by the application to run. Each time a container is built based on that image, everything that is needed is automatically installed. Even small features such as this help teams be more efficient and keep the environment more stable by avoiding inconsistencies. The alternative would be to use an image from a registry and add on dependencies each time a container is built from that image. Databases are another important component of the CI/CD pipeline. In the past, applications would have to rely on relational databases. They are typically complex to install, very costly and inflexible in the way that data is managed and stored. Tables are used in relational databases; entities have assigned attributes. Each record in the table must have a value for each attribute and they cannot store unstructured data. Although they are not very flexible, they do offer more reliability. Non-relational databases are much more flexible, cost-effective and can be scaled out. They use collections to store data in documents. Each document does not need to have the same fields. The flexibility of non-relational databases like MongoDB, Couchbase and Cassandra allows applications to ingest unstructured data faster and store it in JSON format . This lets developers read and pull from the database easily. Non-relational databases can be easily created in the public cloud when they are deployed through the CI/CD pipeline making them an excellent fit. Testing the scripts created by developers is a major part of the CI/CD pipeline. Ideally, after developers meet with their business partners to understand the requirements of the project, they should begin their development process by writing test cases, rather than actual application code. There are different third-party applications available on the market like pytest and cucumber io that help developers create automated test cases. The test cases are run each time the code is modified. If the DevSecOps engineer doesn’t get the result he or she was looking for, it’s much easier to determine where the break down occurred. Automated testing is also an important efficiency tool for developers saving them time from running manual tests each time they make changes and run their code. If tests were run manually each time the code was changed, engineers would never be able to match the volume of tests that can be completed when they are automated. It’s often very clear to spot when testing has not been included as part of the development process. Applications are typically very buggy; they don’t work the way they were intended to run and result in too many errors. The lack of quality in the application development process can lead to the breakdown of the relationship between the developers and the business partners. In order to get from scripts stored within the software repository to deploying infrastructure on the public cloud, another phase needs to be added to the CI/CD pipeline. Tools like Terraform are responsible for provisioning the infrastructure in the cloud. A connection from the local environment is made to the public cloud through a secure token. Once that is configured, infrastructure like servers with specific network requirements and object storage buckets can be deployed in specific regions in the cloud by running a script.

Putting each of these technology pieces together to create the CI/CD pipeline that is used in conjunction with the public cloud creates a process for infrastructure engineering and operations teams and developers to work together to deploy cutting-edge hardware and software solutions on a much shorter timeline through a repeatable process. The drastic reduction in time to stand up infrastructure and deploy homegrown applications brings enhanced and new products and services to the market more efficiently and gives the organization a huge competitive edge to increase their customer base and revenue.

III. DEVSECOPS

It’s common for employees on compute, storage, network and security teams to only focus on the needs of their own environment and work in a siloed manner in traditional IT environments. This leads to very slow deployments of infrastructure and application development. That has a big impact on the products and services a company can offer as well as their profitability. Having separate infrastructure, development and security teams come together to create integrated solutions became the origin of the DevSecOps engineering role. DevSecOps has had incredible impacts on the way the engineering, operations, security and development teams work together to implement their solutions. As a result, it can have even greater positive impacts on the company’s profitability. The cloud and CI/CD pipeline offer DevSecOps engineers more tools to work with that help implement more customized solutions. With the access to these different tools, DevSecOps engineers can right size their solutions with regard to the number of servers, containers, databases and network bandwidth needed to run their workload. This means that they avoid under or over provisioning their hardware and software needs. Over provisioning leads to wasted resources that are added expenses. This cuts into profits, although it doesn’t impact performance of the systems and applications. Under provisioning resources can have huge impacts when there are traffic spikes. Performance suffers in situations like this because there isn’t enough bandwidth to handle the traffic and it ends up reducing productivity and profitability. It ultimately ends up damaging the company’s brand which can certainly have long term consequences. Customers can lose confidence in organizations that have very public failures which can cause even more losses in revenue. That’s why it is so consequential to implement the CI/CD pipeline managed by DevSecOps engineers to evolve into a modern and efficient IT department. Understanding how DevSecOps engineers do their jobs is crucial to understanding the benefits they bring to an organization. They leverage the steps of the CI/CD pipeline to deploy customized agile solutions to the cloud. This helps them design the best technology solutions for the applications their business partners want to deploy. Because of the number of resources in the cloud, they can develop a process that maintains the correct levels of resources to run their applications at any given time. When new application requests are made to the DevSecOps teams, a review of the requirements takes place first. The engineers research the different options available in the cloud and run test workloads on different platforms to prove which one is the best fit. The development of the actual application itself is now completely revolutionized as well. The speed of the automation qualities of the CI/CD pipeline allows developers to create code, run automated tests, push to the code repository, and have all different types of infrastructure provisioned (servers, containers, databases, etc.) in minutes. DevSecOps engineers can promote changes to production, get feedback from testers and business partners and incorporate even more changes in the code. The changes could either be fixes to the behavior of the application or new feature requests. This process of incorporating changes quickly facilitated by the cloud and the CI/CD pipeline removes the development process as a bottleneck and brings new products and service to the market for the company. It streamlines the development process in a way that brings increased value to the organization by being a catalyst for change. The process also places a lot of importance on the testing aspects of development. Because of this, the number of test cases that can be created and run each time the code is revised is groundbreaking. It is far beyond anything that could be done manually. Being surprised by an unexpected behavior in production that needs to be corrected is rare. The integration of the infrastructure and development teams in the DevSecOps role gives engineers more control over the size of the workload that can run at any given time. With the resources in the cloud, the CI/CD pipeline can be used to auto-scale all the resources that are needed to run workloads. Configurations can be added to the pipeline to add more servers, containers, databases, storage and more when certain thresholds are met - like the number of client connections, CPU and memory usage and high latency. Cloud deployments through CI/CD help DevSecOps engineers manage tasks by breaking them down into smaller more manageable chunks. Each phase of the pipeline is important because each piece has its own objectives that need to be accomplished, so they can all work together to make the entire process a success. From coding the application and managing it in a repository, building automated test cases, designing yaml files to build and start containers, or designing infrastructure config files to stand up systems in the cloud with specific server, storage, container, database, network, and security requirements - they all become more manageable when they are broken down into smaller pieces. The process would be much more daunting if all these parts had to managed, designed and implemented as a whole solution. The security aspect of the DevSecOps role is a little different compared to development and operations tasks that are included in this methodology. The security tasks that must be included to ensure the confidentially, integrity and availability of the environment act more as a wrapper around the entire DevOps process. Security needs to be taken into consideration in all areas of the development and operations phases. This includes things like security scans for known threats, vulnerabilities inadvertently created in code, and scheduled periodic updates to operating systems, packages, and libraries. Security principals relating to user accounts and permissions, group permissions as well as service accounts are also included in the DevSecOps responsibilities. Unused user accounts need to be deactivated and deleted and excessive permissions should be avoided. Regular audits need to be performed on a consistent basis to supplement routine clean-up activities. Security groups should be created and used for specific technology areas and systems. The levels of access should also factor into the permissions that will be granted to the group, such as read only, read and write and root access. The number of users added to these groups should be kept to a bare minimum. Service accounts should have limited access as well, such as disabling interactive system logins (“Security Best Practices”). All these accounts should be authenticated through the chosen authentication provider implemented by the organization. Without this step, rogue local accounts can be created and used without being regulated.

Adding the DevSecOps engineering role to an IT team’s pool of resources is important to maximize the benefits of the public cloud through the CI/CD pipeline. This opens the organization up to faster, more frequent and consistent hardware and software implementations. It helps the company remain up to date with current trends, grows its confidence and popularity in the market and increases its customer base and profitability.

IV. CHALLENGES

There are a number of challenges associated with running an IT environment in the modern era. The public cloud and CI/CD pipeline are no different. The biggest concern surrounding the implementation and maintenance of the cloud and CI/CD solution is making sure the environment stays safe and secure. The central tenants of IT security, confidentiality, integrity, and availability (CIA) need to be maintained in the implementation and management of cloud-based solutions deployed through the CI/CD pipeline. Because there are so many connections between all the different technology components included in the pipeline and the connection to the cloud, each of the junction points creates a weakness in the environment. They are potential ingress locations for hackers to infiltrate the environment. These weaknesses have to be reinforced to make sure the environment isn’t compromised because of the desire to bring new capabilities to the company. Compatibility between the various stages is also something that needs to be taken into consideration. Updates and security patches become available at a quick rate causing administrators to constantly upgrade various parts of the pipeline, such as code libraries and packages, container images and OS versions. When hardening the environment, it’s important to make sure the systems remain compatible with each other and that they either maintain or improve the performance of the applications. Cloud and CI/CD technologies require companies to hire resources that are knowledgeable about the different components and the extensive number of solutions available. Having resources that can administer all these components are critical. If resources lack the knowledge to manage these processes, systems and applications, it’s likely that the most fitting solution won’t be implemented - and more importantly, security precautions won’t be included. Any adverse event that the organization is presented with most likely wouldn’t be handled efficiently in order to limit the impact. Therefore, a lack of knowledge and understanding will impact the overall value added to the organization.

I. SECURITY

Security is a huge branch of any successful IT department. There are plenty of security aspects to consider in order to maintain a healthy and safe environment. An IT security professional’s day-to-day activities include tasks such as monitoring network traffic for irregularities, administering identity and access management, and secret management systems (“IT Security Roles and Responsibilities of Cyber Security Professionals”). However, they also need to define a security strategy for their cloud and CI/CD implementation to avoid any unforeseen impacts to confidentially, integrity or availability. The strategy needs to include potential threats, vulnerabilities and attacks. It should also contemplate things like, who the attackers can be, how the environment needs to be secured and what to do if vulnerabilities are exploited. Parts of this strategy would include proactive tasks; however, reactive plans need to be in place in the event that an exploit is successful. Resources need to know what needs to be done to reduce as much of the impact as possible. Threats to the CI/CD pipeline and public cloud solutions are similar to those that affect traditional environments. These can come in the form of malware, ransomware and code injection attacks (“Cyber Security Threats”). One of the biggest threats to an organization is the potential that data could become compromised. This would include data such as secret and proprietary code developed by the organization, secret credentials and tokens as well as customer data that has been entrusted to the organization. If any alterations are made to these data sets, if they become inaccessible or they are made available to unauthorized users, the impact to the organization could be severe.

II. ATTACKS

Attackers can present themselves as different types of individuals and organizations. They can exist inside and outside an organization. Many are groups that look for vulnerabilities to exploit for financial gain commonly known as ransomware attacks. These types of attacks occur when hackers gain unauthorized access to systems belonging to an organization. They end up encrypting large amounts of the organization’s data and threaten to release the sensitive data. They demand a ransom to be paid in order to stop the data from being published and in return provide an encryption key to the organization to unencrypt the data. Victims of these attacks often struggle with the decision to pay the ransom. It's possible to become the target of another attack especially after attackers know a ransom has already been paid. Some also refuse to pay ransoms on principal and instead work to recover the data on their own. Foreign governments also engage in hacking activates that use tactics like denial-of-service attacks (DOS) to inflict harm on entities and organizations. These are instances where excessive amounts of network data are sent to specific endpoints to overload the network in order to make websites inaccessible. An example of this type of attack was just prior to the Russian invasion of Ukraine in 2022. A massive number of attacks were aimed at Ukrainian government, communication and financial websites (Lewis). Knowing about the impending invasion, their goal was to paralyze the Ukrainian preparations and defenses. Foreign governments also use IT vulnerabilities to help them in their espionage activities in order to gain sensitive information they can use to their advantage. Recent reports have been published detailing a Chinese effort to infiltrate email accounts of specific individuals in the U.S. Government through a Microsoft cloud email service vulnerability. The extent of the damage is not known because the event occurred in the last several weeks (Page). Activists also use a variety of attacks and vulnerabilities to target organizations and institutions to raise awareness for certain political or social causes. Internal company employees can also attack the organizations they work for themselves. There are examples of employees compromising internal systems by installing malicious software, using unauthorized user accounts to disrupt business activities and selling secret company information for a myriad of reasons (Baker). Some might want to turn on their company for financial gain, others might justify their actions because they are disappointed with the company or how they are being treated. It’s important for companies to understand that threats can come from all types of entities, groups or individuals. None of these threats should be discounted nor the amount of damage that any of them can inflict on an organization.

III. IDENTIFYING AND MANAGING VULNERABILITIES

All areas of the IT footprint are full of potential vulnerabilities that can compromise a company’s secret internal operations and data. This includes the connections that are made to customer-facing portals, VPN networks, proxies, system ports, operating systems, application code, passwords, and much more. Within the CI/CD pipeline, the code that is developed by DevSecOps engineers can also introduce vulnerabilities in the environment. The code repositories are a target, and integrated development environment (IDE) can present potential vulnerabilities. Container images that are used are often full of vulnerabilities that need to be mitigated. Coding languages, modules, packages, libraries, databases and cloud infrastructure all carry various vulnerabilities. Even the location of the hardware both geographically and the physical building structure are potential vulnerabilities to take into consideration. Each of these areas needs to have a plan in place to make sure all possible threats to the health and security of that component are documented with a procedure in place to mitigate the threats. All these software and hardware systems are part of the IT ecosystem and if one of them is susceptible to the vulnerabilities, then all the systems are susceptible.
In a legacy environment, configuring security measures and patching vulnerabilities is usually a manual effort. It is quite possible that an individual employee configuring security settings, like authentication providers or system privilege access using group policies, might make a mistake in their configuration that exposes a higher level of access to many users. They might also have a difficult time scanning for vulnerabilities and patching them manually (“The Definitive Guide to CI/CD Pipelines and Tools”). It’s likely that something might get missed and exploited by bad actors. In the past, massive networks were deployed to support the bandwidth required to run IT operations. Firewalls were added to manage and authorize traffic both internally and externally. VPNs and two-factor authentication were used to authorize users to the internal network. These technologies were typical parts of the attack surface that hackers commonly used as footholds to gain unauthorized access to an organization's IT environment. In cloud-based models, the attack surface isn’t eliminated, however, it is different from the traditional scenario. The attack surface shifts to the cloud, CI/CD tools, as well as connections to the cloud and in between CI/CD pipeline technologies. There are very practical proactive steps that organizations can take to avoid having exploitable vulnerabilities in their environment. Reports of Common Vulnerabilities and Exposures (CVE) can be pulled by IT teams on a regular basis from the Cybersecurity and Infrastructure Security Agency (CISA). This report details the vulnerabilities that exist in COTS and open-source software and firmware solutions that are available on the market and in the IT environment. The vulnerability report ranks the criticality of the vulnerability using a low, medium, high and critical scale. To leverage these types of reports, systems and applications, such as code modules, packages and libraries, should be scanned to see if any of the vulnerabilities on the report are present. Scans should be run at build and run time to avoid introducing vulnerabilities inside the internal environment and as a fail-safe scan before moving things through the production pipeline. An automated process should be in place to run a comprehensive scan of the environment on a schedule. The frequency of the scan should be on a shorter interval for critical vulnerabilities. If the vulnerability is present, automated actions should be developed and implemented to remediate the threat in an appropriate amount of time. Critical events would take precedence and be patched on an expedited schedule. Vulnerability scanning also needs to be done on in-house development projects. Developers can inadvertently create vulnerabilities in their own code that can leave the environment exposed to buffer overflow and inject attacks. Developers also need to make sure input validations are added to support the security of their code. Integers, strings and floats variables need to be validated to make sure users input the required type in each field. Length limits need to be in place for things like name fields and the range should be checked for reasonable pricing or age inputs. Format validations should be checked for phone numbers, birthdates, credit card numbers and more. Security tools can be added to IDEs and application code repositories to scan for the accidental vulnerabilities created by the developers. They can also be used to locate known CVEs within code modules, libraries and packages. Security scans can also be included in the code repositories as a part of the code management process (“Managing Code Scanning Alerts for Your Repository”). CVE reports can be generated and users can test applications after vulnerabilities have been remediated. Container images and non-relational databases are normal components of the CI/CD pipeline and follow a similar process of security scanning and remediation. The connection points between the different technologies included in the CI/CD pipeline and cloud are particularly important. Connections between these systems are usually made by exchanging secrets from one to the other authorizing the communication. Keeping the secrets safe is an important part of maintaining the highest levels of CI/CD security. It’s not uncommon for DevSecOps engineers to skip this step and hard code the secrets in their code. This is an incredibly dangerous shortcut and another example of a vulnerability created by engineers. If a hacker was able to gain access to the internal network, coming across a hard-coded password could unlock access to many more systems creating a massive security event for an organization. It’s even more disappointing if the secrets are used in a ransomware attack since there is a solution that fits in with the CI/CD pipeline to safeguard them in an encrypted vault. The secrets can be called on in a secure way whenever a connection needs to be made. Unfortunately, there have been examples of hard-coded passwords being compromised by hackers that end up leading the hacker deeper into more sensitive areas of the IT environment they breached. Toyota and Equifax are just a few examples of this type of breach. In the Equifax example, the system administrators failed to patch a known vulnerability that hackers used to access the internal systems which allowed them to access hard coded credentials and unlock an incredible number of systems and data. This breach ended up affecting millions of consumers and cost over a billion dollars in settlements and IT security upgrades (Fruhlinger). Toyota’s event wasn’t due to a hacker exploit, but rather human error. An inadvertent commit to a public Github repository included a plain text server password exposing the personal details of thousands of customers (Wadhwani). If secrets and credentials are not safeguarded properly, security breaches will prove to be costly. The confidentially, integrity and availability of the data will be put at risk and companies will have to deal with penalties, fines, serious brand damage and much more.
Vulnerabilities should also be logged as well as the remediations that were performed to create an audit trail for the DevSecOps engineers. Historical data like this is helpful in knowing what changes were made in the environment. This data can also be useful in troubleshooting issues in the future.

IV. COMPLIANCE, POLICIES & BEST PRACTICES

Having a governance team is an important factor in keeping IT departments in compliance with the regulations of their respective industries. Governance teams are responsible for reviewing requests to bring new hardware and software solutions into the environment. Reviews require the requester to document the details and requirements of the solution as well as the reason they need the solution. Requests are taken through reviews to understand the use case the solution solves as well as the impact to existing architecture in the environment. The review would cover high availability (HA), disaster recovery (DR), and security compliance with internal standards. Compliance with external regulations would also be included in the review. This would include the laws and regulations of industries and countries all around the world where they are applicable (“Compliance Resource Center”). This procedure is typically followed in traditional robust IT organizations. This same process must be included when running workloads in the cloud. The various cloud solutions must be vetted through this process as well as all of the tools selected to be included in the CI/CD pipeline. After the review is completed, the findings are added to the IT standards. The IT standards includes all reviews and the results of both the accepted solutions as well as the rejected requests. This ensures that all IT teams and individuals are aware of what is and is not permitted to be in the environment. This process should also be followed when making changes to existing approved systems on the IT standards. This would include reviews of newer versions of hardware and software or the addition of new features. This process helps to reduce security and compliance risk to the organization. In order to keep cloud and CI/CD tools and systems running smoothly without impacts, IT teams need to be extremely diligent about the way they setup and administer their environments. Policies and best practices need to be implemented to support this effort. Teams should begin their CI/CD implementation by first researching several different vendor solutions for each stage of the pipeline. This will give them a sense of what features each includes and which tool is best aligned with the requirements of their organization. A more diligent research phase will focus the efforts of the engineers during the testing phase. This will help streamline the implementation efforts and avoid unnecessary solutions that slow down operations. An example of this would be including an overly complex solution like Kubernetes when many of the cloud native solutions offer features like autoscaling. When developing new code, DevSecOps engineers should break down the project into small manageable chunks and commit to the repository often (Miles). Everyone’s code should also be peer reviewed before it is pushed to the main software repository. Team members can review the proposed changes before approving and merging into the main repository and pushing to production. Features like this are included in repositories and can easily be enabled. Another important best practice is to maintain two identical environments, one for testing and the other for production. This allows for all proposed changes to be implemented and tested in pre-production. Only after successfully testing in this environment, should the changes be pushed to production (Sosna). Change management procedures should also be followed prior to implementing changes in production environments. Change records should be opened and must include the implementation details for the change. Backout instructions should also be added in the event that there is an impact to production as a result of the implementation. Lead times for changes are often put in place based on a risk score that is derived from the systems that are being impacted and the potential downtime as a result of the change. Changes should also only be implemented when there is the least potential for disruption to peak production workloads. Since much of the work that the CI/CD pipeline manages is automated, many of these policies and best practices can be included in the workflow. For example, creating the change record should be added as a step to the pipeline by making a secure connection to the IT Service Management (ITSM) portal. DevSecOps engineers can work with business stakeholders and change management teams to negotiate pre-approvals for a set of defined changes. If the requested change and the corresponding requirements meet specific conditions, the change record is generated and initiates the automatic execution based on the change record start time. This process of integrating change management policies and procedures with the CI/CD pipeline brings value to organizations by not only running operations efficiently, but by reducing the number of changes that are done by individuals drastically reducing the potential impacts of human error.

V. WHEN TO USE CI/CD PIPELINE AND THE PUBLIC CLOUD

There are a few things that need to be taken into consideration when contemplating if an organization should use the CI/CD pipeline and the public cloud. The solution as a whole offers great capabilities and makes certain workloads a breeze to deploy, but it might not be right for every company and it might not be the right solution for every problem either. The line of business (LOB), the size of the organization and the specific use cases all need to be taken into consideration. Some businesses value innovation and can handle changes to their environment easily, while others need more security features because rapid changes might introduce too much risk. Also, some workloads don’t need automation, and rapid changes. Including them in the pipeline would not be worth the investment. IT departments need to analyze these concerns in their deliberations in order to avoid setting up a solution that is more complicated than beneficial. It would be difficult to find an organization in this day and age that was not dependent on any technology solutions. Many industries rely on technology to help them in their daily tasks or to bring products and services to the marketplace for their customers. Each industry has different requirements for their technology solutions. Some industries like technology, travel, media and entertainment need more innovative solutions that can be delivered to the market quickly, while others, such as finance and healthcare, need highly secure and reliable solutions. IT departments need to take a close look at the pros and cons of CI/CD and the cloud and compare and contrast them to the solutions they are trying to implement in order to determine what combination of solutions in the cloud is the right fit. If this analysis isn’t done, it’s likely that the right solution won’t be deployed and that could lead to issues like increased security vulnerabilities or loss of revenue. Implementing a solution in the cloud using the CI/CD pipeline does have a substantial attack surface. As a result, security considerations need to play a major part in the implementation process. If security is a high priority for a LOB like finance and healthcare, specific solutions that cater to these types of workloads need to be implemented. Solutions are marketed by cloud providers for these specific LOBs. Google Cloud offers Health Insurance Portability and Accountability Act (HIPAA) compliant solutions that include network security measures, threat detection and response plans (“Google Cloud for Healthcare and Life Sciences”). They offer similar types of specialized solutions for the finance industry as well as government agencies. The financial sector solutions include services like money laundering detection and compliance standards for personally identifiable information (PII) (“Google Cloud for Financial Services”) (“Compliance Offerings”). Government specific solutions include enhanced security services like defense-in-depth principles and threat detection and mitigation (“Google for Government”). They include government compliance standards as well. By contrast, if the goal of an organization is to use cutting edge solutions, such as machine learning, to increase the traffic and sales of an ecommerce website, then the most appropriate components of the cloud and CI/CD pipeline should be included. This scenario thrives on constant feedback to be received that leads to the development of new capabilities through quick updates and releases. Implementing and maintaining solutions in the cloud through CI/CD is complex. The size of an organization and the workloads they need to run are things to consider before diving into implementing this solution. Small organizations with even smaller IT departments and teams wouldn’t be able to make good use of the cloud through the CI/CD pipeline because of the complexity of the solution. It would be far too costly to hire resources that could support the strategy. Smaller organizations also wouldn’t be able to take advantage of features like scaling IT resources based on traffic. It would be a poor investment and a waste of financial resources that could be better utilized in other aspects of the organization.
It’s important to understand the different workloads that an organization needs to run and decide whether it would be appropriate to deploy them in the cloud through CI/CD. The cloud can be a suitable solution for many workloads, but it is not a solution that is suitable for all. There are several questions organizations should ask themselves before blindly deploying to the cloud. Are there any region or country specific requirements for the workload in question? In certain circumstances, there are regulations by countries in place that require the physical infrastructure running transactions and workloads for their country to be present within the boundaries of the country itself. If the workloads are performed outside of the country, the organization can be subjected to compliance violations and heavy penalties and fines. This isn’t a technical limitation of cloud capabilities; however, cloud infrastructure isn’t currently hosted in every country around the world. This would eliminate the cloud as a potential solution. There are many workloads that are appropriate for the cloud and the capability of those workloads are actually elevated as a result of being deployed to the cloud through the CI/CD pipeline. Simply deploying solutions to the cloud isn’t the best way of making a success of a cloud deployment. If companies use cloud server, storage and network solutions, but still have engineers provisioning virtual machines, volumes and VLANs manually, they are not taking advantage of the CI/CD framework to enable efficient and agile operations. Using manual processes can take longer to fulfill requests and it is more likely that a mistake could be made due to human error. Automating workloads of repeatable tasks in the cloud gives companies a huge advantage in how they run their operations. Customers often make requests for services such as a new storage volume, an increase in capacity of an existing volume, a new VM or the opening of a new firewall port through an internal IT catalog. Automation workflows that have been developed by DevSecOps engineers are kicked off after the approval of the requested service. The workflow validates the requirements of the request provided by the customer and proceeds in provisioning if there are no errors. Due to the unlimited resources in the cloud, automating workloads becomes less complicated since capacity validations are no longer a blocker to move forward. Service level objectives (SLOs) can be shortened giving project teams a more realistic schedule to follow in their project plans. The probability that a service level objective would be missed is low since the automated workflow can process requests quickly and efficiently. It also provides more robust validations of the requirements that notifies the requester if the ticket did not pass validations and provides a detailed rejection response. The user could then correct the errors and re-submit the request again.

VI. USE CASE ANALYSIS

With the dominance of various public cloud providers in the market, more and more customers are making the leap to move their workloads to the cloud through the CI/CD pipeline. Companies from all different industries are leveraging the vast number of solutions and features offered by the cloud to achieve many different goals. Some companies need to enhance their technical capabilities like scalability. Others are reacting to budgetary constraints and are trying to capitalize on cost savings. Some companies are focusing on innovative features and solutions to gain a bigger share of their industry profits. These are some of the high-ranking reasons companies move to the cloud, however, there are many more as well. Below are several examples of use cases from AllSaints, Capital One and Dropbox which discuss the impact the cloud and CI/CD pipeline had on the organizations.

I. USE CASE #1: ALLSAINTS

One of the biggest reasons companies end up moving to the cloud is the ability to deploy microservices to support their business operations. The retail company AllSaints was one of the many companies that opted to move in this direction for that very reason. Companies often struggle with managing capacity in traditional IT environments. Trying to predict peak workloads and procure the capacity they needed became too difficult. Even with their best guess predictions to avoid wasted resources, the additional capacity needed to handle the seasonal peaks of their business was too costly. Finding a more economical, efficient and performant solution was important to maintaining the status of their luxury brand. They were able to strategically run certain workloads in the cloud and following the successful outcome, made the decision to move many of their services in a coordinated effort to Google Cloud Platform (GCP). This helped them to move an entire ecosystem of connected systems and services to the cloud. One of AllSaint’s major reasons for migrating so much of their workload within one project was to avoid latency issues from having split operations. Following their move to GCP, AllSaints developers leveraged the CI/CD pipeline choosing Jenkins and Terraform from the many options on the market and started deploying new capabilities to their customers in a fraction of the time. The identical development and production environments gave them the confidence to make swift code changes and deploy updated services and bug fixes to production quickly and efficiently (“AllSaints: Soaring to Heavenly Site Speeds and Savings with Google Cloud”).

II. USE CASE #2: CAPITAL ONE

Capital One was also another organization that was successful in its adoption of the public cloud by opting to move their workloads off their multiple on-premises datacenters. They leveraged EC2, S3, Relational Amazon Database Services (RDS) and more, all provided by AWS. These services allowed Capital One to think and act more like a tech company giving them the opportunity to embrace their status as a digital bank in the industry. This put them more in tune with their customer base and enabled them to provide real-time personalized services through their website and mobile application, in addition to the traditional banking services (Perkel). Capital One was also able to bring mutual value to their own organization as well as their customers by developing applications through the use of machine learning that helped to significantly decrease fraudulent activity. Their customer centric approach allows Capital One to get feedback from customers and implement feature changes and enhancements quickly by reducing development to production timelines from 3 months to several minutes (“Capital One Completes Migration from Data Centers to AWS, Becomes First US Bank to Announce Going All In on the Cloud”). By embracing the cloud, CI/CD and DevSecOps, Capital One has also given their employees more opportunities to be exposed to many technology services and actively encourages employees to take training classes and gain AWS certifications. This will help them keep their employee base happy knowing their employer is investing in their future. It will help keep Capital One’s workforce stable and reduce debilitating attrition.

III. USE CASE #3: DROPBOX

In contrast to many large organizations, Dropbox started its operations in the AWS cloud. After approximately eight years, they decided to migrate away from the cloud and move their IT operations to an on-premises solution (Metz). The main achievement of this change was a cost savings of 75 million dollars (Krazit). They did not consider the various technology capabilities they had access to in the cloud in order to provide innovative solutions to their existing customer base or to attempt to grow their customer base using them either. Instead, they chose to have their employees work on the tedious task of building out networks and implementing storage and server solutions in their own data center. However, after several years, they quickly found themselves in trouble again seeing that they would soon reach the capacity limits of their own environment. They had no choice but to resume some of their operations in the AWS cloud again (“Dropbox Saves Millions by Building a Scalable Metadata Store on Amazon DynamoDB and Amazon S3”). Ultimately, it can be very defeating to employees that work several years on a project like this and see it fail because the goal was not achievable in the first place.
Without a clear IT strategy to stay current in the market and optimize business strategies to create growth, companies can fail to keep customers and employees happy. This will eventually erode their customer base, reduce their profits, and force the business to potentially go out of business.

VII. BENEFITS

Moving to the cloud and using the CI/CD pipeline can have beneficial effects for organizations and their employees. Companies are able to keep their eye on the horizon of their business and operations. They can stay current with emerging technology, adapt quickly to changes, deploy new and innovative solutions and operate more efficiently. By leveraging the features of the CI/CD pipeline and the cloud, companies gain the opportunity to reestablish their IT presence inside and outside of their organizations. DevSecOps engineers can be assigned to manage specific business partners, becoming a one-stop shop for their application development and infrastructure needs. It is a system that avoids silos, extended lead times, SLOs and handoffs. Organizations also have confidence knowing that they have the extensive support of cloud providers to help them run their operations as these companies and their employees are industry leaders in their field.
Employees of these organizations gain motivation by knowing that they are not working for an organization that is stuck in the past. Instead, they are getting opportunities to work on new technology. They get to test with the many products and solutions included in the CI/CD pipeline and all the cloud services as well. Employees also have the opportunity to research and vet the various vendors and solutions included in this area of technology. This gives the engineers a sense of confidence, pride and satisfaction, which ultimately leads to more employees that are happier in their jobs and with their companies. The benefits associated with employees that enjoy their work and have pride in their organization is invaluable for such businesses.

VIII. CONCLUSION

The CI/CD pipeline and the public cloud have many solutions to offer organizations in all types of industries. It is apparent there are many complexities associated with the implementation and management of this type of environment. Following the analysis of the questions posed at the outset of this research, the benefits ultimately outweigh the complexities of deploying and maintaining the CI/CD framework and public cloud solutions. Without the implementation of this modern technology, IT teams struggle with agility and often have difficulty developing and releasing new solutions. They become bottlenecks for their organizations by operating inefficiently, railing against modernization and inhibiting innovative efforts. The management of a traditional environment is done through manual tasks that can lead to inconsistencies and errors. In contrast, CI/CD and the cloud keep organizations and their employees focused on the future, emerging technology and market trends. The CI/CD pipeline and public cloud offers organizations opportunities to automate solutions allowing them to focus on development activities and expedite deployments. Shifting focus to development with CI/CD and the cloud can quickly bring new products and services to the market for an organization, increasing the customer base and revenues. There are many more solutions available from the three major public cloud providers (e.g., AWS, Google Cloud Platform and Microsoft Azure) than could ever be brought on-premises to be tested and implemented in a traditional environment. Cloud providers have a much larger footprint around the world making international deployments much easier. The capacity in the cloud allows organizations the ability to scale resources up and down on demand based on traffic spikes avoiding over or under procuring capacity issues. This shift from capital expenditure to operational expenditure avoids wasting money on idle systems and protects from insufficient IT resources that can cause outages during increased workloads. The automation of repeatable tasks within the CI/CD pipeline creates a reliable environment for the administrators to manage and results in a positive experience for customers of the organization. The CI/CD pipeline also places a focus on the testing aspects of code development, which helps create high quality solutions for the organization. CI/CD and the cloud also gives engineers an opportunity to transition into DevSecOps roles allowing them to break away from the traditional server, storage, network and security silos they previously operated in. They now have the ability to manage all aspects of their applications and line of business resulting in better relationships with their business partners. DevSecOps engineers are able to implement a more well-rounded approach when designing solutions since the development, infrastructure and security tasks are broken down into small and manageable pieces. Security is a challenge in traditional IT environments as well as the CI/CD pipeline and cloud. The risks, threats and vulnerabilities still need to be identified and managed to avoid as many impacts to the environment as possible. Vulnerabilities such as CVEs, code injections, excessive access and rogue employees need to be taken into consideration. Plans need to be developed to ensure engineers know what actions to take to reduce risks. However, scans of the environment to identify CVEs in operating systems, repositories, images, databases and more can be added to the automated efforts of the CI/CD pipeline as opposed to traditional operations. Code created by developers should be scanned within the pipeline to identify buffer overflows and injection attacks. Periodic audits of system access should also be performed to avoid unnecessary privileges that could result in service impacts. The CI/CD pipeline also includes methods within the framework to handle secrets properly to reduce the risks of security-related incidents like ransomware attacks. Even oversight policies like change management that are developed to minimize risk to IT environments can benefit from CI/CD and the cloud. The CI/CD pipeline is able to partner with IT catalogs and ITSM systems to implement standard changes by automating requirement validations and deploying standard changes with blanket approvals. Automated workflows, innovative services and expansive capacity included in the CI/CD pipeline and public cloud provides organizations with opportunities to grow and become more profitable in their industries. The focused attention of DevSecOps engineers on their projects brings a fresh approach to IT operations by creating an environment that enables innovation, with consistency and stability. Technical concepts like automated testing, scaling and deployments all contribute to the success of this modern methodology. A flexible environment like this allows for rapid development changes and provides the ability to quickly deploy new capabilities. Typical pitfalls such as siloed teams, slow development and capacity issues are avoided by organizations that shift to the DevSecOps model using the CI/CD framework and public cloud. Organizations are able to distinguish themselves from their competitors by providing them an advantage in the market. Due to the vast number of benefits that result from the implementation of systems in the public cloud through the CI/CD pipeline, organizations should make every effort to work through the complexities in order to deploy this modern solution.