devops beware: your servers are open for businesswow.intsights.com/rs/071-zwd-900/images/devops...

Threat Intel l igence Real ized.

1 IntSights Cyber Intelligence Ltd. Copyright © All Rights Reserved 2018.


April 2018


DevOps Beware: YOur servers are Open fOr Business




Thousands of organizations have DevOps servers that are open to the web, scarcely protected, giving attackers easy access to sensitive information.


executive summaryIn recent years, DevOps, the culture and practice of automating and monitoring the development life cycle, has enabled teams to deliver software increasingly faster and shorten the time to market. At the same time, it’s also led to the creation of a wide range of DevOps tools, most of which are cloud-based, meaning cyber criminals have a new attack vector they can use. This causes massive security risks to organizations because they are quite frequently (and unknowingly) exposing sensitive information and making it easy for hackers to access DevOps servers.

DevOps teams are always looking for new tools to help accelerate product delivery, but they don’t always involve their security team when setting up and configuring these tools. So as new tools are incorporated into DevOps environments, it becomes increasingly complex to manage all the different security and configuration settings for each tool. Additionally, the transition to cloud-based tools and operations increases an organization’s digital footprint, leaving more and more breadcrumbs for attackers to collect and utilize in an attack. Companies invest thousands of dollars each year to accelerate the development lifecycle, but in an effort to increase speed and agility, security gets tossed aside, simply due to oversight.

If you’re not sure how your DevOps servers are setup and configured, you may be leaving sensitive information wide open for anyone to access.

In this report, you’ll learn how many DevOps servers may be exposed based on a study done by the IntSights research team, how cyber criminals typically access open DevOps servers, and what you can do to protect yourself and your data from a DevOps cyber attack.

focus and methodsThis research focuses on the security configuration of DevOps platforms, tools, and servers, and the ease with which misconfigurations can lead to enormous security breaches. We set out to check how wide this phenomenon of open and accessible DevOps tools and servers are.

To achieve that, we compiled a list of 25,876 URL’s and subdomains of known DevOps tools of different organizations, by searching the name of the tool under the domain of the company. Most of the DevOps tools names are pretty unique, so any DevOps tool name under a domain (Jenkins.example.com) was considered a probable DevOps server. On that list, we wanted to see how many servers will return an HTTP 200 OK response. From our initial 25,876 servers list, 5967 servers (23.06%) returned a positive response. We also took into account 401 (unauthorized) and 403 (forbidden) responses, as they too signal a somewhat exposed server.




Short hiStory of the DevopS role


The DevOps role started to emerge as early as 2009 and aimed at helping developers with building their development infrastructure, a job traditionally reserved for the IT department. As DevOps infrastructure is more complex and integrated, DevOps tools deployment requires a high level of development cycle understanding, good relations with Dev and IT alike, problem-solving capabilities, and technical capabilities – capabilities not every IT staff possesses. As companies switched to agile methodology and wanted to establish an agile working environment, the need increased for qualified DevOps engineers.

The DevOps position required a mix of technical knowledge in deploying software infrastructure tools, combined with the understanding of the development process as a developer sees it. Moreover, it required taking decisions on tools, methodologies, and processes more related to R&D roles than to IT roles. That’s why the ideal candidate was and still is, very hard to find. Some research shows that the #1 hardest tech job to fill in North America is that of the DevOps engineer. But as this job was a middle ground between Dev and IT, the DevOps starting point wasn’t so security minded.

The DevOps role was to build the systems, operate them, and keep the whole process flowing. As the business grew, engineers needed to keep the system scalable and ready for rapid growth and development. This process was very demanding, with a steep learning curve, focused on learning to operate and deploy. Once that was achieved, things moved forward to production without giving any real thought for security, as it wasn’t anybody’s direct responsibility. The CISO could direct and give instructions, but the needs of R&D for an up and running system was stronger. Security was part of the process, but as a lot of the infrastructure being built was new, most engineers focused on getting systems up and running and weren’t mindful of security issues and configurations.




Figure 1 The Multitude of DevOps tools (Credit: XebiaLabs)

There are currently about 120 different tools and technologies in the DevOps landscape (see Figure 1). The DevOps engineer usually comes from either the IT department or from R&D. The engineer needs to be knowledgeable of both domains to bridge the gap between the departments and move the development process forward. Each of those traditional departments faces security challenges of their own that have their own solutions. But for the DevOps engineer, the situation becomes a lot more complicated, as each tool presents its own learning process, demands, and integration steps.

Here is a short list, just to mention a few of these tools:

1. Source Code Management - Git, GitHub, Bitbucket, Subversion, TFS.

2. Continued Integration (CI) - Jenkins, Bamboo, Travis CI, Codeship, TeamCity.

3. Build Servers - Maven, Gradle, Ant, CMake.

4. Config / Provisioning - Chef, Puppet, Ansible, Vagrant.

5. BI / Monitoring - Kibana, New Relic, Nagios, Zabbix.

6. Collaboration - Trello, Jira, Slack, HipChat.

7. Cloud Services - Amazon, Azure, Google Cloud, OpenStack, Heroku, Rackspace.

8. Containerization - Docker, rkt, Mesos, Swarm, Kubernetes, Nomad.

Each of these tools has their own design, configurations, credentials, filters, API’s, protocols, user lists security configurations and security holes. Some of them don’t even have basic authentication to access them. Although no organization uses all these tools simultaneously, managing even a fraction of these tools can give a DevOps engineer a very large headache.

In a rush to deploy and operate these tools, their security aspect was sometimes overlooked. A DevOps engineer who is constantly struggling to maintain and keep the system in a production state doesn’t have the time and focus to weigh the security features of each product.

Most of the time, the IT or Dev environment does not contribute to the secure implementation of DevOps products, as they have security aspects different from traditional IT software or development systems. Furthermore, the sheer volume of tools to deploy and configure, and the speed at which things need to be done, leave room for a great deal of human error and oversight. Moreover, you have the CISO’s perspective on DevOps tools security with its clashing priorities: Although the CISO is aware of the possible security holes from DevOps platforms, power struggles with R&D, and the need to keep the development process flowing without hindrance, often creates a vacuum in which these misconfigurations can occur.

the DevopS univerSe of toolS anD challengeS




According to a SANS survey, companies invest thousands of dollars in securing their development process. More than 60% of responders have mentioned the securing of the development process as their number one spending area, with companies spending between $500K to $10M on security each year. With so much money going into securing the apps and programs companies produce, it should have minimized the risk of vulnerable apps. But the fast migration to the cloud has made this process a lot harder.

In the past, the development tools and release cycles were more or less constant and gave the CISO the time to blend security objectives into the process. But today, the move to cloud-based tools together with fast release cycles makes the CISO’s mission near impossible. Each new tool needs to be learned thoroughly in order to evaluate its risk factor and prepare a security plan for it.

But today’s methodology is “develop first, ask questions later.” New tools are adopted and deployed faster and faster. And as each new tool goes into production, it becomes harder to tweak and change it, because of the constant fear of breaking a production system. As the old saying goes: “if it’s not broken, don’t fix it.” Moreover, as developers themselves are using these tools from a user’s perspective, they do not know the inner workings of the tools they use, and how they can be compromised. In addition, DevOps engineers themselves don’t have the time to study each guide or drill down to the best security practices of each tool. In large corporations, it’s not an uncommon sight to see different development groups and divisions using completely different tools. Sometimes, developers will even start using a web-based tool, without the help of DevOps or the knowledge of the CISO, making it very hard to know where, and how, they are vulnerable.

DevOps tools are 99.9% web-based. Most of them utilize the cloud as the platform of choice for developing, deploying, storing, and running software. In typical cloud-based product development, there are at least six to seven of the above tools involved in the process. Each of these tools holds valuable company information, which when left exposed, can pose a serious risk to company assets. These include source code, username lists, development server naming conventions, internal IPs and network structure. This is beside the potential for malware to penetrate (as with the CCleaner supply-chain attack). An attacker who searches a way into the company network doesn’t need to look too far -- these tools are accessible from the web, ripe for the taking.

In our research, we have gathered a list of 25,876 URLs of different DevOps tools and servers from various organizations. What we did was simply to try connecting to them through a browser. No fancy attack tools, or port scanning, or any preliminary data, except for using open OSINT tools and websites to create the list. The list was based on the name of the tool appearing as a subdomain of the company. Every server that answered with a live and valid response (HTTP 200 response), we considered as an open server. Later, we drilled down into these open servers to check what was presented on the page, and which type of security was present - if any. We also included responses such as 401 (unauthorized), as they still count as an open login page, but with somewhat fewer implications, as we can’t tell what type of security and authentication sits behind the preliminary authentication request. But in our opinion, they can pose as much a risk as a HTTP 200 responses, as they could be just a simple username/password combination. We also considered 403 (forbidden) responses, as these servers still validate their existence, although requiring more investigation into how to connect to them.

The rush To The Cloud




Here we see an open Kibana server. Kibana is a search and visualization utility, part of the very popular Elasticsearch stack (ELK stack). Access to Kibana means that an attacker can run queries on the company’s Elasticsearch database. It can retrieve user data, internal coding conventions and names, and any other data stored in the database (Figure 4):

We found several wide-open servers, relying only on knowing the server URL to access them. Some servers were protected with a simple username/password combination, which poses no real defense against a determined hacker. But in today’s world, even an undetermined hacker has dozens of other ways to acquire basic authentication information or employee naming conventions, drastically narrowing down the amount of brute force needed to crack these login pages. Even without a noisy brute force attack, there are vulnerabilities that allow an attacker to bypass these login pages.

Figure 2 – Open DevOps Servers – By HTTP Response

Figure 4 – Open Kibana server

Figure 3 - Open DevOps Servers - By Tool

15.6%

17.5%

10.0% 11.6%

8.6%

5.7%

5.9%7.4%7.6%

mavenTrelloAPMBitBucketjira

bamboonagiosicingaSlackSplunk

chefPuppetJenkinsKibana

HTTP 200 Response

HTTP 401 Response

HTTP 403 Response

Kibana

We found that 5967 out of the 25,876 that we tested were accessible from the web (23.06%). The level of access differed from organization to organization. Some were totally exposed without any user/password combination, exposing company data, user lists, internal server names, etc. Most were protected with a simple login page, and a minority with a more robust cloud access security broker (CASB). From the servers we did find exposed, we divided our findings according to the responses we got (Figure 2):

19.1%

17.1%

63.7%



Threat Intel l igence Real ized.Threat Intel l igence Real ized.

Here we see a Nagios login page. Nagios is an IT infrastructure monitoring server. Access to the data inside these servers can expose the whole IT structure of a company to an attacker. In one case we saw, in addition to leaving the server accessible from the web, the admin added a comment to remind users to “use your Windows account,” which gives an attacker a very specific set of credentials to use (Figure 5):

Jenkins is one of the more popular CI/CD automation servers in use. It holds information on all the build data of a company’s software, such as versions, users, and links to repositories. In our research, we found totally exposed Jenkins platforms (Figures 6 and 7):

nagioS

jenKinS

Figure 6 – Jenkins open server list

Figure 5 - Nagios Login page




Splunk is a tool for big data analytics. It’s a tool for managing, monitoring, and analyzing huge datasets that come from multiple sources. It can contain any data for which the company needs analytics. Below is an example of an open Splunk login page of a government authority (Figure 8). Even if this database is exposed on purpose for development or productivity needs, this must be done in a more secure manner. Open pages like this will tempt any hacker to check what it may contain.

Our findings present an alarming picture, as the data from numerous organizations are open to the public, and this is from a simple search for known DevOps tools, a feat that even the most novice attackers can achieve. A determined and knowledgeable attacker can use these open tools to stage a larger attack, or use smaller, more hackable companies as part of supply chain attacks to leverage the attack on larger organizations.

Figure 7 – Jenkins open workspace

SplunK

Figure 8 – .GOV open Splunk server




Elasticsearch is part of the ELK stack and in its default form, requires no authentication, allowing everyone who knows the server’s address to connect, view database names and structures, or even delete whole databases (Figure 9).

elaSticSearch

Figure 9 – Exposed Elasticsearch console

Opendev Tools findings By industry

Figure 11 - Open Dev Tools - By Industry

Software

/IT

Constr

uctio

n

Telec

om

Finan

ce

Educati

on

Retail

Automoti

ve

Infras

tructu

re

Govern

ment

Health

care

Others

Hotel +

Enterta

inmen

t

Food

Manufa

cturin

g

Figure 10 – Exposed Elasticsearch Database

Open source Dev tools are usually used by technology companies, in our research we have tested about 25,000 random and different domains, and as we can see most of our findings were in software and IT services companies. The following chart (Figure 11) is a breakdown of the research findings based to the various industries.

50%50%

60%

30%

40%

10%

20%

.50% .50%1%3% 3% 3%7%10% 10%

4% 4% 4%




In our research, we saw all kinds of security levels for DevOps tools. Some sites were totally inaccessible from the outside, while others were totally exposed. Most sites used only the login page for the tool itself. The problem with this approach is that the default login page is just not safe enough. Through simple internet research, an attacker can narrow down the username/password combinations to a relatively small number of options and build a dictionary that will minimize the time required to access the page from years and weeks to days and hours. In our regular research for our customers, we find numerous leaked user/password combinations. In this research, we did just a small sampling of the different sites, and we still found numerous username/password combinations for some of the sites we’ve tested.

A small percentage of sites referred us to a login page with other types of accounts such as Google, GitHub, Microsoft, and even Facebook accounts. These types of authentication methods are relatively more safe then just the server login page. Just a handful of sites used a third-party authentication solution (CASB), or a multi-factor authentication system.

This itself is alarming, as some of the tools do not supply an out-of-the-box multi-factor authentication system. They rely on plug-ins, or third-party add-ons to strengthen the login process, making it harder for DevOps engineers to secure their systems. The developers of these tools – as well as the developers using them – focused on making the tools work, not securing them. And DevOps engineers followed suit, making the tools accessible and working while neglecting to secure access to them.

1. Don’t use DevOps tools names for web facing servers – as trivial as it might sound, when you’re choosing the name of your server, don’t use the real tool name, like Jenkins, Kibana, Trello, Jira, etc. It might make your life a little less convenient, but it will make the intelligence gathering a lot more difficult. A development server name under your domain name (Jenkins.example.com) will surely attract unwanted attention and will make a nice target for an attack. The same goes for obvious and interesting systems (Dev.* Staging.* QA.* DB.* etc.).

2. enable Multi-factor authentication – most of these tools don’t have a built-in Multi-Factor Authentication. Use third-party plug-ins, or third-party vendors to strengthen the login process to your servers– a simple login page won’t stop hackers from getting to your data.

3. use a proxy server – instead of giving direct access from the web to your development servers, add a proxy server to which users will connect, and from there route the traffic to your development servers. The proxy can be combined with another layer of authentication, or left as is for production needs, but it will still be a gateway to your servers and will help in monitoring traffic and in hiding your servers from the public eye.

4. Don’t use Default ports – The default ports of these servers are a target for scanning. Using unusual ports will lower your attack surface and will make your servers harder to get. This can be combined with a proxy server, where internally, the default ports will be used for convenience, but any outside communication to the servers will be forwarded from other, less known ports.

5. patching – DevOps tools are in no way immune to bugs and vulnerabilities. It’s always frightening to upgrade a production system, but as these servers are accessible from the web, patching them is even more critical than tools and systems inside your organization. If you absolutely can’t minimize the servers’ exposure to the web, at least keep them patched against the latest vulnerabilities.

6. Block access to the servers from the web – while not feasible for every organization, if possible, restrict access to these servers from a defined range of IP addresses, such as your corporate network.

7. use an external Monitoring service – As opposed to managing all of your DevOps servers through internal tools and manual processes, an external monitoring service will enable you to keep track of which servers may be openly accessible on the web. Using a third-party monitoring solution that scans sources outside of your organization helps you find your open servers the same way an attacker would be able to. This provides you with a full visualization of your external digital footprint, allowing you to see which of your servers are vulnerable, in which ports, and if there are any leaked user credentials that could compromise your servers.

Levels Of security

recommendations for Mitigation




intSightS cyber threat intelligence Our platform can show you your organization’s digital footprint, its weak spots, and how easy it is to target you. It lets you see your organization through the eyes of an attacker. Our platform can automate the process of discovering and alerting on open Dev platforms, login pages, and much more, enabling your organization to mitigate weakness and cyber attacks before they are exploited or executed, by giving you the most advanced cyber threat intelligence tailored to your organization, market, and needs.

The move into cloud tools and methodologies has proven to be very valuable and is here to stay. The learning curve of new technologies and software is on a constant rise, and the challenges in security this shift brings needs to be addressed. In DevOps, as we’ve shown, some challenges are very simple to mitigate, and some need more fundamental changes in thinking and culture to solve. But in order not to lose the value of what already has been achieved, we need to solve the simple and mundane challenges, before moving on to more complex and sophisticated security concerns.

about the analystAriel Ainhoren is a Security Researcher at IntSights, focused on discovering new cyber trends, threats, hacker strategies and vulnerabilities. He is a seasoned security professional with over 8 years of experience in the cyber industry, with expertise in computer forensics, malicious programs, vulnerability management and Microsoft Products. Ariel enjoys solving cyber puzzles, preferably byte by byte.

concluSion




about intsightsIntSights is redefining cyber security with the industry’s first and only enterprise threat management platform that transforms tailored threat intelligence into automated security operations. Our ground-breaking data-mining algorithms and unique machine learning capabilities continuously monitor an enterprise’s external digital profile across the surface, deep and dark web, categorize and analyze tens of thousands of threats, and automate the risk remediation lifecycle — streamlining workflows, maximizing resources and securing business operations. This has made IntSights’ one of the fastest growing cyber security companies in the world. IntSights has offices in Tel Aviv, Amsterdam, New York and Dallas and is backed by Glilot Capital Partners, Blumberg Capital, Blackstone and Wipro Ventures. To learn more, visit www.intsights.com.


devops beware: your servers are open for businesswow.intsights.com/rs/071-zwd-900/images/devops...

Documents