Security Walls in Linux Environment: Practice, Experience, and Results

Mykola PerehinetsI&O, IS Application Administrator

SoftServe Inc., 11/02/2016System-Part1

Agenda Vision of our problems Searching for solutions

Practical software Some more ideas Analysis of results Literature Questions and answers

Vendors Vision of Situation

GNU/Linux distribution for ALL people

Cruel Reality and Other Issues

Dirty COW (CVE-2016-5195) is a privilege escalation vulnerability in the Linux Kernel

Why is it called the Dirty COW bug?"A race condition was found in the way the Linux kernel's memory

subsystem handled the copy-on-write (COW) breakage of private read-only memory mappings. An unprivileged local user could use this flaw to gain write access to otherwise read-only memory mappings and thus increase their privileges on the system."

Y2007 Y2016

Cruel Reality and Other Issues

Dirty COW (CVE-2016-5195) is a privilege escalation vulnerability in the Linux Kernel

Why is it called the Dirty COW bug?"A race condition was found in the way the Linux kernel's memory

subsystem handled the copy-on-write (COW) breakage of private read-only memory mappings. An unprivileged local user could use this flaw to gain write access to otherwise read-only memory mappings and thus increase their privileges on the system."

Y2007 Y2016Hackers

Vulnerability

Rootkits

Trojans Human factors

Our Vision of Situation Y2076

Our Vision of Situation Y2016

Distribution for YOUR PRODUCTION!!!

8. And conspired all of them together to come and to fight against Jerusalem, and to hinder it.14. And I looked, and rose up, and said unto the nobles, and to the rulers, and to the rest of the people, Be not ye afraid of them: remember the Lord, which is great and terrible, and fight for your brethren, your sons, and your daughters, your wives, and your houses.17. They which builded on the wall, and they that bare burdens, with those that laded, every one with one of his hands wrought in the work, and with the other hand held a weapon.18. For the builders, every one had his sword girded by his side, and so builded. And he that sounded the trumpet was by me.

Nehemiah 4:8-18

Your Vision of Situation

Our Vision of Situation Y2016

Distribution for MY PRODUCTION!!!

Your Vision of Situation

Real Way for Us

We Really Need Solutions

We Really Need Solutions

Real Way for Us

Practices of Security

Internal Audit

Protection of Communications

Protection of File Systems and Data

Protection of Configuration

Files

Practices of Security

Protection of Kernel

Internal Audit

Protection of Communications

Protection of File Systems and Data

Protection of Configuration

Files

Practices of Security (Software)1. Etckeeper - is a revision

control system for your /etc directory using bzr, git, hf, or darcs as a back-end. https://github.com/joeyh/etckeeper

2. AIDE - (Advanced Intrusion Detection Environment - Host Based IDS) is a file and directory integrity checker. It creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files. http://aide.sourceforge.net/

3. Tripwire Software - can help to ensure the integrity of critical system files and directories by identifying all changes made to them. http://www.tripwire.com/

Protection of Configuration

Files

Practices of Security4. Spacewalk is an open source

Linux systems management solution that allows you to: manage and deploy configuration files to your systems, distribute content

across multiple geographical sites in an efficient manner, inventory your systems. http://spacewalk.redhat.com/ https://fedorahosted.org/spacewalk/wiki/HowToInstall#SettingupSpacewalkrepo5. Setup a Local Mail Server and Create Server Mail Group. [root@ua /]# cat /etc/aliases root: SecurityOperators@softserveinc.com 6. Use LogWatch is a log parsing program that analyzes and

generates daily reports on your system’s log activity.

Protection of Configuration

Files

Practices of Security (Software)1. Chkrootkit - locally checks for

signs of a rootkit. http://www.chkrootkit.org/

2. Rkhunter - scanner tool for Linux systems (+need update).

3. ClamAV - antivirus engine for detecting trojans, viruses, malware & other malicious threats. http://www.clamav.net/

4. Available Repositories Provided by CentOS - these repositories have varying levels of stability, support and cooperation within the CentOS community. Please Verify Your Repo List! https://wiki.centos.org/AdditionalResources/Repositories

5. Install additional plugin yum-cron - The package that allows us to do automatic updates via yum (auto-update mechanism). Please Always Update Your Systems!

Protection of File Systems

and Data

Practices of Security6. Spacewalk - is a systems

management solution allows you to: install and update software on your systems, collect and distribute your custom software packages.

7. Bacula/Bareos - is a set of Open Source, computer programs that permit you (or the system administrator) to manage backup, recovery, and verification of computer data across a network of computers of different kinds. Bacula is relatively easy to use and very efficient, while offering many advanced storage management features that make it easy to find and recover lost or damaged files. Please Backup Your Systems! http://blog.bacula.org/source-download-center/ http://download.bareos.org/bareos/release/latest/

Protection of File Systems

and Data

Practices of Security8. Bacula File Integrity Check is a

feature can be used for detecting changes to critical system files similar to what a file integrity checker like Tripwire does.

9. OSSEC - is a Open Source HIDS SECurity. OSSEC watches it all, actively monitoring all aspects of Unix system activity with file integrity monitoring, log monitoring, rootcheck, and process monitoring, writing scripts that take actions in response to security alerts. http://ossec.github.io/ https://atomicorp.com/ , http://wazuh.com/ https://www.alienvault.com/

Protection of File Systems

and Data

Practices of Security10. Secure Partition Mount Options

please use in /etc/fstab: noatime,nosuid,noexec,nodev

11. Use Secure Disk Partitioning use for your server: “/boot”, “/”, “/home”, “/var”, “/tmp”, “/usr”, “/opt”

Protection of File Systems

and Data

12. Prevent Mounting USB Storage in your servers echo "install usb-storage /bin/false" > /etc/modprobe.d/usb- storage.conf13. Mount “/boot” partition in ‘read-only’ mode use for this in /etc/fstab next options for “/boot”: defaults,nosuid,nodev,ro (manually re-mount as ‘read-write’ for system update)

[root@ua /]# df -ThФ. система Тип Розм Вик Дост Вик% змонтований наdevtmpfs evtmpfs 16G 0 16G 0% /devtmpfs tmpfs 16G 84K 16G 1% /dev/shmtmpfs tmpfs 16G 410M 16G 3% /runtmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup/dev/mapper/system--lvm-root xfs 60G 657M 59G 2% //dev/mapper/system--lvm-usr xfs 60G 6,9G 53G 12% /usrtmpfs tmpfs 16G 4,0M 16G 1% /tmp/dev/sda1 xfs 1014M 402M 613M 40% /boot/dev/mapper/system--lvm-var xfs 30G 5,8G 25G 20% /var/dev/mapper/system--lvm-RW xfs 334G 58G 277G 18% /RW/dev/mapper/system--lvm-home xfs 15G 48M 15G 1% /hometmpfs tmpfs 3,2G 16K 3,2G 1% /run/user/42tmpfs tmpfs 3,2G 0 3,2G 0% /run/user/0[root@ua /]#

Practices of Security

Protection of Kernel

Internal Audit

Protection of Communications

Protection of File Systems and Data

Protection of Configuration

Files

Practices of Security (Software)1. Edit sysctl.conf - is an tweaking feature that reads and modifies

the attributes of the system kernel such as its version number, maximum limits, and security settings.

2. Use nscd - is a daemon that provides a cache for the most common name service requests.

4. NTP Client (Chrony) to synchronize the time of your local Linux client machine with NTP server, edit the /etc/ntp.conf file on the client side. Comparison of NTP implementations.

5. Configure Rsyslog with Any Log File Forwarding to other server!

Protection of Kernel

3. Configure DNS Client - to configure Linux as DNS client you need to edit or modify /etc/resolv.conf file.

[root@ua /]# cat /etc/sysctl.conf# Kernel sysctl configuration file for Red Hat Linux…# Controls IP packet forwardingnet.ipv4.ip_forward = 0…# Controls source route verificationnet.ipv4.conf.default.rp_filter = 1net.ipv4.conf.default.arp_filter = 1net.ipv4.conf.all.rp_filter = 1net.ipv4.conf.all.arp_filter = 1…# Log Martian Packetsnet.ipv4.conf.all.log_martians = 1

vm.swappiness = 0net.ipv4.tcp_congestion_control = htcpnet.ipv4.tcp_window_scaling = 1net.ipv4.tcp_timestamps = 1net.ipv4.tcp_sack = 1net.ipv4.tcp_fack = 1net.ipv4.tcp_low_latency=1…

Practices of Security6. Security-Enhanced Linux (SELinux) - is an implementation of a

Mandatory Access Control mechanism in the Linux kernel, checking for allowed operations after standard discretionary access controls are checked. SELinux can enforce rules on files and processes in a Linux system, and on their actions, based on defined policies.

7. Applications optimization – Java:Huge Pages, Lan:Multipathing.8. ELRepo - is a community repository for Enterprise Linux

distributions. ELrepo-kernel channel provides the latest Stable Mainline Kernels. http://elrepo.org/tiki/kernel-ml

Protection of Kernel

SELinux is enabled by default in Red Hat Enterprise Linux. Please use option enforcing or permissive!

Practices of Security9. Write Custom System Audit Rules (in SELinux) - by default, the

audit system records only a few events in the logs such as users logging in, users using sudo, and SELinux-related messages. It uses audit rules to monitor for specific events and create related log entries. It is possible to create personal audit rules!

Protection of Kernel

[root@ua rules.d]# cat /etc/audit/rules.d/audit.rules# This file contains the auditctl rules that are loaded# whenever the audit daemon is started via the initscripts.# The rules are simply the parameters that would be passed# to auditctl.

[root@ua rules.d]# cat /etc/audit/rules.d/audit.rules…-w /etc/localtime -p wa -k time-change-a always,exit -F arch=b64 -S adjtimex -S settimeofday -S stime -k time-change-a always,exit -F arch=b64 -S clock_settime -k time-change-w /etc/group -p wa -k identity-w /etc/passwd -p wa -k identity…-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged-a always,exit -F path=/usr/libexec/abrt-action-install-debuginfo-to-abrt-cache -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged…-w /etc/sudoers -p wa -k scope-w /var/log/sudo.log -p wa -k actions-w /sbin/rmmod -p x -k modules-w /sbin/modprobe -p x -k modules-a always,exit -F arch=b64 -S init_module -S delete_module -k modules-e 2

Practices of Security (Software)

5. On-Line System Monitoring - for SSH sessions – use Glancesis a cross-platform curses-basedsystem monitoring tool writtenin Python. https://github.com/nicolargo/glances

1. Service Management - Systemd is an init system and system manager that is widely becoming the new standard for Linux machines. Verify your services and DISABLE UNNEEDED!

2. Enable Firewall - Firewalld provides a dynamically managed firewall with support for network/firewall zones to define the trust level of network connections or interfaces.

3. How do I disable IPv6? (Daniel Walsh not recommends)4. Use Multiple IP Network Interfaces/cards for prevent network

performance bottlenecks and improved security.

Protection of Communications

Practices of Security

Practices of Security For WEB sessions - real-time performance monitoring, done right! This is the default dashboard of NetData: real-time, per second updates, snappy refreshes! 300+ charts out of the box, 2000+ metrics monitored! zero configuration, zero maintenance, zero dependencies! https://github.com/firehol/netdata https://github.com/firehol/netdata/wiki/Installation

Protection of Communications

For FULL TIME monitoring – use monitoring with Collectd, InfluxDB & Grafana or The InfluxData Platform is the first purpose-built, end-to-end solution for collecting, storing, visualizing and alerting on time-series data at scale.

Practices of Security

Protection of Communications

Practices of Security

Protection of Communications

https://influxdata.com/get-started/sending-data-to-influxdb-with-telegraf/

Practices of SecurityThis Platform Based on the TICK stack, all of the components of the platform are designed to work together seamlessly.http://www.vishalbiyani.com/graphing-performance-with-collectd-influxdb-grafana/http://grafana.org/https://dbiers.me/setup-grafana-influxdb-collectd-centos-7-x/ https://influxdata.com/get-started/what-is-the-tick-stack/https://influxdata.com/get-started/download-and-install-influxdb/

Check_MK is comprehensiveIT monitoring solution in the tradition of Nagios. http://mathias-kettner.com/check_mk.html

Protection of Communications

Practices of Security

9. Suricata Engine is an Open Source, high performance Network IDS, IPS and Network Security Monitoring engine. https://oisf.net/suricata/

6. Protect with Fail2Ban(+setup) - this solution scans log files (e.g. /var/log/error_log) and bans IPs that show the malicious signs - too many password failures, seeking for exploits, etc. http://www.fail2ban.org/wiki/index.php/Main_Page

7. ‘Hang’ all Production Services/Demons to the Separate Network Adapters and/or Ports. (+setup Your Firewall Rules)

8. Certbot, is an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your webserver to USE HTTPS! https://certbot.eff.org/about/

Protection of Communications

[root@ua /]# netstat -ntulpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.xxx.xxx:8000 0.0.0.0:* LISTEN 26610/python tcp 0 0 0.0.0.0:4545 0.0.0.0:* LISTEN 3359/rhnmd tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 4518/varnishd tcp 0 0 192.168.xxx.xxx:6789 0.0.0.0:* LISTEN 26446/ceph-mon tcp 0 0 0.0.0.0:9102 0.0.0.0:* LISTEN 3291/bacula-fd tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 2474673/netdata tcp 0 0 192.168.xxx.xxx:80 0.0.0.0:* LISTEN 4518/varnishd tcp 0 0 0.0.0.0:8083 0.0.0.0:* LISTEN 3277/influxd tcp 0 0 0.0.0.0:8086 0.0.0.0:* LISTEN 3277/influxd tcp 0 0 192.168.xxx.xxx:22 0.0.0.0:* LISTEN 3296/sshd tcp 0 0 192.168.xxx.xxx:3000 0.0.0.0:* LISTEN 3279/grafana-server tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 3277/influxd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 4523/master tcp 0 0 0.0.0.0:6556 0.0.0.0:* LISTEN 3292/xinetd udp 0 0 0.0.0.0:8096 0.0.0.0:* 3277/influxd udp 0 0 172.xxx.xxx.xxx:123 0.0.0.0:* 2481262/ntpd udp 0 0 192.168.xxx.xxx:123 0.0.0.0:* 2481262/ntpd udp 0 0 127.0.0.1:123 0.0.0.0:* 2481262/ntpd udp 0 0 0.0.0.0:123 0.0.0.0:* 2481262/ntpd

Practices of Security

Protection of Kernel

Internal Audit

Protection of Communications

Protection of File Systems and Data

Protection of Configuration

Files

Some More Ideas for UsSending alerts to administrators:[root@ua /]# cat /etc/profile…echo “ALERT on `hostname`: Shell access to your server! Detail information: incident time - '`date` `who`'.” | mail -s "ALERT from `hostname`: Access to your server from IP: `who | cut -d"(" -f2 | cut -d")" -f1`! Please verify this issue and approve (if need)!" SecurityOperators@softserveinc.com…Improve SSH protocol security:[root@ua /]# cat /etc/ssh/sshd_config…# Specifies the ciphers allowed for protocol version 2Ciphers aes128-ctr, aes192-ctr, aes256-ctr, arcfour256, arcfour128, arcfour

Some More Ideas for Us# Specifies the MAC (message authentication code) algorithmsMACs hmac-sha1, umac-64@openssh.com, hmac-ripemd160, hmac-sha2-256, hmac-sha2-512…Disable reboot using ‘CTRL+ALT+DELETE’ keys:[root@ua /]# systemctl mask ctrl-alt-del.target

The CIS-CAT Benchmark Assessment Tool:CIS-CAT is a host-based configuration assessment tool. A Java-based tool that compares the configuration of target IT systems to CIS Benchmarks and reports conformance scoreson a scale of 0-100.https://benchmarks.cisecurity.org/downloads/audit-tools/ The OpenSCAP Family Tools: https://www.open-scap.org/tools/

Some More Ideas for UsMonitoring users activity using ‘psacct’ or ‘acct’ tools:If you have lot of users who access your servers frequently in your company and if you wanna to keep an eye on what data they are accessing, what commands they are issuing, how long they have been accessing servers and how much system resources are consumed by them, then psacct or acct are the tools that you should have (starting psacct or acct as service)! Display Statistics of Users Day-wise:[root@ua /]# ac -d Display Time Totals for each User:[root@ua /]# ac -p Print All Account Activity Information:[root@ua /]# sa

Use iPerf - The ultimate speed test tool for TCP, UDP and SCTP.

Practices of Security

Protection of Kernel

Internal Audit

Protection of Communications

Protection of File Systems and Data

Protection of Configuration

Files

Analysis of Results (Software)

4. Security Content Automation Protocol (SCAP) Validation Program is designed to test the ability of products to use the features and functionality. https://scap.nist.gov/ https://www.open-scap.org/

1. Nmap - ("Network Mapper") is a free and open source (license) utility for network discovery and security auditing. https://nmap.org/

2. Wireshark - is the world’s foremost and widely-used network protocol analyzer. https://www.wireshark.org/

3. Nessus(+plugins) - prevents network attacks by identifying the vulnerabilities and configuration issues that hackers use to penetrate your network. http://www.tenable.com/

Internal Audit

Analysis of Results5. Tcpdump - dump traffic on a network.

http://www.tcpdump.org/ http://www.winpcap.org/windump/

6. Elastic Stack (Beats, Logstash, Elasticsearch, Kibana, X-Pack) - Elastic's open source solutions solve a growing list of search, log analysis, and analytics challenges across virtually every industry. https://www.elastic.co/ https://www.elastic.co/downloads/x-pack

Internal Audit

7. Logscape - is a big data analytics tool, which allows you to turn your data into knowledge. http://logscape.github.io/ http://logscape.com/

Analysis of Results

10. Splunk (+plugins) makes it simple to collect, analyze and act upon the untapped value of the big data generated by your technology infrastructure, security systems and applications. https://www.splunk.com/

8. Lynis - is an open source security auditing tool. It runs on the host itself, so it performs more extensive security scans than vulnerability scanners. https://cisofy.com/lynis/

9. OSSIM - AlienVault’s Open Source Security Information and Event Management (SIEM) product, provides you with a feature-rich open source SIEM complete with event collection, normalization and correlation. https://www.alienvault.com/products/ossim

Internal Audit

Analysis of Results11. HTM Studio - Find Real-Time Anomalies in your Streaming

Data. HTM Studio allows you to test whether our Hierarchical Temporal Memory (HTM) algorithms will find anomalies in your data. With just one click, you can uncover anomalies other techniques cannot find in your numeric, time-series data, in minutes. http://numenta.com/htm-studio/

Internal Audit

Analysis of Results The Center for Internet Security (CIS) is a organization dedicated to enhancing the cybersecurity readiness and response among public and private sector entities. The CIS Security Benchmarks program provides vendor-agnostic, consensus-based best practices to help organizations assess and improve their security. Resources include:• secure configuration benchmarks• automated configuration assessment tools and content• security metrics• security software product certifications

The Security Benchmarks program is an independent authority that helps both public and private industry experts collaborate and find consensus on practical cybersecurity solutions. Our resources are used by organizations worldwide to help meet compliance requirements for FISMA, PCI, HIPAA and more.

Analysis of Results (Example) Overview This document, CIS CentOS Linux 7 Benchmark,

provides prescriptive guidance for establishing a secure configuration posture for CentOS version 7.0 running on x86 and x64 platforms. To obtain the latest version of this guide, pleasevisit http://benchmarks.cisecurity.org.

Analysis of Results (Example)

Analysis of Results (Example)1.1.1 Create Separate Partition for /tmp (Scored)

Profile Applicability: Level 1 Description: The /tmp directory is a world-writable directory used for temporary storage by all users and some applications. Rationale: Since the /tmp directory is intended to be world-writable, there is a risk of resource exhaustion if it is not bound to a separate partition. In addition, making /tmp its own file system allows an administrator to set the noexec option on the mount, making /tmp useless for an attacker to install executable code. It would also prevent an attacker from establishing a hardlink to a system setuid

Analysis of Results (Example)program and wait for it to be updated. Once the program was

updated, the hardlink would be broken and the attacker would have his own copy of the program. If the program happened to have a security vulnerability, the attacker could continue to exploit the known flaw. Audit: Verify that there is a /tmp file partition in the /etc/fstab file. # grep "[[:space:]]/tmp[[:space:]]" /etc/fstab Remediation: For new installations, check the box to "Review and modify partitioning" and create a separate partition for /tmp. For systems that were previously installed, use the Logical Volume Manager (LVM) to create partitions. References: AJ Lewis, "LVM HOWTO", http://tldp.org/HOWTO/LVM-HOWTO/

Analysis of Results (Example)

Real Way for Us

Security Walls in Linux Environment

Protection of Kernel

Internal Audit

Protection of Communications

Protection of File Systems and Data

Protection of Configuration

Files

Literature1) CIS CentOS Linux 7 Benchmark 2) Kernel sysctl configuration file for Li

nux 3) SELinux User's and Administrator's

Guide 4) Multipathing5) How To Use Systemctl to Manage Sy

stemd Services and Units 6) FirewallD7) Security Harden CentOS 78) System Settings in Linux Server 9) Hacker Tools Top Ten Y201610)Defining

Persistent Audit Rules and Controls

Literature11)Bossie

Awards 2016: The best open source networking and security software

12)Host Based IDS 13)Open

Source Host-based Intrusion Detection System (OSSEC)

14)How to Install Splunk on CentOS 715)Penetration Testing Framework

Questions and Answers

Thank you!Mykola Perehinets

I&O, IS Application AdministratorSkype: mykola.perehinets

Cell: +380 67 772 6910