Art of Information Security

Time is critical in security systems; specifically, having systems know the time  is very important. Adequate clock synchronization is important for:

• Operational Integrity (things happen when they are supposed to happen – backups, tasks, etc.)
• Reproducibility of events (meaningful logs and records)
• Validation of SSL certificate expiration (or other tokens, etc.)
• Correct application of time restricted controls
• Etc.

So, the big question is, what is “adequate clock synchronization”, and how do we achieve it ?

But First, What Time Is It ?

Time itself is of course a natural phenomenon. Just like distance, volume, and weight, the measurements for time are artificial and man-made.  The dominant time standard (especially from a computer and therefore Information Security perspective) is Coordinated Universal Time (UTC). This could probably have been called Universal Compromise Time, as it turns out that getting the whole world to drop their cultural biases, deployed technology, etc. and move to a single time system has been a long and complicated road (and it isn’t over yet).

One major component of UTC is an agreement on what time it in fact is, and how that is determined. Also, there are  questions surrounding how to adjust leap seconds, leap years,  and other “measurement vs reality” anomalies.  Time (and its measurement) is quite complex in itself, but for the purposes of Information Security (system operation, log correlation, certificate expiration, etc.), the good news is that UTC provides a solid time standard.

Now, all we need to do is synchronize our clocks to UTC !
(and adjust for our local time zone…)

Network Time Protocol (NTP)

Network Time Protocol (NTP) is a well established, but often misconfigured and misunderstood, internet protocol. NTP utilizes Marzullo’s Algorithm to synchronize clocks in spite of the fact that:

• The travel time for information passed between systems via a network is constantly changing
• Remote clocks themselves may contain some error (noise) vs UTC
• Remote clocks may themselves be using NTP to determine the time

In spite of this, a properly configured NTP client can synchronize its clock to within 10 milliseconds (1/100 s) of UTC over the public internet. Servers on the same LAN can synchronize much more closely . For Information Security purposes, clock synchronization among systems and to UTC, within 1/5 or 1/10 of a second, should be sufficient.

Classic Misconfiguration Mistakes (and how to avoid them)

The misconfiguration mistakes that folks make tend to be the result of:

• Overestimating the importance of Stratum 1 servers
• Over-thinking the NTP configuration

NTP Servers are divided into Stratums based on what time source. A Stratum 1 server is directly connected to a device that provides a time reference. Some examples of reference time sources include:

• Atomic Clocks
• GPS
• CDMA
• WWVB, DCF77, MSF60

NTP servers which synchronize with a Stratum 1 time source are Stratum 2 servers, with the Stratum number increasing by one for each level.

Big Mistake – Using a Well Known NTP Reference

The most frequent mistake people make when configuring NTP on a server is assuming that they need (or will get the best time synchronization) by using one of the well known atomic clock sources. This tends (thought not always) to be a bad idea because it overloads a small number of servers. Also, a server with a simpler network access path will generally provide better synchronization than a more remote one.

When configuring the NTP protocol, it is a good idea to specify several servers. The general rule of thumb is 2-4 NTP servers. If everyone specifies the same servers, then those servers become overloaded and their response times become erratic (which doesn’t help things). In some cases, an unintended denial of service attack is caused.

Both Trinity College of Dublin, Ireland and the University of Wisconsin at Madison experienced unintended denial of service attacks caused by misconfigured product deployments. In the case of the University of Wisconsin at Madison, NETGEAR shipped over 700,000 routers which were set-up to all pull time references from the university’s servers. NETGEAR is not the only router or product manufacturer to have made such an error.

Enter the NTP Pool…

“The pool.ntp.org project is a big virtual cluster of timeservers striving to provide reliable easy to use NTP service for millions of clients without putting a strain on the big popular timeservers.” quoted from pool.ntp.org

Basically, the NTP pool is a set of over 1500 time servers, all of which are volunteering to participate in a large load-balanced virtual time service. The quality and availability of the time service provided by each of the NTP servers in the pool is monitored, and servers are removed if they fail to meet certain guidelines.

Unless a system itself is going to be an NTP server, then use of the NTP Pool is your best bet 100% of the time. It is a good idea to use the sub-pool that is associated with your region on the globe. Here is ta sample configuration: (/etc/ntp.conf file)

server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org

It may not be necessary for your to run the NTP service itself. Running the ntpdate command at boot and then in a cron job once or twice a day may be sufficient. The command would look like:

ntpdate 0.us.pool.ntp.org 1.us.pool.ntp.org 2.us.pool.ntp.org 3.us.pool.ntp.org

If you do need to install ntp on Ubuntu, the commands are:

sudo apt-get install ntp

and then edit the /etc/ntp.conf file and add the server lines from above. On my OSX workstation, the entire /etc/ntp.conf file is:

driftfile /var/ntp/ntp.drift

server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org

Overthinking the Configuration

The “server” parameter in the configuration file has a number of additional directives that can be specified. These are almost never needed, but can generate a lot of extra traffic on the NTP server. Avoid over thinking the server configurations and avoid using prefer, iburst, or burst.

When Should I Run NTP Service Rather Than Use The NTPDate Command ?

There is almost no downside to running the NTP service. It is very low overhead and generates almost no network traffic. That being said, the only downside to running the ntpdate command a few times a day, is that the clock can drift more. If I were performing an audit, and the shop-practice was to use ntpdate on everything except infrastructure service machines (directory servers, syslog concentrators, etc.), I would accept that practice. I would be more concerned about how time synchronization was being managed on HSMs, directory services, NIDS, firewalls, etc.

When Should I Run My Own NTP Server ?

There are two cases when you should consider running your own server:

• You have a large number of machines that need time services
• You wish to participate in NTP Pool

A Stratum 1 time server appliance or a GPS/CDMA card can be purchased for costs similar to a rack mounted server (of course you will need two). If that is just out of the (budgetary) question, then I would look for the time servers to use authenticated time sources. NIST and several other Stratum 1 NTP providers have servers which are only available to folks who have requested access, and are authenticating to the server. If time accuracy is critical to risk management, and GPS/CDMA is not available, then I would push for authenticated NTP.

Option 3 is acceptable in the vast majority of situations, including cases where logs and events are only correlated locally, or where no compelling need exists.

NTP and Network Security

NTP uses UDP on port 123. This traffic should be restricted in DMZ or other secure network zones to only route to authorized NTP servers. Tools like hping can be used to turn any open port into a file transfer gateway or tunnel.

One option is to set-up a transparent proxy on your firewalls and to direct all 123/UDP traffic to your NTP server or to one you trust. (The risk of the open port involves providing a data path out of the organization, not rogue clocks…)

Resources and More Information

• Wikipedia on NTP
• A Brief History of NTP Time: Confessions of an Internet Timekeeper
• Marzullo’s Algorithm

Cheers,

Erik

The Art of Information Security continues our interview with Michael Rash, Network Security expert and the driving force behind several open source security tools including PSAD, FWSnort, and FWKnop.

In Part 2 of the interview Michael discussed how network threats, and network counter measures have been evolving. He also touched on the development of his book. Here goes the final installment in this series…

Erik: What would be your recommendations for folks who are adopting Linux (either enthusiasts or corporations) in terms of properly protecting their hosts and networks from network attacks?

Michael: I think that deploying host and network firewalls is a great first step here, and iptables functions admirably. Many people in corporate environments are concerned about the questions of performance, manageability, scalability, and support, and iptables together with some third party software have decent answers to these concerns. For example, the fwbuilder project provides good graphical support for the display and manipulation of iptables policies, and large Linux distributions such as Red Hat and SuSE offer commercial support.

Beyond having proper firewalls deployed, intrusion detection systems are a critical piece to point the way to attempted (and sometimes successful) compromises. Also, strong security mechanisms such as SELinux can provide a powerful barrier to attempted malicious usages of hosts. Finally, patch early and patch often.

Erik:  Do you have any tool or reference recommendations for debugging IP tables firewalls?

Michael: For debugging iptables policies and maintaining tight controls on the type of packets that are allowed to traverse those policies, one of the best techniques is to use tcpdump either on the end points or on the firewall itself (and these may be the same system) and watch how network traffic is allowed to progress. For example, a SYN packet to a port that is filtered will not respond either with a SYN/ACK or a RST, and seeing this behavior with tcpdump is quite easy. At the same time, understanding where in an iptables policy packets are getting dropped (or otherwise messed with) is usually made clear by watching how packet and byte counters are incremented on particular iptables rules. Use ‘iptables -v -n -L’ for this, and couple this with the ‘watch’ command to see how things change. Beyond this, if you have a kernel compiled with support for the iptables TRACE target, then you can use an iptables TRACE rule that causes all packets hitting this rule to be logged. Lastly, for really advanced debugging of iptables code itself, the nfsim project provides a simulator for running Netfilter code within userspace (and hence the ability to test code before running it within the kernel itself where a bug can have dire consequences). The nfsim project can be found here:

http://ozlabs.org/~jk/projects/nfsim/

Erik: So, you obviously are deeply connected to all things Network IDS/IPS. What kinds of trends have you seen in 2008? Were there any new attack styles that surprised you? Do you have any ideas about what 2009 may hold?

Michael: Well, 2008 will certainly go down in history as the year that people were forced to really pay attention to DNS by the Kaminsky attack. One thing Dan did really well is make it clear just how important DNS is for literally everything on the Internet, and how a flaw there has implications that are difficult to over estimate. Online banking, acquiring SSL certificates, SMTP, “forgot my password links”, and countless other infrastructures depend on DNS information being correct. But, then there were also serious issues in 2008 with BGP and with SSL, so if there was any trend in 2008 I would say that it was the year of security flaws in big Internet infrastructures. In 2009, it will be interesting to see whether this trend remains true for as-yet undiscovered vulnerabilities in other important systems.

Erik: Has your support for open source helped you professionally?

Michael: Absolutely. My current position as a Security Architect on the Dragon IDS/IPS developed by Enterasys Networks is a role that my open source work helped me to acquire. Many forward looking innovations are created by the open source community, and understanding this community helps to guide many companies and the products they develop. Companies are recognizing the power of open source software more and more, and this translates to better professional positions for open source developers and technology enthusiasts.

Many Thanks to Michael !

Thanks a ton for the time and energy you put into this, the first of what I hope will be many, interviews with notables from around the Information Security community.

Thanks, Erik

Quick update to Part 2 of the AoIS Secure Your Linux Host Series on SSH.

I noticed a great article today on  Xavier Mertens‘ /dev/random blog (which by the way has several great posts that have caught my eye…), on SSH tunneling -> “Keep an Eye on SSH Forwarding“.

In addition to providing a solid introudction to SSH Port Forwarding Xavier also discusses:

• Using SSH as a SOCKS Proxy via the SSH Server
• Logging port forwarding
• Restricting  ports that can be forwarded

Check it out.

Cheers, Erik

The Art of Information Security continues our interview with Michael Rash, Network Security expert and the driving force behind several open source security tools including PSAD, FWSnort, and FWKnop.

In Part 1 of the interview Michael discussed how he came to be involved in Network Security and Intrusion Detection system design. Here in Part 2 we get a little deeper into Michael’s philosophy on Network Intrusion Protection and discuss more open source tools that he is involved with the develop and support of.

Erik: How do you see network based attacks changing ?

 Michael: Over time, I think network based attacks will continue to be more automated and therefore accessible and deployable by more people. When it comes to educating oneself on the details of network insecurity, excellent projects such as Metasploit, Nessus, and Nmap point the way – and this is essential also for people trying to defend networks too. We will see more attacks delivered over IPv6, and we will see ever more clever ways to exploit the natural tendency of people to trust data in ways they shouldn’t. For me as a person trying to protect networks, the later is the most worrisome. A good example of a new and clever attack is “in-session phishing” as described here (Arstechnica link).

Erik: The firewalls that I run are utilized as host based protection. As you see network security becoming increasingly important, do you see the firewall “concept” become a hybrid of network protection layered over host based network controls?

Michael: With good firewall implementations (such as iptables) that do not place undue burdens on network processing that takes place on hosts, I do believe that firewalls will be viewed more and more as an essential protection mechanism for the host. The network perimeter will also continue to be an important deployment point for large firewalls to enforce global policy, but limiting the damage a successful exploit against an internal system is a problem that such an external firewall is not well-suited to address. Having a hardened network security stance on each host can provide an important benefit in this area. Further, as firewalls offer more application layer processing features, hosts can deploy customized policies that define sets of application layer data (derived from Snort rules) that are unfit for communicating with local sockets.

There are challenges though regarding managing all of those host-level firewall policies, and this is where some patience and scripting ability can play a roll.

Erik: And then came FWSnort? What were the principles that drove the development of FWSnort ?

Michael: The fwsnort project was inspired originally by the snort2iptables script written by William Stearns. This was back in the Linux 2.4 days when the string match extension was still distributed within the patch-o-matic system from the Netfilter project. Being interested in intrusion detection and firewalls at the same time, it was a goal of mine to see how far iptables could be taken in the direction of detecting (and blocking) malicious traffic. The snort IDS had a well-developed signature language, and at that time the signatures were still free and released under the GPL. So, it was natural to try and extend the snort2iptables code, and fwsnort was created.

The main goal of fwsnort is to use facilities provided by iptables to recast Snort signature sets within iptables policies. A clean translation is not always possible particularly with complex Snort signatures that use regular expression matching (because no regex engine is available to the iptables code running in the kernel), but many Snort signatures can faithfully be translated.

 Erik: Was your vision that PSAD and fwsnort teamed up as host IDS dynamic duo, or more as services that strengthen network firewalls?

Michael: Ideally I would say both here. The difference between the two types of deployments is negligible from psad and fwsnort’s perspectives – both can be deployed just as effectively against the iptables INPUT chain (for packets directed at the local system) as the FORWARD chain (for packets directed through a network firewall). The effect of not deploying host firewalls is that the outside of the network may be protected by a crunchy shell, but the inside is a chewy center. If any system can be compromised internally on such a network, an attacker is presented with few barriers to additional actions once the perimeter is breached.

 Erik: But wait – there’s more ! You are also the driving force behind FWKnop !

Michael: Thanks for mentioning fwknop. This project has received a large percentage of my attention in the last year or so. It was started originally in 2004 as the first port knocking system that added passive OS fingerprinting as an authentication parameter, but in 2005 Single Packet Authorization was added. SPA solves many of the protocol limitations that are built into port knocking (ease of replay attacks, lack of decent data transmission, and difficulty of scaling to many users), and takes the idea of “default-drop” to a new level. That is, a service such as SSH is itself made completely inaccessible before the lightweight SPA packet is passively sniffed and the firewall is reconfigured to allow access only if the SPA packet is valid. This essentially combines techniques from the IDS world (passive packet sniffing) with techniques from the authentication and authorization world (encryption and the like).

Erik: And how did the book come to be ?

Michael: I have generally tried to capture my thoughts on computer security by writing them down. In 2001 I started writing articles, and wrote a few for the Linux Journal after working with Jay Beale on the Bastille Linux project. From there, I joined Jay with writing material for Snort books for Syngress. My open source development interest has always remained in IDS and firewall technologies, so I eventually decided to write a book about the two together. The result was the No Starch book. Let me just mention here that if any of your readers is interested in writing a book, I can wholeheartedly recommend No Starch as an absolutely fantastic publisher to work with.

Stay Tuned for Part 3

Part 3 of this series is coming soon, with more discussion about network security as well as the impact that contributing to open source tools has had on Michael professional opportunities.

Cheers, Erik

Two Quick updates to Part 2 of the AoIS Secure Your Linux Host Series on SSH.

Interesting Series by ISS X-Force on SSH

Just this morning I ran across a three part series on SSH published last year in IBM’s Internet Security Systems X-Force Threat Insight in the following issues:

• X-Force Threat Insight Monthly – April 2008
• X-Force Threat Insight Monthly – May 2008
• X-Force Threat Insight Monthly – June 2008

X-Force expresses a slightly different set of concerns, and solutions. One topic that I did not touch on was the use of ssh agents for the management of sessions. Part 3 (June) is almost entirely focused on that.

Logwatch Samples

One of the great things about the script kiddies is they are keep testing your security for you ! Below is a mash-up and edit-down of the last few days of ssh related itms from my logwatch logs. Logwatch really has become one of my favorite tools. I don’t have tons of attacks on my servers, but there is always enough activity in the logs to let me know that the controls and countermeasures are up and running. After installing fail2ban, I always have some activity in 24 hour period of time. 

And a tip for the paranoid – if you have Failed logins and Illegal users but no fail2ban activity – then fail2ban has stopped running (or worse…).

——————— fail2ban-messages Begin ————————
Banned services with Fail2Ban:
Bans:Unbans  
ssh: [ 6:6 ]  
ssh: [ 4:7 ]  
ssh: [ 6:5 ]
ssh: [ 5:3 ]
———————- fail2ban-messages End ————————-

——————— SSHD Begin ————————
Failed logins from:
75.xxx.109.82 (75-xxx-109-82-Indianapolis.hfc.comcastbusiness.net): 1 time
79.xxx.248.27 (host27-xxx-static.38-79-b.business.telecomitalia.it): 1 time
200.xxx.209.156 (dedint-200-xx-209-156.mexdf.axtel.net): 3 times
59.xxx.92.26: 6 times
88.xxx.16.23 (…): 7 times
119.xxx.154.57: 6 times
203.xxx.198.3 (…): 6 times

Illegal users from:
60.xxx.249.90 (…): 3 times
75.xxx.109.82 (…): 3 times
79.xxx.248.27 (…): 3 times
200.xxx.209.156 (…): 2 times
202.xxx.28.244 (…): 3 times
85.xxx.133.177: 4 times
193.xxx.161.136: 4 times
———————- SSHD End ————————-

Cheers, Erik