Information Security with a twist

Recently I was shoring up the defenses of BSD hosts in the DMZ and enabling the IPFW firewall. The goal of that effort was two-fold: 1) make certain services are only available to ip addresses or ranges that are require, say for example I need SSHd open, but I only want the trusted LAN to be able to access,and 2) prevent the BSD box from inadvertently broadcasting anything I don’t want it to, through a simple egress ruleset.

I have a straightforward stateful ruleset, I have no delusions about IPFW, but it certainly serves a purpose and I am pretty confident in the setup. One of my final explicit rules it a deny log all rule that I use to help further tune the rules and that I rely on as a cheap sniffer of sorts. By tailing the security log, occasionally or for periods at a time, I get an idea of what I have broken with my rules, what other hosts are hitting the box and for what reason, and in this case, which other hosts in the DMZ are overly chatty.

Source: wikimedia

As some can likely guess, the BSD hosts are not alone in the DMZ and while the BSD systems may be content to serve in silence, the Redmondian hosts seem to chatter nervously, perhaps overly aware of the peril they are in being bastion hosts. Looking at the firewall log, I see frequent UDP broadcast traffic, ports 137/138. It seems they are going out of their way to get noticed, this make me cringe a bit and I set out to accomplish 2 thing. First I need to see if I can silence the Windows servers to some degree, certainly they should require UDP broadcasts in order to function. Second, I think I need to do a port scan and vulnerability assessment from within the DMZ. I think the latter is important because should any bastion host be compromised, an attacker will often look into lateral movement to other DMZ hosts before making the push to get to the LAN. If your security measures are such that you are: relying on the perimeter firewall only to protect the DMZ hosts; and, believe that the DMZ itself provides any true, magical protection for your trusted network – you are likely in for a rude awakening. All too often, network admins are requested to created too many lax DMZ-to-LAN ACLs for things like database connections, easy file sharing (SMB), remote access (RDP etc.) effectively converting the DMZ into an attacker’s dream. All the more reason to harden each host in the DMZ as much as possible.

Also in larger organizations, weak change management could easily allow for a system manager to install VNC on a host in the DMZ so that he/she can readily hit a desktop without having to sign into a busy management server. Lack of egress rules on the perimeter firewall means nothing gets in the way of the LAN-to-DMZ connection, and by default DMZ-to-DMZ will also be able to utilize VNC. One quick scan from a compromised host in the DMZ shows an open port 5900 and the attacker has his second victim system with little effort.

A quick scan with NMap and my cringing returns:

PORT      STATE SERVICE       VERSION
80/tcp    open  http          redacted
135/tcp   open  msrpc         Microsoft Windows RPC
139/tcp   open  netbios-ssn
443/tcp   open  ssl/http      redacted
445/tcp   open  netbios-ssn
3389/tcp  open  ms-wbt-server Microsoft Terminal Service
5357/tcp  open  http          Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP)
8192/tcp  open  redacted        redacted
8193/tcp  open  tcpwrapped
8194/tcp  open  redacted    redacted
49152/tcp open  msrpc         Microsoft Windows RPC
49153/tcp open  msrpc         Microsoft Windows RPC
49154/tcp open  msrpc         Microsoft Windows RPC
49155/tcp open  msrpc         Microsoft Windows RPC
49156/tcp open  msrpc         Microsoft Windows RPC

In all fairness, I also scanned a *nix host and found more open ports than I expected, like Postfix listening intently on TCP/25 when it should be a null host. The lesson here is that placing hosts in the DMZ does not instantly secure them, and being overly reliant on the perimeter firewall only bolsters a false sense of security. Take the time to harden each bastion host, they need to be protected from each other as much as they do from outside attackers.

One of the WordPress-powered blogs I admin is currently under attack, not this one mind you, but another. It’s a fairly typical brute force password guessing attempt hitting the admin account. From what I am seeing, it appears it’s somewhat automated and it’s using a botnet of sorts, so that as soon as one ip gets blocked, another attempt on the account occurs from another ip address.

So I’m logging all these attempts and not really actively trying to thwart the aggressor, but they have been so persistent, going on months now, that I’ve decided to investigate a bit. I grabbed a handful of ip addresses and I was surprised that so far every host attacking my system is running some flavour of Linux. To be honest, I was expecting home users and Windows OS to be the culprit. Most of these hosts I’m finding are running ftp, ssh and web servers. If I visit the websites they seem fairly legit, I expect the site owners have no clue their systems are being coerced into a life of crime.

I have half a mind to run a vulnerability scan against the attacking hosts to see if I can figure out how they were exploited in the first place. Maybe they were running WordPress themselves, wouldn’t that be interesting.

This all doesn’t lend much credence to the overblown belief that Linux systems are somehow inherently more secure than Windows. The same thing holds true for any OS, if you aren’t locking the gate and manning the walls, someone is going to take advantage. In addition to the WordPress login activity, there is also currently an SSH brute force going on, it too is from a Linux host, but I do not think it’s necessarily related.

I recommend the WordPress Limit Logins plugin, at the very least it can be configured to give you a heads-up when something nefarious is afoot. Of course, I also recommend keeping thing patched, especially any public facing services that could potentially have known exploits.

NMAP results below are fairly typical of attacking hosts:

PORT      STATE  SERVICE    VERSION
21/tcp    open   ftp        PureFTPd
22/tcp    open   ssh        OpenSSH 4.7 (protocol 2.0)
43/tcp    closed whois
80/tcp    open   http       Apache httpd
443/tcp   open   ssl/http   Apache httpd
8021/tcp  closed ftp-proxy
8080/tcp  closed http-proxy

UPDATE: (4/13/2013)

Dumped some of the offending ip addresses from a few hosts and compiled a list, posted to PasteBin http://pastebin.com/eCjGyyBR – you could use the list for null routes, .htaccess, firewall ACL, TCPWrappers etc. Some of the addresses are serious repeat offenders, while others appear only once.

Just read a great article about a small American company (Solid Oak Software/CyberSitter) raising the ire of the Chinese, and the on-going attack that ensued. The article does a good job of giving a high level overview of the ongoing attack and provides an opportunity for analysis – what could Solid Oak have done to help mitigate the attack, what did they do right?

In Brian Milburn’s defense, hindsight is 20/20 and it’s easy to sit here and go on about what could have been done. Brian made a valiant effort to keep his network and business afloat despite being up against what may have been one of China’s most elite hacking groups. However, his reluctance to involve security professionals may have been the reason the attacks persisted for so long and were so successful in compromising his network.

Let’s start with the firewall, I’m going to guess it might have been a TZ series, but I don’t have data so it could have been a more advanced model. SonicWall (now owned by Dell) makes a decent product but even the best firewall can let you down, and a firewall configuration can make the difference between a false sense of security and real protection. It is hard to determine from the article whether the firewall died of its own accord or if the attackers managed to compromise it. Suffice it to say, every device, application or operating system out there has known vulnerabilities; they also have unknown vulnerabilities that may be known only to a select few who can exploit those systems with impunity. Sometimes this knowledge belongs to an individual, sometimes a group and sometimes to a nation-state. You can never consider a single item secure, which is why layers are so important – defense-in-depth is a very sound practice.

Even the simplest of firewalls can offer great protection, in the case of Solid Oak (and many others) I would bet there was no egress filtering. Early on it would have been prudent to alter the explicit outbound rule to log all traffic. It would make for some large logs, but would be of huge benefit to those who could analyze the data. Following that, close the hole with a deny all outbound and start creating explicit ACLs for outbound traffic. Tedious sure, but it gives a simple SPI firewall a tremendous boost when it comes to securing the perimeter. This can still be combined with logging of the denied traffic to allow you to see which internal hosts are trying to communicate to outside systems.

Another easy firewall trick is the country blocking approach – many firewalls are now capable of denying traffic to and from blocks of ip addresses that belong to certain countries. If Solid Oak does 95% if its business in North America – block the rest. Certainly, Chinese hackers can still use compromised systems in the US to attack, but when you identify the attacking hosts, contacting a US ISP to complain about abuse is likely to give better results than trying to do the same with a Chinese (state owned) ISP.

Brian eventually discovered files / applications residing on his systems that amounted to a hacker’s toolkit. This is a common hacker tactic – once you compromise a system, you copy your tools to it to allow you to start attacking other nearby systems, in addition to tools you might use for remote control, eavesdropping, info gathering etc. Had Brian employed file integrity checking on the more vulnerable or more critical systems (exposed web / mail servers) these files may have been detected early. Something like OSSEC, which also includes Host Intrusion Detection, could have been beneficial – or Tripwire, which is purely file integrity checking, would have been good and inexpensive choices. Tripwire, would also have caught the single missing apostrophe that prevented his potential customers from completing a purchase online.

A sound logging strategy may also have been of much benefit to Solid Oak during the attacks, using something like Splunk or ELSA to consolidate all the server logs to a single point would at the very least provide strong forensic data for later analysis. Coupled with a few good alert rules, these solutions could also have helped provide early warning of malicious activity.

IDS/IPS – Intrusion Detection/Prevention would be another layer that is wise to deploy. Despite the name however, there is no guarantee this technology would detect or prevent what was happening to Solid Oak. IDS is sometimes integrated with some of the newer UTM firewalls, it can also be deployed as a standalone system. There are many options in this space and some like SNORT, Suricata and BRO do not come with prohibitive price tags, although they may be difficult to deploy for the uninitiated. Yet, the Security Onion project provides an easy to deploy solution with a large number of open source security projects that gives you a solid Network Security Monitoring foundation with a few clicks.

I have intentionally left endpoint security out, it’s widely know that endpoint is the last line of defense, but also often the weakest. Anymore it seems trivial for attackers to circumvent, disable or just sneak past endpoint security – there is no magic bullet in that space unfortunately. That is not to say I recommend avoiding endpoint altogether, it has its benefits, just know it for what it is and don’t rely on it too much.

Happily, Solid Oak’s problems did resolve, although it appears their adversary gave up rather than being beaten. This story is a bit of a cautionary tale, especially when dealing with China, not only does one not simply walk into Mordor, but one also does not simply raise the ire of the dragon. China’s reputation for hacking and industrial espionage is well-earned and even if they aren’t on your radar, you might be on theirs. If this tells us anything, it’s that you are never too small to be a target for malicious attackers. In this day and age, if you are doing business online, lock down and layer up, you could be in for a wild ride.

CVE-2012-2122 : Serious Mysql Authentication Bypass Vulnerability
A serious security bug in MariaDB and MySQL Disclosed, According to Advisory All MariaDB and MySQL versions up to 5.1.61, 5.2.11, 5.3.5, 5.5.22 are vulnerable.

The auth bypass is insanely trivial – basically try to gain root access enough times and you will just get in. However this exploit is limited in scope affecting only specific platforms. So far in all my testing on FreeBSD, I have not gained access using the python script found here.

I have hit MySQL instances from version 5.0.55 to 5.1.61 and some 5.5 instances on both Intel and AMD platforms, on both 32 and 64-bit OSes.

At this point I’m pretty comfortable saying it appears MySQL on FreeBSD is not affected by this particular bug – but I would patch anyway when it’s available.

Flamer (aka SkyWiper) is getting much attention right now. The recently discovered toolkit (I refuse to even call it malware) has infosec types and security vendors all excited and the mainstream media has jumped in with both feet. The ensuing explosion of coverage, both electronic and conventional is grossly disproportionate to the scope and danger Flamer represents to the public. Simply stated, 99.99% (or greater) of the population needs not be concerned with this toolkit – unless you are enhancing uranium in your basement and selling it to Iran.

Flamer is a lot of things, and it certainly is an interesting study for AV and infosec professionals, like Stuxnet it represents a new breed of offensive APT tool. It’s huge by comparison with most malware and extremely complex, it is also extremely focused to a singular task. This is not a banking trojan, or some run-of-the-mill botnet, it may fit the broad definition of malware, but I would not call it that. This is an example of things to come – weaponized code designed to penetrate and disrupt the digital infrastructure of an enemy while evading detection. While Flamer is capable of basic espionage, it has teeth and is more than just a passive intelligence gathering app.

I digress, much smarter people than I have already analyzed this thing to death, the point I want to make is that this code is not targeting the general population. Even if it has spread beyond the borders of the target countries, that could very well be intentional to muddy the waters to make it more difficult to discern where it started and who is behind it.

The hysteria is unwarranted and I would not lose any sleep over this, I will however, disconnect my gas centrifuges from the home wi-fi later. (Damn, they were so easy to manage from the XBox 360 too!)