Chatty Cathys

Recently I was shoring up the defenses of BSD hosts in the DMZ and enabling the IPFW firewall. The goal of that effort was two-fold: 1) make certain services are only available to ip addresses or ranges that are require, say for example I need SSHd open, but I only want the trusted LAN to be able to access,and 2) prevent the BSD box from inadvertently broadcasting anything I don’t want it to, through a simple egress ruleset.

I have a straightforward stateful ruleset, I have no delusions about IPFW, but it certainly serves a purpose and I am pretty confident in the setup. One of my final explicit rules it a deny log all rule that I use to help further tune the rules and that I rely on as a cheap sniffer of sorts. By tailing the security log, occasionally or for periods at a time, I get an idea of what I have broken with my rules, what other hosts are hitting the box and for what reason, and in this case, which other hosts in the DMZ are overly chatty.

server racks

Source: wikimedia

As some can likely guess, the BSD hosts are not alone in the DMZ and while the BSD systems may be content to serve in silence, the Redmondian hosts seem to chatter nervously, perhaps overly aware of the peril they are in being bastion hosts. Looking at the firewall log, I see frequent UDP broadcast traffic, ports 137/138. It seems they are going out of their way to get noticed, this make me cringe a bit and I set out to accomplish 2 thing. First I need to see if I can silence the Windows servers to some degree, certainly they should require UDP broadcasts in order to function. Second, I think I need to do a port scan and vulnerability assessment from within the DMZ. I think the latter is important because should any bastion host be compromised, an attacker will often look into lateral movement to other DMZ hosts before making the push to get to the LAN. If your security measures are such that you are: relying on the perimeter firewall only to protect the DMZ hosts; and, believe that the DMZ itself provides any true, magical protection for your trusted network – you are likely in for a rude awakening. All too often, network admins are requested to created too many lax DMZ-to-LAN ACLs for things like database connections, easy file sharing (SMB), remote access (RDP etc.) effectively converting the DMZ into an attacker’s dream. All the more reason to harden each host in the DMZ as much as possible.

Also in larger organizations, weak change management could easily allow for a system manager to install VNC on a host in the DMZ so that he/she can readily hit a desktop without having to sign into a busy management server. Lack of egress rules on the perimeter firewall means nothing gets in the way of the LAN-to-DMZ connection, and by default DMZ-to-DMZ will also be able to utilize VNC. One quick scan from a compromised host in the DMZ shows an open port 5900 and the attacker has his second victim system with little effort.

A quick scan with NMap and my cringing returns:

80/tcp    open  http          redacted
135/tcp   open  msrpc         Microsoft Windows RPC
139/tcp   open  netbios-ssn
443/tcp   open  ssl/http      redacted
445/tcp   open  netbios-ssn
3389/tcp  open  ms-wbt-server Microsoft Terminal Service
5357/tcp  open  http          Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP)
8192/tcp  open  redacted        redacted
8193/tcp  open  tcpwrapped
8194/tcp  open  redacted    redacted
49152/tcp open  msrpc         Microsoft Windows RPC
49153/tcp open  msrpc         Microsoft Windows RPC
49154/tcp open  msrpc         Microsoft Windows RPC
49155/tcp open  msrpc         Microsoft Windows RPC
49156/tcp open  msrpc         Microsoft Windows RPC

In all fairness, I also scanned a *nix host and found more open ports than I expected, like Postfix listening intently on TCP/25 when it should be a null host. The lesson here is that placing hosts in the DMZ does not instantly secure them, and being overly reliant on the perimeter firewall only bolsters a false sense of security. Take the time to harden each bastion host, they need to be protected from each other as much as they do from outside attackers.

SSH Tunneling – Security on the Road

I’m on vacation, taking a break with the family. Of course, I was up before 6AM and on my laptop shortly thereafter checking emails and servers and doing all the usual stuff I do when not on vacation. It’s the curse of the North American work ethic. I can’t help it, if I have connectivity, I’m going to use it.

There is free wi-fi where I’m staying – it’s wide-open, which means network traffic is overly susceptible to eavesdropping and interception. To combat this I tunnel my web traffic over SSH, it’s a beautiful solution both in simplicity and effectiveness. Ideally though, you require a trusted host to make the connection to, in my case I have a server in my residence at home. The security of my connection is further enhanced because I use SSH keys rather than username / password and an alternate port. The command is straight-forward:

Ricks-MacBook-Pro:~ rickc$ ssh -p 11111 my.home.server -L localhost:8080:localhost:3128

The above command connects to my home server on port 11111 (not the actual port #) and then creates a local listening port 8080 and then tunnels all that traffic to port 3128 on the home server. As some of you will have figured out, I’m tunneling http to a squid proxy. On my local system, I just configure Firefox proxy settings to use 8080 and then all my browsing activity whether it is http or https, is now tunneled over the SSH connection and then goes out my home Internet connection.

@kairoer mentioned on Twitter that a one-click solution for the everyday user would be great, and I agree. Having a secure tunnel to virtually guarantee safe browsing from wi-fi hotspots and hotels etc. while you travel would be great. Unfortunately, it is a bit of an advanced technique that also requires one to have a trusted SSH server somewhere to tunnel to. This creates the requirement for a trusted SSHd service to be run by whomever creates the one-click tunnel software. The developer could easily leverage OpenSSH and wrap a nice interface around it, but again, a trusted back-end is required for the process to be complete.


Dirty Penguins!

One of the WordPress-powered blogs I admin is currently under attack, not this one mind you, but another. It’s a fairly typical brute force password guessing attempt hitting the admin account. From what I am seeing, it appears it’s somewhat automated and it’s using a botnet of sorts, so that as soon as one ip gets blocked, another attempt on the account occurs from another ip address.

So I’m logging all these attempts and not really actively trying to thwart the aggressor, but they have been so persistent, going on months now, that I’ve decided to investigate a bit. I grabbed a handful of ip addresses and I was surprised that so far every host attacking my system is running some flavour of Linux. To be honest, I was expecting home users and Windows OS to be the culprit. Most of these hosts I’m finding are running ftp, ssh and web servers. If I visit the websites they seem fairly legit, I expect the site owners have no clue their systems are being coerced into a life of crime.

I have half a mind to run a vulnerability scan against the attacking hosts to see if I can figure out how they were exploited in the first place. Maybe they were running WordPress themselves, wouldn’t that be interesting.

This all doesn’t lend much credence to the overblown belief that Linux systems are somehow inherently more secure than Windows. The same thing holds true for any OS, if you aren’t locking the gate and manning the walls, someone is going to take advantage. In addition to the WordPress login activity, there is also currently an SSH brute force going on, it too is from a Linux host, but I do not think it’s necessarily related.

I recommend the WordPress Limit Logins plugin, at the very least it can be configured to give you a heads-up when something nefarious is afoot. Of course, I also recommend keeping thing patched, especially any public facing services that could potentially have known exploits.

NMAP results below are fairly typical of attacking hosts:

21/tcp    open   ftp        PureFTPd
22/tcp    open   ssh        OpenSSH 4.7 (protocol 2.0)
43/tcp    closed whois
80/tcp    open   http       Apache httpd
443/tcp   open   ssl/http   Apache httpd
8021/tcp  closed ftp-proxy
8080/tcp  closed http-proxy

UPDATE: (4/13/2013)

Dumped some of the offending ip addresses from a few hosts and compiled a list, posted to PasteBin – you could use the list for null routes, .htaccess, firewall ACL, TCPWrappers etc. Some of the addresses are serious repeat offenders, while others appear only once.

Java Zero Day

boomThe latest Java 0day exploit is causing a bit of a stir and getting some mainstream media attention. This is what you need to know and what you need to do.

  • it affects Java 1.7, older java not affect by this particular exploit
  • since it is Java, it can be cross-platform, there is some evidence this exploit also affects Mac OS X / Safari
  • java is rarely required by the everyday user on the Internet today
  • java can be unplug from the browser and sometimes even removed completely from the system

How to render Java safe on your system:

More info:

When China Attacks

dragonJust read a great article about a small American company (Solid Oak Software/CyberSitter) raising the ire of the Chinese, and the on-going attack that ensued. The article does a good job of giving a high level overview of the ongoing attack and provides an opportunity for analysis – what could Solid Oak have done to help mitigate the attack, what did they do right?

In Brian Milburn’s defense, hindsight is 20/20 and it’s easy to sit here and go on about what could have been done. Brian made a valiant effort to keep his network and business afloat despite being up against what may have been one of China’s most elite hacking groups. However, his reluctance to involve security professionals may have been the reason the attacks persisted for so long and were so successful in compromising his network.

Let’s start with the firewall, I’m going to guess it might have been a TZ series, but I don’t have data so it could have been a more advanced model. SonicWall (now owned by Dell) makes a decent product but even the best firewall can let you down, and a firewall configuration can make the difference between a false sense of security and real protection. It is hard to determine from the article whether the firewall died of its own accord or if the attackers managed to compromise it. Suffice it to say, every device, application or operating system out there has known vulnerabilities; they also have unknown vulnerabilities that may be known only to a select few who can exploit those systems with impunity. Sometimes this knowledge belongs to an individual, sometimes a group and sometimes to a nation-state. You can never consider a single item secure, which is why layers are so important – defense-in-depth is a very sound practice.

Even the simplest of firewalls can offer great protection, in the case of Solid Oak (and many others) I would bet there was no egress filtering. Early on it would have been prudent to alter the explicit outbound rule to log all traffic. It would make for some large logs, but would be of huge benefit to those who could analyze the data. Following that, close the hole with a deny all outbound and start creating explicit ACLs for outbound traffic. Tedious sure, but it gives a simple SPI firewall a tremendous boost when it comes to securing the perimeter. This can still be combined with logging of the denied traffic to allow you to see which internal hosts are trying to communicate to outside systems.

Another easy firewall trick is the country blocking approach – many firewalls are now capable of denying traffic to and from blocks of ip addresses that belong to certain countries. If Solid Oak does 95% if its business in North America – block the rest. Certainly, Chinese hackers can still use compromised systems in the US to attack, but when you identify the attacking hosts, contacting a US ISP to complain about abuse is likely to give better results than trying to do the same with a Chinese (state owned) ISP.

Brian eventually discovered files / applications residing on his systems that amounted to a hacker’s toolkit. This is a common hacker tactic – once you compromise a system, you copy your tools to it to allow you to start attacking other nearby systems, in addition to tools you might use for remote control, eavesdropping, info gathering etc. Had Brian employed file integrity checking on the more vulnerable or more critical systems (exposed web / mail servers) these files may have been detected early. Something like OSSEC, which also includes Host Intrusion Detection, could have been beneficial – or Tripwire, which is purely file integrity checking, would have been good and inexpensive choices. Tripwire, would also have caught the single missing apostrophe that prevented his potential customers from completing a purchase online.

A sound logging strategy may also have been of much benefit to Solid Oak during the attacks, using something like Splunk or ELSA to consolidate all the server logs to a single point would at the very least provide strong forensic data for later analysis. Coupled with a few good alert rules, these solutions could also have helped provide early warning of malicious activity.

IDS/IPS – Intrusion Detection/Prevention would be another layer that is wise to deploy. Despite the name however, there is no guarantee this technology would detect or prevent what was happening to Solid Oak. IDS is sometimes integrated with some of the newer UTM firewalls, it can also be deployed as a standalone system. There are many options in this space and some like SNORT, Suricata and BRO do not come with prohibitive price tags, although they may be difficult to deploy for the uninitiated. Yet, the Security Onion project provides an easy to deploy solution with a large number of open source security projects that gives you a solid Network Security Monitoring foundation with a few clicks.

I have intentionally left endpoint security out, it’s widely know that endpoint is the last line of defense, but also often the weakest. Anymore it seems trivial for attackers to circumvent, disable or just sneak past endpoint security – there is no magic bullet in that space unfortunately. That is not to say I recommend avoiding endpoint altogether, it has its benefits, just know it for what it is and don’t rely on it too much.

Happily, Solid Oak’s problems did resolve, although it appears their adversary gave up rather than being beaten. This story is a bit of a cautionary tale, especially when dealing with China, not only does one not simply walk into Mordor, but one also does not simply raise the ire of the dragon. China’s reputation for hacking and industrial espionage is well-earned and even if they aren’t on your radar, you might be on theirs. If this tells us anything, it’s that you are never too small to be a target for malicious attackers. In this day and age, if you are doing business online, lock down and layer up, you could be in for a wild ride.


Increase Your Sodium Intake

When I was much younger I remember my Dad salting things, he would sprinkle a bit of salt on his apple as he ate it, as well as raw carrots and quite often his beer. What I did not realize at the time was that my Dad was onto something that many database architects should pay heed to – salt. Breach after breach, headline after headline we hear about username and passwords being dumped from compromised systems. A thousand facepalms later it appears we still have not learned some basic password security – i) hash the password ii) SALT the damn hash. I mean even MD5 provides some protection if salted, especially for us that might use 8+ character passwords.

It’s one thing if it’s the local minor hockey association website that gets hacked and poor password security is found to be evident, but when it’s Yahoo!, LinkedIn or another major brand – it’s just ridiculous. I’m not sure when folks are finally going to catch on and take client PII seriously, but I hope it’s soon.

Personally, I put my faith in my abnormally long passwords and hope that when a database containing my information is attacked at least I won’t be one of the victims whose hash is cracked and posted to the web in the first 15 minutes!

MySQL Auth Bypass and FreeBSD

CVE-2012-2122 : Serious Mysql Authentication Bypass Vulnerability
A serious security bug in MariaDB and MySQL Disclosed, According to Advisory All MariaDB and MySQL versions up to 5.1.61, 5.2.11, 5.3.5, 5.5.22 are vulnerable.

The auth bypass is insanely trivial – basically try to gain root access enough times and you will just get in. However this exploit is limited in scope affecting only specific platforms. So far in all my testing on FreeBSD, I have not gained access using the python script found here.

I have hit MySQL instances from version 5.0.55 to 5.1.61 and some 5.5 instances on both Intel and AMD platforms, on both 32 and 64-bit OSes.

At this point I’m pretty comfortable saying it appears MySQL on FreeBSD is not affected by this particular bug – but I would patch anyway when it’s available.

Attack of the Bingbot

I have a new special friend, Microsoft’s Bingbot which has taken quite an unhealthy interest in one of my websites. In particular, Bingbot just can’t get enough of the RecentChanges Mediawiki URL.

http access log entry: – – [05/Jun/2012:15:33:53 -0400] “GET /wiki/index.php?title=Special:RecentChanges&hidebots=0&days=30&limit=250&from=20120517204216 HTTP/1.1” 200 4254 “-” “Mozilla/5.0 (compatible; bingbot/2.0; +”

Just how much of a digital stalker is Bingbot? Well, lets look:

At the time of writing this Bingbot has hit an impressive 182119 times since June 1. For comparison, in the same time frame GoogleBot only stopped by to say hi 3188 times.

To add to the fun – Bingbot is coming in from 45 unique ip addresses:

Remember – this is all for a single URI – seems like overkill, seems excessive, seems fundamentally broken… I guess it seems to be typically Microsoft.

I pulled the whois record for a few of those IPs, it lists as an admin contact – of course they don’t really expect you to contact them, so emailing that address gets you a 504 error and a rejected email. Next try is – have not heard back, but at least it did not bounce.

Anyone else ever see anything like this?

UPDATE 6/7/12

Now at 261067 hits and climbing…

UPDATE 2 6/8/12

264360, looks like it’s slowing down. Bingbot guy tells me they have fixed it.

Great Balls of Fire (Flamer low-down)

Flamer (aka SkyWiper) is getting much attention right now. The recently discovered toolkit (I refuse to even call it malware) has infosec types and security vendors all excited and the mainstream media has jumped in with both feet. The ensuing explosion of coverage, both electronic and conventional is grossly disproportionate to the scope and danger Flamer represents to the public. Simply stated, 99.99% (or greater) of the population needs not be concerned with this toolkit – unless you are enhancing uranium in your basement and selling it to Iran.

Flamer is a lot of things, and it certainly is an interesting study for AV and infosec professionals, like Stuxnet it represents a new breed of offensive APT tool. It’s huge by comparison with most malware and extremely complex, it is also extremely focused to a singular task. This is not a banking trojan, or some run-of-the-mill botnet, it may fit the broad definition of malware, but I would not call it that. This is an example of things to come – weaponized code designed to penetrate and disrupt the digital infrastructure of an enemy while evading detection. While Flamer is capable of basic espionage, it has teeth and is more than just a passive intelligence gathering app.

I digress, much smarter people than I have already analyzed this thing to death, the point I want to make is that this code is not targeting the general population. Even if it has spread beyond the borders of the target countries, that could very well be intentional to muddy the waters to make it more difficult to discern where it started and who is behind it.

The hysteria is unwarranted and I would not lose any sleep over this, I will however, disconnect my gas centrifuges from the home wi-fi later. (Damn, they were so easy to manage from the XBox 360 too!)

Is Antivirus Really Any Good?

There is a pretty strong debate presently underway regarding the usefulness and effectiveness of today’s antivirus products against the modern threat landscape. It’s definitely not cut-and-dry, nor black and white and the best answer to the topic requires one to be as vague and nebulous as possible. Let me sum up my professional opinion thusly: maybe – maybe not.

There. It’s in writing for all to see, as concrete and accurate an answer you will be able to find anywhere in the blogosphere / twitterverse.

Allow me to qualify my stance. Personally, I currently have 8 or so systems at home that I manage – Windows, Mac, Linux and BSD. The Linux and BSD run around naked all day, all night – they have no shame, but what can I do, they are giddy children and wont for such things. The Macs and Windows systems are currently covered by Sophos, just because I use it at work. Over the years my Windows systems have run a plethora of AV products, some I’ve liked more than others, but in the end they really have had little to do. I don’t really get viruses – I cannot remember the last “OMG-my-system-is-so-pwn3d-by-malware” moment I’ve had. I mean, I think I had more virus incidents via floppy disk than Internet, if that tells you anything.

One thing I can say, in my experience, once a user, friend or family member gets malware on their computer (and invariably dials my number in a knee-jerk reaction), it does not matter what AV they run, the removal is never automatic, never easy and likely never complete. With the right user behind the mouse, any system, regardless of AV product – or even OS, can be infected with malware.

In my professional life, I’m responsible for the security of around 700 endpoints, and as with my home systems I rely on more than just endpoint security to mitigate the risk of a few hundred users with Internet access. As far as AV goes – we deploy it on nearly every workstation and server and I would not want to be without that security blanket. I’m sure it prevents some infections outright, while at other times it at least yelps loud enough when something is amiss that we can respond and clean up or format the host.

AV is no panacea – which despite the marketing hyperbole the vendors are all guilty of – they would be the first to admit. In the enterprise, it’s part of a well-rounded, defense-in-depth strategy. At home for the average user, AV will likely provide more malware-free days than one would have otherwise. Savvy, power-users however, may do just as well without AV, if they are cautious and avoid the common pitfalls that often lead to infection.

I guess in the end, the big question is – does AV give you enough ROI to make a paid product worthwhile. It’s very difficult to answer yes to that – today’s free AV offerings are fairly decent and likely do 80-90% of the job a major paid product does. I know @mikko has been chirping back and forth on this subject with guys like @dakami and even some of the Sophos tweeps have jumped into the conversation. It’s definitely a conversation worth having as I think the vendors need to do more for the money and the industry is starving for innovation these days.

Another take on the subject: