Wednesday, February 25, 2009

Forensic Acquisition Simplified

Check out the Raptor forensic tool.

Tuesday, February 17, 2009

Internet Outage

The company I work for provides virtual server hosting services. So, when the Internet goes down, it's gonna be a bad day at the office.

Yesterday, our provider's Internet connectivity went down, for about 1 1/2 hours. An outage of this duration causes us to lose customers. So, it was a very bad Monday!

Our data center provider has not been very forthcoming with information on the cause of the outage. But, I believe, it was related to the BGP overflow caused by an ISP in the Czech.

The unfortunate consequence is that the arm chairs quarterbacks, like myself, get to determine that the cause of this issue was absolutely avoidable. Isolated to Cisco IOS, two things could have prevented this catastrophic outage for us, and our clients.
  1. Maintain patching levels by running a newer IOS on the routers.
  2. Implement the 'bgp maxas-limit command. http://tinyurl.com/cygzbz
The offending information:

%BGP-6-ASPATH: Long AS path 3549 3257 29113 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868
47868 47868 47868 47868 47868 47868 47868 received from xxx.xxx.xxx.xxx:
Has more than 255 AS


This issue was resolved in IOS release 12.1(3a)E1 (http://tinyurl.com/db6v2d)
A Border Gateway Protocol (BGP) UPDATE contains Network Layer Reachability Information (NLRI) and attributes that describe the path to the destination. Each path attribute is a type, length, value (TLV) object.

The type is a two-octet field that includes the attribute flags and the type code. The fourth high-order bit (bit 3) of the attribute flags is the Extended Length bit. It defines whether the attribute length is one octet (if set to 0) or two octets (if set to 1). The extended length bit is used only if the length of the attribute value is greater than 255 octets.

The AS_PATH (type code 2) is represented by a series of TLVs (or path segments). The path segment type indicates whether the content is an AS_SET or AS_SEQUENCE. The path segment length indicates the number of autonomous systems in the segment. The path segment value contains the list of autonomous systems (each autonomous system is represented by two octets).

The total length of the attribute depends on the number of path segments and the number of autonomous systems in them. For example, if the AS_PATH contains only an AS_SEQUENCE, then the maximum number of autonomous systems (without having to use the extended length bit) is 126 [= (255-2)/2]. If the UPDATE is propagated across an autonomous system boundary, then the local Abstract Syntax Notation (ASN) must be appended and the extended length bit used.

This problem was caused by the mishandling of the operation during which the length of the attribute was truncated to only one octet. Because of the internal operation of the code, the receiving border router would not be affected, but its iBGP peers would detect the mismatch and issue a NOTIFICATION message (update malformed) to reset their session.

The average maximum AS_PATH length in the Internet is between 15 and 20 autonomous systems, so there is no need to use the extended length. The failure was discovered because of a malfunction in the BGP implementation of another vendor. There is no workaround. This problem is resolved in Release 12.1(3a)E1. (CSCdr54230)
Source information was obtained from the following locations:
Renesys – Reckless Driving on the Internet - http://tinyurl.com/cd2eq9
Merit Networks – North American Network Operators Group - http://tinyurl.com/dmexp2
Data Center Knowledge – Router Snafu: A ‘Global Internet Meltdown’ - http://tinyurl.com/atmo6h
Slashdot - One Broken Router Takes Out Half the Internet - http://tinyurl.com/bsqhk6

Tuesday, February 3, 2009

ITEXPO Keynote

The keynote session consisted of two presentations. The first was by John Frederiksen, GM of Microsoft Response Point. The second was Danny Windham, CEO of Digium (Asterisk).

John's well prepared presentation centered around Microsoft's Response Point and their software centric philosophy. No offense John, but honestly, my attention was glued to the presentation. So, I didn't manage to write down very many notes. One note of interest was how Microsoft sees the requirements of the current business climate. Identified as business requirements include:
  1. 'Save me money'
  2. 'Save me time'
  3. 'Increase productivity'
  4. 'Help us grow our business'
No debating these requirements. Budgets are tight, and there is no patience for an unstable phone system, when every call counts.

Included in John's presentation was a quote from Bill Gates in 2007. Paraphrasing, he said the power of software will transform communications. With Microsoft's R&D budget, we should see solutions that could very well do that.

Danny's presentation provided some very interesting statistics about OSS (Open Source Software) IPT.

Using a statistic from IDC, OSS is expected to be a $5.8B industry by 2010/2011. There were 10's of millions of downloads of open source software in 2009.

The IP PBX market continues to increase in market share and is expected to reach $7.9B by 2010.

There were a total of 1.75 million downloads of open source IPT software. Of that, 1.5M downloads were for Asterisk software.

It is estimated that there are over 86,000 IP PBX's with an average of 32.8 end points per system.

Another slide presented market share by PBX vendor.
  1. Open Source
  2. Nortel
  3. Cisco
  4. Avaya
  5. Mitel
  6. NEC
The list changes slightly, when looking at total number of end points by vendor.
  1. Nortel
  2. Open Source
  3. Cisco
  4. Avaya
There is no shortage of innovation in the IPT market. It will be exciting to see how products from both Microsoft and the Open Source community mature over the next couple of years.

ITEXPO Service Provider Roundtable

It's been a busy couple of weeks.

Last week I was in Las Vegas for Network Instruments Observer training. This week, I'm in Miami attending the Internet Telephony Conference and Expo.

In between, I've been fighting a cold. Oh well, I guess there has to be a negative for hanging out in the hot spots of the country. No pun intended.

Below are some of the highlights I scribbled down during the 'Service Provider Roundtable'.

The following service providers participated:

8X8, Inc. - Hosted PBX services for the SMB market
Broadvox - Wholesale SIP trunk provider
inPhonex - Consumer Internet phone service provider with strong Latin America presence
MagicJack - Consumer Internet phone service provider
Telefonica - Global corporate communications provider, mainly in Latin America
TW Telecom - Provider of corporate fiber communications infrastructure in 75 US markets


This was a nice discussion about industry issues from a provider perspective.

Mentioned by the panel was the need for the providers to continually innovate. The last major innovation in the industry was 10-15 years ago when DSL was released.

Another issue the SIP providers faced five years ago was their lackluster 99.9% availability. Still not as good as traditional land line solutions, most SIP providers will guarantee 4 nines.

Most enterprises readily adopt NG (Next Generation) products, but not necessarily NG services like SIP. However, budgets will drive NG service adoption because they are more cost effective.

However, like most things, efficiency and reliability depend on proper implementation. This is complicated when these services are delivered over 'plug-and-play' Ethernet connectivity - complicating troubleshooting and blurring the DMARC.

The main take away, regarding SIP services, was to focus on function, instead of features. SIP reliability is very important. Nobody wants to tell their CXO that their phone services are down, again.