DOCSIS 3.0 Tutorial – Upstream Channel Bonding

Posted on August 22, 2010 by Brady

DSAM DOCSIS 3.0 Throughput TestSince its inception, the DOCSIS 3.0 specification has supported upstream channel bonding. Most CMTS vendors have not supported this in their firmware until the past year. In most cases the firmware that is available is still in beta testing in cable operator labs to ensure cable modems and networks can support it – why? The bonding of two, three or four upstream channels in an RF DOCSIS network creates a lot of challenges such as finding enough RF spectrum, resolving overloading return path lasers, isolation issues and much more. Some of these items have been addressed in the Speeding Upstream Articles written by John Downey and me.

The focus of this article will be on the mechanics of upstream channel bonding and how it works more from a DOCSIS protocol perspective.  Much more detailed information can be found in the DOCSIS 3.0 MULPIv3.0 document located in the Library, but this will provide a high level overview for the layman who is curious about the basics.  First lets understand that it is the cable modem that is doing the channel bonding, remember in the upstream the cable modem transmits data to the CMTS.  Per DOCSIS 3.0, the CM can bond from two to four channels in the upstream as coordinated by the CMTS.  The CM is always controled by the CMTS.

Upstream channel bonding has a lot in common with downstream channel bonding.  Like downstream channel bonding, upstream bonding consists of two or more channels active as an Upstream Bonding Group (UBG).  In order to enable multiple upstream channels in a cable modem, a special mode is turned on in the cable called Multiple Transmit Channel mode (MTC mode).  This is the basic distinction of whether the cable modem is functioning in DOCSIS 2.0 mode or DOCSIS 3.0 mode.  Once in DOCSIS 3.0 mode, all of the power level variations described in Figure 3 of Speeding Upstream Part II are in effect.

At a very basic level, the CMTS assigns each upstream channel with a service flow.  The Best Effort service flow is a just one of many “SID Clusters” that assigns this type of service flow to each channel in the bonding group.  Queuing in the cable modem buffers data and then bandwidth REQs on any upstream from the CM allow the CMTS to load balance the CM by sending back MAP Grants to any upstream channel.  Why would load balancing be necessary?  For one the CMTS is managing many cable modems, some of which could be legacy DOCSIS 1.x/2.0 modems using only one of the available channels in the bonding group.  Further, there could be impairments on some upstream channels seen by some modems.  Higher levels of service flows will be established for QoS by the CMTS, this will be another SID Cluster and spread across the bonding group so that load balancing can also occur for that particular QoS service flow to achieve the best possible performance in the case of traffic or impairments.

A channel in a bonding group may become unusable due to say upstream impairments.  In this case, if the CMTS becomes aware of the outage, it will stop providing opportunities for CMs to transmit on that channel (i.e. MAP grants).  If the CM becomes aware, it will stop sending REQs for data.  Both the CM and CMTS will continue to do Initial Ranging (i.e. the cable modem will continue to establish communication with the CMTS) until the channel become available.  So basically, if an UBG channel becomes unavailable, the DOCSIS 3.0 network will continue to work with the remaining upstream bonded channels until the down channel is brought back up.  Its smart enough to know not to transmit any channel on the down channel.

So upstream channel bonding can be pretty straight forward.  If you dig deeper you will find that a host of advanced features are supported similar to DOCSIS 2.0 such as Dynamic Channel Change, fragmentation, concatenation, etc.  Of course these are beyond the scope of this blog and I encourage you to read the spec if you are interested in this level of detail.

Posted in DOCSIS 3.0 | Tagged , , , , , , , , , , , , , , , | Leave a comment

Hacking DOCSIS Cable Modems

Posted on August 1, 2010 by Brady

Hacking Cable ModemsFundamental Precautions You  Should Take to Secure Your Network

I hope the title of this post caught your eye.  DOCSIS security holes are a serious problem, even if you are a major MSO (Multiple System Operator).  Look at this from the subscriber’s point of view, you have a coax cable coming into your house.  Your paying for some level of DOCSIS cable modem service and you just want to get a little more speed our of your modem.  If I’m already getting 12 Mbps, what is the difference if I up it to 20 Mbps?  Hey, who of us can honestly say we have never broken the speed limit?  Have you never driven 50 MPH in a 40 MPH speed zone?  Most people can self-justify a lot of small “law bending infractions”, but what hacker’s may not realize are the impacts they have on other users in the network.  Further there are hackers who do not pay for service at all and connect cloned modems to get free service.  Finally, with DOCSIS 3.0, un-capping modems could result in hackers accessing download speeds in excess of 150 Mbps+.  Depending upon what they are doing, this could have significant impacts on other services being offered to subscribers.

Recently a reader contacted me and said that theft of service, especially uncapping cable modems via hacking, was still impacting his network.  Not surprisingly, one vendor’s CMTS was able to ward off the hacker’s while another vendor’s CMTS was unable to prevent the uncapping and subsequent theft of service.  I will protect the vendor’s identities because I believe that the CMTS is the first line of defense.  Vendors have put into place very effective, CMTS specific techniques, such as Cisco’s TFTP-Enforce which prohibits a cable modem from registering and coming on line if there is no matching TFTP traffic through the CMTS preceding the registration attempt.  But often individual techniques are “hacked” (such as in the TFTP-Enforce bypass method found on hacker sites).  What this indicates is that any reliance on a single point or method of hack-proofing your network WILL NOT WORK.  You must implement a layered approach consisting of a number of CMTS, DHCP, TFTP and potentially SNMP and Kerberos related methods.  The later would apply for MTAs and set top boxes.  For now we will just focus on cable modems and the realm of CMTSs and DHCP/TFTP servers.  Here are is the bare minimum of what you should be doing:

CMTS Hacking Safety

Layer 2 vs Layer 3 – In a previous DOCSIS 3.0 article, I briefly discussed how a CMTS can be configured as a either a Layer  2 or a Layer 3 device.  Consider your Layer 2 device to be a basic switch.  From a hacking standpoint, let’s look at it as a direct connection from the hacker’s computer to your network.  As we move up to a Layer 3 (router) configuration, we now have something that we can configure as a very rudimentary firewall, but the base configuration of any CMTS does not look like a firewall from a hacker’s point of view.  It is more like a speed bump.  So let’s make certain that we have enabled the basic security features within the DOCSIS specifications starting with the Baseline Privacy Plus Interface Specification (BPI+).

BPI+

BPI+ should be considered a MUST do in every DOCSIS network.  Why?  Because it protects subscriber data from being viewed by malicious users in addition to thwarting potential hackers by using 56-bit or 40-bit DES encryption (56-bit DES is standard).  There are two types of encryption under the BPI umbrella, BPI and BPI+.  BPI was first released in 1999 under DOCSIS 1.1.  This was rather weak encryption because it did not authenticate cable modems with unique certificates.  BPI was later enhanced to BPI+ which incorporates digital certificates issued by VeriSign Corporation which are uniquely chained to the individual MAC address of each cable modem.  For every cable modem’s MAC address there can be only on uniquely created digital certificate in existence.  VeriSign and vendors work closely together to ensure that the trust is not broken during the creation of the certificates and the manufacture of the cable modems.  Breaking this trust would be cause for a vendor to have all of their previous, current and future certificates revoked in addition to suffering significant financial penalties.  The end result is assurance to the cable operator that enforcing BPI+ should ensure that a BPI+ verified cable modem is a paying customer.  How do hackers defeat this?  There are a couple of key ways:

  1. The biggest hole in BPI+ is that cable operators turn on the “allow self-signed certificates” in their CMTS.  Why do they do this?  Because they are using legacy test equipment, outdated test equipment or non-conforming test equipment that does not support BPI+ certificates.  If your hand-held test equipment vendor cannot upgrade your equipment to BPI+, find a new vendor, because you are enabling hackers in your network to create their own self-signed certificates, install them in their own cable modems with “valid MAC addresses” sniffed from your network and steal your service.
    • Disable self-signed certificates and plug the hole
  2. Another hole in BPI+ is that many systems still have old cable modems that do not support BPI+ and so operators will enable BPI+ in its most limited mode.  In this case, modems that support BPI+ will be required to register with BPI+, but modems that do not support BPI+ will register in BPI mode or with no encryption at all.  This is an open door once again for hackers.
    • Require “bpi-plus-enforce” on all CMTSs – this means only modems that support BPI+ will be able to register
    • Monitor “cloned MAC Addresses” across your network – a good hacker will still be able to clone a cable modem MAC along with its digital certificate, but this will show up on your CMTS logs.  Automatic monitoring of your CMTS logs for this event will rapidly identify this hacker and enable you to give him the boot.

Modem Registration and Uncapping

We will assume now that you have adequately enforced BPI+ to the point that all modems on-line are valid in the provisioning system because only valid MAC addresses can pass the stringent tests that are part of BPI+ registration.  So your next major weakness is going to be in your provisioning system.  You see, now hackers will be forced into paying for at least basic data service.  Hey, this is at least a start.  You have some money coming in right?  Oh but wait.  If your IP network is not properly secured, these guys and gals are going to intercept the standard TFTP file your systems download to the cable modem during registration, alter the TFTP file and then configure their modem to download their modified TFTP file.  What will this do?  If all they are interested in doing is fast downloads, then they will max out the download and upload speed of their modem.  If your system is really open (meaning no PacketCable security), then they could even give themselves fast speed and guaranteed QoS (Quality of Service), which means that their traffic would have priority over everyone else.  This type of QoS is called a Static Service Flow as compared to the Dynamic Service Flows provided in a PacketCable environment.  A smart hacker with a Static Service flow is a serious issue in your network.  So what do you do?

Lets first look at the problem graphically:

DOCSIS Cable Modem Registration using On the Fly TFTP

Cable Modem Registration using OTF

The above diagram (click on it to get a better view) shows the DOCSIS cable modem registration process after ranging (for more information on ranging see http://bradyvolpe.com/docsis-101/docsis101_modem-registration).  Steps 2-4 shows where the DHCP server checks with the back-office server(s) to make certain that the cable modem sending the DHCP Discover message is a paying customer.  If so, then in step 5, the DHCP server continues its normal mode of providing IP addresses, options, etc.  In step 7, the cable modem requests the TFTP configuration file as normal.  Typically the TFTP server would just send the TFTP file directly to the cable modem without any encryption.  This is where hackers have a field day, intercept the file, modify it, etc. and then your happy hacker has an uncapped modem.  So what are you to do?

Well there are number of options.  The picture above shows one good example employed by the folks at Incognito who has a quite secure provisioning system for up to DOCSIS 3.0 and PacketCable Multimedia  (www.incognito.com).  The Incogito server does at least two  things (that I am aware of):

  1. It dynamically generates the configuration file On-The-Fly (OTF) and downloads it the cable modem.
  2. It randomly generates  a 64-byte name (last time I checked the length was 64-ASCII characters), so the file name changes everytime.

Why is this important?  Because one of the techniques a hacker will use is to is to substitute the TFTP download with their own local copy of their modified TFTP file.  The problem that now occurs is that the file the cable modem is asking for is completely different than the one the hacker originally obtained because the DHCP server generates a new file name each time.  The hacker will never be able to replicate a randomized file name.  So even though the file sitting on your TFTP server is call “silver.cm”, your DHCP server is setting a proxy for that file and telling your cable modem to fetch “kjdhie8&^$lsk(uej&$nmJHEuI8&^yhNM,>Lkhy6tGftRFgt%$rgbnM<.lkIJuy7” and the next time this sequence will completely change.  Pretty simple, but for the hacker its a real headache.

To add one more layer of security, you should always add your “shared secret” to every CMTS and use password-encryption.  Let’s assume that hacker is very good and has their own TFTP server and plans on loading in the TFTP file locally during registration, thus bypassing altogether the need for a TFTP file name that was causing them a problem with the Incognito TFTP server.  The shared secret is a simply a unique “word” that you put in the running-configuration of the CMTS.  You should keep this word private and limited to as few people as possible within your security team.  This is why I recommend encrypting it on the CMTS with the vendor specific password encryption.  Remember, your weakest link can also be those people internal to your network.

The DOCSIS specification requires that the CMTS generates a Message Integrity Check (MIC) based upon a Message Digest 5 (MD5) using a number of parameters, including the “shared secret”.  The MD5 is a one-way (non-invertible) hash—meaning that the input cannot be recovered from the output—and the output is considered unique for a specific input. If the MIC is not correct, the cable modem registration process fails and it will not be allowed to come on line.  So if the hacker tries to modify the TFTP file and reboot the modem, it is mathematically unfeasible that the hacker will be able to generate a MIC that will pass the MD5 hash in the CMTS and the cable modem will fail CMTS registration, even the hacker is paying for some level of service.  This will force the hacker to go back to his/her standard level of service.

Summary

I have just touched on the basics of securing a DOCSIS network.  I am sure that many of your systems have security that vastly supersedes the recommendations I have provided here, in which case please feel free to provide comments and tips that other readers may learn.  At the same time, there are some basic fundamentals that I have seen overlooked in even big systems which open themselves to theft of service.  You have to no more than Google “hacking cable modems” to see a thriving network of individuals who are not just hacking, but are also profiting from providing the capabilities to others.  Although it is almost one year since Ryan Harris of TCNISO, one of the best known cable modem hacking sites, was arrested there are still numerous hacking sites and forums out there that have taken his place.  Secure your network externally and internally or expect that you will have theft of service and possibly worse.

Posted in Troubleshooting DOCSIS | Tagged , , , , , , , , , , , , , , , | 3 Comments

DOCSIS 3.0 Tutorial – Downstream Channel Bonding

Posted on July 18, 2010 by Brady

DSAM DOCSIS 3.0 Throughput TestDownstream Channel Bonding is perhaps the ball bearings of DOCSIS 3.0, enabling subscriber data speeds in excess of 160 Mbps (4 times that of previous DOCSIS versions).  While conceptually simple, the principle of combining multiple downstream DOCSIS channels together to carry the same user data must have tight constraints in order to preserve the integrity of the data and have the data arrive at the correct subscriber’s device and in sequence.  This article will cover both the physical layer aspects and DOCSIS protocol aspects that enable channel bonding.

Physical Layer

Channel bonding simply means that the CMTS knows that there are four or more RF signals within a 60 MHz passband (greater if more than four channels are bound).  The 60 MHz window is defined in section 6.3 of the DOCSIS 3.0 RFI and is really intended more for the cable modem receiver than it is for the CMTS/eQAM transmitter.  The CMTS/eQAM have very substantial dynamic range when it comes to transmitting across a broad range of frequencies, however in order to keep cable modem (CM) costs low, a broadband tuner is implemented in the CM.  It was determined that a 60 MHz bandwidth would be reasonable for cost effectiveness based upon existing hardware at the time of the specification.  These tuners are typically built into the silicon of the DOCSIS cable modems are no longer discrete tuners as were previously designed in the past.  If more than four downstream channels are tranmistted as part of the Downstream Bonding Group (DBG), then more than 60 MHz is permitted, but the cable modem must be able to tune to at least four channels in a 60 MHz bandwidth, support of additional channels outside of the 60 MHz bandwidth is optional and now becoming more of a necessity for cable operators who are commonly using eight channels in their DBGs – Note, remember the term Downstream Bonding Group or DBG as it is common lingo for DOCSIS 3.0.

Another change in DOCSIS 3.0 from DOCSIS 1.x and 2.0 is the downstream operational frequency range.  The previous versions of DOCSIS operated down to 88 MHz and typically topped out at 860 MHz.  DOCSIS 3.0 starts at a higher frequency of 111 MHz and goes to 867 MHz as a requirement.  Additionally, the specification has a recommendation that 999 MHZ should be the high end frequency of DOCSIS 3.0, fully utilizing a 1 GHz plant.  In a competitive market, these recommendations are usually taken as “we had better do it to stay in business”, which has been the case.  The reason that DOCSIS 3.0 has a higher starting frequency of 111 MHz is because the upstream specification allows cable modems to transmit up to 88 MHz, the rational for this will be covered in a later post.  Otherwise, other physical characteristics of DOCSIS 3.0 are similar to DOCSIS 1.x and 2.0.  Any channel in the DBG can operate in either 64-QAM or 256-QAM mode.  BER must be equal to or better than 1×10-8 and codeword error rate (CER) must be less than or equal to 9×10-7.  Oh, maybe you have not heard about the new spec. for code error rate before?  Well this is new to the DOCSIS 3.0 specification and is something that is quite over due since BER is nearly impossible to measure in a live DOCSIS plant while CER can be obtained right from the CMTS.  CER can be computed as follows:

{R_{c}=\frac{(E_{u}-E_{u0})}{(E_{u}-E_{u0})+(E_{c}-E_{c0})+(C-C_{o})}},

Where:

  • Eu is the value of the count of code words with uncorrectable errors;
  • Ec is the value of the count of code words with correctable errors and;
  • C is the value of the count of code words without errors.

Keep an eye on CER to be a new metric for equipment manufacturers and test vendors to be using moving forward in addition to BER and MER.

DOCSIS 3.0 Protocol

In a DOCSIS 3.0 network implementing downstream channel bonding, the DOCSIS CMTS dynamically balances the data across the Downstream Bonding Group (DBG), which can consist of four or more downstream channels.  The reason this is done is to offer subscribers the best quality of service across downstream channels with changing impairments and changing congestion at the receive side.  Each outgoing packet from the CMTS is tagged with a sequence number.  The sequence number becomes important for a number of reasons.  Packets can be dispersed across different downstream channels and can have different time delays in arriving at the receiving cable modem.  It is then the cable modem’s responsibility to re-synchronize the incoming packets based upon the sequence numbers.  TCP/IP windowing acknowledgments will take care of any lost packets at Layer 3, however for UDP flows, such as voice and video, those packets will be forever lost.  Further, by dynamically distributing the packets across downstreams, the CMTS can take advantage of statistical gains of many cable modems connected to the DBG.  This becomes especially critical when your system has a mixture of legacy DOCSIS 1.x, 2.0 and 3.0 cable modems.  The 1.x and 2.0 modems will all be receiving data from only the Primary Channel of the DBG which the DOCSIS 3.0 modems will be able to receive data from all four+ downstream channels in the DBG.  So this dynamic prioritization is in effect acting like upstream load balancing in the downstream.

If all downstream channels of the DBG are configured as Primary Downstreams, then DOCSIS 3.0 has another capability to load balance all legacy cable modems across the DBG.  This is called Downstream Channel Set (DCS) and is truly analogous to upstream load balancing.  It is highly recommended that when you are first turning on a DOCSIS 3.0 network with few DOCSIS 3.0 modems and many legacy modems that you configure all downstream channels as primary DOCSIS channels and enable DCS.  There is a secondary benefit to configuring all downstreams as primaries and that is you will be able to fully test each DOCSIS primary channel using your legacy DOCSIS 2.0 hand-held test meters.  What is the drawback then?  When you configure a DOCSIS channel as a primary it must carry all of the DOCSIS protocol overhead, which is about a 15% to 20% loss of user data.  So when you have the ability to later configure a DOCSIS downstream channel to a secondary channel, the secondary channel no longer carriers the excessive DOCSIS overhead, except for some minimal synchronization information, and so you can now transport significantly more data to your end subscribers.

So let’s review the downstream terminology as a recap:

  • Local Downstream – Always Primary
    • Comes from the CMTS line card in M-CMTS architecture
    • Comes from the CMTS in I-CMTS architecture
  • Primary Downstream – Can come from Local Downstream or eQAM
    • Carries UCD, timing Sync, MAPs, etc.
    • Required for CMs to register on network Also carries PDU (subscriber data)
    • Has DOCSIS overhead, so you loose some subscriber data utilization
  • Secondary Downstream – Comes from eQAM or other bonded channel on I-CMTS
    • Only carries PDU – No UCD, timing Sync, MAPs, etc.
    • Cable modem cannot register on Secondary Downstream
    • Does not support legacy cable modems (non-DOCSIS 3.0)
    • Best subscriber data utilization capability

Some final take aways from this post

Downstream Channel Bonding has been the heart of DOCSIS 3.0 and is what has made it a huge success.  There have been a number of deployment issues along the way, but that is for another post.  From the physical layer standpoint, DOCSIS 3.0 is very similar to DOCSIS 1.x and 2.0.  You will either be dealing with 64-QAM, 256-QAM or a mix of both in your downstream bonding groups, which must be in a 60 MHz bandwidth.  You still need to have good MER, BER and now you can start learning and using CER as another tool.

Just as you may have been getting acquainted with upstream load balancing, this is a new a valuable feature in DOCSIS 3.0.  Not only are cable modems load balanced in 3.0, but the very data packets themselves.  At first this was done from a statistical standpoint to take advantage of which modems may be pulling more traffic than others.  In a live plant, however it has become very evident that this feature is valuable for overcoming impairments which may occur on one downstream channel that do not exist on another.  Consider it a self-healing mechanism that can come into play that gives you an opportunity to resolve a problem on that particular frequency.  It does not fix the plant, because your codeword errors and thus CER will go up considerably on that channel, your overall data rate will drop for the DBG, but data will still go through on the other channels at a maximum rate.

Posted in DOCSIS 3.0 | Tagged , , , , , , , , , , , , , , , | Leave a comment