bradyvolpe.com

Downstream Channel Bonding is perhaps the ball bearings of DOCSIS 3.0, enabling subscriber data speeds in excess of 160 Mbps (4 times that of previous DOCSIS versions).  While conceptually simple, the principle of combining multiple downstream DOCSIS channels together to carry the same user data must have tight constraints in order to preserve the integrity of the data and have the data arrive at the correct subscriber’s device and in sequence.  This article will cover both the physical layer aspects and DOCSIS protocol aspects that enable channel bonding.

Physical Layer

Channel bonding simply means that the CMTS knows that there are four or more RF signals within a 60 MHz passband (greater if more than four channels are bound).  The 60 MHz window is defined in section 6.3 of the DOCSIS 3.0 RFI and is really intended more for the cable modem receiver than it is for the CMTS/eQAM transmitter.  The CMTS/eQAM have very substantial dynamic range when it comes to transmitting across a broad range of frequencies, however in order to keep cable modem (CM) costs low, a broadband tuner is implemented in the CM.  It was determined that a 60 MHz bandwidth would be reasonable for cost effectiveness based upon existing hardware at the time of the specification.  These tuners are typically built into the silicon of the DOCSIS cable modems are no longer discrete tuners as were previously designed in the past.  If more than four downstream channels are tranmistted as part of the Downstream Bonding Group (DBG), then more than 60 MHz is permitted, but the cable modem must be able to tune to at least four channels in a 60 MHz bandwidth, support of additional channels outside of the 60 MHz bandwidth is optional and now becoming more of a necessity for cable operators who are commonly using eight channels in their DBGs – Note, remember the term Downstream Bonding Group or DBG as it is common lingo for DOCSIS 3.0.

Another change in DOCSIS 3.0 from DOCSIS 1.x and 2.0 is the downstream operational frequency range.  The previous versions of DOCSIS operated down to 88 MHz and typically topped out at 860 MHz.  DOCSIS 3.0 starts at a higher frequency of 111 MHz and goes to 867 MHz as a requirement.  Additionally, the specification has a recommendation that 999 MHZ should be the high end frequency of DOCSIS 3.0, fully utilizing a 1 GHz plant.  In a competitive market, these recommendations are usually taken as “we had better do it to stay in business”, which has been the case.  The reason that DOCSIS 3.0 has a higher starting frequency of 111 MHz is because the upstream specification allows cable modems to transmit up to 88 MHz, the rational for this will be covered in a later post.  Otherwise, other physical characteristics of DOCSIS 3.0 are similar to DOCSIS 1.x and 2.0.  Any channel in the DBG can operate in either 64-QAM or 256-QAM mode.  BER must be equal to or better than 1×10^-8 and codeword error rate (CER) must be less than or equal to 9×10^-7.  Oh, maybe you have not heard about the new spec. for code error rate before?  Well this is new to the DOCSIS 3.0 specification and is something that is quite over due since BER is nearly impossible to measure in a live DOCSIS plant while CER can be obtained right from the CMTS.  CER can be computed as follows:

,

Where:

• E_u is the value of the count of code words with uncorrectable errors;
• E_c is the value of the count of code words with correctable errors and;
• C is the value of the count of code words without errors.

Keep an eye on CER to be a new metric for equipment manufacturers and test vendors to be using moving forward in addition to BER and MER.

DOCSIS 3.0 Protocol

In a DOCSIS 3.0 network implementing downstream channel bonding, the DOCSIS CMTS dynamically balances the data across the Downstream Bonding Group (DBG), which can consist of four or more downstream channels.  The reason this is done is to offer subscribers the best quality of service across downstream channels with changing impairments and changing congestion at the receive side.  Each outgoing packet from the CMTS is tagged with a sequence number.  The sequence number becomes important for a number of reasons.  Packets can be dispersed across different downstream channels and can have different time delays in arriving at the receiving cable modem.  It is then the cable modem’s responsibility to re-synchronize the incoming packets based upon the sequence numbers.  TCP/IP windowing acknowledgments will take care of any lost packets at Layer 3, however for UDP flows, such as voice and video, those packets will be forever lost.  Further, by dynamically distributing the packets across downstreams, the CMTS can take advantage of statistical gains of many cable modems connected to the DBG.  This becomes especially critical when your system has a mixture of legacy DOCSIS 1.x, 2.0 and 3.0 cable modems.  The 1.x and 2.0 modems will all be receiving data from only the Primary Channel of the DBG which the DOCSIS 3.0 modems will be able to receive data from all four+ downstream channels in the DBG.  So this dynamic prioritization is in effect acting like upstream load balancing in the downstream.

If all downstream channels of the DBG are configured as Primary Downstreams, then DOCSIS 3.0 has another capability to load balance all legacy cable modems across the DBG.  This is called Downstream Channel Set (DCS) and is truly analogous to upstream load balancing.  It is highly recommended that when you are first turning on a DOCSIS 3.0 network with few DOCSIS 3.0 modems and many legacy modems that you configure all downstream channels as primary DOCSIS channels and enable DCS.  There is a secondary benefit to configuring all downstreams as primaries and that is you will be able to fully test each DOCSIS primary channel using your legacy DOCSIS 2.0 hand-held test meters.  What is the drawback then?  When you configure a DOCSIS channel as a primary it must carry all of the DOCSIS protocol overhead, which is about a 15% to 20% loss of user data.  So when you have the ability to later configure a DOCSIS downstream channel to a secondary channel, the secondary channel no longer carriers the excessive DOCSIS overhead, except for some minimal synchronization information, and so you can now transport significantly more data to your end subscribers.

So let’s review the downstream terminology as a recap:

• Local Downstream – Always Primary
  • Comes from the CMTS line card in M-CMTS architecture
  • Comes from the CMTS in I-CMTS architecture
• Primary Downstream – Can come from Local Downstream or eQAM
  • Carries UCD, timing Sync, MAPs, etc.
  • Required for CMs to register on network Also carries PDU (subscriber data)
  • Has DOCSIS overhead, so you loose some subscriber data utilization
• Secondary Downstream – Comes from eQAM or other bonded channel on I-CMTS
  • Only carries PDU – No UCD, timing Sync, MAPs, etc.
  • Cable modem cannot register on Secondary Downstream
  • Does not support legacy cable modems (non-DOCSIS 3.0)
  • Best subscriber data utilization capability

Some final take aways from this post

Downstream Channel Bonding has been the heart of DOCSIS 3.0 and is what has made it a huge success.  There have been a number of deployment issues along the way, but that is for another post.  From the physical layer standpoint, DOCSIS 3.0 is very similar to DOCSIS 1.x and 2.0.  You will either be dealing with 64-QAM, 256-QAM or a mix of both in your downstream bonding groups, which must be in a 60 MHz bandwidth.  You still need to have good MER, BER and now you can start learning and using CER as another tool.

Just as you may have been getting acquainted with upstream load balancing, this is a new a valuable feature in DOCSIS 3.0.  Not only are cable modems load balanced in 3.0, but the very data packets themselves.  At first this was done from a statistical standpoint to take advantage of which modems may be pulling more traffic than others.  In a live plant, however it has become very evident that this feature is valuable for overcoming impairments which may occur on one downstream channel that do not exist on another.  Consider it a self-healing mechanism that can come into play that gives you an opportunity to resolve a problem on that particular frequency.  It does not fix the plant, because your codeword errors and thus CER will go up considerably on that channel, your overall data rate will drop for the DBG, but data will still go through on the other channels at a maximum rate.

Now that we have established the two primary architectures available in DOCSIS 3.0, I-CMTS and M-CMTS (though hybrids do exist), and the hardware  components of these architectures, it is time to delve into the protocol of the DOCSIS specifications that make up DOCSIS 3.0.  There are five primary specifications that I will be drawing upon from here on out listed below and located in the Library and also on the CableLabs website.  Here are documents to focus on if you really want some heavy reading:

DOCSIS 3.0 Series of Specifications (found in Library)

• CM-SP-PHYv3.0 Physical Layer Specification
• CM-SP-MULPIv3.0 MAC and Upper Layer Protocols Interface Specification
• CM-SP-OSSIv3.0 Operations Support System Interface Specification
• CM-SP-SECv3.0 Security Specification
• CM-SP-CMCIv3.0 Cable Modem CPE Interface Specification

I will be taking key excerpts from these documents and boiling them down into a more legible format in my blogs from here on out to focus on the key enhancements in the DOCSIS 3.0 specification that separate it from previous versions of the DOCSIS standard.  Here are some areas to start with that will be explored in much more detail later:

• Downstream Channel Bonding with Multiple Receive Channels: DOCSIS 3.0 introduces the concept of a cable modem (CM) that receives simultaneously on multiple downstream channels. Downstream Channel Bonding refers to the ability (at the MAC layer) to schedule packets for a single service flow across those multiple downstream channels.  It is Downstream Channel Bonding that gives DOCSIS 3.0 the ability to provide throughput speeds from 150 Mbps and up, depending on the number of channels bonded together.
• Upstream Channel Bonding with Multiple Transmit Channels: DOCSIS 3.0 introduces the concept of a CM that transmits simultaneously on multiple transmitting upstream channels. Upstream Channel Bonding, refers to the ability to schedule the traffic for a single upstream service flow across those multiple upstream channels. Upstream Channel Bonding offers significant increases in the peak upstream data rate that can be provided to a single CM.  It is Upstream Channel Bonding that enables upstream data rates of over 100 Mbps.
• IPv6: DOCSIS 3.0 introduces built-in support for the Internet Protocol version 6. DOCSIS 3.0 CMs can be provisioned with an IPv4 IP address, an IPv6 IP address, or both. Further, DOCSIS 3.0 CMs can provide transparent IPv6 connectivity to devices behind the cable modem (CPEs), with full support for Quality of Service and filtering.  IPv6 is not just new technology, but a requirement for many MSO who are running out of IPv4 IP addresses for the numerous subscribers and IP devices in the network.
• Source-Specific Multicast: DOCSIS 3.0 supports delivery of Source-Specific IP Multicast streams to CPEs.  Rather than extend the IP multicast protocol awareness of cable modems to support enhanced multicast control protocols, DOCSIS 3.0 takes a different approach. All awareness of IP multicast is moved to the CMTS, and a new DOCSIS-specific layer 2 multicast control protocol between the CM and CMTS is defined which works in harmony with downstream channel bonding and allows efficient and extensible support for future multicast applications.  This becomes important for network efficiency as well as multicast storm suppression when devices have problems.  In effect it puts a VLAN in the DOCSIS network for DOCSIS Multicast traffic.
• Multicast QoS: DOCSIS 3.0 defines a standard mechanism for configuring the Quality of Service for IP multicast sessions. It introduces the concept of a “Group Service Flow” for multicast traffic that references a Service Class Name that defines the QoS parameters for the service flow.  This is critical for enabling QoS for clients behind a cable modem and not just cable modem attached clients, such as an eMTA.  It greatly increased the ability of cable operator to improve QoS for new IP services in the home.

CMTS Forwarding Model

I have had many comments on previous posts about DOCSIS CMTSs and whether or not they are Layer 2 or Layer 3 devices, so I believe it is important to clarify this on the DOCSIS 3.0 specification, which has not changed much from previous revisions of the spec.  The CMTS relays upstream and downstream data to a forwarder in the CMTS.  The forwarder can function as a Layer 2 or a Layer 3 device (per the OSI model).  As a Layer 2 device, the CMTS forwarder is acting similar to an IP switch and transparently moving packets based without altering the packets in any way.  As a Layer 3 device, the CMTS forwarder is now a Router and has value add.  It looks at MAC address, IP address, metrics to least-cost-path and routes packets accordingly.  In this mode, the CMTS may even bridge one IP domain to another and so the source IP and MAC addresses can be changed when they arrive at the destination.  The DOCSIS specification leaves it up to the vendor (and sometimes the end user) as to whether Layer 2 or Layer 3 forwarding is used.  It is my recommendation that the CMTS forwarder always be deployed as a Layer 3 device to segregate your DOCSIS network from your IP network for many reasons, security being one of the biggest.  One thing that does change in DOCSIS 3.0 is that the MAC Domain is not considered to forward data packets from its upstream to its own downstream channels; all upstream data packets are considered to be delivered to a CMTS forwarder. DOCSIS 3.0
leaves most details of CMTS forwarder operation to CMTS vendor-specific implementation.

The MAC Domain

Another important concept that I have not covered before is that of the MAC domain.  A MAC domain consists of at least one downstream on one upstream port and contains all of the functions necessary to operate a DOCSIS network (DOCSIS protocol, IP connectivity, etc.).  Now your typical MAC domain would be one downstream channel and four or more upstream channels on a line card.  Most line cards have more than one MAC domain because they have more than one downstream and eight or more upstreams (say a 2×8 card for example would have two MAC domains). It’s that simple.  So if you have been confused when people were talking about MAC domains, now you know the jargon.

Downstream MAC Domain Terminology – DOCSIS 3.0

A MAC Domain provides downstream DOCSIS data forwarding service using the set of downstream channels associated with the MAC Domain.  A downstream channel is defined as either:

• A “Downstream (RF) Channel”, representing a single-channel downstream RF signal on a Downstream RF Port of an Integrated CMTS; or
• A “Downstream M-CMTS Channel”, representing a single-channel downstream RF signal at a remote Edge QAM that is reached via a DEPI tunnel from an M-CMTS Core.

Upstream MAC Domain Terminology

An “upstream channel” can be used to refer to either:

• A “Physical Upstream Channel”; or
• “Logical Upstream Channel” of a Physical Upstream Channel.

A “Physical Upstream Channel” is defined as the DOCSIS RF signal at a single center frequency in an upstream carrier path.  Multiple “Logical Upstream Channels” can share the center frequency of a Physical Upstream Channel, but operate in different subsets of the time domain. Transmit opportunities for each Logical Upstream Channel are independently scheduled by the CMTS.

So I have run out words for this article, but I have just touched on the beginning of DOCSIS 3.0 terminology, especially in the downstream.  It is critical that you learn the differences in naming conventions because a downstream is not just a downstream in DOCSIS 3.0 and you will definitely want to keep your physical, primary, secondary and logicals straight when communicating with others DOCSIS 3.0 channels.

Before DOCSIS 3.0 and before modular CMTS architectures, a CMTS existed in one chassis.  Life was much simpler for everyone.  Inside the chassis existed a 10.24 MHz clock or oscillator.  This was a master time keeper that kept every event in synchronization with every other event.  Timing is very important in communications networks, especially when dealing with the microsecond timing calculations necessary for DOCSIS transport – remember the “tick” (6.25 usec).  This article is going to address the DOCSIS Timing Interface Specification (DTI) and DTI time servers that have arisen due to the distributed architectures in M-CMTSs and DOCSIS 3.0 CMTSs.  In these architectures, it is possible to have the CMTS core in say the headend, with the eQAM and upstream receivers in remote hubsites.  Suddenly the single 10.24 MHz clock keeping the system in synchronization is no longer an option.  Three separate, free running 10.24 MHz clocks would also not work because they would not be in phase and would not be running at the same frequency, causing the entire system to out of synchronization – there would packet collisions and lost data and VoIP packets all over the place.  It would be chaos!  So the smart folks at Cablelabs put together the DTI specification to resolve these issues.  Here are some of the details.

The following is taken from Figure 1-1 in the DOCSIS Timing Interface Specification located both on the Cablelabs website and in the Library.

DOCSIS Timing Interface Specification - DTI

This diagram illustrates several important details.  First, there is a primary DTI server, called the Root Server.  The DTI root server provides a 10.24 MHz clock source.  Second, there is (actually it is strongly recommended) a DTI Slave Server.  The DTI slave server is nothing more than a redundant 10.24 MHz clock that monitors the root server.  In the event that the root server should fail, the slave server will automatically provide a 10.24 MHz reference clock to all sources.  The current vendor of DTI servers has the provisions in their firmware to monitor root and slave, providing for minor and major alarms when there are problems with the servers.  The DOCSIS DTI Specification also also defines the method by which the root and slave communicate with each other over the IP network, so be certain to provision all timing servers. Next, the DTI servers provide 10.24 MHz clock sources to the eQAM, the CMTS and the upstream receivers.  As indicated at the beginning of this article and in this diagram, all three of these devices can be physically located in different buildings.  Since it is not practical to run very long CAT5 cables between Headends and Hub sites, DTI time servers are installed in each location.  Synchronization of time servers between the locations is recommended (though not required).  The way that the time servers are synchronized is via the GPS (Global Positioning Satellite) antenna port on the time servers or network streams.  The time servers synchronize themselves to the global time server signal originating from the GPS satellites, ensuring that every time server connected to a GPS is synchronized to within 100 nsec of every other time server in the network.  If the GPS option is not used, it is recommended to use a network-based source such as PDH, SONET, or equivalent.

In addition to being a highly accurate time source, the DTI server can also function as an NTP (Network Time Protocol) server for the cable modems and other devices in your network.  This will provide IP devices with the time of day during the DHCP process.  This is a function of the Symmetricom time servers, which at the time of this writing, is the only game in town for getting time servers.  But their equipment works quite well and I have found both their technical support and RMA departments quite competent.

To dive a little deeper into the implementation end of things, here are some considerations that one should consider when setting up DTI servers:

• Traditional CAT5e cables works fine for interconnects
• Lengths of cables may not exceed 200 meters
• When using GPS mode, it can take up to three (3) days, worse case, to synchronize the time server clock with the GPS clock.  This is called Time of Coincidence (ToC) and is explained in more detail in the DTI Specification.  Symmetricom’s Time Creator 1000 makes this process a little smoother, but don’t get concerned if it takes a while
• Shutdown all unused DTI ports on the time servers as if they are not connected they will give you errors

Modes of Operation

The DTI Server can support three modes of operation:

1. Free running Master: no external source is provided to control frequency or timestamp accuracy
2. GPS Traceable: source of timestamp and frequency traceability is GPS
3. Network Traceable: a standard network PDH, SONET, or SDH interface is used to provide traceability to a Stratum 1 frequency reference

In M-CMTS, deployment where all the elements are located in the same area GPS traceability is not required.  The DTI server should be connected to all devices.
For distributed applications, the DTI Specification states (in items #2 and #3 above) that a GPS or network traceable source must be used if the network shows degradation as measured over a 35 second period of time.  What types of degradation is the DOCSIS specification looking for?  At the very minimum, the specification states a BER of <1e-8, but it further details requirements for synchronization jitter, timing, skew, etc.  So it is important to understand that just good BER is not an indication that the system is meeting adequate timing.  Other timing impairments could be present which will impact voice and video that will not impact data only applications.

In summary, DOCSIS 3.0 with DTI servers has upgraded the timing in DOCSIS networks from a Tier 3, low end timing service to a much higher end, telco quality service.  Using a DTI server in collocated M-CMTS architecture provides for a Tier 2 timing service with 5 nsec of timing resolution between the devices.  Using multiple DTI servers between remotely distributed devices in an M-CMTS architecture traceable to network or GPS timing elements provides for a Tier 1 timing service with 100 nsec accuracy between each device.  This is unprecedented timing accuracy in our continuously improving DOCSIS networks enabling evolving service offerings with enhanced quality of service and quality of experience.