Characterizing Residential Broadband Networks
Characterizing Residential Broadband Networks
Marcel Dischinger
Andreas Haeberlen
MPI for Software Systems
MPI for Software Systems, Rice University
mdischin@mpi-sws.mpg.de
ahae@mpi-sws.mpg.de
Krishna P. Gummadi
Stefan Saroiu
MPI for Software Systems
University of Toronto
gummadi@mpi-sws.mpg.de
stefan@cs.toronto.edu
A large and rapidly growing proportion of users connect to
1.
INTRODUCTION
the Internet via residential broadband networks such as Dig-
Residential broadband networks such as Digital Subscriber
ital Subscriber Lines (DSL) and cable. Residential networks
Lines (DSL) and cable are increasingly being used to access
are often the bottleneck in the last mile of today’s Internet.
the Internet. More than 158 million people use these net-
Their characteristics critically affect Internet applications,
works worldwide [39], and this number is expected to rise
including voice-over-IP, online games, and peer-to-peer con-
to 477 million by 2011 [51]. In the United States alone,
tent sharing/delivery systems. However, to date, few studies
more than half of all Internet users connect via residential
have investigated commercial broadband deployments, and
broadband networks [38]. In addition, many governments
rigorous measurement data that characterize these networks
are adopting policies to promote ubiquitous broadband ac-
at scale are lacking.
cess [18, 48].
In this paper, we present the first large-scale measurement
Residential broadband networks provide the critical ‘last
study of major cable and DSL providers in North America
mile’ access to the Internet infrastructure.
It is widely
and Europe. We describe and evaluate the measurement
thought that the bottlenecks in the performance of the In-
tools we developed for this purpose. Our study character-
ternet lie in its access networks [1]. So the reliability and the
izes several properties of broadband networks, including link
performance of Internet applications – including voice-over-
capacities, packet round-trip times and jitter, packet loss
IP (VoIP), video on demand, online games, and peer-to-peer
rates, queue lengths, and queue drop policies. Our analysis
content delivery systems – depend crucially on the charac-
reveals important ways in which residential networks differ
teristics of broadband access networks.
from how the Internet is conventionally thought to operate.
Despite the widespread deployment of residential broad-
We also discuss the implications of our findings for many
band networks and their importance to emerging applica-
emerging protocols and systems, including delay-based con-
tions, they remain relatively unexplored by the academic
gestion control (e.g., PCP) and network coordinate systems
community. Although many measurement studies have fo-
(e.g., Vivaldi).
cused on the Internet’s core [6,26,40] and academic/research
edge networks [5, 35], rigorous measurement data that char-
Categories and Subject Descriptors
acterize residential network deployments at scale are lacking.
C.2.2 [Computer Systems Organization]: Computer-
In the absence of systematic studies, knowledge about res-
Communication
Networks—Network Operations;
C.2.5
idential broadband networks is based on anecdotal evidence,
[Computer
Systems
Organization]:
Computer-
hearsay, and marketing buzzwords. Although broadband
Communication Networks—Local and Wide-Area Networks;
networks are known to have very different characteristics
C.4 [Computer Systems Organization]: Performance of
from academic networks [5, 43], there have been no large-
Systems
scale studies quantifying these differences. As a result, re-
searchers today are left to second-guess how well protocols or
General Terms
systems evaluated in academic networks would work in the
Measurement, Performance, Experimentation
commercial Internet, where broadband networks are widely
deployed.
Keywords
One reason for the lack of large-scale measurement stud-
Broadband access networks, DSL, cable, network measure-
ies on residential networks is that researchers have limited
ment
access to broadband environments. Most academic insti-
tutions and research laboratories do not access the Inter-
net over broadband. Even state-of-the-art research network
testbeds, such as PlanetLab [41] and RON [2], have only a
Permission to make digital or hard copies of all or part of this work for
handful of broadband nodes. We overcame this problem by
personal or classroom use is granted without fee provided that copies are
developing tools that can measure broadband networks re-
not made or distributed for profit or commercial advantage and that copies
motely and without cooperation from end hosts connected
bear this notice and the full citation on the first page. To copy otherwise, to
to the broadband links.
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
In this paper, we present the first large-scale measurement
IMC’07, October 24-26, 2007, San Diego, California, USA.
study examining 1,894 broadband hosts from 11 major com-
Copyright 2007 ACM 978-1-59593-908-1/07/0010 ...$5.00.
I I
nn
t t
ee
r r
nn
ee
t t
CMTS
Internet
Modem
CM
CM
Phone
DSLAM
Splitter
network
(a)
(b)
Figure 1: A typical setup of (a) cable and (b) DSL access networks
mercial cable and DSL providers in North America and Eu-
2.1
Cable networks
rope. To conduct this study, we developed tools that enable
Cable networks use the cable television infrastructure to con-
us to measure a large number of remote broadband links.
nect home users to the Internet. In these networks, a master
We performed a detailed characterization of an extensive set
headend connects to several regional headends using fiber-
of properties of broadband links. Our analysis was driven
optic cables. Each regional headend serves a set of customers
by three questions:
(up to 2,000 homes). A single coaxial cable, carrying both
1. What are the typical bandwidth, latency, and loss char-
television and data signals, links these customers to their
acteristics of residential broadband links?
headend.
DOCSIS [10] is the most common specification defining
2. How do the characteristics of broadband networks dif-
the interface requirements of cable modems. In DOCSIS,
fer from those of academic or corporate networks?
each cable modem (CM) exchanges data with a cable modem
3. What are the implications of broadband-network prop-
termination system (CMTS) located in a regional headend.
erties for future protocol and system designers?
In the downstream direction, the CMTS broadcasts data
to all cable modems that are connected to it. The cable
Our study reveals important ways in which cable and DSL
modems filter all received data and forward only the bytes
networks differ from the conventional wisdom about the In-
destined for their customer’s host. In the upstream direction,
ternet, accumulated from prior studies of academic networks.
the access channel is time-slotted – a cable modem must
For example, many cable links show high variation in link
first reserve a time slot and wait until the CMTS grants the
bandwidths over short timescales. Packet transmissions over
reservation. When the time slot has been granted, the cable
cable suffer high jitter as a result of cable’s time-slotted ac-
modem can transmit data upstream. Figure 1(a) illustrates
cess policy. DSL links show large last-hop delays and con-
a typical setup of a cable access network.
siderable deployment of active queue management policies
There are several important differences between cable and
such as random early detection (RED). Both cable and DSL
other access networks. First, cable links typically have asym-
ISPs use traffic shaping and deploy massive queues that can
metric bandwidths: their downstream bandwidth is much
delay packets for several hundred milliseconds.
higher than their upstream bandwidth. Second, customers
Our findings have important implications for emerging
cannot use the full raw capacity of their cable links. Instead,
protocols and systems. For instance, the high packet jit-
cable networks use traffic shaping to restrict users from con-
ter in cable links can affect transport protocols that rely
suming more bandwidth than their contract stipulates. Al-
on round-trip time (RTT) measurements to detect conges-
though cable networks currently allow raw data rates of up
tion, such as TCP Vegas [9] and PCP [3]. Further, the large
to 40 Mbps, the contracts of individual customers specify
queue sizes found in cable and DSL ISPs can be detrimental
much lower rates, between 128 Kbps and 10 Mbps. Further,
to real-time applications such as VoIP when they are used
some ISPs over-subscribe their cable access networks. In this
concurrently with bandwidth-intensive applications such as
case, the level of service experienced by customers can vary
BitTorrent [8].
depending on the amount of competing network traffic.
The rest of the paper is organized as follows: Section 2
Finally, cable modems can concatenate multiple upstream
provides an overview of residential cable and DSL networks.
packets into a single transmission, which results in short
Section 3 describes our measurement methodology, including
bursts at high data rates. Thus, the upstream latencies can
the tools we built for gathering data over remote broadband
fluctuate heavily, depending on the allocation policy, and the
network links. Section 4 then presents an in-depth analysis of
amount of signaling and concatenation used by the CMTS.
our data set, characterizing the bandwidth, latency and loss
properties of broadband networks. In Section 5, we discuss
the implications of our findings for the designers of future
2.2
Digital Subscriber Line networks
protocols and systems. Section 6 presents related work, and
DSL access networks use existing telephone wiring to con-
Section 7 summarizes our conclusions.
nect home users to the Internet [13]. Unlike cable customers,
DSL customers do not share their access link. Each cus-
2.
BACKGROUND
tomer’s DSL modem uses a dedicated point-to-point con-
Two types of broadband access networks are popular to-
nection to exchange data with a Digital Subscriber Line
day: cable networks and DSL networks. In this section,
Access Multiplexer (DSLAM). The connection carries both
we present a brief description of their architectures, and we
data and telephone signals, which are encoded in different
point out differences to other access networks, such as cor-
frequencies. On the customer side, a splitter separates the
porate and academic networks.
two signals and forwards the data signal to the DSL mo-
DSL
Cable
BT
Ameritech
BellSouth
PacBell
Qwest
SWBell
Charter
Chello
Comcast
Road Runner
Rogers
Broadband
Charter
Company
AT&T
AT&T
BT Group
AT&T
Qwest
AT&T
UPC
Comcast
TimeWarner
Rogers
Comm.
S+SW
Region
S+SW USA
SE USA
UK
S+SW USA
W USA
USA
Netherlands
USA
USA
Canada
USA
Hosts
113
155
173
158
97
397
114
120
118
301
148
measured
768K,
256K,
768K,
384K,
Offered BWs
768K, 1.5M, 768K, 1.5M,
3M, 5M,
128K, 1M,
2-8M
1.5M, 3M,
1.5M,
1.5M, 3M,
1.5M, 3M,
6M, 8M
5M, 8M
(bps)
3M, 6M
3M, 6M
10M
5M, 6M
6M
7M
6M
6M, 8M
Table 1: Measured hosts: We measured 1,894 broadband hosts from 11 major commercial cable and DSL providers in North
America and Europe.
dem. Figure 1(b) illustrates a typical setup of a DSL access
stream and upstream. This approach requires support from
network.
only one endpoint of an Internet path, but obtaining ac-
There are two important differences between DSL net-
curate measurements is more challenging than with tools
works and other access networks.
First, like cable net-
that require support from both endpoints or with tools that
works, DSL networks often have asymmetric bandwidths;
have been explicitly designed to measure one specific prop-
their downstream bandwidth is higher than their upstream
erty [17, 29, 33].
bandwidth. Second, the maximum data transmission rate
In the remainder of this section, we present our measure-
falls with increasing distance from the DSLAM. To boost
ment methodology in more detail. We describe how we se-
the data rates, DSL relies on advanced signal processing and
lected broadband hosts from different ISPs. We list the types
error correction algorithms, which can lead to high packet
of probe trains used to gather data. And we describe how we
propagation delays. Consequently, the properties of DSL ac-
inferred the characteristics of the broadband links. Finally,
cess links vary depending on the length or the quality of the
we present how we validated the assumptions of our method-
wiring between a modem and its DSLAM.
ology, and we discuss potential concerns and limitations of
our tools.
3.
MEASUREMENT METHODOLOGY
3.1
Selecting residential broadband hosts
The goal of our study was to perform a rigorous characteri-
We used techniques similar to those described in [23] to se-
zation of broadband access networks. For this, we measured
lect 1,894 broadband hosts from 11 major cable and DSL
their link bandwidths, latencies, and loss rates. We also
providers in North America and Europe.
We identified
characterized the properties of broadband queues, including
IP address ranges of popular residential ISPs from IP-to-
queue sizes and packet drop policies. Finally, we examined a
DNS mappings (e.g., BellSouth’s DNS names are adsl-
physical property specific to the cable transmission medium:
*.bellsouth.net), and we scanned for IP addresses re-
the time-slotted access policy of the upstream channel. We
sponding to our probes.
measured the effects of this access policy on latency and
Table 1 summarizes high-level information about the ISPs
jitter. Because broadband access links are asymmetric, we
we measured. Our study includes five out of the top ten
measured the properties of the upstream and downstream
largest broadband ISPs in the U.S. [27]1, the largest cable
directions separately.
provider in Canada [28], the second-largest cable provider in
For our measurements to be generally applicable, the
the Netherlands [50], and the largest DSL provider in the
study needed to be performed at large scale. Previous stud-
U.K. [42]. From each ISP, we chose approximately 100 hosts
ies of broadband [14, 32, 33] used measurement tools that re-
randomly and measured them.
quired cooperation from the remote broadband hosts. Such
Table 1 also shows the bandwidths advertised by ISPs on
a methodology restricts the scale of the measurement study.
their web sites. Although a range of speeds is available, all
Instead, we developed a different methodology for conduct-
advertised bandwidths are lower than 10 Mbps. We took
ing large-scale detailed broadband measurements. Our ap-
advantage of this property by using 10 Mbps probe streams
proach requires minimal cooperation from the remote hosts,
to saturate these broadband links and their routers.
allowing our measurements to scale to thousands of broad-
band links.
3.2
Probe trains to measure broadband links
Remote hosts need to cooperate only in two simple ways.
First, they must respond to ICMP echo request packets with
We used five types of probe packet trains to measure each
ICMP echo responses. Second, they must send TCP re-
broadband link.
Each probe train was sent from well-
set (RST) packets when they receive TCP acknowledgments
connected hosts located in four academic networks (Fig-
(ACK) that do not belong to an open TCP connection. Both
ure 2). The academic networks used are dispersed geograph-
responses are mandated by the corresponding protocols, and
1
previous work shows they are supported widely [23].
During the recent consolidation of the U.S. telecom indus-
try, many large ISPs merged with each other. Four of the
At a high level, our technique is simple – we probe the
eleven ISPs we measured are owned today by AT&T, a single
broadband link with packet trains of different rates, using
company. However, our measurements show that their net-
packets of various types and sizes. We use the responses re-
works have very different characteristics. For the purposes
ceived to infer a broad range of characteristics, both down-
of this study, we treat them as independent ISPs.
Measurement
3.3
Measured broadband link properties
Broadband link
hosts
Our measurements rely on a simplifying assumption: that
the broadband access link is the only bottleneck along the In-
ternet path between our measurement hosts and the remote
Residential
broadband hosts (Figure 2). We validate this assumption in
network
Modem
the next section. This section describes how we measured
the properties of the broadband links based on this assump-
tion.
Link bandwidth: To estimate the allocated downstream
bandwidth, we calculated the fraction of answered probes in
Broadband
the large-TCP flood, which saturates the downstream link
host
only. For example, we estimate the downstream bandwidth
of a link to be 6 Mbps when 60% of packets in our 10 Mbps
Figure 2: Experimental setup
large-TCP flood are answered. We used the same technique
to estimate upstream bandwidths from the symmetric large-
ICMP flood. The behavior of the large-ICMP flood is driven
ically – three in North America (in the south, northwest, and
by the bandwidth of the slower link, which for cable and DSL
northeast) and one in Europe. We also probed the last-hop
is the upstream link.
router before each broadband link. We used traceroute to
Our techniques yield incorrect estimates in the presence of
discover these routers.
cross-traffic. We use IPID-based techniques described in [23]
We sent probe trains at different rates. We refer to our
to identify and eliminate all measurement probes affected by
high-rate probe trains as floods, and to our low-rate probe
cross-traffic.
trains as trickles. All floods were sent at 10 Mbps to saturate
Packet latencies and jitter: We characterized three types
the broadband links. Consequently, packet floods measure
of packet delays and their variation (jitter) for each link:
the network under congestion. By contrast, all packet trick-
queueing delay, propagation delay, and transmission delay.
les were sent at a rate of a few tens of Kbps, so they char-
We estimated the maximum possible queueing delays (or
acterize the broadband network under normal operational
queue lengths) by calculating the variation in RTTs of pack-
conditions.
ets in our floods. To determine downstream queue lengths,
We limited the packet floods to at most 10 s, whereas we
we calculated the difference between the 95th percentile
allowed trickles to last from several hours to several days. To
highest RTTs and minimum RTTs of packets in the large-
capture diurnal variations of network properties, we repeated
TCP flood, which overflows only the downstream router
the floods every half hour for one week.
queues.
A similar calculation for the large-ICMP flood,
Asymmetric large-TCP flood: We sent large (1,488-
which overflows queues in both directions, estimated the sum
byte2) TCP ACK packets, and the remote host responded
of downstream and upstream queue lengths. We subtracted
with small (∼40-byte) TCP RST packets. The ACK pack-
the downstream queue length from this estimate to obtain
ets saturated the downstream links and router queues, but
the length of the upstream queue.
the responses, being smaller and fewer, did not saturate the
To study propagation delays of broadband links, we esti-
upstream links or queues.
mated their last-hop delays. We calculated last-hop delay
Symmetric large-ICMP flood: We sent large (1,488-
as the difference between the latencies of small-TCP trickle
byte) ICMP echo request (PING) packets, and the remote
probes to the broadband host and to its last-hop router. By
host responded with ICMP echo response packets of the same
comparing the last-hop delays for different packet sizes, we
size. This packet train saturated the links and router queues
were able to infer the transmission delays in broadband links.
in both downstream and upstream directions.
We discuss transmission delays in more detail in Section 4.2.
Symmetric small-TCP flood: We sent small (40-byte to
Packet loss: We estimated typical packet loss rates in
100-byte) TCP ACK packets, and the remote host responded
broadband networks by calculating the fraction of lost pack-
with small (∼40-byte) TCP RST packets. Like the symmet-
ets in the small-TCP trickle. To detect packet loss due to
ric large-probe flood, this packet train saturated the network
queue management policies, such as RED, we examined how
in both downstream and upstream directions but with much
the loss rate varies with the latencies of the packets. We
smaller packets.
discuss the details of RED detection in Section 4.3.
Symmetric large-ICMP trickle: We sent large (1, 488-
byte) ICMP echo request packets spaced at large intervals
3.4
Validating our assumptions
randomly chosen between 10 ms and 30 ms, and the remote
Next, we discuss five important concerns about our method-
host responded with ICMP echo response packets of the same
ology:
size. Unlike the above probe trains, this packet train did not
saturate the downstream or upstream links.
1. To be accurate, our probes must traverse the entire In-
Symmetric small-TCP trickle: We sent small (40-byte)
ternet path reaching the broadband host and not be
TCP ACK packets spaced at large random intervals between
answered by an intermediate router. Do our measure-
10 ms and 30 ms, and the remote host responded with small
ments reflect accurately the properties of broadband ac-
(∼ 40-byte) TCP RST packets. This packet train did not
cess links?
saturate the downstream or upstream links.
2. We assumed that the broadband links are the bottle-
necks in the measured Internet paths. How often are
2 We used 1,488-byte probes because some DSL links running
broadband links the bottlenecks along the measured In-
PPPoE or PPPoA have an MTU of less than 1,500 bytes.
ternet paths?
1
1
1
t
s 0.8
t
s 0.8
Last-hop
t
s 0.8
Last-hop
os
Broadband hosts
os
routers
Broadband hosts
os
routers
h 0.6
h 0.6
h 0.6
of
Last-hop
of
of
0.4
routers
Broadband hosts
t
i
on
t
i
on 0.4
t
i
on 0.4
r
ac
r
ac
r
ac
F 0.2
F 0.2
F 0.2
0
0
0
0
2,000
4,000
6,000
8,000
10,000
12,000
0
100
200
300
400
500
0%
20%
40%
60%
80%
100%
Available bandwidth (Kbps)
Increase in RTT (milliseconds)
Packet loss rate
Figure 3: The broadband link is the bottleneck: Comparison of the paths to the residential broadband hosts and their
corresponding last-hop routers. The former include the broadband link, while the latter do not. The two sets of paths have
very different characteristics, which validates our assumption that broadband links are the bottlenecks along the Internet paths
to broadband hosts.
3. We assumed broadband hosts respond to all probes
estimates, we used our access to the end hosts to measure
without any delay. In practice, end hosts could drop
the upstream and downstream queue lengths separately and
or rate limit their responses. How often do broadband
accurately. The measurements matched the estimated queue
hosts delay or drop response packets?
lengths very well. The close match suggests that both our
bandwidth and queue length measurements are accurate.
4. Our probes can be interpreted as port scans or attacks.
What are the best practices we used in our measure-
3.4.2
How often are broadband links the bottlenecks
ments?
along the measured Internet paths?
5. Large-scale Internet studies suffer from limitations and
Our methodology assumes the broadband link is the bottle-
shortcomings. What are some of the limitations of our
neck on the Internet path measured. Because our probes
study?
are sent from well-connected academic hosts, the broadband
links are likely to be the bottlenecks in these paths. To
3.4.1
Do our measurements reflect accurately the
validate this assumption, we sent a large-TCP flood probe
properties of broadband access links?
train to the broadband host and another train to its last-hop
We ran controlled experiments using five broadband hosts
router. Comparing these two probe trains revealed that the
(two cable and three DSL) under our control, located in
broadband links are in fact the bottlenecks.
North America and Europe. These experiments were per-
Figure 3 compares the available bandwidth, the RTT in-
formed on a small scale because they required end-host co-
creases, and the packet loss rate of the two traces for 1,173
operation. Although we hoped to recruit more volunteers,
randomly selected broadband hosts. Most paths to the last-
the effort required to setup our experiments made it difficult
hop routers achieved the full 10 Mbps throughput, experienc-
to convince users to perform them. Our experiments require
ing almost no losses or RTT fluctuations. By contrast, the
root access and manual changes to the modems’ firewalls.
paths including the broadband link had much lower through-
First, we checked whether the probe packets were being
put, considerable RTT increases, and high packet loss. This
sent over the broadband link or whether they were being
suggests that these variations are caused by the last hop (i.e.,
answered by a router in the middle of the network. We
the broadband link).
found that in all cases the probes were being responded to
by the NAT-enabled modems in the customers’ premises.
3.4.3
How often do broadband hosts delay or drop
By configuring the modems to forward any arriving probe
response packets?
packets to end hosts, we were able to receive the probes at
Our methodology assumes broadband hosts respond to
our end hosts (Figure 2). Note that the probes must cross
probes without any delay.
Several factors could prevent
the broadband link to reach the modems.
hosts from responding to some or all of our probes. For
Second, we checked whether the NAT-enabled modems
example, a firewall may block certain types of probes, such
affected the measurements by delaying or rate-limiting their
as PINGs. Some routers add a delay between the arrival of
responses. We gathered two traces for each link: one when
a probe and the departure of the response [21]. Also, a host
the modem responded to the probes, and another when the
with limited processing power might delay or drop packets
modem forwarded all probes to the broadband hosts. We
arriving at high rates.
configured the broadband hosts to respond to the probes
We removed all hosts that did not respond to our probes.
without any delay (less than 100 µs) or rate-limiting. We
We also removed the broadband hosts that rate-limited their
compared the two traces with respect to latencies and losses
probe responses. We identified such hosts by checking for
of probes and responses. The two traces matched closely in
large loss episodes occurring periodically.
all cases, suggesting that the modems do not adversely affect
Finally, we performed the following experiment to check
our measurements.
whether our probe trains were too aggressive for the process-
Finally, we verified the accuracy of our bandwidth and
ing power of some hosts. We sent probe trains at 10 Mbps
queue length measurements. We compared the measured
but with varying packet sizes. Although the trains consumed
bandwidths of the access links with the rate speeds ad-
the same bandwidth, their packet sending rates were differ-
vertised by their ISPs. We found that these bandwidths
ent. We checked whether hosts experienced higher losses at
matched very closely – the average difference in downstream
faster sending rates. A higher loss rate suggests that an end
bandwidths was less than 3%. To validate our queue length
host cannot process packets at fast rates. We checked how
1
1
Qwest
Charter
0.8
SWBell
0.8
BT Broadband
Comcast
t
s
t
s
os
os
h 0.6
PacBell
h 0.6
of
BellSouth
of
Chello
t
i
on 0.4
t
i
on 0.4
r
ac
r
ac
Rogers
F
F
Ameritech
0.2
0.2
Road Runner
0
0
0
1,000
2,000
3,000
4,000
5,000
6,000
0
2,000
4,000
6,000
8,000
10,000
Allocated link bandwidth (Kbps)
Allocated link bandwidth (Kbps)
(a) DSL (downstream)
(b) Cable (downstream)
1
1
BellSouth
Qwest
0.8
0.8
Ameritech
t
s
t
s
Rogers
os
os
Comcast
h 0.6
h 0.6
of
BT Broadband
PacBell
of
Charter
Road
t
i
on 0.4
t
i
on 0.4
Runner
r
ac
r
ac
F
F
Chello
SWBell
0.2
0.2
0
0
0
500
1,000
1,500
2,000
0
500
1,000
1,500
2,000
2,500
3,000
Allocated link bandwidth (Kbps)
Allocated link bandwidth (Kbps)
(c) DSL (upstream)
(d) Cable (upstream)
Figure 4: Allocated downstream and upstream link bandwidths: Most ISPs offer upstream bandwidths of 500 Kbps
or less, even when the downstream bandwidths exceed 5 Mbps.
losses vary with packet sending rates for all broadband hosts
4.
CHARACTERIZING BROADBAND
in our study. The loss rates remained constant for over 99%
LINKS
of the hosts in our study, suggesting that the end hosts have
sufficient processing power to handle our probing rates.
In this section, we analyze the data gathered from sending
probe packet trains to a large number of residential broad-
band hosts in several major ISPs (see Table 1). We exam-
3.4.4
What are the best practices we adopted?
ine three important characteristics of broadband networks,
namely link bandwidths, packet latencies, and packet loss.
Performing active measurements on the Internet raises im-
Analyzing these properties is important because they af-
portant usage concerns. Although it is difficult to address
fect the performance of protocols and systems running over
and eliminate all such concerns, we adopted a set of precau-
broadband.
tions to mitigate these concerns. We restricted our high rate
probe trains to no more than 10 s each. We also embedded
4.1
Allocated link bandwidth
a custom message in each of our probe packets, which de-
scribed the experiment and included a contact email address.
Allocated link bandwidth refers to the bandwidth reserved
To date, we have not received any complaints.
by a provider to a single broadband user. In cable networks,
Another cause for concern was that users with a per-byte
allocated link bandwidth is the portion of the shared link’s
payment model end up paying for our unsolicited traffic. To
capacity assigned to an individual user, whereas in DSL net-
mitigate this concern, we only measured hosts in ISPs that
works it is the ISP’s cap on a user’s traffic rate. Characteriz-
offer flat-rate payment plans, and we restricted the total
ing allocated link bandwidths in broadband networks helps
amount of data sent to any single broadband host over our
to predict the maximum throughput any transport proto-
entire study.
col (such as TCP Reno or TCP Vegas) or application (such
as BitTorrent) can achieve. As described in Section 3.3, our
probe streams measured allocated bandwidths by saturating
3.4.5
What are some of the limitations of our study?
the broadband links.
Two important limitations affect our measurements. First,
we studied only major cable and DSL ISPs in North Amer-
4.1.1
What are the allocated link bandwidths?
ica and Europe. Our conclusions are unlikely to generalize
Figures 4(a) and (b) show the cumulative distributions of
to high-speed fiber-based broadband ISPs, such as those in
downstream link bandwidths for the different DSL and ca-
Japan or South Korea [12]. Second, we removed all hosts
ble ISPs. For many ISPs, the distributions jump sharply
that did not respond to our probes or that were rate-limited,
at distinct bandwidth levels, such as 256 Kbps, 384 Kbps,
which could introduce some unknown bias.
512 Kbps, and 1 Mbps. Only two cable ISPs (Rogers in
1
7,000
s
)
DSL
p
b 6,000
0.8
Unstable (Rogers cable host)
(
K
t
s
t
h 5,000
os
i
d
h 0.6
w
d 4,000
of
an
b
t
i
on
k 3,000
0.4
r
ac
l
i
n
F
2,000
Cable
e
d
0.2
at
1,000
l
l
oc
A
Stable (PacBell DSL host)
0
0
0
2
4
6
8
10
12
14
0
1
2
3
4
5
6
7
8
9
10
Ratio (downstream bandwidth / upstream bandwidth)
Time (seconds)
Figure 5:
The ratio of downstream to upstream
Figure 6: Stable and unstable link bandwidths: The
link bandwidths: The gap between downstream and up-
allocated link bandwidth is stable for the PacBell DSL host.
stream bandwidths is much wider for cable networks than
For the Rogers cable host, the access link bandwidth varies
for DSL networks.
greatly over time.
Canada and Comcast in the United States) allocate band-
t
h
1
widths distributed along a continuous spectrum.
i
d
w
d
By comparing these measured allocated bandwidths to the
an
0.8
advertised link speeds from Table 1, we can confirm and
l
e
b
quantify some commonly held opinions. We find that most
0.6
t
ab
s
DSL ISPs have bandwidth rates corresponding to those ad-
i
t
h
0.4
vertised. By contrast, major cable ISPs, such as Comcast
t
s
w
os
and Rogers, show rates different from those advertised (both
0.2
h
higher and lower). We consider that this discrepancy is due
of
0
to the nature of the two technologies – cable is a shared
t
i
on
r
ac
ell
est
ell
ast
medium, whereas DSL is not. Our data also shows that
B
ritech
w
Q
hello
harter
C
ogers
unner
F
ellSouth
SW
C
omc
R
many cable ISPs have significantly higher downstream band-
me
C
A
B
roadband PacB
B
oad R
T
R
widths than DSL.
B
Figures 4(c) and (d) show the cumulative distributions of
Figure 7: Fraction of hosts with ‘stable’ downstream
upstream link bandwidths. Upstream bandwidths are strik-
link bandwidths: Most DSL links show stable bandwidths,
ingly different from downstream bandwidths – with the ex-
whereas most cable links do not. The results for upstream
ception of a few ISPs, most upstream bandwidths are lower
bandwidth are similar.
than 500 Kbps, even when their downstream bandwidths
exceed 5 Mbps. To examine this difference, we plotted the
ratio of downstream to upstream link bandwidths in Fig-
ure 5. Most DSL hosts have much smaller ratios than ca-
the PacBell link shows stable bandwidth, the Rogers link
ble hosts, because compared to cable, DSL hosts have lower
weaves above and below its average bandwidth of 3 Mbps.
downstream but similar upstream bandwidths. For over half
Figure 7 shows the fraction of DSL and cable links that
of the cable hosts, the downstream bandwidths exceed up-
exhibit stable bandwidths in the downstream direction. We
stream bandwidths by a factor of more than 10.
classified a link as stable if at least 90% of the 100 ms in-
The highly asymmetric nature of bandwidths does not
tervals show a bandwidth estimate within 10% of the aver-
align well with the requirements of emerging peer-to-peer
age bandwidth. Although most DSL ISPs show stable link
systems [8, 24], whose workloads tend to be symmetric. De-
bandwidths, we found that most cable ISPs have bandwidths
spite all the excitement surrounding user-driven content gen-
that vary significantly even within the short 10 s duration of
eration and distribution, residential networks continue to be
our probes. We also found that upstream bandwidths have
predominantly optimized for client-server workloads.
unstable short-term characteristics. We have omitted these
results because of space constraints.
4.1.2
How stable are the allocated link bandwidths?
This large short-term variation in cable bandwidths poses
Next, we studied the short-term and long-term stability of
new challenges to transport protocol designers. Tradition-
link bandwidths. Understanding the stability of link proper-
ally, transport protocols have been developed to achieve sta-
ties is useful for designing network protocols that can quickly
ble throughput and to avoid reacting to short-term events
adapt to changing link conditions.
(on timescales less than one RTT) [19]. However, when run-
We examined the stability of the allocated link bandwidths
ning in a cable network environment, protocols need to ad-
over the 10 s duration of our packet floods. For this, we
just quickly to rapidly changing link conditions. Slow react-
divided the 10 s into 100 ms intervals (the RTT of a typi-
ing protocols might not achieve good throughput in cable
cal Internet path), we estimated the bandwidth within each
networks.
interval, and we compared the different estimates across in-
We now turn our focus to the long-term diurnal stability
tervals. Figure 6 shows how bandwidths for a PacBell link
of link bandwidths. We took measurements of the upstream
(DSL) and a Rogers link (cable) vary over time. Whereas
and downstream bandwidths every half an hour for one week
100%
3,000
t
h
i
d
s
)
p
w
b
d
BT Broadband
2,500
80%
an
(
K
t
h
ax b
i
d 2,000
w
m
60%
d
/
t
h
an 1,500
i
d
Rogers
b
k
w
40%
d
l
i
n 1,000
an
e
d
b
d
at
20%
500
i
e
ve
l
l
oc
A
c
h
A
0
0%
Tue
Wed
Thu
Fri
Sat
Sun
Mon
0
1
2
3
4
5
6
7
8
9
10
0:00
0:00
0:00
0:00
0:00
0:00
0:00
Local time
Time (seconds)
Figure 8: Long-term link bandwidth stability: Whereas
Figure 9: Traffic-shaped downstream: The bandwidth
BT Broadband has stable bandwidths over time, Rogers’s
of this link is initially 2.5 Mbps, but it drops to 1.5 Mbps
allocated link bandwidths show diurnal patterns.
after one second.
from 70 randomly chosen hosts from each ISP.3 Figure 8
shows an example link from Ameritech DSL, whose band-
shows the diurnal variation in bandwidths for one DSL ISP
width drops from 2.5 Mbps to 1.5 Mbps (its long-term rate)
(BT Broadband) and one cable ISP (Rogers). Each curve
after the first second.
shows the bandwidth variation averaged across all measured
We found similar downstream traffic-shaping techniques
links within one ISP. To account for links with different
used by three ISPs – Ameritech, Comcast, and Chello. 11%
bandwidths, we normalize each link’s bandwidth by using
of the Ameritech links, 26% of the Comcast links, and 67%
the maximum measured bandwidth of that link during the
of the Chello links provide an initial burst of bandwidth to
entire measurement period.
speed up short transfers. The burst rates are typically more
We found that most ISPs have high long-term stability
than 1 Mbps above the long-term bandwidth. However, in
(not shown). As the curve for BT Broadband illustrates,
many cases, we were unable to quantify precisely the burst
their bandwidths do not vary with the time of the day. By
rates because they exceeded the rate of our probe train. In
contrast, a small number of ISPs, such as Rogers, show a
the upstream direction, we found no evidence of traffic shap-
clear diurnal trend in link bandwidths. Rogers’s end hosts
ing or bandwidth throttling of our probe stream. The short
see significantly lower bandwidths (almost a 25% reduction)
duration of our probe trains (10 s) could have prevented us
in the evening (between 4 PM and 7 PM) than in the early
from detecting upstream traffic shaping.
morning (between 1 AM and 5 AM). In the upstream di-
History-based bandwidth prediction is a popular technique
rection, we find stable bandwidths (not shown) for all ISPs,
used in several transport protocols [3, 9, 19] and content dis-
including Rogers. These findings seem to contradict the pop-
tribution systems [25]. Although our traffic-shaping anal-
ular idea that competing traffic affects the bandwidths of
ysis is preliminary, it suggests that using past bandwidth
broadband hosts. For most ISPs, we found little evidence
estimates to predict future bandwidth conditions might not
that competing traffic affects link bandwidths during the
work well over broadband links.
day.
4.2
Packet latencies
4.1.3
Is there evidence of traffic shaping?
We analyzed each of the three components of packet laten-
Traffic shaping is likely to be one of the factors leading to the
cies: propagation delays, transmission delays, and queueing
bandwidth instability encountered in broadband networks.
delays.
Some ISPs allow an initial burst of bandwidth that is often
many times greater than the advertised bandwidth. For ex-
4.2.1
Do broadband links have large propagation
ample, Comcast’s PowerBoost feature [15] doubles the cus-
delays?
tomer’s allocated bandwidth for a short time. This provi-
A link’s propagation delay is the time elapsed between send-
sioning reduces the download times of relatively small files,
ing a bit at one end and receiving it at the other end. On one
such as MP3s. Other ISPs throttle the bandwidth allocated
hand, broadband propagation delays could be short because
to long running transfers to discourage the heavy hitters
the links themselves are short. On the other hand, sophisti-
from consuming a disproportionate share of the bandwidth.
cated signal processing and error correction algorithms could
Because our probe floods were limited to 10 s, we could
increase broadband propagation delays.
only detect the traffic shaping associated with short-duration
Our methodology prevents us from directly measuring the
flows. To do this, we performed the following experiment.
propagation delay of a broadband access link. Instead, we
We used our packet streams to compute the allocated link
were able to estimate the round-trip delay of the last-hop of
bandwidth of each 100 ms interval. To detect the presence
the path between our measurement hosts and the broadband
of traffic shaping, we checked for a consistent and signifi-
hosts. This last-hop delay roughly approximates the sum of
cant drop in bandwidth after some initial period. Figure 9
downstream and upstream broadband propagation delays.
3 To minimize DHCP effects, we discarded any host that went
To do this, we sent small-TCP trickle probes to both the
offline (i.e., did not respond to probes) during this period.
broadband host and its last-hop router. The trickle consisted
We also excluded measurements when we detected cross-
of several hundred widely spaced small probes and their re-
traffic.
sponses. We calculated the last-hop RTT by subtracting the
1
1
DSL
jitter
Cable
0.8
0.8
t
s
Cable
t
s
DSL
os
last hop
os
h 0.6
h 0.6
of
Cable
of
jitter
t
i
on 0.4
t
i
on 0.4
r
ac
r
ac
F
F
0.2
0.2
DSL
last hop
0
0
0
5
10
15
20
25
30
0
5
10
15
20
25
30
Latency (milliseconds)
Minimum RTT distance from the host to the router (milliseconds)
Figure 10: Last-hop delay and jitter in cable and DSL
Figure 11: Difference in transmission delays between
networks: DSL shows higher last-hop RTTs than cable,
large and small packets: DSL shows longer transmission
while cable exhibits higher jitter than DSL.
delays than cable.
minimum RTT to the last-hop router from the minimum
in Figure 10. By contrast, the transmission delays for cable
RTT to the broadband host. We used the minimum RTT
are surprisingly low – 99% of hosts show an increase of less
estimates to avoid transient jitter as a result of queueing at
than 1 ms to send an extra 1,388 bytes.
intermediate routers.
We believe that the time-slotted nature of cable links is re-
Figure 10 shows our results for last-hop RTTs for cable
sponsible for these short transmission delays. All our probes,
and DSL networks. DSL hosts exhibit considerably higher
both large and small, experience similar waiting times for a
propagation delays than cable hosts. 75% of all DSL hosts
time slot. When a slot has been granted, packets are trans-
have last-hop delays larger than 10 ms, while 15% have prop-
mitted at the full link speed (10.24 Mbps according to the
agation delays larger than 20 ms. These delays are surprising
DOCSIS 1.0 specification). This matches our data very well;
because many last-hop routers are located in the same city
our measured transmission delays correspond to an upstream
as their end hosts.4
link speed of 11 Mbps.
Figure 10 also shows the jitter in our latency measure-
Next, we examined transmission delays under high net-
ments. We used the RTTs of the small-TCP trickle to esti-
work load. In this case, packets have to wait longer to re-
mate the jitter of the broadband link. We calculated jitter
serve a time slot. When the reservation is granted, multiple
by subtracting the 10th percentile RTT from the 90th per-
waiting packets can be concatenated (see Section 2.1) and
centile highest RTT. Compared to cable, DSL links have
sent in a single burst. Although concatenation reduces the
higher last-hop delays but lower jitter. We believe that the
overhead of scheduling many small packets, such as TCP
characteristics of the upstream cable links are responsible for
ACKs, it introduces a systematic jitter, which we refer to as
these differences. We examine this next.
the concatenation jitter.
We used the small-TCP flood to examine the effects of
4.2.2
How do cable’s time-slotted policies affect
concatenation because it saturates the upstream link with a
transmission delays?
large number of small packets, which are well suited for con-
Transmission delay refers to time elapsed between a router
catenation. We clustered probe responses received in very
starting to transmit a packet and ending its transmission.
close succession (separated by less than 100 µs) as part of
It is usually calculated by dividing the packet length by the
a single bursty transmission, and we calculated the number
link bandwidth. However, cable links use a reservation policy
of packets in the largest cluster. Because there is no known
to transmit packets in the upstream direction. This policy
concatenation feature for DSL, we expected these links to
can cause additional delays to a packet’s transmission. We
show only minimal burst sizes.
examined the effects of such transmission policies under both
Figure 12(a) shows the extent of packet concatenation in
low and high network loads.
DSL and cable ISPs. As expected, DSL links show only very
First, we studied transmission delays under low network
short bursts, whereas 50% of cable links can concatenate
loads. We used the large-ICMP trickle to calculate the last-
19 packets or more in a single burst. We used the link’s
hop delays, similar to the experiment conducted in the previ-
speed and the number of packets in a burst to estimate a
ous section. We compared these last-hop large-packet delays
lower bound on the amount of concatenation jitter when the
to the last-hop small-packet delays measured in the earlier
link is saturated. Figure 12(b) shows the results. Whereas
experiment. The differences in the last-hop delays between
the mean concatenation jitter for cable networks is about
large (1
5 ms, many links experience 10 ms or more of jitter due to
,488-byte) and small (100-byte) packets are mostly
due to the additional transmission delays incurred by send-
concatenation.
ing larger packets.
In cable networks, the concatenation jitter under high net-
Figure 11 shows the difference in transmission delays be-
work load can be higher than the end-to-end jitter over the
tween large and small packets for cable and DSL hosts. We
entire path under normal load (shown in Figure 10). The
found that the transmission delays for DSL are large, on the
presence of high jitter in cable networks has important con-
same order of magnitude as their propagation delays, shown
sequences for protocols such as TCP Vegas [9] and PCP [3],
which interpret changes in RTT as a sign of incipient con-
4 We inferred the locations of hosts and routers from their
gestion. High jitter could cause these protocols to enter con-
DNS names as suggested in [47].
gestion avoidance too early, leading to poor performance.
1
1
Cable
DSL
DSL
0.8
0.8
Cable
t
s
t
s
os
os
h 0.6
h 0.6
of
of
t
i
on
0.4
t
i
on 0.4
r
ac
r
ac
F
F
0.2
0.2
0
0
0
5
10
15
20
25
30
0
5
10
15
20
Number of packets
Jitter (milliseconds)
(a) Maximum number of packets per burst
(b) Lower bound estimate of concatenation jitter
Figure 12: Cable links show high RTT variation: In addition to a high level of basic jitter, cable modems can send small
packets in a single burst and thus cause additional jitter.
1
1
PacBell
Chello
0.8
0.8
SWBell
t
s
Rogers
Charter
Qwest
t
s
Road Runner
os
BellSouth
os
h 0.6
h 0.6
of
of
Comcast
BT Broadband
t
i
on 0.4
t
i
on 0.4
r
ac
r
ac
F
F
0.2
0.2
Ameritech
0
0
0
100
200
300
400
500
0
100
200
300
400
500
Queue length (milliseconds)
Queue length (milliseconds)
(a) DSL (downstream)
(b) Cable (downstream)
1
1
SWBell
Comcast
0.8
PacBell
0.8
t
s
Qwest
t
s
Charter
os
os
h 0.6
h 0.6
of
BT Broadband
of
Chello
BellSouth
t
i
on 0.4
t
i
on 0.4
r
ac
r
ac
F
Road Runner
Ameritech
F
0.2
0.2
Rogers
0
0
0
500
1,000
1,500
2,000
2,500
0
1,000
2,000
3,000
4,000
5,000
Queue length (milliseconds)
Queue length (milliseconds)
(c) DSL (upstream)
(d) Cable (upstream)
Figure 13: Downstream and upstream queue length in milliseconds: Some downstream queue lengths follow the
recommendation for voice calls (150 ms), but most are significantly longer. The upstream queue length can be massive,
especially for cable links.
4.2.3
How large are broadband queueing delays?
stream link. We calculated the difference between the mini-
mum RTT and the 95th percentile highest RTT. To estimate
Sizing router queues is a popular area of research (e.g., [4]).
upstream queue lengths, we first measured the difference
A common rule of thumb (attributed to [49]) suggests that
between the minimum RTT and the 95th percentile high-
router queues’ lengths should be equal to the RTT of an av-
est RTTs of large-ICMP flood probe trains. This difference
erage flow through the link. Larger queues lead to needlessly
corresponds to the sum of downstream and upstream queue
high queueing delays in the network. We investigated how
lengths. We then subtracted the estimate of the downstream
well this conventional wisdom holds in broadband environ-
queue length to obtain the length of the upstream queue.
ments.
Figures 13(a) and 13(b) show the cumulative distributions
We measured queue lengths in milliseconds by calculating
of downstream queue lengths for different cable and DSL
the RTT variation of our probe streams’ packets. To esti-
providers. Across most cable ISPs and two DSL ISPs (Pac-
mate downstream queue lengths, we used large-TCP flood
Bell and SWBell), the curves show a sharp rise at 130 ms.
probe trains, which saturate the downstream but not the up-
This value is consistent with that recommended by the ITU
1
3%
0.8
Ameritech
t
s
Cable
os
DSL
2%
h 0.6
e
of
Chello
s
r
at
t
i
on 0.4
os
L
r
ac
F
1%
0.2
0
0%
0.0%
0.2%
0.4%
0.6%
0.8%
1.0%
Sun
Mon
Tue
Wed
Thu
Fri
Sat
Sun
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
Round-trip loss rate
Local time
Figure 14: Observed round-trip loss rate for residen-
Figure 15: Packet loss over time: The loss rate is gener-
tial broadband paths: DSL and cable paths show similar
ally low and shows heavy diurnal variations with intermit-
loss rates. 95% of all DSL and cable hosts have loss rates of
tent spikes. Note that this graph includes both upstream
less than 1%.
and downstream losses; the time axis shows local time (EST
for Ameritech and CET for Chello).
G.114 standard for maximum end-to-end latency in a net-
work running interactive traffic – 150 ms. Nevertheless, these
loss rate is below 1% for more than 95% of all DSL and ca-
queue lengths are significantly higher than a typical flow’s
ble paths. Overall, we found that the packet loss rates for
average delay, which ranges between 50 and 75 ms within
broadband access networks are similar to those observed in
North America or Europe. By contrast, we observed queue-
academic network environments [11, 40].
ing delays of up to 2 s for a significant number of Comcast
We also examined how loss rates varied over the course of
and Qwest hosts and up to 6 s for some BT Broadband
the week. Figure 15 shows our measurements for two typical
hosts (not shown). Our findings show diverse queue con-
providers: a DSL ISP (Ameritech) and a cable ISP (Chello).
figurations for broadband links, with most hosts exhibiting
The horizontal axis shows the local time for the ISPs. The
queue lengths significantly higher than 130 ms.
loss rates shown along the vertical axis are averaged over
Figures 13(c) and 13(d) show the cumulative distributions
intervals of 120 minutes. We found that loss rates exhibit
of upstream queue lengths for the different cable and DSL
diurnal patterns with occasional spikes. Both ISPs follow
providers. Compared with downstream queues, the lengths
similar diurnal patterns, showing lower loss rates in the early
of upstream queues are very large. Most DSL links exhibit
morning than in the evening.
queues of 600 ms or higher, and many cable links allow their
upstream queues to grow to several seconds. Although some
4.3.2
Do ISPs use active queue management?
of the upstream queues’ build-up results from the low up-
When packets are sent very quickly, they begin to fill up
stream link bandwidths, the excessive lengths will negatively
queues, and the routers must eventually drop some of the
affect interactive traffic like VoIP whenever users upload con-
packets. The most common queue management policy is
tent, such as when using BitTorrent.
tail-drop – i.e., all packets arriving after the queue is full are
discarded. More active queue management policies, such
4.3
Packet loss
as RED [20], proactively drop packets using probabilistic
In this section, we characterize packet loss in residential
schemes when the queue starts to fill up but before the queue
broadband networks. We contrast our results with the com-
is full. Active queue management has been extensively stud-
monly held idea that broadband networks have high packet
ied, but relatively little is known about the extent to which
loss rates. Our tools cannot measure the access links’ loss
it is deployed in practice.
rates. Instead, we examined the packet loss rates of the In-
We performed the following experiment to infer whether
ternet paths between our well-connected measurement hosts
the broadband ISPs are using active queue management poli-
and the broadband hosts. Because the broadband access
cies. We used the small-TCP flood to overflow both down-
links are part of these Internet paths, our measured loss rates
stream and upstream links, and we used IPIDs to distin-
provide an upper bound on the broadband links’ loss rates.
guish between losses occurring upstream and those occurring
downstream [36]. For each successfully received response, we
4.3.1
Do broadband links see high packet loss?
recorded the RTT, and we calculated the average loss rate
We used the small-TCP trickle probe trains to calculate the
over a sliding window of 40 packets. We examined the cor-
loss rates along the round-trip paths to remote broadband
relation between the loss rates and the corresponding RTTs.
hosts. We sent widely spaced trickle probes at a very low
On the basis of this correlation, we can infer whether routers
rate for a week, and we measured the fraction of probes for
use tail-drop or more active queue management policies. A
which the broadband hosts did not respond. This includes
tail-drop policy will result in a steep increase in loss rate
losses on both the upstream and the downstream paths, and
when the queue is full (i.e., for a large RTT value); if an
it measures the loss rate under normal operating conditions
active queue management policy such as RED is used, then
of the network. Note that the loss rate we measured might
the loss rate will increase proportionally to the RTT after a
differ from the loss rate that application traffic (e.g., TCP
certain threshold.
flows) saturating broadband links would suffer.
Figure 16 shows how the loss rates increase with the RTT
Figure 14 presents our results. We found that both ca-
for two broadband hosts, one in PacBell and one in SWBell.
ble and DSL have remarkably low packet loss rates. The
For the PacBell host, the loss rate increases steeply around
100%
All ISPs deploy queues that are several times larger
than their bandwidth-delay products. Whereas downstream
80%
queues can delay packets by more than 100 ms, the upstream
t
e
queueing delays can exceed several hundreds of milliseconds
s
s
r
a
60%
l
o
Active queue management (probably RED)
and, at times, a few seconds.
m
(SWBell)
40%
Packet loss: Both DSL and cable ISPs exhibit surprisingly
s
t
r
e
a
p
U
low packet loss. We also found that many DSL hosts use ac-
20%
tive queue management policies (e.g., RED) when dropping
Tail-drop
packets.
(PacBell)
0%
0
100
200
300
400
500
600
700
800
900
Round-trip time (milliseconds)
5.
IMPLICATIONS OF OUR FINDINGS
Figure 16: Tail-drop and active queue management:
We consider that our observations about broadband net-
When a tail-drop queue overflows, the loss rate increases
works’ characteristics can help researchers to understand
sharply. If the loss rate increases proportionally to the queue
how well existing protocols and systems work in the com-
length after a threshold, then this suggests that active queue
mercial Internet. Our findings offer useful insights for the
management (probably RED) is being used.
designers of future applications. To illustrate this, we briefly
discuss the potential implications of our measurements for
three popular Internet-scale systems.
an RTT of 850 ms, which suggests that a tail-drop queue
Transport Control Protocols: Our bandwidth and la-
is used. The loss rate for the SWBell host shows a different
tency findings have several implications for transport pro-
trend; after 500 ms, it increases almost linearly with the RTT
tocol designs.
For example, protocols such as TCP Ve-
before stabilizing at around 85%. This behavior matches the
gas [9] and PCP [3] use RTT measurements to detect incip-
description of the RED active queue management policy.
ient congestion. In the presence of the high jitter found in
To quantify the extent of RED deployment in broadband
our measurements, this mechanism might trigger congestion
networks, we tested whether the increases in RTT and loss
avoidance too early. Bandwidth-probing techniques, such as
rates are strongly correlated. If the correlation coefficient is
packet-pair [31], could return incorrect results in the pres-
high (≥ 0
ence of traffic shaping or packet concatenation. This could
.9) beyond a threshold loss rate of 5%, we conclude
that the link may be using RED as its drop policy. We
be detrimental to transport protocols that rely on probing
did not calculate the correlation coefficient for low loss rates
to adjust their transfer rates, such as PCP.
(below 5%) because these loses might be sporadic and not
Network coordinate and location systems: Many IP-
representative of the broadband router’s queue policy.
to-geolocation mapping tools [22, 52] use latency measure-
We found that 26
ments to determine a host’s location. The large propagation
.2% of the DSL hosts show a RED-style
drop policy on their upstream queues. The three providers
delays and high jitter found in broadband networks are likely
owned by AT&T (i.e., Ameritech, BellSouth, and PacBell)
to seriously interfere with the accuracy of these systems.
exhibit deployment rates between 50
Similarly, network coordinate systems [16, 37] use latency
.3% and 60.5%, whereas
all other DSL providers’ deployment rates are below 23
estimates to assign a set of coordinates to their participating
.0%.
The partial deployment of RED-style policies within ISPs
hosts. A recent study [34] found that network coordinate sys-
could be due to heterogeneity in the ISPs’ equipment. We
tems do not perform well when deployed in BitTorrent net-
did not detect RED in any of the cable ISPs measured.
works, because RTTs between nodes can vary by up to four
orders of magnitude. Our measurements explain and pro-
4.4
Summary
vide insights into these findings: BitTorrent networks typi-
We have presented an in-depth characterization of the prop-
cally include many residential links, which have very large
erties of residential broadband networks. Our analysis re-
RTT variations as a result of their long queues. BitTorrent
veals important ways in which these networks differ from
traffic compounds these variations because it tends to fill up
academic networks, and it quantifies these differences. We
the queues.
summarize our key findings below:
Interactive and real-time applications: Recently, the
Allocated link bandwidths: Our results show that down-
popularity of VoIP and online games has grown considerably.
stream bandwidths exceed upstream bandwidths by more
Our data shows that real-time applications will be negatively
than a factor of 10 for some ISPs. In contrast to popular be-
affected by the broadband links’ large queueing delays. Be-
lief, for most ISPs, the measured bandwidths matched well
cause queueing delays increase in the presence of competing
with the advertised rates at all times of day, and we found
traffic, these time-sensitive applications are likely to experi-
little evidence of competing traffic affecting their links. Al-
ence degraded service when they are used concurrently with
though link bandwidths remain stable over the long term,
bandwidth-intense applications, such as BitTorrent.
they show high variation in the short term, especially for ca-
ble hosts. For some ISPs, link bandwidths change abruptly
as a result of traffic shaping.
6.
RELATED WORK
Packet latencies: Many DSL hosts show large (≥ 10 ms)
There is a large body of previous measurement work charac-
last-hop propagation delays. Cable hosts suffer higher jitter
terizing Internet paths other than broadband. Paxson [40]
than DSL hosts as a result of time-slotted packet transmis-
studied network packet dynamics among a fixed set of Inter-
sion policies on their upstream links. Packet concatenation
net hosts located primarily in academic institutions. More
on the upstream links can add another 5 − 10 ms of jitter in
recently, several studies have examined the characteristics of
cable links.
the network paths connecting the PlanetLab testbed [5, 43].
Although our paper uses similar measurement techniques,
9.
REFERENCES
the network environment we study is different.
[1] A. Akella, S. Seshan, and A. Shaikh. An empirical
Compared with other parts of the Internet, broadband ac-
evaluation of wide-area Internet bottlenecks. In
cess networks have received relatively little attention. Clay-
Proceedings of IMC’03, Oct 2003.
pool et al. [14] performed a measurement study of access
[2] D. G. Andersen, H. Balakrishnan, F. Kaashoek, and
networks’ queue sizes using 47 volunteering broadband hosts.
R. Morris. Experience with an Evolving Overlay
They found that the median queue size was 350 ms in DSL
Network Testbed. ACM Computer Communication
networks and 150 ms in cable networks, and they showed
Review, 33(3):13–19, July 2003.
in simulation that large queue sizes are detrimental to net-
[3] T. Anderson, A. Collins, A. Krishnamurthy, and
J. Zahorjan. PCP: Efficient endpoint congestion
work traffic from interactive applications. Our results are
control. In Proceedings of NSDI’06, San Jose, CA,
consistent with these earlier findings, but they are based on
May 2006.
a set of hosts that is more than two orders of magnitude
[4] G. Appenzeller, I. Keslassy, and N. McKeown. Sizing
larger. Similarly, Jehaes et al. [30] observed a large increase
router buffers. In Proceedings of ACM SIGCOMM’04,
in round-trip delays over saturated broadband links. Their
Aug 2004.
experiments were limited to one DSL and one cable link.
[5] S. Banerjee, T. G. Griffin, and M. Pias. The
Some recent studies have examined the traffic generated
interdomain connectivity of PlanetLab nodes. In
Proceedings of the 5th Passive and Active
by residential customers in Japan [12] and France [46]. These
Measurement Conference (PAM), Apr 2004.
results complement ours, because we examined the proper-
[6] J. Bellardo and S. Savage. Measuring packet
ties of the networks themselves rather than the properties of
reordering. In Proceedings of the 2nd ACM SIGCOMM
traffic traversing them. A comprehensive view of residential
Internet Measurement Workshop (IMW), Marseille,
networks requires a good understanding of both.
France, Nov 2002.
Lakshminarayanan and Padmanabhan [32] performed a
[7] R. Bhagwan, S. Savage, and G. M. Voelker.
network measurement study from 25 broadband hosts to dif-
Understanding availability. In Proceedings of 1st
ferent Internet hosts, covering several application-level met-
International Workshop on Peer-to-Peer Systems
(IPTPS), Boston, MA, March 2002.
rics such as TCP throughput and latency. Our study con-
[8] Bittorrent. http://bittorrent.com.
firms some of their findings but at a much larger scale. In
[9] L. S. Brakmo and L. Peterson. TCP Vegas: End to
their later work, Lakshminarayanan et al. [33] outlined pit-
end congestion avoidance on a global Internet. IEEE
falls in measuring link capacities of cable and DSL networks
Journal on Selected Areas in Communication,
by using existing bandwidth estimation tools. In particular,
13(8):1465–1480, Oct 1995.
the accuracy of these tools is greatly influenced by the rate
[10] CableLabs. DOCSIS 1.1 interface specification, 2006.
regulation schemes used in cable and DSL networks. Our
http://www.cablemodem.com/specifications/
measurement methodology does not suffer from such inac-
specifications11.html.
[11] B. Chandra, M. Dahlin, L. Gao, and A. Nayate.
curacies because it relies on saturating the links for a short
End-to-end WAN service availability. In Proceedings of
duration.
the 3rd USITS, 2001.
Many previous studies have measured network proper-
[12] K. Cho, K. Fukuda, H. Esaki, and A. Kato. The
ties of hosts participating in file-sharing peer-to-peer sys-
impact and implications of the growth in residential
tems [7, 44, 45]. Because a large fraction of peers in these
user-to-user traffic. In Proceedings of SIGCOMM’06,
systems access the Internet over cable and DSL networks, all
Sep 2006.
these studies indirectly include measurements of broadband
[13] Cisco. Internetworking technology handbook, 2006.
access networks. However, these results cannot be compared
[14] M. Claypool, R. Kinicki, M. Li, J. Nichols, and H. Wu.
Inferring queue sizes in access networks by active
directly with ours because the focus of these studies is pri-
measurement. In Proceedings of the 4th Passive and
marily on application-level performance and not on the link
Active Measurement Workshop (PAM), Antibes
level characteristics of broadband networks.
Juan-les-Pins, France, Apr 2004.
[15] Comcast PowerBoost press release, Jun 2006.
7.
CONCLUSIONS
http://www.cmcsk.com/phoenix.zhtml?c=147565&p=
In this paper, we presented the first large-scale measurement
irol-newsArticle&ID=890297.
study of major cable and DSL providers in North Amer-
[16] F. Dabek, R. Cox, F. Kaashoek, and R. Morris.
Vivaldi: A decentralized network coordinate system.
ica and Europe. Our study characterized several important
In Proceedings of SIGCOMM’04, Aug 2004.
characteristics of broadband networks, including available
[17] C. Dovrolis, P. Ramanathan, and D. Moore. Packet
link capacities, packet transmission policies, jitter, packet
dispersion techniques and a capacity estimation
drop policies, and queue lengths. Our analysis revealed im-
methodology. IEEE/ACM Transactions on
portant ways in which residential networks differ from the
Networking, Dec 2004.
conventional wisdom about the Internet. We also discussed
[18] Federal Communications Commission ComCom,
the implications of our findings for many emerging protocols
Switzerland. Broadband in the universal service, Sep
2006.
and systems, such as delay-based congestion control (e.g.,
[19] S. Floyd, M. Handley, J. Padhye, and J. Widmer.
PCP) and network coordinate systems (e.g., Vivaldi).
Equation-based congestion control for unicast
applications. In Proceedings of SIGCOMM’00, Aug
8.
ACKNOWLEDGMENTS
2000.
We thank Steve Gribble and Dan Sandler for generously
[20] S. Floyd and V. Jacobson. Random early detection
gateways for congestion avoidance. IEEE/ACM
hosting measurement servers for our experiments. We are
Transactions on Networking, 1(4):397–413, Aug 1993.
grateful to our shepherd Guillaume Urvoy-Keller, Peter Dr-
[21] R. Govindan and V. Paxson. Estimating router ICMP
uschel, and our anonymous reviewers for providing detailed
generation delays. In Proceedings of Passive and Active
and helpful feedback on this paper.
Measurement (PAM), Fort Collins, CO, USA, 2002.
[22] B. Gueye, A. Ziviani, M. Crovella, and S. Fdida.
[36] R. Mahajan, N. Spring, D. Wetherall, and
Constraint-based geolocation of Internet hosts.
T. Anderson. User-level Internet path diagnosis. In
IEEE/ACM Transactions on Networking,
Proceedings of the 19th Symposium on Operating
14(6):1219–1232, 2006.
Systems Principles (SOSP), Bolton Landing, NY,
[23] A. Haeberlen, M. Dischinger, K. P. Gummadi, and
USA, Oct 2003.
S. Saroiu. Monarch: A tool to emulate transport
[37] T. Ng and H. Zhang. Predicting Internet network
protocol flows over the Internet at large. In Proceedings
distance with coordinates-based approaches. In Proc.
of IMC’06, Rio de Janeiro, Brazil, Oct 2006.
INFOCOM 2002, New York, NY, USA, June 2002.
[24] Y. hua Chu, A. Ganjam, T. S. E. Ng, S. G. Rao,
[38] Nielsen/NetRatings. U.S. broadband connections reach
K. Sripanidkulchai, J. Zhan, and H. Zhang. Early
critical mass, 2004. http:
experience with an Internet broadcast system based on
//www.nielsen-netratings.com/pr/pr_040818.pdf.
overlay multicast. In USENIX Annual Technical
[39] OECD broadband statistics, Dec 2005.
Conference, Jun 2004.
http://www.oecd.org/sti/ict/broadband.
[25] Y. hua Chu, S. G. Rao, S. Seshan, and H. Zhang.
[40] V. Paxson. End-to-end routing behavior in the
Enabling conferencing applications on the Internet
Internet. IEEE/ACM Transactions on Networking,
using an overlay multicast architecture. In Proceedings
5(5):601–615, Oct 1997.
of SIGCOMM’01, Aug 2001.
[41] PlanetLab. http://www.planet-lab.org/.
[26] G. Iannaccone, C. nee Chuah, R. Mortier,
[42] Point Topic Ltd. UK broadband market monitor,
S. Bhattacharyya, and C. Diot. Analysis of link
2006. http://point-topic.com/content/ukplus/
failures in an IP backbone. In Proceedings of the 2nd
email%20archive/wukbbmm7050602.htm.
ACM SIGCOMM Workshop on Internet measurement
[43] H. Pucha, Y. C. Hu, and Z. M. Mao. On the impact of
(IMW), 2002.
research network based testbeds on wide-area
[27] ISP Planet. Top 24 U.S. ISPs by subscriber: Q3 2005,
experiments. In Proceedings of IMC’06, Oct 2006.
2006. http://www.isp-planet.com/research/
[44] S. Saroiu, K. P. Gummadi, and S. D. Gribble. A
rankings/usa.html.
measurement study of peer-to-peer file sharing
[28] ISP Planet. Top seven ISPs in Canada by subscriber:
systems. In Proceedings of the Multimedia Computing
Q4 2006, 2006. http://www.isp-planet.com/
and Networking (MMCN), San Jose, CA, Jan 2002.
research/rankings/2006/canada_q42006.html.
[45] S. Sen and J. Wang. Analyzing peer-to-peer traffic
[29] M. Jain and C. Dovrolis. End-to-end available
across large networks. In Proceedings of the 2nd ACM
bandwidth: Measurement methodology, dynamics, and
SIGCOMM Workshop on Internet Measurement
relation with TCP throughput. IEEE/ACM
(IMW), Marseille, France, Nov 2002.
Transactions on Networking, Aug 2003.
[46] M. Siekkinen, D. Collange, G. Urvoy-Keller, and
[30] T. Jehaes, D. D. Vleeschauwer, T. Coppens, B. V.
E. Biersack. Performance limitations of ADSL users:
Doorselaer, E. Deckers, W. Naudts, K. Spruyt, and
A case study. In Proceedings of the 8th Passive and
R. Smets. Access network delay in networked games.
Active Measurement Conference (PAM), Apr 2007.
In Proceedings of the 2nd workshop on Network and
[47] N. Spring, R. Mahajan, D. Wetherall, and
system support for games (NetGames), 2003.
T. Anderson. Measuring ISP topologies with
[31] S. Keshav. A control-theoretic approach to flow
Rocketfuel. IEEE/ACM Transactions on Networking,
control. In Proceedings of SIGCOMM’91, Zurich,
12(1):2–16, Feb 2004.
Switzerland, Sept 1991.
[48] United Kingdom e-Minister and e-Envoy. UK online:
[32] K. Lakshminarayanan and V. N. Padmanabhan. Some
The broadband future, 2001.
findings on the network performance of broadband
[49] C. Villamizar and C. Song. High performance TCP in
hosts. In Proceedings of IMC’03, Miami, FL, USA, Oct
ANSNET. ACM Computer Communication Review,
2003.
24(5):45–60, 1994.
[33] K. Lakshminarayanan, V. N. Padmanabhan, and
[50] Wikipedia. Broadband Internet access in the
J. Padhye. Bandwidth estimation in broadband access
Netherlands, 2006. http://en.wikipedia.org/w/
networks. In Proceedings of IMC’04, Taormina, Italy,
index.php?title=Broadband_Internet_access_in_
Oct 2004.
the_Netherlands&oldid=152236688.
[34] J. Ledlie, P. Gardner, and M. Seltzer. Network
[51] Windsor Oaks Group, LLC. Market outlook report,
coordinates in the wild. In Proceedings of NSDI’07,
May 2005. http:
Apr 2007.
//www.broadbandtrends.com/Report%20Summary/
[35] S.-J. Lee, P. Sharma, S. Banerjee, S. Basu, and
2006/BBT_GlobalBBOutlook2006_061110_TOC.pdf.
R. Fonseca. Measuring bandwidth between PlanetLab
[52] B. Wong, I. Stoyanov, and E. G. Sirer. Octant: A
nodes. In Proceedings of the Passive and Active
comprehensive framework for the geolocalization of
Measurement Workshop (PAM), March 2005.
Internet hosts. In Proceeedings of NSDI’07, Apr 2007.
