[e2e] performance of BIC-TCP, High-Speed-TCP, H-TCP etc

[Douglas Leith]

I suggest you take a closer look Injong - there is a whole page of data 
from tests covering a wide range of levels of background traffic.  These 
results are all new, and significantly strengthen the conclusions I 
think, as is the expanded explanatory discussion of the observed 
behaviour of the various algorithms (the result of a fair bit of 
detective work of course).  Your claim that "Your report mostly ignores 
the effect of background traffic" is simply not true.

I can't really comment on your own tests without more information, 
although I can say that we went to a good bit of trouble to make sure 
our results were consistent and reproducible - in fact all our reported 
results are from at least five, and usually more, runs of each test.  We 
were also careful to control for differences in kernel implementation so 
that we compare congestion control algorithms rather than other aspects 
of the network stack implementation.  All of this is documented in the 
paper.  The kernel we used is available on the web.  Our measurements 
are also publicly available - the best way forward might be to pick one 
or two tests and compare results of them in detail with a view to 
diagnosing the source of any differences.

General comments such as "our experience tells that the RTT variations 
of mid size flows play a very important role in creating significant 
dynamics in testing environments" are not too helpful.  What do you mean 
by a "mid-sized flow" ?  What do you mean by "significant dynamics" ? 
What do you mean by "important role" - is this quantified ?  Best to 
stick to science rather than grandstanding.  This is especially true 
when dealing with a sensitive subject such as the evaluation of 
competing algorithms.

Re FAST, we have of course discussed our results with the Caltech folks. 
  As stated in the paper, some of the observed behaviour seems to be 
associated with the alpha tuning algorithm.  Other behaviour seems to be 
associated with packet burst effects that have also been reported 
independently by the Caltech folks.  Similar results to ours have since 
been observed by other groups I believe.  Perhaps differences between 
our results point to some issue in your testbed setup.

Doug

Injong Rhee wrote:
> 
> 
> This is a resend with fixed web links. The links were broken in my 
> previous email -- sorry about multiple transmissions.
> 
> --------------------------------------------------------------------------------- 
> 
> 
> Hi Doug,
> 
> Thanks for sharing your paper. Also congratulations to the acceptance of 
> your journal paper to TONs. But I am wondering what's new in this paper. 
> At first glance, I did not find many new things that are different from 
> your previously publicized reports. How much is this different from the 
> ones you put out in this mail list a year or two ago and also the one 
> publicized in PFLDnet February this year 
> http://www.hpcc.jp/pfldnet2006/? In that same workshop, we also 
> presented our experimental results that shows significant discrepancy 
> from yours but i am not sure why you forgot to reference our 
> experimental work presented in that same PFLDnet. Here is a link to a 
> more detailed version of that report accepted to COMNET 
> http://netsrv.csc.ncsu.edu/highspeed/comnet-asteppaper.pdf
> 
> The main point of contention [that we talked about in that PFLDnet 
> workshop] is the presence of background traffic and the method to add 
> them. Your report mostly ignores the effect of background traffic. Some 
> texts in this paper state that you added some web traffic (10%), but the 
> paper shows only the results from NO background traffic scenarios. But 
> our results differ from yours in many aspects. Below are the links to 
> our results (the links to them have been available in our BIC web site 
> for a long time and also mentioned in our PFLDnet paper; this result is 
> with the patch that corrects HTCP bugs).
> 
> [Convergence and intra protocol fairness]
> 
> without background traffic: 
> http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/intra_protocol/intra_protocol.htm 
> 
> 
> with background traffic: 
> http://netsrv.csc.ncsu.edu/highspeed/1200/bk/intra_protocol/intra_protocol.htm 
> 
> 
> [RTT fairness]:
> 
> w/o background traffic: 
> http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/rtt_fairness/rtt_fairness.htm 
> 
> 
> with background traffic: 
> http://netsrv.csc.ncsu.edu/highspeed/1200/bk/rtt_fairness/rtt_fairness.htm
> 
> [TCP friendliness]
> 
> without background traffic: 
> http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/tcp_friendliness/tcp_friendliness.htm 
> 
> 
> with background traffic: 
> http://netsrv.csc.ncsu.edu/highspeed/1200/bk/tcp_friendliness/tcp_friendliness.htm 
> 
> 
> After our discussion in that PFLDnet, I puzzled why we get different 
> results. My guess is that the main difference between your experiment 
> and ours is the inclusion of mid-sized flows with various RTTs -- our 
> experience tells that the RTT variations of mid size flows play a very 
> important role in creating significant dynamics in testing environments. 
> The same point about the importance of mid size flows with RTT 
> variations has been raised in several occasions by Sally Floyd as well, 
> including in this year's E2E research group meeting. You can find some 
> reference to the importance of RTT variations in her paper too [ 
> http://www.icir.org/models/hotnetsFinal.pdf]. Just having web-traffic 
> (all with the same RTTs) does not create a realistic environment as it 
> does not do anything about RTTs and also flow sizes tend to be highly 
> skewed with the Pareto distribution-- but I don't know exactly how you 
> create your testing environment with web-traffic -- I can only guess 
> from the description you have about the web traffic in your paper.
> 
> Another puzzle in this difference seems that even under no background 
> traffic, we also get different results from yours..hmm...especially with 
> FAST because under no background traffic, FAST seems to work fairly well 
> with good RTT fairness in our experiment. But your results show FAST has 
> huge RTT-unfairness. That is very strange. Is that because we have 
> different bandwidth and buffer sizes in the setup? I think we need to 
> compare our notes more. Also in the journal paper of FAST experimental 
> results [ 
> http://netlab.caltech.edu/publications/FAST-ToN-final-060209-2007.pdf ], 
> FAST seems to work very well under no background traffic. We will verify 
> our results again in the exact same environment as you have in your 
> report, to make sure we can reproduce your results....but here are some 
> samples of our results for FAST.
> 
> http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/rtt_fairness/1200--2.4_FAST-2.4_FAST-NONE--400-3-1333--1000-76-3-0-0-0-5-500--200000-0.6-1000-10-1200-64000-150--1/ 
> 
> 
> In this experiment, FAST flows are just perfect. Also the same result is 
> confirmed inthe FAST journal paper [ 
> http://netlab.caltech.edu/publications/FAST-ToN-final-060209-2007.pdf -- 
> please look at Section IV.B and C. But your results show really bad RTT 
> fairness.]
> 
> Best regards,
> 
> Injong
> 
> ---
> 
> Injong Rhee
> 
> NCSU
> 
> On Sep 22, 2006, at 10:22 AM, Douglas Leith wrote:
> 
> For those interested in TCP for high-speed environments, and perhaps 
> also people interested in TCP evaluation generally, I'd like to point 
> you towards the results of a detailed experimental study which are now 
> available at:
> 
>  
> 
> http://www.hamilton.ie/net/eval/ToNfinal.pdf 
> <http://www.hamilton.ie/net/eval/ToNfinal.pdf>
> 
>  
> 
> This study consistently compares Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP 
> and H-TCP performance under a wide range of conditions including with 
> mixes of long and short-lived flows. This study has now been subject to 
> peer review (to hopefully give it some legitimacy) and is due to appear 
> in the Transactions on Networking.
> 
>  
> 
> The conclusions (see summary below) seem especially topical as BIC-TCP 
> is currently widely deployed as the default algorithm in Linux.
> 
>  
> 
> Comments appreciated. Our measurements are publicly available - on the 
> web or drop me a line if you'd like a copy.
> 
>  
> 
> Summary:
> 
> In this paper we present experimental results evaluating the
> 
> performance of the Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP and
> 
> H-TCP proposals in a series of benchmark tests.
> 
>  
> 
> We find that many recent proposals perform surprisingly poorly in
> 
> even the most simple test, namely achieving fairness between two
> 
> competing flows in a dumbbell topology with the same round-trip
> 
> times and shared bottleneck link. Specifically, both Scalable-TCP
> 
> and FAST TCP exhibit very substantial unfairness in this test.
> 
>  
> 
> We also find that Scalable-TCP, HS-TCP and BIC-TCP induce significantly 
> greater RTT unfairness between competing flows with different round-trip 
> times. The unfairness can be an order of magnitude greater than that 
> with standard TCP and is such that flows with longer round-trip times
> 
> can be completely starved of bandwidth.
> 
>  
> 
> While the TCP proposals studied are all successful at improving
> 
> the link utilisation in a relatively static environment with
> 
> long-lived flows, in our tests many of the proposals exhibit poor
> 
> responsiveness to changing network conditions. We observe that
> 
> Scalable-TCP, HS-TCP and BIC-TCP can all suffer from extremely
> 
> slow (>100s) convergence times following the startup of a new
> 
> flow. We also observe that while FAST-TCP flows typically converge
> 
> quickly initially, flows may later diverge again to create
> 
> significant and sustained unfairness.
> 
>  
> 
> --Doug
> 
>  
> 
> Hamilton Institute
> 
> www.hamilton.ie <http://www.hamilton.ie/>
>