[e2e] performance of BIC-TCP, High-Speed-TCP, H-TCP etc

[Injong Rhee]

This is a resend with fixed web links. The links were broken in my previous email -- sorry about multiple transmissions.

---------------------------------------------------------------------------------

Hi Doug,


Thanks for sharing your paper. Also congratulations to the acceptance of your journal paper to TONs. But I am wondering what's new in this paper. At first glance, I did not find many new things that are different from your previously publicized reports. How much is this different from the ones you put out in this mail list a year or two ago and also the one publicized in PFLDnet February this year http://www.hpcc.jp/pfldnet2006/? In that same workshop, we also presented our experimental results that shows significant discrepancy from yours but i am not sure why you forgot to reference our experimental work presented in that same PFLDnet. Here is a link to a more detailed version of that report accepted to COMNET http://netsrv.csc.ncsu.edu/highspeed/comnet-asteppaper.pdf


The main point of contention [that we talked about in that PFLDnet workshop] is the presence of background traffic and the method to add them. Your report mostly ignores the effect of background traffic. Some texts in this paper state that you added some web traffic (10%), but the paper shows only the results from NO background traffic scenarios. But our results differ from yours in many aspects. Below are the links to our results (the links to them have been available in our BIC web site for a long time and also mentioned in our PFLDnet paper; this result is with the patch that corrects HTCP bugs). 


[Convergence and intra protocol fairness] 

without background traffic: http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/intra_protocol/intra_protocol.htm



with background traffic: http://netsrv.csc.ncsu.edu/highspeed/1200/bk/intra_protocol/intra_protocol.htm


[RTT fairness]: 

w/o background traffic: http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/rtt_fairness/rtt_fairness.htm

with background traffic: http://netsrv.csc.ncsu.edu/highspeed/1200/bk/rtt_fairness/rtt_fairness.htm


[TCP friendliness]

without background traffic: http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/tcp_friendliness/tcp_friendliness.htm

with background traffic: http://netsrv.csc.ncsu.edu/highspeed/1200/bk/tcp_friendliness/tcp_friendliness.htm


After our discussion in that PFLDnet, I puzzled why we get different results. My guess is that the main difference between your experiment and ours is the inclusion of mid-sized flows with various RTTs -- our experience tells that the RTT variations of mid size flows play a very important role in creating significant dynamics in testing environments. The same point about the importance of mid size flows with RTT variations has been raised in several occasions by Sally Floyd as well, including in this year's E2E research group meeting. You can find some reference to the importance of RTT variations in her paper too [ http://www.icir.org/models/hotnetsFinal.pdf]. Just having web-traffic (all with the same RTTs) does not create a realistic environment as it does not do anything about RTTs and also flow sizes tend to be highly skewed with the Pareto distribution-- but I don't know exactly how you create your testing environment with web-traffic -- I can only guess from the description you have about the web traffic in your paper. 


Another puzzle in this difference seems that even under no background traffic, we also get different results from yours..hmm...especially with FAST because under no background traffic, FAST seems to work fairly well with good RTT fairness in our experiment. But your results show FAST has huge RTT-unfairness. That is very strange. Is that because we have different bandwidth and buffer sizes in the setup? I think we need to compare our notes more. Also in the journal paper of FAST experimental results [ http://netlab.caltech.edu/publications/FAST-ToN-final-060209-2007.pdf ], FAST seems to work very well under no background traffic. We will verify our results again in the exact same environment as you have in your report, to make sure we can reproduce your results....but here are some samples of our results for FAST. 


http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/rtt_fairness/1200--2.4_FAST-2.4_FAST-NONE--400-3-1333--1000-76-3-0-0-0-5-500--200000-0.6-1000-10-1200-64000-150--1/


In this experiment, FAST flows are just perfect. Also the same result is confirmed inthe FAST journal paper [ http://netlab.caltech.edu/publications/FAST-ToN-final-060209-2007.pdf-- please look at Section IV.B and C. But your results show really bad RTT fairness.]


Best regards,

Injong


---

Injong Rhee

NCSU




On Sep 22, 2006, at 10:22 AM, Douglas Leith wrote:

For those interested in TCP for high-speed environments, and perhaps also people interested in TCP evaluation generally, I'd like to point you towards the results of a detailed experimental study which are now available at: 



http://www.hamilton.ie/net/eval/ToNfinal.pdf 



This study consistently compares Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP and H-TCP performance under a wide range of conditions including with mixes of long and short-lived flows. This study has now been subject to peer review (to hopefully give it some legitimacy) and is due to appear in the Transactions on Networking. 



The conclusions (see summary below) seem especially topical as BIC-TCP is currently widely deployed as the default algorithm in Linux. 



Comments appreciated. Our measurements are publicly available - on the web or drop me a line if you'd like a copy.



Summary:

In this paper we present experimental results evaluating the

performance of the Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP and

H-TCP proposals in a series of benchmark tests.



We find that many recent proposals perform surprisingly poorly in

even the most simple test, namely achieving fairness between two

competing flows in a dumbbell topology with the same round-trip

times and shared bottleneck link. Specifically, both Scalable-TCP

and FAST TCP exhibit very substantial unfairness in this test.



We also find that Scalable-TCP, HS-TCP and BIC-TCP induce significantly greater RTT unfairness between competing flows with different round-trip times. The unfairness can be an order of magnitude greater than that with standard TCP and is such that flows with longer round-trip times 

can be completely starved of bandwidth.



While the TCP proposals studied are all successful at improving

the link utilisation in a relatively static environment with

long-lived flows, in our tests many of the proposals exhibit poor

responsiveness to changing network conditions. We observe that

Scalable-TCP, HS-TCP and BIC-TCP can all suffer from extremely

slow (>100s) convergence times following the startup of a new

flow. We also observe that while FAST-TCP flows typically converge

quickly initially, flows may later diverge again to create

significant and sustained unfairness.



--Doug



Hamilton Institute

www.hamilton.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20060922/75c36169/attachment-0001.html