Hi All,<br><br> I was able to identify the size of the buffer in the Router's interface . It was found to be 2040 <br> bytes and the buffer management scheme used is FIFO .<br> Then based on this statement by Lachlan , I did the following tunings.
<br><br> ""<br> <span style="color: rgb(255, 0, 0);" class="q">The general rule-of-thumb for Reno is that the send buffer should be<br> at least twice the bandwidth*RTT. For BIC is is probably reduced to
<br> about 120% of the BDP (because it reduces its window by a smaller
<br> factor when there is a loss). The receive buffer should still be at<br> least equal to the BDP plus the router buffer.</span><br> ""
<br> The tunings are as follows.<br><br> Send buffer = BDP + 120 % of BDP <br> = 921600 + 184320 <br> = 1105920 bytes<br><br> Receive buffer = BDP + Router's buffer size
<br> = 921600 + 2040<br> = 923640 bytes<br> <br> After that I tried the same earlier experiments with iperf . I got a average throughput of 66
<br> Mbits/s . <br> <br> Next , I tried some more tuning . I increased the the send buffer size to twice of BDP.<br> The receive buffer is same as earlier. This is according to what Lachlan suggested for TCP
<br> Reno.<br> Based on these tunings , I got a throughput of average 78 Mbits/s using iperf. More improvement <br> but not fully utilise the link capacity.<br><br> But if I tune the window size to twice the size of BDP , I got a average throughput of
<br> around 88 Mbits/sec which I feel very much O.K for a 100 MBits/sec link . But it can be further <br> improved. Also , earlier, I have written in this mailing list that when I tuned the window size to twice <br>
of BDP , I was getting a throughput of 95 Mbits /s . That I was referring to the maximum <br> throughput . <br> For all these experiments, I was using BIC. <br><br> I also tried with the following tunings as suggested . But I didnot get any difference.
<br><br> /sbin/ifconfig eth0 txqueuelen 1000<br> /sbin/sysctl -w net.ipv4.tcp_adv_win_scale=3 <br><br> <br> So , based on the result of all these experiments , I reach the following conclusion.
<br><br style="font-weight: bold;"><span style="font-weight: bold;"> </span><span style="font-style: italic;"> The </span><span class="q"><span style="font-style: italic;">receive buffer should be at least equal to the BDP plus the router buffer .
</span><br style="font-style: italic;"><span style="font-style: italic;"> The send buffer should be 20% more than BDP if you are using BIC. </span><br style="font-style: italic;"><span style="font-style: italic;"> These tunings will probably try to utilise the maximum link capacity provided
</span><br style="font-style: italic;"><span style="font-style: italic;"> the buffer size in the intermediate router is equal to the BDP. <br> This I cannot prove practically because it is not possible fot me to increase the buffer
<br><span style="font-style: italic;"> size to the size of BDP , because the network is not under my administration. <br><br style="font-style: italic;"></span></span><span style="font-style: italic;"> But since in most cases the network is under the control of ISP , so it might not
</span><br style="font-style: italic;"><span style="font-style: italic;"> be possible for end users to know the size of the router's interface.</span><br style="font-style: italic;"><span style="font-style: italic;">
In that case , the end to end send and receive buffer size should be atleast </span><br style="font-style: italic;"><span style="font-style: italic;"> equal to twice the size of BDP to obtain maximum throughput. This statement
<br> was reflected by my findings.<br><br></span> It will be helpfull to me if everybody give there views on my conclusion. If you <br> have any contradiction , please write. <br><br> With Thanks and Regards
<br> zaman<br><span style="font-style: italic;"> </span><br style="font-weight: bold;"><br style="font-weight: bold;"><br style="font-weight: bold;"><br style="font-weight: bold;"><span style="font-weight: bold;">
</span><br><br></span> <br> <br> <br><br> <br><br><br><br><br><br><br><br><br><div><span class="gmail_quote">On 7/6/07, <b class="gmail_sendername">query</b> <<a href="mailto:query.cdac@gmail.com">query.cdac@gmail.com
</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>T<span style="color: rgb(204, 102, 0);">hanks a lot Andrew . It helped me to understand and I feel that my Tunings are
O.K. </span><br style="color: rgb(204, 102, 0);"><div><span class="q"><span style="color: rgb(204, 102, 0);" class="gmail_quote">
</span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> I was doing some Bandwidth measurement test on a 100 mbs link with a RTT<br>> of about 70ms.<br>> Based on that, I calculated the BDP as follows.<br>><br>> BDP = Bandwidth * RTT
<br>> = 921600 bytes<br>> I did the following adjustments. I increased the above calculated BDP by<br>> nearly<br>> half of the value . The TCP settings now look like this.<br>><br>> /proc/sys/net/core/rmem_max 175636
<br>> /proc/sys/net/core/wmem_max 175636<br>> /proc/sys/net/ipv4/tcp_rmem 4096 87380<br>> 175636<br>> /proc/sys/net/ipv4/tcp_wmem 4096 87380<br>> 175636<br>
><br>> After these settings , I find the link utilisation to be nearly 95 mbs.<br>><br>> According to many papers that I read , I found that the BDP should be<br>> equal<br>> to the product of Bandwidth * RTT .
<br><br>The papers probably said that *router* buffers need to equal the<br>bandwidth*RTT. You are adjusting the sender/receiver buffers. These<br>need to be significantly larger, as you have found.</blockquote></span>
<div><br>
<span style="color: rgb(153, 51, 0);"> The papers or rather articles are talking of sender and receiver </span><br style="color: rgb(153, 51, 0);"><span style="color: rgb(153, 51, 0);"> buffers . Here is one such link where I find it.
</span><br style="color: rgb(153, 51, 0);"><span style="color: rgb(153, 51, 0);"> </span><a style="color: rgb(153, 51, 0);" href="http://www.psc.edu/networking/projects/tcptune/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://www.psc.edu/networking/projects/tcptune/
</a><span style="color: rgb(153, 51, 0);"> .</span><br style="color: rgb(153, 51, 0);"><br> </div><span class="q"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
In order to allow retransmissions, the sender buffer needs to be able
<br>to store all "packets in flight", which include both those in the in<br>the router buffers and those "on the wire" (that is, in the nominal<br>RTT of the link).<br><br>In order to be able to provide in-order delivery, the receiver buffer
<br>needs to be able to hold even more packets. If a packet is lost, it<br>will receive an entire RTT (plus router buffer) worth of data before<br>the first retransmission of that packet will arrive. If the first<br>retransmission is also lost, then it will need to store yet another
<br>RTT worth of data.<br><br>The general rule-of-thumb for Reno is that the send buffer should be<br>at least twice the bandwidth*RTT. For BIC is is probably reduced to<br>about 120% of the BDP (because it reduces its window by a smaller
<br>factor when there is a loss). The receive buffer should still be at<br>least equal to the BDP plus the router buffer.</blockquote></span><div><br> <span style="color: rgb(51, 153, 153);"> <span style="color: rgb(204, 102, 0);">
What I understand from your reply, is that It is not necessary that TCP Window should be</span></span><br style="color: rgb(204, 102, 0);"><span style="color: rgb(204, 102, 0);"> equal to BDP in all cases . Had the router buffer size is equal to BDP , then I think I
</span><br style="color: rgb(204, 102, 0);"><span style="color: rgb(204, 102, 0);"> should equal link utilisation equal to the capacity of the link .</span><br style="color: rgb(204, 102, 0);"><span style="color: rgb(204, 102, 0);">
Since , In Internet it will not be possible to know the router buffer size , so the best thing one </span><br style="color: rgb(204, 102, 0);"><span style="color: rgb(204, 102, 0);"> can do , is to make the TCP window size twice to BDP as you have suggested.
</span><br style="color: rgb(204, 102, 0);"><br style="color: rgb(204, 102, 0);"><span style="color: rgb(204, 102, 0);"> I am finding another problem. The UDP transmission rate on that link is decreased. I changed</span>
<br style="color: rgb(204, 102, 0);"><span style="color: rgb(204, 102, 0);"> to the default settings , but it is showing the exact readings after tuning. It seems it is reading some fixed value from something and based on that it is transferring data .
<br><br></span><span style="color: rgb(204, 102, 0);">The readings are like this..........<br>
<br>
iperf -u -c <a href="http://192.168.60.62" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 192.168.60.62</a> -t 300 -l 1460 -i 2<br>
------------------------------------------------------------<br>
Client connecting to <a href="http://192.168.60.62" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 192.168.60.62</a>, UDP port 5001<br>
Sending 1460 byte datagrams<br>
UDP buffer size: 108 KByte (default)<br>
------------------------------------------------------------<br>
[ 3] local <a href="http://10.128.0.2" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 10.128.0.2</a> port 32785 connected with <a href="http://192.168.60.62" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font>
192.168.60.62</a> port 5001<br>
[ ID] Interval Transfer Bandwidth<br>
[ 3] -0.0- 2.0 sec 257 KBytes 1.05 Mbits/sec<br>
[ 3] 2.0- 4.0 sec 257 KBytes 1.05 Mbits/sec<br>
[ 3] 4.0- 6.0 sec 255 KBytes 1.05 Mbits/sec<br>
[ 3] 6.0- 8.0 sec 257 KBytes 1.05 Mbits/sec<br>
[ 3] 8.0-10.0 sec 255 KBytes 1.05 Mbits/sec<br>
[ 3] 10.0-12.0 sec 257 KBytes 1.05 Mbits/sec<br>
[ 3] 12.0-14.0 sec 255 KBytes 1.05 Mbits/sec<br>
[ 3] 14.0-16.0 sec 257 KBytes 1.05 Mbits/sec<br>
[ 3] 16.0-18.0 sec 257 KBytes 1.05 Mbits/sec<br>
<br>
</span> <span style="color: rgb(204, 102, 0);">The result is for the following tuning.</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> net.core.rmem_default = 110592</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> net.core.wmem_default = 110592</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> </span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> After that I changed the tuning to </span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> net.core.rmem_default = 196608</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> net.core.wmem_default = 196608 </span><br style="color: rgb(204, 102, 0);">
<br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);"> The readings for the tuning is like this...</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">iperf -u -c <a href="http://192.168.60.62" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 192.168.60.62</a> -t 300 -l 1460 -i 2</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">------------------------------------------------------------</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">Client connecting to <a href="http://192.168.60.62" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 192.168.60.62</a>, UDP port 5001</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">Sending 1460 byte datagrams</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">UDP buffer size: 192 KByte (default)</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">------------------------------------------------------------</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] local <a href="http://10.128.0.2" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font> 10.128.0.2</a> port 32785 connected with <a href="http://192.168.60.62" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font color="red"><b>MailScanner warning: numerical links are often malicious:</b></font>
192.168.60.62</a> port 5001</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ ID] Interval Transfer Bandwidth</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] -0.0- 2.0 sec 257 KBytes 1.05 Mbits/sec</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] 2.0- 4.0 sec 257 KBytes 1.05 Mbits/sec</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] 4.0- 6.0 sec 255 KBytes 1.05 Mbits/sec</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] 6.0- 8.0 sec 257 KBytes 1.05 Mbits/sec</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] 8.0-10.0 sec 255 KBytes 1.05 Mbits/sec</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] 10.0-12.0 sec 257 KBytes 1.05 Mbits/sec</span><br style="color: rgb(204, 102, 0);">
<span style="color: rgb(204, 102, 0);">[ 3] 12.0-14.0 sec 255 KBytes 1.05 Mbits/sec<br><br style="color: rgb(204, 102, 0);"></span>
<span style="color: rgb(204, 102, 0);"> </span>Kindly please help me to rectify the problem. It is the same link on which I<br>
performed the TCp test.<br>
<br>
regards<br>
zaman<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br></blockquote><span style="color: rgb(204, 102, 0);"> </span><br> <br> <br>
<br><br> </div><span class="q"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I hope this help,<br>Lachlan<br><br>--<br>Lachlan Andrew Dept of Computer Science, Caltech
<br>1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA<br>Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603<br></blockquote></span></div><br>
</blockquote></div><br>