[Tsvwg] Re: [e2e] Are you interested in TOEs and related issues
Sunay.Tripathi at eng.sun.com
Wed Mar 3 23:10:06 PST 2004
> > 3) For the up and coming 10Gb NICs, TOE will help saturate
> > the link. Some
> > vendors assert that TOE will be required to support 10Gb NICs.
> TOE will bring CPU utilization down of course, but 10GbE NIC will
> saturate the link with or without a TOE - much like GbE NICs do it now.
I think the TOE vendors meant that single CPU won't be able to saturate
the 10Gb NIC on a more meaningful workloads than just a simple throughput
tests. Even a 2 CPU machine which wants to do some work also might also
> > 4) Performance reasons. Just the LSO aspect of TOE (sending
> > large chunks of
> > data and letting the TOE split it up in mss size pieces) and ack
> > coalescing gives a pretty good boost (our own prototypes
> > indicates that
> > this is true).
> Just curious - in your testing, how much CPU is freed up by the ack
We actually always try to run the CPUs to saturation (keep increasing
the load) and we can get a double digit percentage boost on a web
like workload (multiple simultaneous connections). I didn't verify
exactly where the gains were coming from but my feeling was that
more efficient data movement and better locality was contributing to
bulk of the gains.
> >The gains are by optimizing data movement and not by
> > offloading protocol processing.
> LSO is a 'stateless' offload (much like checksum offload) so one doesn't
> need a TOE to support it;
It depends. If the kernel does a write based on open window, then yes
you are right. But if you have full TOE and you pass large chunks of
data to TOE (does wonder when sendfile/sendfilev is in play) and let
TOE figure out when to send it, the gains increase. In the later case,
TOE needs teh TCP state.
> most modern NICs (including our 10GbE Adapter) do LSO. Also, for normal
> frames LSO has quite dramatic effect at 10GbE rates, but for Jumbo
> frames performance gain diminishes to ~7%.
Do you have any insight on why you would see a degradation with Jumbo
frames? As long as you are stateless, the jumbograms should show
> > 5) TOE is necessary for RDMA, iSCSI etc. for layering
> > reasons. I am not
> > involved with RDMA so someone who is an expert can
> > probably comment on
> > this part.
> > 6) TOE based NIC are already making pretty good headway in
> > embedded space.
> > The technology is already maturing so why not use it in
> > broader market.
> > Note that the above claims are in no particular order of
> > importance and made my TOE vendors in general. Of these, I
> > personally do agree with 1 and 4 but that iteself doesn't
> > mean that TOE will make it in general purpose networking.
> > It would be interesting to see if you and others in the list
> > agree or disagree with these claims.
> > Thanks,
> > Sunay
> > --
> > Sunay Tripathi
> > Senior Staff Engineer,
> > Solaris Kernel Networking,
> > Sun MicroSystems Inc.
> > email: sunay at eng.sun.com Phone: 650-786-6007 (W)
> Leonid Grossman
> > _______________________________________________
> > tsvwg mailing list
> > tsvwg at ietf.org
> > https://www1.ietf.org/mailman/listinfo/tsvwg
Solaris Kernel Networking,
Sun MicroSystems Inc.
email: sunay at eng.sun.com Phone: 650-786-6007 (W)
More information about the end2end-interest