The main difference between TCP and UDP, as this programmer discovered, relates to quality of realtime service.
Times to use UDP over TCP
* When you need the lowest latency
* When LATE data is worse than GAPS (loss of) in data.
* When you want to implement your own form of error correction to handle late/missing/mangled data.
TCP is best when
* You need all of the data to arrive, period.
* You want to automatically make a rough best estimate use of the available connection for /rate/ of transfer.
For a videogame, having a general control channel to the co-ordination server in TCP is fine. Having interactive asset downloads (level setup) over TCP is fine. Interactive player movements /probably/ should be UDP. Very likely with a mix of forward error correction and major snapshot syncs for critical data (moving entity absolute location, etc).
People talk like most games use UDP, but they don't.
Pretty much the only genre that does is First Person Shooters.
MMOs (like world or warcraft)
MOBAs (like Dota)
RTSs (like Starcraft)
Action RPGs (like Diablo)
All of these are action games and use TCP.
In most genres people would rather have the simulation pause when there is packet loss, and then run fast to catch up rather than have unreliable data about player locations.
Due to the large number of geographical locations you can have your servers in these days, it's common for most of your players to have pings <30ms. With that kind of latency and a few tweaks, it's possible for a single packet lost to be recovered very quickly with TCP.
> MMOs (like world or warcraft) MOBAs (like Dota) RTSs (like Starcraft) Action RPGs (like Diablo)
Dota 2 definitely does not use TCP. The Starcraft games I assume can also work with TCP but I doubt that they use it as a primary choice of communication. I would love a source for this.
Starcraft uses a mix of TCP and UDP[1]. I can't find the link now but I remember something about a late switch from TCP to UDP in the beta of the first StarCraft 2 game and some pretty unstable network gameplay while they were tweaking the error handling.
starcraft II uses TCP and UDP. I have verified this by reviewing my firewall, I have not gone as far to take pcaps and see what packets are doing what but I thought this was relevant enough to say.
as the parent mentioned -- UDP where time is more important than reliability. I disagree with his last comment on building your own reliability -- use TCP for that.
tcp/1119, udp/dynamic (listed app definitions for app-based firewall)
True but you can create a reliable UDP implementation and get the same TCP features you need (resending/ordering/ACK). Many games use UDP and then some form of reliable UDP where important messages are ACK'd (game start, game end, global events etc).[1] Almost every multiplayer game I have worked on has a flavor of this, where there is a bit or flag on important messages that need reliable tracking over UDP. Everything else is just broadcast like player positions, local action and more where packets can be missed and extrapolated/interpolated, predictive methods and lag compensation.
There can also be some packet loss increase when using TCP mixed with UDP[2]. Most games built recently probably use more RUDP than TCP. But when you need to use any UDP at all you may as well just go reliable UDP. TCP definitely works for non-realtime games and turn based especially but if you ever want some non essential systems updating where packet loss is ok (maybe world physics, or events that are just decorative/fun) then reliable UDP is best in my opinion.
SCTP was supposed to fix much of this but really reliable UDP does almost everything you need for realtime and not, so if investing in a network lib/tech that is the best choice.
enet[3] is a popular reliable UDP framework that was the base or basis for many others. RakNet[4] also has some nice implementations of it. There are many others now but some larger common systems based their systems on these, Unity is one.
This is terrible advice. These games use TCP because they can get away with the latency and often have game designs where latency is a non-issue (300ms in WoW won't ruin your raids or quests). Latency spikes are also very noticeable in these games. Starcraft adds latency locally to your inputs so they're played on all computers simultaneously.
I'd argue that none of these games require time-critical data and often have latencies much higher than they would with UDP. None of these games have networked physics either, TCP breaks down badly in that case. I'd also argue that great network developers come in very short supply and therefore most companies are left to their own misconceptions. (Then again, most game developers grossly overestimate their own skills, nothing new here :D)
I don't know where you get that "people would rather have the simulation pause when there is packet loss, and then run fast to catch up" idea but my experience is the exact opposite.
"Unreliable data about player locations" is a myth. Its just as "unreliable" in TCP with the difference being that in TPC the protocol will arrange for the packet be resent; at which point its no longer relevant to the current frame - you've wasted CPU and bandwidth, crippling your game designers in the process, often without realizing so. You rarely ever want a reliable world state, just reliable actions.
With UDP you actually rely on unreliability to only ever use the latest game state; actions you want reliable or ordered can be implemented very easily on top of UDP or left in TCP. Client-side prediction takes care of dropped frames such that missing a datagram (and therefore that frame's player positions) isn't noticed by the player. A delayed TCP packet will do the exact same, but waste CPU/bandwidth in the process.
I've seen my share of "senior" network programmers who couldn't make a networked engine that is actually stable. I can understand why some teams look at TCP and naively claim "problem solved!" But it doesn't make TCP the best choice because you can appeal to authority with big titles using it :)
I don't know why it lists all three ports under both UDP and TCP for WoW. WoW uses 3724 over UDP for it's voice chat, but the other ports are only used with TCP.
>> In most genres people would rather have the simulation pause when there is packet loss, and then run fast to catch up rather than have unreliable data about player locations.
I thought that modern games use interpolation to compensate for this, because network lags are always present in WAN.
It's easier and more robust to do NAT traversal with UDP than with TCP. The Xbox game network services implement TCP within the Microsoft domain on top of UDP!
I always wondered why nobody used UDP for bulk data transfer: Ship all the chunks over with a sequential number identifying each, then when you've reached the end, have the client request any lost packets, then repeat the cycle until all are transferred. After all, the client doesn't need everything in sequential order if it knows the size, and it can deal with holes as long as they're eventually repaired.
This gives you (in theory, I've never tried it) better throughout, and you get the equivalent of TCP's retransmission without the window semantics, since you move all the retransmissions to the end. On the other hand, it's probably bad for everyone else, since there's no congestion control.
> ship all the chunks over with a sequential number
Except that if you ship them too fast, they will at best simply build up in a buffer somewhere (e.g. near a bottleneck link), or more likely they will overflow causing many of your packets to be lost. So not only did the recipient not get them (and you have to figure this out and resend them later), but you also wasted a bunch of resources sending packets only to have them dropped.
And if you send them too slow, you are leaving some potential capacity on the table. So you need to send them fast, but not too fast.
So you add some additional logic to implement flow control and congestion control. And that's TCP, pretty much.
Exactly. The core feature provided by TCP is not reliability, that is trivially easy. What TCP gives us is robust flow control that can detect and adapt to current network capacity.
CNN has a video transfer tool for b2b subscribers that uses multiple UDP streams to increase transfer speeds. Instead it saturated and brought down our company firewalls, which were older and our main bottleneck. All for just a potentially few more megs of speed on TB sized videos. Gotta be careful with UDP.
That just sounds like your firewalls are made of pentium pros and duct tape. TCP can oversaturate a connection too, and a firewall should be able to handle large packets at line rate.
Yeah the firewalls were ancient and already scheduled for an upgrade. This helped accelerate the schedule for sure. The company had about 800 employees doing all kinds of things, watching Netflix, watching videos on YouTube and Facebook, downloading whole ISO's, not to mention many business processes transferring files around, but the second one person started using this video downloader tool it brought everything to a crawl.
I believe something like that was the underlying tech. I researched it at the time but I just can't remember 100% now, although I think that was it. It actually was implemented via a Java applet in the browser, so there was no way we could block or restrict a specific exe with group policy. The firewalls were already scheduled for replacement/upgrade, so for a few weeks we ended up just watching traffic and slapping wrists if more than one person ever tried to use the tool at the same time.
Sure. And not even the best known flow control mechanism. But nonetheless that is the important innovation of TCP, not reliability. The post by lobster_johnson upthread implied that you could use UDP for "bulk data transfer", which basically doesn't work without reinventing something akin to TCP.
There is actually such a protocol - UFTP. It was designed for high-latency connections such as satellite links. It can use multicast to allow multiple recipients over such a link. If memory serves, it was designed by Gannet to help transfer news content nationally for USA Today.
I've tested it on "long fat pipes" when evaluating it for a project a number of years ago. The numbers don't sound that impressive now. I was able to saturate the 100Mbps NIC of the slower server in the transfer. The transfer was from Northern VA to LA over the Internet. This was from data center to data center and the network capacity was such that I knew that the transfer rate would not cause any serious packet loss.
Using simple TCP over the same link, same computers was limited to around 40Mbps. More modern TCP implementations have a larger window and are able to better saturate high-throughput, high-latency links.
UDP messages are limited to 64K. It can be a pain to reassemble. lots of ugly corner cases.
We had need to send one type of message split due to size with udp. We were on the same subnet sending and receiving so it worked almost always. We were also using multicast.
Its very very hard to figure out whats going on when things don't work ( You don't know if the control messages are getting through, when you ask again or if the message just isn't being sent). At some point you are reinventing tcp/ip.
I always liked the way udp messages are received though, all or nothing. Not the byte stream that is tcp which needs to be broken into messages manually.
Picking a smallish, fixed chunk size should work, and it could even be made to self-regulate. Reassembly is trivial -- allocate the file first, write chunks to their right location. And for the control stuff (i.e. requesting chunks) you just use a TCP connection. UDP for the data only.
If you do the obvious work to convert UDP to a reliable streaming mechanism, you will generally more-or-less reinvent TCP, only probably more poorly, and would probably be better off just using TCP. UDP is a good choice when you have options or requirements that TCP can't accommodate. For instance, the classic case of streaming audio has both; it both requires timely delivery and has some very sophisticated techniques for dealing with missing packets with a lot of local intelligence about human perception and such built into the codecs. If you can drop packets without issue, if future packets completely supercede past packets (FPS network gaming for instance), there's a lot of possibilities, but if you're just going to recreate reliable streaming you probably be net losing on account of the effort spent doing that instead of something useful, and bugs in the implementation.
Why are so many comments here ignoring the purpose of the article?
The purpose of this is for games, not web traffic or torrent2.0.
In real time games where you're syncing physics states or player locations data >200ms isn't just useless it's actually negative.
This article is 2+ years old, and is the foundation I suspect for libyojimbo which hybridizes UDP low latency communication for large player real time games and reliable messaging (basically custom TCP?) for less time critical things.
There are many protocols that use this approach, Aspera FASP is possibly the most advanced one. It uses a TCP control channel to manage the rate on the UDP data channel.
On LAN, this is basically how Symantec Ghost works; it multicasts the image over UDP, and each client figures out what they missed. Those chunks get retransmitted at the end.
Although to be fair, given that with TFTP each packet must be acknowledged before the next one is sent, you're actually achieving a way worse throughput than TCP. Not a very good example on how to use UDP efficiently IMO.
What it really comes down to is file block size granularity. TCP is simply using a much finer block size (~1500 bytes) which means a higher likelihood of arriving than for a large UDP packet. Furthermore, you'd still have to implement your own checksumming algorithm to ensure data integrity as UDP doesn't require a packet checksum for IPv4. I suppose you could offload file-block integrity validation to another thread so you can receive and verify in parallel, but at some point you have to checksum every block anyways which means the full file must be processed to figure out where corruption might be before sending retry requests for the missing packets.
I see what you're getting at by trying to avoid processing the packets in order if it wastes time, but ultimately I think its a micro-op that introduces other bottlenecks. That's why bit-torrent still typically uses TCP even though they could potentially receive file parts in any order.
But bittorrent breaks the files in to chunks it can receive out of order and sends those over TCP, so TCP only preserves ordering within a chunk. It might even get chunks from several different people.
The real answer is that chunking over TCP is efficient enough that little is gained by chunking over UDP.
That sounds like a TCP congestion control mechanism, except worse! TCP leaves much to be desired in terms of performance, largely because their congestion control fails miserably on lossy links (TCP RENO). There are better backoff mechanisms that can be added on many distros, like Westwood. And then there are altogether more efficient protocols like QUIC. But designing one for general purposes from UDP is easier said than done.
NFS used to default to UDP for the same reasons that games still would use UDP, it just needed the data as quickly as possible since it was blocking IO. Now a days, we don't need UDP for NFS because networks are a lot faster and the advantages of TCP start to over rule UDP.
The original reason was that using UDP was much faster CPU-wise back in the 80's. There was a requirement to go fast because it was used for transferring data in bulk, not events like games.
When Linux came around it just followed suit. TCP support came to NFS only later with other bells and whistles in later NFS versions, and Linux stuck with basic NFSv2 for a good while.
NFS was designed for local-area networks, so it didn't need to worry about working over the Internet.
In addition to the network being much faster relative to CPUs back then, the TCP stacks hadn't seen the tuning they have now. Making TCP fast came later.
NFS (was) stateless and idempotent. Lose a packet? Send it again! Which is mostly what you want, since there can be dozens of processes performing IO and they're all independent. Last thing you want is head of line blocking.
> I always wondered why nobody used UDP for bulk data transfer
1) Bulk data transfer doesn't work over UDP.
There is no flow control whatsoever with UDP and no guarantee, it sends stuff without distinction between a 56k uplink and a 10Gigabit LAN. The packet loss and lack of reliability are atrocious when you send non negligible amount of data.
2) You'd need to create a protocol on top of UDP to handle the basics (controlflow/error/retransmission), that's equivalent to re-inventing TCP. Don't re-invent your own TCP, just use TCP ;)
UDP is usable for this when used to transport a file transfer protocol. I've seen for instance Kermit over UDP.
Basically, if you transport data using a protocol that already takes care of integrity, using TCP is more or less redundant and just adds annoyances (connection management).
Back in the day of hard drives, the random access I/O would be nasty and you'd have to be careful about creating sparse files, so you'd end up double writing the missing packets. you'd also still need congestion control so might as well just use TCP to retransmit.
Bad for everyone else, and it's up to the implementation to handle flow-control. Congestion-control handles network related loss/latency/load. Flow-control helps with receiver related loss/latency/load.
This isn't really true, or at least not the main reasons to choose UDP or TCP .
In modern days UDP is really only used in situations where you can afford packet loss and CPU is expensive, or when you need to bypass the built-in behavior of TCP on your OS. It's not as much about latency or data loss, usually UDP is used in places where processing power is expensive or load is too extreme to justify TCP.
Sometimes, like with DNS, it's better to just resend to request than burden a massively parallel server with maintaining TCP state and checksums.
The main issue I have with the above comment is that UDP absolutely does not guarantee better latency or have higher priority than TCP packets. The packets usually queue in the NIC buffers and downstream identically regardless of which protocol you use. The main draw of TCP besides data and ordering guarantee (which are CPU costly) is that (depending on your operating system) it follows some kind of automatic rate limiting algorithm be it TCP Vegas(Linux, delay based) or Reno (windows, loss based). These algorithms attempt to limit your connection "stream" to the line rate automatically and play somewhat nicely with each other on large scale.
In contrast, the main problem with using UDP is figuring out a smart way to rate the limit to avoid excessive packet loss. In some protocols like DNS anycast type things where only raw throughout matters, who cares if responses drop. Other times, like with LEDBAT or Microsoft's BITS, your main reason for using UDP is to roll your own rate limiting protocol, since otherwise you're stuck with what the OS gives you on TCP.
I'm not sure I follow your argument. Say I want to communicate from player A the position of player A to player B. The position is really only relevant for a 200 ms time frame or so (the game might support teleports, jumping, dashing, etc). In a TCP setting, even if a packet is old (>200ms), I have no choice but to send it regardless, and all new packets must queue behind it. The whole point behind using UDP for games is that I can control what the resend policy is, which determines what ends up in the buffers you are referring to in your post. TCP gives me not only little to no choice in the matter, but often does the worst thing in a critical situation, especially in a game where input latency is 6 ms and the tolerance for network latency is <50 ms.
You're right in that UDP will let you send the most updated data without requiring everything to be sent. My main issue was that UDP does not give you lower latency. TCP algorithms are extremely aggressive to the point I doubt you could get more recent packets on the wire with UDP.
The problem is, the OS throws everything into the same buffer, and unless you're clairvoyant you won't be able to skip sending old packets into the buffer until it's too late. Basically by the time you know the data is outdated it's already on the way to the NIC buffer and you can't stop it, making TCP and UDP equal in this regard
It's way more rediculous than you think. OS network stacks try to do the reasonable thing in most cases if possible. In the case of Linux TCP Vegas is very unlikely to drop any packet since the main algorithm is RTT based not drop.
Red and hopefully codel can combat buffer bloat to some point but in the end all packets go to the same place
I'm getting the impression that you're mostly only taking the endpoints into account. There's tons of stuff happening in between the peers: having worked on specialty routing equipment, I can add that the prevailing philosophy was that generic UDP packets were lower quality-of-service than TCP. That is, if the router was under high load, UDP would be dropped before TCP.
Another article on this site addresses TCP plus UDP mix. The tl;dr is that they should not be used at the same time because TCP retransmission will interfere with UDP (and routers give TCP priority).
As long as TCP happens outside of the "hot" loop, it's fine.
What routers? Is this something that stupid routers did in the '90s and people are still worried about, or are modern routers actually behaving this way? (And if so, why?)
TCP retransmission interfering with UDP sounds like something that shouldn't be a problem on any sane network, especially if the application authors are smart and use something like LEDBAT for their TCP traffic when they're also doing latency-sensitive UDP communication.
The internet is generally built on the implicit contract of "TCP-friendly flow control". If your homegrown protocol backs off slower than TCP, some router configurations will drop your packets first.
There is no general policy against non-TCP packets of course, as lots of important internet services are non-TCP. Just flows that don't respond to congestion.
It's not a vendor specific thing, all the major router vendors provide ways of doing this.
> If your homegrown protocol backs off slower than TCP, some router configurations will drop your packets first.
I think that's only if you're being less responsive to congestion signals (drops, ECN marks), and you're using more than a fair share of bandwidth or trying to use any available bandwidth. Fixed but low rate flows (eg. VoIP) shouldn't be penalized until the link is congested with so many flows that the VoIP is trying to use more than 1/N of the bandwidth.
I'm well aware of how dumb home routers can be, but a general de-prioritizing of the transport protocol that DNS uses is both really dumb and pretty easy to detect. Are you aware of any router vendors that have actually shipped such a configuration? (Or for that matter, any consumer router that has shipped with any prioritization rules enabled out of the box?)
DNS isn't VoIP, but most DNS traffic is very much latency sensitive. When you click a link to a typical modern web page, you trigger multiple HTTP connections to load the page and its many resources hosted on dozens of domains. The DNS lookups are on the critical path for all of those requests.
Yes, but the HTTP requests themselves dominate the overall load time, even if DNS takes a bit longer. Plus there's caching to reduce somewhat the need for the DNS requests.
(Of course, I'm pulling all of this out of my nether regions without a lot of thought, and you may well be right, the sheer volume of such requests might lead to problems.)
It looks like on Windows a typical DNS retry timout is 1 second, with backoff if multiple retries are needed. If a DNS packet needed for loading a web page gets dropped, it's very likely to increase the overall page loading time. Re-ordering DNS to be delivered only when there's a lull in the large TCP packets would almost serialize the loading of resources from different domains.
DNS caching is a good thing, but it certainly doesn't eliminate this problem. Web browsing still produces a lot of DNS requests, and any cache upstream of the bottleneck (ie. any cache operated by your ISP rather than in your own router) doesn't help against loss during congestion.
Without active queue management, a single bulk upload can break DNS. When the bottleneck link (usually your cable/DSL modem) is saturated by the bulk upload, it will start dropping new packets that come in while the queue is full.
When a single packet worth of space frees up, the chances are that the bulk flow will instantly take it, such that nearly all outbound DNS packets get dropped, and DNS queries time out.
The solution is active queue management (CODEL is a good choice) at every bottleneck.
I'm a network engineer and I think this is the first time I've ever heard the claim that "routers give TCP priority" (ICMP is often de-prioritized but that's on the control plane).
Exactly which routers are alleged to be doing this?
There are quite a few "reliable UDP" implementations out there - I wonder if the same considerations would hold. Guess it might depend on the implementation of reliability using UDP.
Use UDP only if you satisfy all the following conditions:
* Don't care about loosing data
* Don't care about receiving data out of order
* Not sending stream of data but individual messages ( < 580 bytes)
* Don't care about the other side of the connection being dead or terminated, and not knowing about it
TCP establishes a connection when it starts and it has build-in timeouts (and various mechanisms) to detect dead connections. TCP will signal to the application if the destination is unavailable or lost.
The requirements make sense and the common usages for UDP fit these requirements.
It's correct that [almost] nobody ever uses UDP. It's a niche protocol for a very few specific use cases, one of which is real-time FPS/RTS/MOBA games (which fits all the criteria I listed).
The sensical comparison would be TCP vs whatever protocol you build or use. It might be your homegrown protocol, but if you have problems with TCP, you should probably look at other ready-made protocols first. For games, there's eg ENet.
(Not to say that TCP is generally out of the question for games, eg. WoW uses it apparently successfully)
A datagram-like protocol with the reliability of TCP might be nice.
I once implemented a communications protocol that worked with variable-length independent messages over TCP. It felt a little silly pushing distinct datagrams into a stream, that would be chopped up into a packets, sent to another PC, where the OS would then reassemble those packets back into a stream, only to be chopped up again in messages.
Implementing the system for chopping up the stream really felt like it should have been done by a lower level as well, not in an application
I think that rule is a little too simple, because it might be correct to retransmit but not to have the retransmission block processing of later packets. I'd say one should probably use TCP when any of the following are true:
1) The correct interpretation of later packets depends on side effects of earlier packets (e.g. commands in an SSH session, OpenGL commands)
2) Raw bytes-per-second throughput is a major consideration (e.g. file transfer, streaming video)
3) The application readily tolerates high latency and jitter (e.g. email, text chat)
also wherever you need real time there is mostly UDP in use,
For SIP + RTP,
webrtc is a little bit complicated but the real time part is based on UDP.
all those have a mechanisms implemented for packet loss and how to handle it.
Questions like "more secure" and "less secure" only make sense in some specific context with a well-defined threat model: what it is that you want to keep safe from which classes of threats.
For many (if not most) threat models, the lack of a sequence number does not provide adequate security.
Times to use UDP over TCP
TCP is best when For a videogame, having a general control channel to the co-ordination server in TCP is fine. Having interactive asset downloads (level setup) over TCP is fine. Interactive player movements /probably/ should be UDP. Very likely with a mix of forward error correction and major snapshot syncs for critical data (moving entity absolute location, etc).