It's been a slow process, and hence perhaps underappreciated, but it's very gratifying that over the last 15 years patent-unencumbered media codecs have won. The companies that contributed to this --- Google, Mozilla, Cisco, and others --- and especially Xiph that got the ball rolling --- deserve a lot of credit.
They have? Are we talking images only because in video mp4 and .h264 and .h265 are pretty much it. Apple/Safari still doesn't support vp8/vp9/webm/av1 for video or ogg for audio
> Are we talking images only because in video mp4 and .h264 and .h265 are pretty much it. Apple/Safari still doesn't support vp8/vp9/webm/av1 for video or ogg for audio
h265 is doomed because of the multiple patent pools that have formed and the extreme price hikes compared to h264. It's a risky technology to build on. That's the best ad that AV1 can have, and many members joined the alliance for open media for that reason after it became clear that the patent pool situation wouldn't resolve.
As for your browser support question, yes, Safari has traditionally been anti-ogg, but a few years ago Apple joined the alliance for open media and added opus support to their browsers (although only in the caf container, for some weird unexplainable reason).
Open codecs have won in the lossless audio domain (flac). There is no reason they can't win in the lossy video and lossy audio domains too (not sure about lossless videos but FFV1 seems to have lots of support by archivists who want open technology).
If your world of Video = Youtube. Then HEVC failed.
If your world of Video = Every Video usage on the planet Earth. Then HEVC isn't doing so bad, every shipping TV has had HEVC decode, most ( if not all ) Smartphone, PC, Tablet has hardware Decoder shipped.
Sure, but it was massively pushed as the successor of h264 as the new de facto video codec standard, after almost a decade in use it has clearly failed in that respect.
IMO this codec generation saw no winner, instead h264 remains the undisputed king, the next battle for the crown will likely be between AV2 (or whatever it ends up being called) and VVC (Versatile Video Coding).
Yes. And I think there is something more than licensing. x264 managed to push H.264 way beyond what anyone could imagine. It wasn't until 2017/2018 did x265 or in fact any commercial H.265 encoder had clear advantage over x264 at 8 / 10Mbps+ bitrate. For HEVC 4K it was obvious, but 4K content never really took off, and h.264 for 1080P was good enough at those bitrate.
I dont know much about AV2, but VVC is looking very promising. I just hope they dont mess this up.
Sure, because it is "good enough" for what it does. But recently, with the advent and demand of 2k and 4k videos, people realised the need of, and have started appreciating the capabilities of HEVC.
When it comes to the popularity of video encoders, my unscientific way of judging it is to look at what the pirates are using. If you look at the torrent scene, most of the popular tv series and movies are now also available HEVC encoded and they are popular too (especially the high quality 2k and 4k Blu Ray rips). It's rare to come across any VP9 videos, and HEVC is undoubtedly number 2.
Personally, even I've started re-encoding my video library with HEVC, once I discovered that the "medium" setting takes about the same time as H.264 encoding and gives an acceptable quality for (sometimes) half the file size.
I do keep an eye on encoders, and while it is good to know the work going on this field, practically speaking nearly everyone is moving on to HEVC. Nearly all the other competitors are in development and unusable because of their really slow encoding time.
> practically speaking nearly everyone is moving on to HEVC. Nearly all the other competitors are in development and unusable because of their really slow encoding time.
Forget commercial softwares, even the popular open-source video encoders like Handbrake or AviDemux or FFMpeg do not have AV1 or any other competing encoders in their release. They do support HEVC though. That itself is telling on the state of the competitors.
Note that I am in no way an advocate for HEVC (even if I sound like one :). I am just speaking from a practical point of view, as a user. If tomorrow there comes another encoder that takes less time to encode, and offers better compression, I'll immediately dump H.264 and H.265 for it.
Decoders for which the vendors still face unknown patent licensing fees.
After the HEVC licensing debacle, the industry isn't going to jump on a bandwagon with unlimited unknown downside again, not when royalty-free alternatives like AV1 exist.
It depends on what you are measuring. If we go purely by web traffic volume, then whatever YouTube is doing is what's "won". AFAICT, that's VP9 to everything that supports it which seems to be all desktops and new Androids but not Apple devices. Android devices of the previous generation fallback to VP8 where there's hardware decode support.
I wouldn't lump them together. There is a chasm between the adoption numbers for H.264 and H.265, HEVC failed gain the momentum AVC did --which is exactly the point GP is making.
MPEG-LA is indeed well connected to lobbying parties and Hollywood, which established HEVC as the codec for 4K Blu-ray discs. (I personally don't remember the last time I borrowed an optical disc to watch a movie. I actually don't even own any type of optical disc driver anymore!). However, VP9 dominates over HEVC in streamed media.
This is a major improvement. It's better than WebP by a bigger margin than WebP was better than JPEG.
Lossy WebP couldn't even match JPEG in some aspects, like full-resolution color (4:2:0 only, not even full 8-bit depth). In terms of quality or features AVIF doesn't have such "buts". It can support HDR and high depth we're likely to want in the near future. AV1 has palette blocks, so it's no longer all-or-nothing choice of lossy vs lossless formats.
I am worried that the AVIF spec is complex (like HEIF, it's a bag of image-related features, more like a PowerPoint file than an image), but probably only commonly-used features will survive out of it. Early JPEG also used to have many more modes than are usable today.
If you like questionable hacks, you can use AV1 codec today in Firefox and Chrome by embedding <video> element with a 1-frame AV1 "video".
The JPEG images in the article are much worse than what you can achieve with JPEG export in Preview.app. While they don't look like AVIF, they aren't nearly as trash as the article makes JPEG's out to be.
I worked on a team responsible for image assets. I recall the web team asking for webp support from our image server. I suppose nowadays using `srcset` you can get some optimization in the ratio between file size and quality while still providing fallback for unsupported browsers. Given our image service was dynamic (e.g. it generated then cached images at different sizes, qualities, etc from a single high-res source) adding a new format wasn't that difficult. It meant that the image heavy pages would display more quickly which was a concern especially for mobile. So I am a big supporter for more and better formats.
That being said, this AV1 vs. HEVC / AVIF vs. HEIF stuff feels eerily similar to HLS vs. DASH. It's not like these formats have drastically different properties (e.g. GIF vs. JPG vs. PNG) - they are all quite similar. I just want to fast-forward time until whichever one is going to win is standard. It's like VHS vs. Betamax or Blueray vs. HD-dvd. Please just get it over with.
The difference is that HEVC/HEIF has at least three different patent licensing organizations that all want to be paid. While AV1/AVIF is supposed to be "patent free"
I heard the same about h.264 vs VP9 and it turned out that patent licensing wasn't a long-term issue. For a while you got that license from Flash (if y'all remember when Flash was the way Youtube worked, and I recall mention that it was one of the primary reasons Flash Player was never opened sourced). Then the license for h.264 it was included in every copy of Windows 8 (IIRC). Nowadays no one even mentions the h.264 license.
I'm no expert on the new license but it seems the restrictions on HEVC are relatively light. I'm pretty sure it is free to compress and distribute media but the license affects hardware and software encoders/decoders. So maybe your HEVC enabled video card with built-in decoder will be a dollar more expensive. I know that every Apple device shipped comes with HEVC hardware standard for the last few years. I would wager most Android devices too.
From English Wikipedia article "Advanced Video Coding", with cited sources:
"The commercial use of patented H.264 technologies requires the payment of royalties to MPEG LA and other patent owners. MPEG LA has allowed the free use of H.264 technologies for streaming Internet video that is free to end users, and Cisco Systems pays royalties to MPEG LA on behalf of the users of binaries for its open source H.264 encoder."
(...)
"On August 26, 2010, MPEG LA announced that royalties won't be charged for H.264 encoded Internet video that is free to end users.[74] All other royalties remain in place, such as royalties for products that decode and encode H.264 video as well as to operators of free television and subscription channels.[75] The license terms are updated in 5-year blocks.[76]"
> streaming Internet Video that is Free to End Users
Yeah, doesn't sound like that would include Netflix, etc. Nor would it include a huge amount of other video that is distributed in other ways. It also doesn't cover using your computer to encode h.264 like your webcam almost certainly does.
The fact is you probably have several h.264 enabled devices on your computer right now that are licensed. And just because Cisco is covering the cost for it's open source encoder doesn't mean the license doesn't exist.
My feeling is if HEVC wins the race it will be the same. 90% of people won't even notice since your license will be included in your hardware/OS without you ever knowing.
No, HEVC licensing is an insane mash of several patent pools and a few companies trying to extract a buck out of anyone that uses it. It's significantly less clear and more complex than what H.264 was and the license fees are higher as well... If you even have a legal team that can figure out who needs to be paid.
I'm on the phone so I can't really look up all the resources, but they shouldn't be hard to find. But this licensing mess is also why most oss development stalled and we still don't have an encoder that would be anything close to the quality of libx264. Instead, most important OSS experts have moved on to AV1.
> HEVC licensing is an insane mash of several patent pools
Yes, but they aren't coming after users. Look at the pools, it is mostly device manufacturers like TV, audio systems and mobile phones. They are holding each other in a mexican stand-off. Two of the three largest pools have already agreed to the same $0 royalty for non physical distribution that h.264 has. They want to extract that money from their competitors in the physical device space.
Fair enough, the uncertainty around final pricing is slowing adoption. But that can ossify quickly, although no guarantee it will.
I hate I'm being drawn into a patent argument when my point is: it won't matter either way. Even if HEVC wins 99% of people will never be affected by a license fee. Let them scrap over $0.25 royalty fees on TVs all they want. In the end I don't care if it is HEVC of AV1 that wins. I just want the fight to be over so I don't have the cognitive overload of having to support both.
>No, HEVC licensing is an insane mash of several patent pools and a few companies trying to extract a buck out of anyone that uses it.
The user paid less than $1 per devices for turning on the blocks of Hardware Encoder and Decoder which is on by default for almost of all of the 1.2Billion Annual Smartphone shipment and 250M PC as well as 300M of Tablet.
As far as I am aware, there are no royalty fees to be paid if you are delivering your content on Digital / Internet Streaming. Only if you are on disc such as BlurRay.
I am not a lawyer, and I realize there's a fine line between genuine concern and speading FUD, but in the spring of 2019 I looked into the patent situation around the HEIF container itself [1] -- the container upon which AVIF builds -- and skimmed through the 5 US patents I found, which cover some techniques that can be used in the format.
Most of them can probably be avoided for the purposes of an AVIF file, but patent US20160232939A1 [2] in my reading seems to be about in-container signalling to express relationships between a "static media item" and "one or more entities" that together "form a group", and "indicating, in the file, a grouping type for the group". The patent appears to be written in a way to allow this definition to encompass, say, a thumbnail and a bunch of frames thereafter, or, say a master image and a set of pictures derived from it, or alternate camera angles of the same thing. Some of these techniques sound like stuff we've seen before, but as is common in patents, the precise wording of claims is often key, and this is where patent lawyers come in.
A thorough look of the AVIF specification [3] and the patents registered with the MPEG LA about this format [1] is likely wise before any widespread deployment that makes use of advanced features of the HEIF container; using it to hold exactly 1 'one-layer' still-image is probably fine.
Additionally, in my reading [5], the HEIF reference software released by Nokia [4] includes a patent grant for non-commercial evaluation, testing and academic research only.
I'm not trying to defend Google's behaviour, but patents are useful for defending a technology you want to remain open (eg proof that someone else didn't invent it) as much as they can be used to generate license fees. Defensive patents are effectively a way of ensuring the prior art shows up in a patent search. It's possible that these patents are not going to be used to restrict AV1's use.
>It's possible that these patents are not going to be used to restrict AV1's use.
Except that is not the case. They email its author on whether they could use it. And in a turn of an event they patents it precisely to stop other video codec from using it.
That 420 vs. 440 comparison animation is rather problematic. GIF can only represent 256 colors. Most of the artifacts in those images are due to GIF-specific dithering.
It is benchmarked here against WebP, HEVC, and JPEG2000, as well as JPEG. That's what all the graphs are. AVIF wins quite convincingly on the metrics they've used.
Interesting. Personally, I’ve been using VMAF extensively in my imager.io project. With regards to only testing the luma channel, someone submitted a PR for such some time ago yet it’s never been merged; I was under the impression that the UV channel isn’t significant (in practice). Personally, I think VMAF is great for images. For me, the biggest downside of VMAF (implementation wise) is that it’s not currently thread-safe.
For some weird reason browser manufacturers strongly oppose adding support for any new media formats. They will only accept a new format once they decide that's a good political move. If I were Google/Mozilla I'd add support for everything ffmpeg and imagemagick support.
As soon as they support something, they can never drop support for it because some sites will have started using it, and some sites will never stop using it. Look at how hard it has been to deprecate Flash.
By enabling support, they are commiting to maintain same-or-better compatibility pretty much forever. Thats a big commitment if your code is based of someones 'for lolz' patch to ffmpeg...
If memory serves it used to be somewhat common in Unixy land before .png was added to browsers. There may have been long-lived documentation or academic pages with xbm that they were reluctant to break.
The official avif github [1] references Kagami/avif.js [2]. It's intended as a server-side component to be installed, and uses service workers to on-the-fly repack AVIF images as AV1 videos. The code is licensed CC0, and is easy to read, so you can tweak it to your needs.
If the browser can't natively decode AV1 videos, it calls dav1d.js [3] to decode, which is a webassembly port of dav1d [4].
I'm guessing that Netflix compares not at all about compression time for this; the most used assets are probably downloaded millions of times and change rarely.
I'm no fan of 4:2:2 (2:1 chroma subsampling), but the comparison looks a lot better for JPEG when you use 4:2:2 (which is the usual JPEG default). Using 4:4:4 instead drives the blocking artifacts way up, and isn't fair to JPEG.
I'm going to put my cynical hat on for a bit and notice that the company behind this format, Netflix, has a walled garden. They have a streaming service that is consumed only through a handful of apps that they are in full control of. They're not doing this for the community, for web standards, or the greater good. They're making this format simply to save some bandwidth for themselves, in a way that won't translate to the general community. The second picture in the linked blog article is literally a diagram of this garden. (Titled: "Compressed image assets destined for various client devices...")
I had a similar comment about the JPEG XL format (mostly developed by Google) and the CSS "Display P3" colour space extension (Apple), both recently featured on YCombinator. These mega-corporations are building an ecosystem where in 2020, the future, it's impossible to send a wide-gamut, 10-bit, or HDR still image to anyone via any of the following: web standards, chat, email, or document exchange formats. The best you can do is send an 8-bit SDR sRGB image and hope for the best. PC monitors, tablets, and most phones have no consistent support for colour management, 10-bit, or HDR. Televisions are leaving the entire PC and Mobile ecosystem in the dust. The closest approximation we PC peasants have is to upload a HDR YouTube video, send a link to it, and hope the viewer uses an newish iPhone. That's just sad, isn't it?
The stewardship of web standards by Microsoft (#2 biggest company), Apple (#3), and Alphabet Group (#4) have led to this. Now Netflix (#50) wants to throw their unnecessary format into the fray, almost completely ignoring the presence of JPEG XL and HEIC. They mention these alternatives in passing, and then notably don't compare image quality, features, or the compression ratio of those to their own format. You see, JXL is a Google's thing, HEIC is an Apple thing, and AVIF is a Netflix thing. So we're going to end up with as many image formats as there are walled gardens. I bet you too can't wait for whatever image format Facebook comes up with specifically to reduce their CDN bandwidth utilisation of Instagram pictures... only.
Notice also that their "idea" of making an image format readily available is a docker container, which is the most insane thing I've ever seen. Where's the Photoshop plugin? The Lightroom plugin? The Windows 10 image codec? Oh wait.. those are Adobe and Microsoft things, so... nowhere to be seen. Netflix Pty Ltd is not in this game to help someone else's walled garden.
No still image interchange for you peasant! Sit down, stay in the garden, and stream that content...
Jpeg XL is not a Google thing, and neither is HEIC an Apple thing. Nor the AVIF part.
AVIF is from Open Media which is indeed a Google Thing. Netflix just happen to take side with OM. JPEG XL is done by a Google Employees as well as other contributors put forward to the JPEG committee. i.e It is not a Google only format.
HEIC is really just HEVC Codec in an image format developed by Nokia and submitted to MPEG ( Not to be confused with MPEG-LA ).
So really JPEG XL isn't a walled garden. And support everything you described including HDR. The problem is Jpeg XL doesn't do so well in small image size as compared to HEIF or AVIF.
When you go to 0.1 BPP, every codec will look unacceptable, but the current encoder of JPEG XL looks worse than video codecs.
We will likely be able to extend the scale of handling bad quality down to 0.1 to 0.25 BPP, but it definitely has not been a priority since we don't anticipate that users actually would use it for photographs.
(When an image has a low noise level or less features, then we naturally can compress it down to 0.00x BPP with JPEG XL, too. The 0.5 BPP rule of thumb is for normal vacation photographs and selfies :-)
> Jpeg XL is not a Google thing, and neither is HEIC an Apple thing...<snip>...Employees as well as other contributors put forward to the JPEG committee.
"Technically open" doesn't help much if each of those formats is primarily pushed by one mega-corp, optimised for their specific use-case, and no industry-wide effort materialises to establish widespread interoperability.
Just last week I tried to convert raw photos to HEIF images on Windows so I could share my images in maximum quality with a bunch of iPhone users. It's basically impossible. The only tool I found was Gimp, but it doesn't properly understand colour spaces and the output looked totally wrong. I'd need OSX or IOS to author files in this format, making HEIC an "Apple thing" for all practical purposes.
It's a safe bet that Google will embed JPEG XL support in Chromium, right? Will I be able to email a .JXL file to an IOS user? Will a Windows PC user with Outlook on their desktop be able to see the image embedded in their email without having to save it and open it again in Chrome? Can I paste a .JXL image into a Word document and have its various features be correctly preserved, or will this be restricted to Google Docs viewed exclusively via Chrome?
Look, here's the thing: All of these post-JPEG formats are nice and all, but are worthless to me and a billion other people unless there's proper effort put into mass adoption. Real industry-wide cooperation, not just token openness.
As I said, it is 2020, and nobody has a viable option for distributing still images in anything other than a handful of 20-30 year old formats.
> The problem is Jpeg XL doesn't do so well in small image size as compared to HEIF or AVIF.
So then why isn't NetFlix working with the JPEG XL group to add the AV1 codec as one of the encoding options? Why are they instead creating yet another standard with no adoption likely outside of their walled garden?
JPEG - The "popular one"
JPEG XT - Never heard of it
JPEG-LS - Never heard of it
JPEG 2000 - Heard of it, never used it
JPEG XR - Never heard of it
AIC - Heard of it, never used it
JPEG Systems - Never heard of it
JPEG XS - Never heard of it
JPEG Pleno - Never heard of it
JPEG XL - Not yet useful, will likely never use it
Clearly, whatever they JPEG group is doing, it's not working. They got lucky once, and in the 27 years since have failed to reproduce that success.
This is the core point of what I'm saying: The reason that GIF, PNG, and JPG got widespread support has more to do with luck rather than the standardisation approach. The laundry list of failures and the sad current state are evidence that luck is not a strategy and something different must be tried.
I put this in the same category of failure as IPv6. The IETF got lucky with IPv4 and figured that "more of the same" will work. They didn't get lucky the second time.
I wish I had the "political clout" to bang some heads together at these megacorps, make them sit down together and fix this.
Realistically, I foresee another decade of 8-bit, SDR, sRGB images stretched incorrectly to whatever random gamut the $150 monitor of the recipient is capable of displaying.
>"Technically open" doesn't help much if each of those formats is primarily pushed by one mega-corp,
Then no Format is Open by your standard. Apple or Google refuse to support anything would instantly make that format not Open by your standard. And if Apple drop Jpeg, would Jpeg not be an open standard ?
What you are describing is politics, nothing to do with the standards itself.
>The only tool I found was Gimp
That is the problem with the tools. If Gimp Dev decide to take side in the format battle and not support it properly, then is that the fault of a standard?
>So then why isn't NetFlix working with the JPEG XL group to add the AV1 codec as one of the encoding options?
Because it doesn't work like that.
>Clearly, whatever they JPEG group is doing, it's not working.
> What you are describing is politics, nothing to do with the standards itself.
Standards are about convincing a large group of people to do something for the common good, sometimes going against their most direct best interests. Nearly a textbook definition of the word "politics".
> They are used in space other than the Web.
Yeah, the tiny, borderline non-existent space that is outside of: desktops, phones, tablets, web, email, collaboration, digital cameras, desktop publishing, or just about any mass-market use of computing you care to name that involves imaging.
The only usage of a JPEG standard other than the original JPEG I'm aware of is JPEG 2000. It has some niche uses for digital movie distribution to cinemas. Not streaming, not cable. Cinemas. Only.
The HEIF and HEVC container are the same. So if they use the HEIF container they already have libraries on devices that can handle the format, metadata, etc. I don't believe Netflix is delivering any Matroska content so they have no media pipeline or tooling for the format. It's the same reason they didn't base the container format on RIFF or ASN.1.
Since they are proposing it as a common standard, I don't think it should matter what they already use, in comparison with something that's actually free to use.
If HEIF is free, then fine. But MPEG stuff is not created free by design, so it has to be stated somewhere.
The HEIF container is the MPEG-4 Part 12 ISO container, the same used for MP4 video and JPEG2000. This lets it fit inside existing workflows and media stacks. AVIF just ends up and additional codec in a familiar container.
Anyone needing to pay license fees for MPEG-4 Part 12 format patents (if they exist) is already covered by usage for video or HEIF. Anyone not needing to pay licenses will continue to not need to pay licenses in they use AVIF.
Not sure what that means. Sounds like some still might need to pay for it. That's an unacceptable approach for a standard proposed by AOM which stresses the royalty free focus (in contrast with MPEG), which was exactly my point above.
I don't know of any patents on the MPEG-4 Part 12 format. They might exist but I don't know of them. If you implement AVIF you'd have to pay whatever licenses might exist over the format. If there's no licenses over the format, you pay nothing. If there are licenses but you're already distributing MPEG-4 video content, JPEG2000, or HEIF your licenses you'd pay for those would cover your use of AVIF in the MP4 container.
So to my knowledge the ISO MP4 container doesn't require license fees so there's no real downside for using it as the container for AVIF.
