The HTML serialisation doesn’t do self-closing tags. A trailing slash does precisely nothing: the parser just skips it. It’s dead weight. I strongly recommend against including trailing slashes, because (a) they serve no purpose to the machine; and (b) they mislead people into thinking they mean something, when they don’t—and indeed the vast majority of cases where I see trailing slashes used, they’re not used consistently, so some void tags have them and others don’t (if it was used consistently I wouldn’t mind so much).
(The XML serialisation is still a thing, in various contexts. Serve a document with an XHTML MIME type and the XML parser will be used; or embed HTML in SVG via foreignObject and it’ll be XML serialisation you need. Also note that SVG in HTML invokes… uhh… more XMLy behaviour from the HTML parser, so that the trailing slash does have its XML meaning.)
To add to what you said: the HTML specification has a bunch of rules to say how to parse a tag that doesn't have a matching close tag. These grew organically based on how browsers responded to malformed documents. At some point the rules were enshrined in the HTML specification.
Some tags such as <meta> and <img> are always considered as a leaf node ("void element"). The HTML parser knows that these tags cannot contain other tags.
Other tags, such as <p> or <li> have various rules saying how to automatically insert a closing tag if it's missing. For example, if you leave out the </li> tags in a list the HTML parser will parse it as if you had closed all the list items.
For these tags, what ends up happening if you try to use the XML self-closing notation is that the "/>" is ignored and then the missing close tag is inserted at a later point, which may or may not be where you want.
Slight correction: all tags will get a closing tag if it’s missing; it’s just less likely to be where you want on tags that “require” an end tag than those where it’s explicitly optional. Remember that HTML defines an exhaustive parser: all inputs produce a defined tree. “Invalid HTML” means a nonconforming document, but it doesn’t actually change anything; it’s just a way of making you feel guilty, or pointing out a place where it’s very likely you made a mistake or might make a mistake in further modifying the document.
See, the headings did get closed eventually, even without their proper closing tags; it just wasn’t where you wanted, either time. (The paragraphs, on the other hand, probably ended where you expected.)
Of your example, I think you typed it incorrectly in a couple of places. Here’s what I think you meant to express (plus a third one to make things clearer with the second having a bad trailing slash):
<ul>
<li>first
<li/>second
<li>third
</ul>
… and here’s what that is precisely equivalent to:
> Not sure why the the self closing tags became widely used
For a while adhering to the open/close conventions helped use tooling / libraries built around the assumptions for XML in dealing with HTML, maybe up through the late 00s/early 10s.
Any utility gains there faded as HTML5 conventions and libraries became more solid, but I still have a bit of the habit, especially with <img/> & <br/>.