URL Extractor

Finding a link is trivial. Knowing where it stops is not

Spotting https:// is a single regular expression, and that is why so many of these tools exist. The interesting question arrives one character later and almost nobody answers it well: how far does the address run?

Prose gives you no marker. The sentence ends, and its punctuation sits flush against the last character of the link with nothing between them. See https://example.com/page. reads perfectly to a person and lies to a pattern — the path is /page, the stop belongs to the sentence, and a greedy match hands you a dead link that looks completely fine. You discover the problem later, one click at a time. Commas in a list of sources do the same thing, as do semicolons, colons and the angle brackets an email client wraps around a URL.

Brackets are the case that breaks everything else

Trailing punctuation has an obvious fix: strip it. Brackets do not, because a URL is allowed to contain them and plenty of real ones do. The Wikipedia article about a film that shares its name with something else ends in a closing parenthesis, and that parenthesis is part of the address.

So a rule that removes every closing bracket destroys those links, and a rule that keeps every closing bracket drags in the one Markdown put there. Both rules are wrong, and each is wrong on the cases the other handles. What works is counting: a closing bracket stays if there is an unmatched opening bracket inside the address, and goes if there is not.

The case worth trying yourself is a Markdown link wrapped around one of those addresses — [text](https://ex.org/Foo_(bar)) — where there are two closing brackets and exactly one of them belongs to the link. Counting resolves it; nothing simpler does. The number of addresses that needed any of this trimming is shown next to the results, because it is otherwise invisible — a URL carrying a full stop looks entirely correct until somebody follows it.

Half of what people write has no scheme in front of it

www.example.com is unmistakably a web address to a reader and is not one to any parser: hand it to a browser's own URL machinery and it throws. The same goes for a bare example.com dropped into the middle of a sentence, which is how most people write most domains most of the time.

Both are found here. What they are not is silently rewritten. Choose the as-written output and you get back exactly the characters that were in your text; choose the normalized one and https:// is added along with a lowercased host, so the result can be fed to something else. A guess belongs in a calculation, never in a quotation of what somebody typed. How many were missing a scheme is one of the four counts above the list.

Things shaped like a domain that are not one

Once you accept bare domains you have accepted a hard problem, because README.md, Node.js and index.aspx have precisely the shape of a hostname: a word, a dot, a short run of letters. Filenames are filtered against a hand-picked list of endings that are not top-level domains, and that list is deliberately short — .app, .dev, .sh and .zip are all real endings now, so a sweeping rule about "file extensions" would throw away genuine links.

A second case has nothing to do with files: a sentence that lost the space after its full stop. It ended there.The next one began. contains there.The, which passes every structural test for a domain. The capitalization gives it away — real domain endings are written all lowercase or all uppercase, never with a single leading capital — and that one rule removes an entire class of nonsense from the results. Anything rejected appears underneath the list with its reason, so you can overrule the judgement yourself.

Then there is the boundary with email. ada@example.com contains example.com, and reporting that as a website answers a question nobody asked; those are counted separately and pointed at the email extractor, which handles the whole address. The local half is the subtler trap: ada.lovelace has a dot and a letters-only ending and is therefore indistinguishable from a domain until you look at the character immediately after it.

Just the domains, which is often the real question

"Which sites does this document cite?" is not the same request as "list every link", and it is frequently the one people actually have. The domains output collapses the list to unique hostnames, so a page with forty links to six publishers comes back as six lines. It is a separate mode rather than a filter you apply afterwards, because deduplicating hostnames after the fact means doing it by hand.

If your source is a saved web page rather than plain prose, running it through HTML to Markdown first will flatten the anchors into readable text, and messy pasted text usually behaves better after a pass through the text cleaner.

This URL extractor never visits a single address it finds

No requests are made. Nothing is fetched, no redirect is followed, no shortened link is expanded, and there is no check for whether a page still exists. That is a boundary rather than an omission, and this is the page in the family where it matters most: a tool that already holds a list of URLs is half a step away from crawling with them, and half a step further from being a scraper with a friendlier name.

The practical consequence is that your text stays where it is. There is no endpoint to send it to, so a pasted newsletter, an internal document or a chat export goes no further than this tab. The same holds for the two sibling tools: numbers go through the phone number extractor, and addresses through the email one, and neither of them reaches the network either.