Building a news crawler is not particularly difficult.
Pick a few publications, fetch their pages or feeds, extract the articles, and put them in a database. With today’s tools, a competent developer can get surprisingly far in a few days.
For a small, stable requirement, that may be all you need.
The mistake is assuming the prototype tells you much about the cost of running broad news coverage in production.
It doesn’t.
TL;DR
- A working prototype takes an afternoon. That speed hides how much real coverage actually involves.
- The real cost isn’t the build. It’s the permanent maintenance tax of keeping coverage alive as the web keeps shifting, plus the licensing and lock-in costs that never make the spreadsheet.
- A simple test tells you whether to build or buy. Is your coverage narrow, English-only, and low-volume? Any “no” and the build costs far more than it looks.
The source list is not the hard part
Opoint monitors more than 250,000 sources. On a typical day, around 97,000 distinct domains actually publish something.
That second number is the interesting one.
It is easy to accumulate URLs. Keeping them alive is much harder. Industry source lists routinely run to hundreds of thousands of entries, yet active daily publishers are often a fraction of that — most of the rest dormant, dead, or between publications. A very large source list is not necessarily a very active one.
News sites are messy. Some publish hundreds of articles every day. Some publish twice a week. Some disappear for a month and return. Others quietly stop publishing forever. New sites appear constantly.
And broken does not always mean dead.
One of the more persistent failure modes is a source that normally delivers 150 articles a day and suddenly delivers 30. The crawler still works. Articles still arrive. Your monitoring may be completely green. You are just missing most of the source.
Catching that means learning what normal looks like for each publication and detecting when it drifts. At hundreds of thousands of sources, that is a substantial operation in its own right.
This is one reason source maintenance has occupied us for more than twenty years. The crawler code is only part of the system. The harder job is knowing whether the coverage is still good today.
Then try to make it fast
Latency looks easy at small scale too.
If you follow twenty websites, you can visit them constantly. Nobody cares much about the bandwidth, and you are unlikely to attract serious attention from publishers.
That stops working when your source list grows.
Check too slowly, and you find important stories late. Check too aggressively and sites rate-limit or block you. Some sources publish several times a minute; others twice a week. Treating them alike wastes crawling capacity and irritates publishers for no benefit.
Low discovery delay becomes as much a scheduling problem as a crawling problem. Which sources are likely to have changed? Which should be checked again immediately? Which can wait? When should you back off because a site is unhappy? How do you distinguish a slow source from a broken one?
Throwing more machines at the problem helps surprisingly little if the result is that more publishers block them.
This matters because users rarely ask whether your crawler eventually found an article. They care whether it found the article while it was still news.
Scale adds problems gradually, then all at once
One language is fairly straightforward.
Ten languages mean ten source universes, each with different scripts, encodings, publishing conventions, and edge cases. A large share of the world’s news never appears in English, and stories that move markets often break in the local language hours before the English wires catch up.
Each new language is not a small addition — it’s a fresh maintenance burden stacked on top of the one you already carry.
Duplication is almost invisible at prototype scale. Then a significant event happens, and the same wire story appears on fifty sites, each with a slightly different headline or introduction. A feed that technically contains everything suddenly becomes unpleasant to use.
Websites redesign. RSS feeds disappear. Paywalls appear. URLs move. Extraction that worked yesterday produces navigation menus today.
None of these problems is individually exotic. That is partly why building initially looks so attractive. The cost comes from dealing with all of them, every day, across a large and constantly changing source collection.
So the useful comparison is not: what does an API cost compared with writing a crawler?
It is: what does the data cost compared with running the coverage, monitoring and maintenance operation ourselves for the next five years?
That calculation often looks rather different.
Don't forget the non-technical parts
Licensing is one example.
A prototype can largely ignore copyright, publisher agreements, database rights and redistribution rules. A commercial product normally cannot. Scraping news content and redistributing it inside a product that customers pay for sits on contested legal ground. Copyright, terms-of-use restrictions, and EU database rights all apply, and publishers have become markedly more willing to pursue scrapers and the products built on them.
In compliance and risk contexts especially, where your data comes from—and whether you’re licensed to use it—becomes part of your customer’s due diligence, and therefore part of yours.
Buying data does not make those questions disappear. Depending on the use case, you may still need your own agreements with specific publishers or rights organisations. But they can be addressed before a product has been built around the data, from a provider who can tell you where the content came from and on what basis — rather than something you discover after a product already depends on it.
Vendor dependency is real as well. Providers change prices and APIs, and switching can be painful. But building has its own form of lock-in: once several products depend on your internal crawling infrastructure, somebody has to maintain it indefinitely. Neither choice makes you independent. They leave you dependent on different things.
When does building make sense?
For a few hundred known, reasonably stable sources—perhaps in one or two languages—building is worth serious consideration, especially if occasional gaps or delays are acceptable.
As the requirement expands, three questions matter:
- How broad and changeable is the source coverage?
- How much does it matter if you miss an article or find it late?
- Are you prepared to own the monitoring and maintenance work permanently?
If the answers are comfortable, build it.
If they are not, don’t estimate the project from the prototype. Estimate the operation you will have to run after the prototype is finished. That is where most of the work is.
Opoint publishes live information about source activity at world.opoint.com.
Watching sources appear, disappear and change over even a few weeks gives a fair impression of what maintaining broad news coverage actually involves.
Frequently Asked Questions
Is it cheaper to build or buy news data?
For a small, stable set of sources in one language, building is often cheaper and gives you full control. The economics change as coverage broadens: once you need many sources, several languages, or high volume, the ongoing cost of maintenance, deduplication and licensing usually outweighs the price of a maintained feed. The honest comparison isn't the API price versus a scraper. It's the API price versus running the whole coverage operation yourself for years.
How hard is it to maintain a news scraper?
The scraper itself is the easy part. The hard part is keeping coverage alive: sites redesign and break parsers, feeds go dark, and sources quietly decline without failing outright. At scale, this becomes permanent engineering work: detecting when a source that normally publishes 150 articles a day drops to 30, and doing that across hundreds of thousands of sources.
Do I need a news API, or can I just scrape the sites myself?
Scraping works for a handful of known sources. It struggles when you need broad discovery across countries and languages, low-latency scale, deduplication, and defensible provenance. It also carries copyright and terms-of-use exposure that a licensed API is built to handle. The question isn't which is technically possible, but which you want to own and maintain.
What's the hidden cost of building a news pipeline in-house?
The costs that decide it usually land in year two: permanent source maintenance, deduplication at volume, multilingual coverage, licensing and provenance under audit, and the opportunity cost of engineers maintaining parsers instead of building your product. None is exotic on its own. The cost is dealing with all of them, every day, indefinitely.
Does buying news data solve the licensing problem?
It makes it assessable rather than making it disappear. Depending on your use case and jurisdiction, you may still need your own agreements with specific publishers or rights organisations. What a provider gives you is a clear answer to where the content came from and on what basis, up front, rather than a legal question you discover after a product already depends on the data.
