Watchlists for collecting and matching sources

I have been connecting the Idea Journal to external sources. A watchlist holds the source query, fetch frequency and filtering or matching settings, giving the plugin a repeatable way to look for material rather than depending entirely on manually entered sources.

Fetching and matching

The Reddit connector uses OAuth to retrieve subreddit material and enrich source records. The watchlist runner processes incoming items, applies relevance filtering and looks for existing ideas to associate with them. Keyword and vector matching offer different signals for that comparison; an LLM filter adds another way to assess relevance.

These signals help organise candidates, but they do not make an editorial decision trustworthy on their own. A source still needs to be assessed in context before it becomes an article.

Running in the background

Cron management schedules watchlist runs and provides reporting so that collection is visible in the admin interface. I also worked on the size and handling of Reddit payloads, because moving unnecessary data through a repeated pipeline has a practical cost.

A separate readiness dashboard was introduced around the same period to surface publishing issues. Collection and readiness are related operational concerns, but they are separate checks: successfully fetching an item does not mean it is ready to publish.