Explicit model selections and R environment changes reach an already initialized Python interpreter. Model loading restores both runtimes’ prior environment values, including when a checkpoint fails to load.
Classifiers now return a common result with
input_id, original input, provider, evidence
type, structured outcomes, typed labels, and
source-specific details. Provider-specific category columns
become category; pie_confidence becomes
confidence. Vendor labels and multiple list memberships are
retained in labels. University records, UT1 usage, and
piedomains fetch/model metadata move into details.
Empty domain vectors return typed empty results without I/O.
no_match distinguishes absent memberships from execution
failures. All download helpers return a path invisibly.
rate_limit is renamed delay, measured
in seconds. web_categories() exposes the shared default LLM
vocabulary. API failures retain status, reason and model
identity.
not_news() is replaced by
news_url_signals() with independent soft_news
and hard_news flags. adult_ml1_cat() is
replaced by adult_name_probability(); its fitted model and
training-era feature encoding are unchanged. No compatibility aliases
are retained.
Steven Black results describe membership only:
blocked or no_match. Hostname fragments no
longer imply advertising or malware. UT1 retains every category
membership.
uni_cat() accepts a saved JSON file and caches its
automatic download for the session.
Current-content collection defaults to
archive_fallback = FALSE. Historical and fallback requests
preserve the full URL. Archive outages and robots transport failures
have distinct retryable outcomes; not-found pages and unresolved
redirects do not report success.
Corrupt cache records are misses. Substantive articles mentioning parking marketplaces are not classified as parked by that mention alone. Acquired pages share one processing path.
VirusTotal preserves typed authentication, quota and HTTP failures; invalid vendor labels fail the affected row. Archive resource guards retain their permanent failure reasons.
Inference-only retries preserve caller-restored batch IDs. Mocked model tests restore the session cache, so mocked and real-model checks can run together.
pie_cat(model = ...) selects a checkpoint
explicitly. The pinned default is reused within a session; custom
checkpoints are loaded afresh. Saved results support inference-only
retry.
R 4.5 is the tested minimum. A dedicated R/Python job exercises the bridge and verifies pinned-weight parity both online and offline. The unused XML dependency is removed.
This is an API and correctness release, not a newly trained classifier. The pinned piedomains checkpoint’s drugs/retail limitations and the historical adult model’s lack of current calibration remain documented. No new model-quality claim is made.
pages inputs must be fetched
again.gpt-6-luna and claude-haiku-5-5
with their supported request parameters. Reject invalid/truncated
categories and preserve explicit VirusTotal key precedence. Live
paid-API tests now require explicit opt-in.domains and both
Steven Black and VirusTotal output to domain_name.
VirusTotal credentials now use api_key.no_match; it is not a safety assessment.collect_content() now caches fetched pages. A cache hit
replays the stored provenance – the time the page was
really fetched, the URL it really resolved to – rather than synthesising
a fresh timestamp, so a cached row still says when it is from. Entries
expire (cache_ttl), the cache is pruned to a size budget
(cache_max_size), and a corrupt entry is a miss rather than
an error. Nothing persists past the session unless you pass
rdomains_cache_dir().rdomains_cache_dir() and
cache_clear().collect_content() and pie_cat() take an
archive_date, fetching each domain as it was on or before
that date. Rows carry the realised
snapshot_timestamp, which is the capture found and rarely
the date asked for.collect_content() recovers pages from the Internet
Archive when a host serves an anti-bot interstitial – detection plus a
fallback rather than an evasion arms race. Recovered rows carry
source = "archive" and a snapshot_timestamp,
so the vintage is never hidden. A domain that no longer resolves is
deliberately not recovered this way: answering “what
was this” while looking like “what is this” is the staleness this
package exists to surface.pie_cat(): fetches a domain’s homepage and
classifies its current content with the piedomains
model, rather than looking the domain up in a list written years ago. It
reaches the model through the Python package via
reticulate, so the four fragile pieces of the model input –
the domain prefix, temperature scaling, label projection and the text
cleaner – stay on the Python side rather than becoming a second place
they can drift. parked and unavailable pages
are labelled from the page itself and cost no model call.collect_content()’s max_bytes default is
now 10 MB, matching piedomains. At 2 MB it rejected cnn.com – roughly 6
MB of homepage – as content_too_large while the server
returned 200.ut1_cat() and get_ut1_data(): the
UT-Capitole blacklists, the maintained successor to
Shallalist. Where shalla_cat() answers from a list
that stopped in January 2022, this one is updated continuously.
Categories are UT1’s own and are reported verbatim, with
ut1_usage saying whether UT1 maintains the list for
blocking or for allowing – both describe content.collect_content() honours an explicit
http:// or https:// in the input, keeping its
port and path. Forcing https:// meant http-only hosts were
unreachable.content_too_large rather than
connection_error. curl aborts the transfer, so the limit
surfaced as a request failure – and connection_error is
retryable, so callers would have retried a too-large page forever.webfakes), covering redirects to private addresses,
robots.txt refusal, oversized bodies, bot walls, and whether the cache
actually prevents a second request.Two of the four lookup sources are no longer published: DMOZ closed in March 2017 and Shallalist stopped in 2022. Their labels were correct when assigned, but domains expire and change hands, so a lookup today can return the previous registrant’s category. Until now that answer was presented identically to one from a list updated last week.
source_vintage() reports every category source,
when it was last published, whether it is still maintained, and its
successor where one exists.shalla_cat(), dmoz_cat() and
stevenblack_cat() now return a
source_last_published column. For the two dead lists this
is a constant; for Steven Black’s actively-maintained hosts file it is
the fetched file’s own date.The sibling project piedomains measured what this
confusion costs: its worst class disagreed with its own page content 71%
of the time, and the cause was not bad annotation but roughly a decade
between the label and the page. Only 60% of the domains it trained on
still resolve.
New collect_content() fetches homepage HTML and text, so
a domain can be classified on what it says today rather
than on what a list said years ago. It returns one row per requested
domain, never dropped, each carrying status,
stage, error_code and retryable –
so a transient failure is distinguishable from a permanent one and only
the right rows get retried. fetch_error_codes() documents
the closed set of reasons; fetch_report() summarises a
run.
Supporting functions, all usable on their own if you already hold HTML:
page_signals() reports whether a page is an anti-bot
interstitial, a domain-parking placeholder, a server’s “nothing here”
page, or too thin to classify. Vendor presence alone is not a block:
reddit, walmart and quora all serve real pages while embedding
reCAPTCHA.html_text_content() extracts text, title, description
and language.The crawler identifies itself as
rdomains/<version> with a contact URL, obeys
robots.txt including Crawl-delay, spaces
requests to the same host, caps the response body, follows redirects by
hand so every hop is re-validated, and refuses hosts resolving to
private or link-local addresses.
Static HTML only – no headless browser, so a JavaScript-rendered page comes back thin and says so.
get_dmoz_data() built its output path with
paste0(), so the documented default
outdir = "." produced a hidden
.dmoz_domain_category.csv that dmoz_cat()
would then fail to find. It now uses file.path(), matching
its two siblings.stevenblack_cat(use_file = NULL) re-downloaded roughly
4 MB on every call. The file is now cached for the
session.skip_on_cran() and skip_if_offline() guards,
as CRAN policy requires.get_alexa_data() function (Alexa service
discontinued by Amazon)virustotal_cat() to use VirusTotal API v3
(previously v2.0)virustotal_cat() implementation to properly
extract categories from v3 API response structureclean_domains() - standardized domain cleaningvalidate_domains() - comprehensive input
validationvalidate_data_file() - consistent file validationget_api_key() - unified API key retrievalbuild_categorization_prompt() - LLM prompt
constructionapply_rate_limit() - rate limiting logic:: notation for imported functions
(cleaner code, consistent with @importFrom)get_alexa_data() has been removed (service
discontinued)virustotal_cat() parameter renamed from
domain to domains for consistencyopenai_cat() and claude_cat() functions*_cat()
functions for seamless integration