Protego
Protego: Exponential backtracking ReDoS in robots.txt URL wildcard matching
Protego is a pure-Python robots.txt parser with support for modern conventions. Prior to 0.6.2, protego._urlpattern._URLPattern._prepare_pattern_for_regex translates every asterisk in an Allow or Disallow directive into a lazy regular-expression wildcard, so a directive containing many asterisks creates exponential backtracking. After protego.Protego.parse processes a crafted robots.txt file, protego.Protego.can_fetch can spend an attacker-controlled period matching a near-miss URL and deny service to the crawler. The vulnerable path is src/protego/_urlpattern.py in the _URLPattern match logic. This issue is fixed in version 0.6.2.
Affected versions and mitigation
From the CVE record and the vendor's references.Affected
- < 0.6.2
Fixed in 0.6.2
Mitigation
Upgrade protego to 0.6.2 or later.
References
- Vendor advisory github.com/scrapy/protego/security/advisories/…
- Patch github.com/scrapy/protego/commit/785940181659b…
- Release notes github.com/scrapy/protego/releases/tag/0.6.2
Timeline
Days relative to CVE publicationNo public PoC found yet; not in CISA KEV.
PoC repositories
Found on GitHub.No PoC repositories found yet.