fix(search): parse current Bing result links (#6150)

While looking into #6009 I noticed the Bing fallback in -g never
returns anything, even when Bing itself serves a normal results page.

Two things changed on Bing's side. The result anchor now carries
target="_blank" before href, so '<h2><a href="..." h=' no longer
matches at all. And the href is no longer the target URL but a
tracking redirect (https://www.bing.com/ck/a?...&u=a1<base64url>),
where the real link sits base64url-encoded after the "a1" prefix
in the u parameter.

The regex now allows extra attributes on both tags and captures
either the u payload of a ck/a redirect or a plain href. The payload
is decoded with decodeBase64(), which already deals with the
URL-safe alphabet and the missing padding. Plain hrefs still work
the same as before (with &amp; unescaped), in case Bing serves the
old markup to some clients.

Checked against a saved live results page (0 -> 10 links) and the
old markup, and --smoke-test passes. Google still can't be scraped
since it only serves a JS page now, so this doesn't fully close
#6009, but Bing is usable again as a fallback when DuckDuckGo blocks
the IP.
This commit is contained in:
Raju Ranjan 2026-10-21 14:11:33 +05:30 • committed by GitHub
parent 8461fd402b
commit 5e2a6aeec8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 3 additions and 2 deletions

View file

@ -214,7 +214,7 @@ GOOGLE_CONSENT_COOKIE = "CONSENT=YES+shp.gws-%s-0-RC1.%s+FX+740" % (time.strftim
DUCKDUCKGO_REGEX = r'<a class="result__url" href="(htt[^"]+)'
# Regular expression used for extracting results from Bing search
BING_REGEX = r'<h2><a href="([^"]+)" h='
BING_REGEX = r'<h2[^>]*><a [^>]*href="(?:https?://www\.bing\.com/ck/a\?[^"]*?[?;]u=a1([^&"]+)[^"]*|([^"]+))"'
# Dummy user agent for search (if default one returns different results)
DUMMY_SEARCH_USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64; rv:141.0) Gecko/20100101 Firefox/141.0"