Python 3.13+ reports sqlite3 connections that are garbage collected
without being closed. The unit tests print a run of
"ResourceWarning: unclosed database" lines on Linux/CPython 3.14:
- SQLAlchemy.close() only returned its connection to the engine's
pool, so the underlying DBAPI connection (e.g. sqlite3 for
'-d sqlite://...') stayed open. Dispose the engine on close.
- vulnTest() used "with sqlite3.connect(...)", which commits but does
not close the connection. Use contextlib.closing().
- test_multibit and test_sqlalchemy never closed what they opened.
Co-authored-by: PiyushSatyarthi <244159432+PiyushSatyarthi@users.noreply.github.com>
While looking into #6009 I noticed the Bing fallback in -g never
returns anything, even when Bing itself serves a normal results page.
Two things changed on Bing's side. The result anchor now carries
target="_blank" before href, so '<h2><a href="..." h=' no longer
matches at all. And the href is no longer the target URL but a
tracking redirect (https://www.bing.com/ck/a?...&u=a1<base64url>),
where the real link sits base64url-encoded after the "a1" prefix
in the u parameter.
The regex now allows extra attributes on both tags and captures
either the u payload of a ck/a redirect or a plain href. The payload
is decoded with decodeBase64(), which already deals with the
URL-safe alphabet and the missing padding. Plain hrefs still work
the same as before (with & unescaped), in case Bing serves the
old markup to some clients.
Checked against a saved live results page (0 -> 10 links) and the
old markup, and --smoke-test passes. Google still can't be scraped
since it only serves a JS page now, so this doesn't fully close
#6009, but Bing is usable again as a fallback when DuckDuckGo blocks
the IP.
A scan covering multiple targets inside a single API task (crawl,
'-m' bulk file, forms) ran every target through one StdDbOut keyed by
the single taskid, so _storeData()'s COMPLETE branch deleted each
earlier target's TARGET/TECHNIQUES rows - /scan/<id>/data returned
only the last target's results.
Mirror the --report-json fix for #6123: controller.start() now sets
kb.reportTaskId per target for API scans too, StdDbOut.write() keys
data rows '<taskid>-<n>' per target, and scan_data() folds the
per-target rows back into the task's flat result in target order.
Single-target scans keep the exact same response shape.
Regression tests in tests/test_api.py (TestStdDbOutMultiTarget,
TestScanDataMultiTarget).
JWT_REGEX accepts a signature segment of any length and parseJWT only
validates the header/payload, so a value that matches the JWT pattern but
carries a malformed/truncated signature (e.g. 41 base64url characters,
which is length % 4 == 1 and thus impossible base64) reached crackHMAC.
There decodeBase64(signature) let binascii.Error propagate, aborting the
whole run with an unhandled exception during checkJWT().
Guard the signature decode: a signature that is not valid base64url cannot
be an HMAC we could verify, so crackHMAC now treats it as not crackable
(returns None) instead of raising. The other auditJWT findings are
unaffected. Added a regression doctest.
Co-authored-by: chan770 <tsramachandran770@gmail.com>
