Add a non-blocking workaround for pcaps that buffer packets.

This is designed to solve the following problem: On Solaris 10 (maybe other
platforms), doing a select on a pcap fd works, in that it returns true when
there are frames available to be read. However, after finding the fd selectable
and calling pcap_dispatch (or pcap_next, etc.), libpcap may read more than one
frame and buffer them internally. This means that later calls to select will
return false. So there may be a frame to be read, but you can't know without
calling pcap_dispatch to check, and that blocks indefinitely (on Solaris) if
you're wrong.

The way this works is that we do a non-blocking read on the pcap fd to see if
there is anything available. If not, we do a select with a timeout as usual.
(The select is to enforce the timeout and prevent spinning CPU by repeatedly
trying non-blocking reads.)

I don't know if this phenomenon affects other platforms than Solaris 10
(more specifically, platforms using DLPI for libpcap). This same thing may be
safe or necessary on other platforms. But I have limited it to Solaris for now.
Solaris 11 uses BPF, not DLPI, for libpcap, but we can unconditionally follow
this code path on Solaris because BPF pcap fds can't be selected on.

libnetutil/netutil.cc

@@ -866,6 +866,23 @@ int pcap_selectable_fd_valid() {
 #endif
 }
 
+/* Are we guaranteed to be able to read exactly one frame for each time the pcap
+   fd is selectable? If not, it's possible for the fd to become selectable, then
+   for pcap_dispatch to buffer two or more frames, and return only the first one
+   Because select doesn't know about pcap's buffer, the fd does not become
+   selectable again, even though another pcap_next would succeed. On these
+   platforms, we must do a non-blocking read from the fd before doing a select
+   on the fd.
+
+   It is guaranteed that if pcap_selectable_fd_valid() is false, then so is the
+   return value of this function. */
+int pcap_selectable_fd_one_to_one() {
+#ifdef SOLARIS
+  return 0;
+#endif
+  return pcap_selectable_fd_valid();
+}
+
 /* Call this instead of pcap_get_selectable_fd directly (or your code
    won't compile on Windows).  On systems which don't seem to support
    the pcap_get_selectable_fd() function properly, returns -1,
@@ -3984,11 +4001,28 @@ static int read_reply_pcap(pcap_t *pd, long to_usec,
 #endif
 
     *p = NULL;
+    /* It may be that protecting this with !pcap_selectable_fd_one_to_one is not
+       necessary, that it is always safe to do a nonblocking read in this way on
+       all platforms. But I have only tested it on Solaris. */
+    if (!pcap_selectable_fd_one_to_one()) {
+      int rc, nonblock;
 
-    if (pcap_select(pd, to_usec) == 0)
-      timedout = 1;
-    else
+      nonblock = pcap_getnonblock(pd, NULL);
+      assert(nonblock == 0);
+      rc = pcap_setnonblock(pd, 1, NULL);
+      assert(rc == 0);
       *p = (u8 *) pcap_next(pd, head);
+      rc = pcap_setnonblock(pd, nonblock, NULL);
+      assert(rc == 0);
+    }
+
+    if (p == NULL) {
+      /* Nonblocking pcap_next didn't get anything. */
+      if (pcap_select(pd, to_usec) == 0)
+        timedout = 1;
+      else
+        *p = (u8 *) pcap_next(pd, head);
+    }
 
     if (*p != NULL && accept_callback(*p, head, *datalink, *offset)) {
       break;