How long would it take to cross the English Channel by sailboat?

From Poole or Weymouth the best target port of destination would be Cherbourg. The channel tides hit left to right to left so cross channel navigation is actually quite straightforward although the tides do run fast off cherbourg and weymouth. 4 to 6 survivors means at least a moderate sail boat of 30 feet or a motor boat of 25 feet. Such a sail boat would average 4 to 5 knots unless lucky with the wind, in which case a good broad wind could give them up to 8 knots. That would take them 8 hours fastest to 12 hours slowest. A small motor speed boat may not carry enough fuel to get across and as you are talking about commandeering whatever is available you would not expect a small boat to be there with long range fuel tanks. If they went extreme and managed to get hold of a fast motor cruiser, 30 to 40 feet already fuelled up, then they might get across in 2 to 3 hours.