Three men and a woman, aged 27, 30, 35 and 37 were spotted by the crew of the Royal Charlotte, a charter fishing boat, at approximately 10am on Saturday.
The four migrants, thought to be from Iran, were drifting on Goodwin Sands, a 10-mile sandbank off the town of Deal, not far from Dover.
The man who led the rescue mission, Royal Charlotte skipper Matt Coker, recalled the moments he first approached the dinghy.
He told Sky News: “We could see they were in distress.
“There was so much panic in their faces.”
Explaining one of the migrants started using a red t-shirt to catch the crew’s attention, he continued: “‘Initially they were waving just oars, and then as we got closer they had t-shirts on the oars and they were frantically waving.”
According to the skipper, the dinghy the migrants were on was only meant to carry three people.
Their condition was made worse by the fact the inflatable didn’t have any engine and they had to share three paddles between the four of them.
He added the migrants were “very lucky” the sea has been “flat calm” during the past few days, as their lives would have been seriously endangered on such a small boat with rough sea.
The rescue operation was carried out by Mr Coker and the 12 angler aboard the Royal Charlotte.
Welcoming the migrants on their fishing boat, the crew offered the distressed four water and clothes.
The Border Force were called immediately after the fishermen spotted the four.
Upon their arrival, officers took the migrants onto one of its patrol vessels.
Speaking about what may have driven the four migrants to cross the seas with such an inadequate boat, Mr Coker said: “They must have been desperate to attempt that, especially in the English Channel, one of the busiest shipping lanes in the world.
A Home Office spokesman confirmed: “On Saturday morning Border Force received a call from the UK Coastguard regarding a small inflatable boat off the coast of Dover.
“A coastal patrol vessel was deployed and the four people on board have been brought to Dover, where they will be processed in line with the immigration rules.”