The first crash happened at 1.30am and involved one car on the eastbound carriageway between junctions 17 for Chippenham and 18 for Bath.
One man was pronounced dead at the scene and another was taken to Southmead Hospital in Bristol with life-threatening head injuries.
Two hours later a second crash between two articulated lorries, a car and a van happened on the westbound carriageway in the same area.
One man was pronounced dead at the scene with two more taken to Southmead Hospital.
A vehicle was carrying gas canisters which were leaking, so the fire brigade was called to make this safe.
The westbound carriageway is expected to remain shut for the rest of the day. The eastbound carriageway is expected to reopen around midday, Highways England has said.
Supt Phil Staynings said: “We had to respond to two truly tragic incidents this morning and our thoughts are with the family and friends of the victims.
“This is a complex incident that put huge pressure on all of the emergency services this morning and will continue to take a considerable amount of time to investigate thoroughly.
“We appreciate the public’s patience on this regarding the closure of the M4 and we are working closely with Highways England to ensure we can have the road back open when it is appropriate to do so.
“I would like to take this chance to appeal to anyone who may have witnessed this incident or may have been driving along this stretch of M4 at the same time and could have dash cam footage that could prove helpful to our investigations to please contact us on 101.”
A diversion has been put in place while emergency services are at the scene.