Anscombe suffered a knee injury ahead of the tournament and required surgery.
It was the first time in his Wales stint that Gatland said there had been a player they couldn't replace.
"We gave it everything in that semifinal against South Africa and at 16-16 with 10 minutes to go we were in control of the ball, the momentum had swung our way," Gatland told walesonline.co.uk.
"I thought we were going to win and get ourselves into a World Cup final.
"It wasn't to be but that's just rugby, it's sport. I was gutted.
"That was the first time in my whole time that I felt we really missed a player and that was Gareth Anscombe. He was a big loss for us," he said.
Replacement Rhys Patchell had done 'really well' and Dan Biggar had been excellent.
"But I just felt that the balance we had with Anscombe and Biggar before hand, they complemented each other really well. They're different types of players and you're able to use them differently and use your strengths differently.
"It was the first time in my entire coaching career I felt that the loss of one player had a real effect.
"If you lose someone like [Taulupe] Faletau – and we know how good he is – we've got other players who are able to fill the space.
"A fit Faletau at his best would have been significant but there are other quality loose forwards there, so I felt like we could manage that. Anscombe is someone that we missed," he said.
In the same interview Gatland also revealed the best player he ever coached. There were plenty of candidates including captains Alun Wyn Jones and Sam Warburton, Mike Phillips, Adam Jones, Jonathan Davies, Martyn Williams and Gethin Jenkins.
"Pound-for-pound, the best player I've coached was Shane Williams," he said.
The pint-sized wing defied the odds of bigger, bulkier players in professional rugby to carve his own place in the game, also being selected for the British & Irish Lions. In 87 Tests he created the Wales try-scoring record of 58 tries.
"It was just how strong he was, his footwork, his speed, the tries he scored," Gatland said.
"There were times when he would frustrate you because he's not the biggest man in the world and defensively he would make a few errors.
"But the positives completely outweighed any of the negatives. He was obviously a massive crowd favourite."
Gatland said that while there had been criticism for picking so many big players, it was just a situation that occurred with those bigger players coming through at the same time.
"But Shane provided that balance. It wasn't about picking the biggest players, it was about picking the best players…Sometimes you get players who come along with the physical attributes and they sometimes have an advantage at the highest level," he said.