But what a mission it was. He was appointed to the role in 2017, taking over from Ben Ryan, who coached the team to Fiji's first Olympic gold medal in any sport at Rio de Janeiro in 2016.
It had been a tough five years with constant reminders of what he was appointed to achieve. And that was secured with a 27-12 win over the New Zealand in the final.
However, Covid's outbreak in Fiji made the achievement more meritorious for all the hurdles it threw in the team's path.
Baber said the pandemic's outbreak in Fiji had made life difficult for the team since they gathered to prepare for the Games. They were forced to stay together at Easter instead of returning home as they had expected to do.
"That was the last time they saw their families, and there was a lot of unknowns.
"I can remember a situation where I was with Jerry [captain Jerry Tuwai] and the players. It got to a point where we are in containment zones. They were ready to break out, go home. They had had enough. They were gone.
"If was one of those moments as a head coach you say" 'If you go, that's it – we're done. We can't go to the Olympics.'"
They were able to transfer from Fiji to Australia to prepare with more freedom than at home.
"There were about two or three days things were in a knife edge. It was real for us, very quickly. We made it to Australia, but did not know how we were going to get to Japan," he said.
They eventually made it aboard a cargo plane carrying frozen fish.
"Trying to piece that together has been a logistical nightmare. You have to get those things right, and that was nagging at us all the way, as we could have our preparations derailed in less than 24 hours," he said.
However, the players had been tight and coped well and had their reward with the gold medal.
"I feel quite numb, and have not quite gathered what we have achieved," he told the BBC.
"It feels slightly surreal, perhaps even more so because of the lack of crowds, and there is not much going on in the village, but the celebrations we generally had have been around faith.
"This [Olympic success] has been the vision for a long time. I have lived with that sort of pressure and been reminded of that several times each day as we have progressed through the last four and a half years," he said.
There was always pressure in the job, and he said there had been times when he wondered if he would still have a job after some losses.
"I wouldn't have gone to Fiji if I didn't want a challenge and some people said I was made doing it in the first place. As I sit here, it sort of vindicates why I went. They have looked after me, and I have loved my time here," he said.
The success had hit home to him what rugby and especially Sevens means to the Fijian people.
"The strongest motivational factor was to do the nation proud and give their families, and loved ones, a lift. It was just about playing like they've done for years in the villages and on the beaches.
"Rugby is so intrinsically linked with national identity, a bit like in Wales. That is prevalent in Fiji and linked to how they feel about themselves and how they believe they are perceived across the world.
"They see themselves as people who can fight above their weight, and they do that in rugby generally, particularly in Sevens," he said.
Baber will return to Wales and take time to think about his future. He hasn't seen his family since January.