A lengthy protest followed before Kagawa finally converted from the spot in the sixth minute.
Radamel Falcao is set to lead his side out against Japan on Tuesday afternoon, in what will be his first World Cup game at the age of 32.
The fastest-ever though still remains Jose Batista, who was sent off inside the first minute for fouling Gordon Strachan in 1986 World Cup match between Scotland and Uruguay.
Yuya Osako celebrates scoring Japan's second goal with coach Akira Nishino and teammates.
Yuya Osako's 73rd-minute victor was a huge relief for the Japanese, who were thumped 4-1 by Colombia four years ago during a miserable campaign in which they failed to register a single win. He almost single-handedly guided Colombia into the championship by scoring six goals in only five games.
"To lose one player in the three minutes to lose such a crucial player, is not an easy thing", Colombia's Argentine coach Jose Pekerman told reporters.
In the early minutes of the game, there was controversial drama as Carlos Sanchez was handed the first red card of the tournament for stopping a goal-bound shot of Shinji Kagawa with a raised arm in the box.
Sanchez at fault for the red card and goal yet are happy clapping fans prefer him over toby.
The 22-year-old central defender started the game and played for the entire 90 minutes, as Colombia lost 2-1.
Victory over Colombia will give Japan hope of progressing beyond the group stage for the first time since 2010, while Pekerman will hope that James can return to full fitness in time for their forthcoming clash with Poland.
Japan, 45 places below their opponents in the latest Federation Internationale de Football Association rankings, had to fend off one final attack from Los Cafeteros as James's shot from eight yards was superbly blocked behind by matchwinner Osako.
Colombia defeated Japan 4-1 when the teams met in group play in Brazil four years ago, and the Japanese also began the game without Shinji Okazaki, a Leicester forward who has been trying to overcome a sore calf.
Colombia continued to rally but the effect of playing with 10 men for nearly the entire game took its toll with Japan holding out, leaving Colombia with a mountain to climb in Group H.