The hospital - Shalimar Bagh's Max Super Speciality Hospital - has admitted to the bone-chilling incident, saying it is "shaken and concerned" by the "rare incident".
"He is a premature baby of just 22 weeks".
Delhi's Chief Minister Arvind Kejriwal announced on Twitter that an official inquiry would be ordered, and that the "strongest action" would be taken if the doctors who had pronounced the child's death were found guilty. A few minutes ago, I was crying over the death of my babies and suddenly I saw one of them alive.
Dr Gupta also added that in cases of a premature birth, while sometimes it can get hard to see signs of life, the heartbeat should be tested before declaring a baby dead. The hospital wrapped the two bodies in two separate paper bags and handed them over to the family in a polythene bag.
"We want the Delhi government to take strict action against the hospital authorities for their negligence", the boy's grandfather said.
"When we opened the bag, we found the boy breathing. We rushed the baby to a nearby hospital in Pitampura", Ashish, father of the twins, told reporters.
Max Hospital Shalimar Bagh had handled the delivery of the babies early Thursday after the woman, Varsha, was shifted to the hospital from a nursing home in Paschim Vihar. "An inquiry regarding the said incident is to be initiated and a preliminary report is to be submitted within 72 hours and a final report within a week", a Delhi government order said. On Friday, the family members of the baby alleged that doctors have told them that the baby has developed severe infection after the incident at the Max Hospital. While the girl was stillborn, the baby boy was declared dead later.
Meanwhile, local media reported that the hospital had taken responsibility for the shocking incident and called it "unfortunate".
In a statement, the hospital said the boy was "handed over without any sign of life" by hospital staff.
The statement said a detailed enquiry was initiated, pending which, the concerned doctor was asked to proceed on leave immediately and they were in constant touch with the parents and are providing all the needed support.
Police have registered a case in the matter under Section 308 of the IPC which governs cases related to attempt to commit culpable homicide and can attract imprisonment of up to seven years.