Believe it or not, you can still live even if you have a portion of your brain removed. For people who experience seizures often or those who are epileptic, there's this surgical procedure called Hemispherectomy. What it is it? Well, half of the brain is removed or disabled to cure the problem.
The reason why babies have such huge craniums is that the brain was about as big as it would be now, in your age, that is. In case you're wondering what's a cranium, it's the "brain case" -- the part of the skull that encloses the brain.