Reading the Bible literally or allegorically will get you into all kinds of error. The Bible is both literal & allegorical, sometimes at the same time. Knowing which passage is which is usually just a matter of a little common sense, but sometimes it's more difficult. It can require some study, prayer, and wisdom.
It's also important to keep in mind that all 66 books were written by actual people for the benefit of actual people. This means that it was written using human languages including idioms, cultural and contemporary allusions, puns, sarcasm, and even jokes.
If your interpretation of a passage doesn't take all of this into account, there's a very good chance that your interpretation is wrong. Even if you have considered all factors and put in a lifetime of study, if you aren't willing to entertain the idea that you could be wrong...you probably are.