East Colfax between Quebec and Peoria isn't too bad.
I would suggest the Swansea area, Globeville area, North Washington area as well. Those are definitely interesting parts of Denver in the sense that it is rougher, but by no means "dangerous."
...then again, I got used to north philly before coming to Denver so all of Denver is "safe."