Why did Angel leave the Buffy series so early?? What season should he really had left?

We all know the storyline behind why Angel had to leave, but personally (rewatching season 5 currently) Riley Finn drives me nuts and I find myself wishing Angel had never been written out so early!

Which Season would you say Angel should have stayed until, or if not why not?

I do believe Angel and Buffy should stay away from each other, but I also believe Angel would have been interesting as a witness to spike's love for Buffy. Opinions?