I just built a deck with this and have the same problem. Played a grief 4cmc and was able to target a bribery 5cmc. Imagine casting an Eldrazi titan for 1 cmc with Saruman. Makes him a bit stronger than intended, I'd say.
I also am having issues with Saruman not basing the graveyard trigger off of the second spell you cast. it seems to be triggering off of the cmc of the cards you milled.