Just be honest. I know it so cliche but I had this situation once and it was not easy. I had to tell him look I care for you and you are a good person. But you are not the person for me. This is not where I want to be. He was very hurt and angry at fiirst. Did not even talk to me for awhile. But we later became good friends again. Dont drag it on that sucks more then anything.