(I'm) (a)fraid not
A response used to reluctantly decline an invitation or politely answer a question in the negative, indicating regret that the answer is "no." When the phrase is abbreviated to "'fraid not," an apostrophe is sometimes used in place of the missing letter. A: "Will you be able to attend the meeting tomorrow?" B: "I'm afraid not. I'm going to be out of town." A: "Could you loan me a hundred bucks?" B: "'Fraid not. I'm broke."
(I'm) (a)fraid so
A reponse used to politely answer a question in the affirmative when the speaker regrets that this is the case. When the phrase is abbreviated to "'fraid so," an apostrophe is sometimes used in place of the missing letter. A: "Are you really moving all the way across the country?" B: "I'm afraid so." A: "Are you going to be working late again tonight?" B: "'Fraid so."