Do not listen to these people. Only you can take charge of your own life and what happens to you and your family. Only you know what your family needs, and that is a wife and mother at home taking care of them. Hopefully by this point in your journey you have discussed things with your husband and he is on board with you. If he is not OK with your decision then this may not be the right time, keep praying. The Lord knows the desires of your heart and he will work on your husband's heart.
You can make this huge step of faith with the Lord's help. Remember not to let anything that others say get in the way of your dream. Often those who are negative are those who wish they could be in your shoes and don't know any other way to express themselves. Pray for them and just let them know that this is a decision that you and your spouse have made together. If they are still negative you may have to distance yourself from them for awhile. Make sure to surround yourself with positive people. "What you think about you bring about." This is one of my favorite quotes. Put up pictures, Bible verses, and quotes all over your house so that you have something to focus on to help you reach your goal. Now go ahead and put your plan into action.