A speaking test is not a chat
Many people taking a speaking test make this mistake: they focus on communicating their answer specifically to their test administrator. This is a problem because you are graded on the level of English you use, not on how easy it is to understand you.
Another way of saying this is your goal is not to give information, but to show your English ability. Complex answers are almost always better than simple ones.
You may also notice that sometimes the tone and body language of interviewers don't invite communication. They are probably not going to be interested in what you are saying, but rather the English you are using to say it. Interviewers have been known to cross their arms, use closed body language, and not react to what the test-taker is saying. At first, this can be uncomfortable and off-putting, but it makes sense when you consider that the goal of the test is not to communicate with the test administrator, but rather to showcase your English ability.
As an example, let's consider this question: Have you ever been abroad? Where did you go?
First, let's talk about what not to do. Don't just answer the question:
Yes. I have been to Australia, Singapore and Italy.
This shows only a junior high school level of English; it won't add anything to your score.
A better answer will address the question and add information from the 5Ws & H (who, what, when, where, why, & how):
Yes, I've been abroad a few times actually—to Australia about ten years ago, Singapore last year and then Italy earlier this year. I went to Singapore and Italy on business, so I didn't get a chance to see much of the country, but I did try some truly spectacular food. My trip to Australia was for pleasure: I went with my university friends after we graduated.
This answer uses complex sentences and higher-level vocabulary to add information, and it will add to your overall score.
So when you answer a question in the test, just take a deep breath and try to use as much high-level English as possible.