I would say do lots of ground work with your horse first before even going out on him if hes the nervous type.
Then you both get to know each other better and you will learn more about him.
Just go back to basics,there is no rush to get him out just be patient as i always say low is Best rather than rush him into things,he will then get the confidence from you and you from him!!
How old is he and how long have you owned him?
Even sitting in his field with him or grooming him,lots of hands on
You will get there just takes time
My TB back in the UK was exactly the same and this i found worked for me,also i only fed her on econamy mix chaff and plenty of carrots and as much haylage(hay)
Everyone has different opinions so what ever suits
Good luck mate it will all come good