In order to become really good at anything you need to consistently practice until it becomes second nature.  It needs to become a habit, something you can do without having to analyse or over think.  If you start out trying to ride above your technical ability, you could injure yourself, which can set you back mentally and physically. It is also not fun riding a trail littered with obstacles that you fear. If you learn to master rocks, roots, small drops, etc, it opens up a whole new level of fun and exhileration.

To become a confident rider you need to consistently work on your skills, ideally with someone who can guide with the correct technique, so you don’t learn bad habits. Skills are just like fitness. You need to constantly develop them, or you lose them.