Run an annealing temp curve to optimize the reactions. A higher annealing temperature will improve specificity.

Run a template concentration curve. If you're getting stronger bands with less template, you may be fighting with an inhibitory substance in your template

Try increasing your extension time. We like to extend for 2 minutes plus 1 minute per kb of target.