Once infected, the leaves will often turn yellow and can fall off,’ Morris affirms. Verticillium wilt can also be another fungal issue. Improving air circulation by pruning your olive tree and ...