The reason why the CRA wasn't a factor is two-fold: it does not cover non-bank lenders, such as mortgage and finance companies, which means all the high-risk loans originated from those sources (which were many) were taking place without CRA incentives. They were much more about being able to sell them off to Fannie and Freddie. Second, in theory, the banks to whom the CRA does apply still had to demonstrate their loans were "within the norms of safe and sound operation."
As Jaffee says, there may be lots of other reasons to not like the CRA, but that list should not include any responsibility for the financial crisis.
This verdict comes from Coordination Problem, a blog written by economists who subscribe to the Austrian school. In other words, a blog even Rand Paul could love doesn't think the CRA played a role in the crisis.