The bank, which was accused of neglecting safeguards against illegal activity and of lying to a regulator, agreed to pay $613 million in fines and penalties to resolve the case.