The rental car company says a hedge-fund firm behind the big swings owes it money because of a niche trading rule.