Very smart, to not give up your phone. The clue that they had no right to take it was in the way they asked, 'CAN we have your phone?' This is a trick police use to get evidence and information they don't have the authority to take. Whenever they ask, they don't have the authority.
When they DO have the authority, they order you to give them what they want, or just take it.
Even when they order, you should ask for their warrant. Not just a verbal order