If I asked my partner to provide an argument for why earth is flat, she would do it. She doesn't think (or have to think) the earth is flat to make an argument.
I'd expect an AI trained on human conversation to act the same and I'd be frustrated if it declined to do so, the same way I'd be frustrated if a friend also declined to do so.
Yeah, the humans I'm referring to don't need the hypothetical prefix, nor do they go out of their way to categorically dismiss everything they've said. That's the difference.
But it's not a hill I want to die on, especially when there are other LLMs I can just switch to that act more how I'd hope/expect.
I'd expect an AI trained on human conversation to act the same and I'd be frustrated if it declined to do so, the same way I'd be frustrated if a friend also declined to do so.