True, but it's an assumption based on very consistent phenomenal data, even at the quantum level. If you study a rock at the quantum level, its properties are drastically different from its properties at the macro level and its phenomenality is highly unintuitive at a certain scale. Nevertheless, something is being experienced, and the apparent rules governing that experience will largely be the same for any observer.
You're right, the externality of the rock from my consciousness is an inference. It's a pretty good inference, though. As opposed to what you said next...
So your argument goes that since our only data comes through subjective experience, and we have no means of knowing a rock outside of our consciousness, then the rock must not exist beyond our consciousness.
Yet that's a logical progression with even less data than the conclusion that the rock exists outside of my mind. You keep saying "inference" like it's a bad thing, but inference alone doesn't disqualify an argument.
You are placing a condition on knowledge of the rock that is impossible to meet (knowing it outside of consciousness), and then declaring that since the condition is not met, option B must be the answer. It's an argument from ignorance combined with a false dilemna.
I really don't have anything against the concept of idealism per se. I've always thought it to be a fun idea. It wouldn't change my life if it were true, and I don't deny that it is possible; I'm just not seeing good arguments to select it over naturalism.