They don't really need to form a coalition with them, they could just accept their support in exchange for big changes in the asylum policies. That is what has happened in other countries.
Also, I think it is hard for Merkel to keep power, if she loses big to AFD. I think SPD would then insist to get most of the power and the leadership. If Merkel agrees to support such a coalition, then she will destroy her own party.
As far as the unwritten rules of German politics go, it's true that in a coalition, the largest coalition partner gets the office of Chancellor (or Prime Minister on state level). But for the CDU/CSU falling below the SPD, A LOT would have to happen. Keep in mind the SPD is down to 23%-25%, too. And if/when the AFD continued to rise, the SPD too would lose some more voters to them and fall even deeper.
In 2005, a "grand coalition" was formed too (Merkel's first term), while the CDU/CSU won 35.2% of the votes and the SPD 34.2% -- that was when then Chancellor Schröder (SPD) had his memorable tv appearance, when he questioned Merkel's right to become Chancellor (perhaps he assumed the SPD might end up first, in the end). Yet Merkel got the office, although her party only had an advantage of 1.0% of the votes.
I agree that further "grand coalitions" are not good for the political system. The two large parties would lose more and more voters to the extremes. And many people would feel there is no real alternative; no matter if they vote left or right, there will be a centrist "grand coalition" anyway -- the feeling of having no impact, of not being able to change the outcome, is never good for any free system.
So what are the alternatives? A left coalition (SPD, Greens and Left Party) was unlikely even last year already, because SPD/Greens consider the Left Party too extreme and unreliable (and some far-leftists in the Left Party explicitly say they would rather stay in the opposition than making any compromises), and the emergence of the AFD makes it even more unlikely (mathematically, a new right-wing protest party sucks votes from the left, too). Which, in effect, means any vote for the Left Party is a vote for a "grand coalition".
The AFD, as it stands now, would be the same matter on the right; its relation to the CDU/CSU is analogue to the relation between SPD and Left Party.
But never say never, and something will have to change on the long run. Your idea of a CDU/CSU minority government tolerated by the AFD is certainly an option -- but only IF, big IF, Merkel is gone. She couldn't afford making such a u-turn, she'd fall with her policies. But you never know what a successor would do. CSU chairman (and PM of Bavaria) Seehofer might be such a person, someone who could pull off such a radical change of course; he has been opposing Merkel all along.
Most CDU members are probably reluctant to sacrifice Merkel, as they basically have no other politician who is Chancellor material; topple Merkel, and they risk losing the power. They're not quite at the point of running such a risk yet, as far as I can tell. At best, Finance Minister Schäuble could take the office, but he'd be a provisional candidate at best, as he's well above 70 years old already, and rather considered a very old face, rather than someone who looks forward.
So... some dynamics in German politics, finally. Only one year ago, the domestic situation looked so stale and immovable, I'd never have imagined we'd see something like that anytime soon. As the Chinese would say, "interesting times".