Position a rack in the center of the oven and preheat to 400°F. Using a fork, pierce the potatoes in several spots. (This allows the steam to escape during baking, which keeps the skins from cracking.) Rub the potatoes with oil and place them on a rimmed baking sheet. Bake until the skins are crisp and the potatoes are tender when pierced with a knife, about 1 hour. Set aside until cool enough to handle, about 10 minutes.
Meanwhile, in a medium frying pan, melt the butter over medium-high heat. Add the almonds and sauté, stirring constantly, for 1 minute. Add the garlic, green onions, and rosemary. Sauté, stirring frequently, until the onions are soft but not brown, about 2 minutes longer. Transfer to a plate and set aside.
Cut the potatoes in half lengthwise. Using a spoon, scoop out the flesh into a bowl, being careful to keep the skins intact and leaving a thin layer of flesh attached to the skins so the shells will be sturdy. Arrange the potato shells, hollow-side up, on a large rimmed baking sheet.
Use a potato masher or food mill to mash the potato flesh. Using a rubber spatula, fold in the onion mixture and the Gruyere cheese. Stir in the sour cream and then the yogurt. Add just enough of the milk to create a fluffy mixture that will mound nicely in the shells. Season with salt and pepper.
Spoon the potato mixture into the shells, dividing it evenly and mounding it above the rim of each shell. (For a fancier dish, use a pastry bag fitted with a large star tip to swirl the potato mixture into the shells.) In a small bowl, stir together the bread crumbs and Parmesan cheese. Spoon about 1½ tsp of the mixture on the top of each stuffed potato. (The stuffed potatoes can be prepared up to this point and set aside at room temperature for 2 hours.)
Bake the potatoes until heated through and nicely browned on top, about 20 minutes. To crisp the top, turn on the broiler and broil until the bread crumb mixture is nicely browned, 2 to 3 minutes. Serve immediately.