Grease an 8x8-inch pan and line it with a square of parchment paper.
In a large bowl, cream the butter and brown sugar until fluffy and light, about 3 minutes. Beat in the eggs one at a time until well combined.
In a medium bowl, whisk together the flours, baking soda, baking powder, salt, cinnamon and cardamom. With the mixer on medium speed, beat in half the flour mixture, then the buttermilk, and finish with the flour.
Pour the batter into the prepared pan and smooth the top with a rubber spatula. Arrange the peach slices on top. Let cake rest while you heat the oven to 350° F.
Place the cake in the oven and bake for 45-55 minutes, until cake is set, a tester comes out clean, and edges are just starting to pull away from the sides. Remove from oven, place on a wire rack, and let cake cool for 30 minutes in the pan before turning out onto the rack to cool fully.
Serve cake slightly warm or at room temperature. Cake will keep, wrapped, on the counter for 2 days, or 5 days in the fridge.