1. Preheat the oven to 350°F. Line a baking sheet with parchment paper.
2. In a stand mixer, beat butter, brown sugar, and granulated sugar until combined, about 1 minute.
3. Scrape down the bowl and add the egg, milk, and vanilla. Beat until combined.
4. Add flour, graham cracker crumbs, baking powder, and salt. Beat on low speed until combined.
5. Use a 1 1/2 ounce cookie scoop to portion dough, roll into balls, and place 2 inches apart on the baking sheet. Flatten slightly and press a divot in the center of each.
6. Bake cookies for 5 minutes.
7. While baking, mix pumpkin puree, brown sugar, egg, evaporated milk, flour, and pumpkin pie spice in a bowl.
8. Remove cookies from the oven. Redefine divots and fill each with 2 teaspoons of pumpkin filling.
9. Return to oven and bake until golden brown, about 6 to 8 minutes.
10. Cool completely. Top with whipped cream if desired.