Place the potatoes and 1-2 teaspoons of salt in a large pot of water. Bring the water to a boil, then lower the heat and simmer until the potatoes are barely tender when pierced with a knife, about 10-15 minutes depending on the size of your potatoes. Drain the potatoes in a colander and place over the empty pot. Cover the colander with a clean, dry kitchen towel. Allow the potatoes to steam for 15 to 20 minutes.
In a small bowl, whisk together the mayonnaise, buttermilk (or heavy cream), Dijon, whole grain mustard, dill, 1 teaspoon of salt, and 1 teaspoon of pepper. Set aside.
When the potatoes are cool enough to handle, cut them in quarters or in half, depending on their size. Place them cut in a large bowl and pour enough dressing over them to moisten. Add the celery, red onion, and salad olives (if using), salt to taste and 1 teaspoon of pepper. Toss well, cover, and refrigerate for a few hours to allow the flavors to blend. Serve cold or at room temperature.