Heat oil in a large heavy pot over medium heat.
Add leek and celery and cook, stirring often, until vegetables are soft, 5-8 minutes.
Add chicken and broth; season with salt and pepper. Bring to a boil, cover, reduce heat, and simmer until chicken is cooked through, 15-20 minutes.
Transfer chicken to a plate. Let cool, then shred chicken into bite-size pieces.
Meanwhile, return broth to a boil.
Add orzo and cook until al dente, 8-10 minutes.
Remove pot from heat. Stir in chicken and dill. Serve with lemon halves for squeezing over.