There are no conditions on the workflow, so it will execute for all records. The delay simply indicates that if a task should be done, don't do it now, but in "delay" days, it is not a condition.
Add a condition on your workflow to get only those leads that have not been modified in the last 7 days or less: