Uhm, let's see:
1) the delete_old_backup_dirs() function is used to delete OLD (4 hours old) backup information from the $CFG->dataroot/temp/backup directory.
2) so it is NOT related with current backup/restore/import/duplicate operation but with some OLD execution that left files there.
Said that, the most common causes for any file not being deleted uses to be
A) some incorrect names in the files being deleted.
B) some sort of wrong permissions files in the filesystem.
The A) cause shouldn't be a problem at all, as far as, since Moodle 2.0 we don't use "incorrect names" for anything, but safe ones all the time.
The B) cause is somehow more probable. If in your site you have executed any cron/cli script and it has been executed with one user different from the the web server one, then you can end with files in $CFG->dataroot/temp/backup having one wrong owner. And that wrong owner prevents them to be deleted by the "duplicate" tool, that is always executed by the web server user (apache/www/nobody or whatever).
So I think that the correct solution for this is:
1) In your site, to confirm B), check which files/dirs are present, more than 4 hours old in the $CFG->dataroot/temp/backup directory and check who is the owner of all them. Can you confirm that you had some temp stuff belonging to another user?
2) Change all your cron/cli scripts to be executed by the same user running the web server process
3) Delete completely the contents of $CFG->dataroot/temp/backup. The problems should not happen if 2) has been fulfilled.
If, after 1, 2, 3, the problem continues... then we'll need to investigate a bit more, surely by running some extra debugging in the site, to see the exact files/dirs causing this.
I'll keep this open some days, before closing as not a bug. Ciao
Edited: Under 21_STABLE the dir is $CFG->dataroot/temp/backup, under master (upcoming 2.2) the dir is $CFG->tempdir/backup.