There's various issues here and I'm having trouble teasing out which are related so here's a bunch of them together in one bug.
For context, what I'm trying to achieve is to always have the same maximum number of items displayed, counting the "..." that get put in at the start and end when there's too many links to display, so if both of those are displayed I'd like two less links to compensate.
This is crying out for some unit tests as it's a pain to test manually with a live site.
I think the fundamental issue is that you can't currently change $maxdisplay of paging_bar lower (and possibly higher) without it breaking things. On line 2146 of lib/outputcomponents the $maxdisplay variable of the paging_bar class is set to 18 by default. It looks like some other decisions are based on this number, but not explicitly, so they don't change when you change this number, either by changing the default or setting it in your code.
So, first issue, line 2257 checks:
which means that no matter how few pages you want displayed via $maxdisplay it won't start truncating until you're on page 17 or higher, so if you have 18 pages of logs say, and are on page 16 it'll show pages 1 all the way through to 18. Click on next to go to 17 and it'll then show 1 ... 7 8 9 10 11 12 13 14 15 16 17 18. This is a fairly big jump from displaying 18 to 13 items and neither of those numbers are actually related to the "18" set in $maxdisplay even though one is coincidentally the same. The setting of $startpage as this->page - 10 also sets a lower bound of 10 on the number of items displayed.
There also seems to be an off-by-one error where if you have 19 pages, it'll display 18 ... 19 as if it was skipping a page but without actually skipping a page.