Very much along the right lines. Nice work. but...
1. Please comment out the
debugging('print_table() has been deprecated. Please change your code to use $OUTPUT->table().');
with a TODO to uncomment, until almost all calls in core code have been fixed. See, for example, admin_externalpage_print_footer in lib/adminlib.php
2. I think most of the ifs at the top of function table() should be in html_table::prepare()
3. Same for all the fix-up ifs in the header loop. That should be in prepare too.
4. html_table::__construct should not take arguments. We had a debate about this and decided on a consistent API with argument-less constructors.
5. Please can you move the __set magic for $class to the moodle_html_component class. Very nice and we should have that everywhere.
6. However, I think that it needs to be clearly defined what happens when you do $table->somethingunknown - and of course the PHP docs at http://php.net/manual/en/language.oop5.overloading.php are hopelessly vague. OK, based on a quick experiment it silently does nothing. I think we should either do $this->$name = $value at the end of __call if nothing else matches, or a throw new coding_exception. I am not sure which option is better.
7. I think the comments for the properties on html_table should be made more verbose, to make it clearer how to use the class (basically, give an example of a typical value for each argument). However, I see that you have just copied the existing comments from the old function, which is a natural place to start.
The justification for 2. and 3. is that if someone overrides moodle_core_renderer, they should not have to copy-and-paste this clean-up code, and they will almost certainly have to. That is why we have the prepare method, to avoid that sort of code duplication.
Sorry to be so picky (and I know I have been missing out some PHPdoc comments myself, recently). Anyway, if you fix all (or most) of these, then I am happy for you to commit without further review.