When both filters are both "on" and are actually in effect, then MathJax filter does not work. There are 2 case scenarios.
1) Mathjax filter is configured higher in the order of filters (ie. above HTMLtidy), then the formulas are not rendered at all.
2) Mathjax filter is configured lower in the order of filters (ie. below HTMLtidy), then the formulas are rendered, but it is not possible to enter formulas in the equation editor as the HTMLtidy filters out the key HTML5 attributes "data-tex" in the ajax post.
When testing this please doublecheck that the HTMLtidy filter is really active in the PHP info page and actual HTML output (ie. the PHP PECL extension installed) as it is not indicated anywhere else in the Moodle (See MDL-47531).