Abstract

Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use. In response, previous work combines decision trees with deep learning, creating a dilemma:

1. sacrifice interpretability to maintain accuracy OR
2. sacrifice accuracy to maintain interpretability.

We forgo this dilemma by proposing Neural-Backed Decision Trees (NBDTs), modified hierarchical classifiers that use trees constructed in weight-space. Our NBDTs achieve (1) interpretability and (2) neural network accuracy: We preserve interpretable properties -- e.g., leaf purity and a non-ensembled model -- and demonstrate interpretability of model predictions both qualitatively and quantitatively. Furthermore, NBDTs match state-of-the-art neural networks on CIFAR10, CIFAR100, TinyImageNet, and ImageNet to within 1-2%. This yields state-of-the-art interpretable models on ImageNet, with NBDTs besting all decision-tree-based methods by ~14% to attain 75.30% top-1 accuracy. Code and pretrained NBDTs are on Github.

Takeaways

Our work culminates in three key contributions that you can takeaway for future research:

• Our Neural-Backed Decision Tree achieves accuracy within 2% of state-of-the-art neural network EfficientNet on ImageNet, maintaining interpretable properties, showing accuracy and interpretability can be jointly attained.
• Hierarchical softmax is known to hurt accuracy. We thus instead develop and use a Tree Supervision Loss, demonstrating hierarchical softmax is not the only way to train a hierarchical classifier.
• Our Induced Hierarchy, built from the weights of a pretrained neural network, outperforms existing taxonomies (e.g., WordNet) and data-based hierarchies (e.g., using information gain), demonstrating a hierarchy built-in weight space is most effective.

Getting Started

Installation is just one line.

pip install nbdt

Run on any image of your choosing.

nbdt https://bit.ly/3eiuCId