Ygtjki

I had a discussion with a friend and we were talking about the advantages of random forest over linear regression.

At some point, my friend said that one of the advantages of the random forest over the linear regression is that it takes automatically into account the combination of features.

By this he meant that if I have a model with

• Y as a target


• X, W, Z as the predictors

then the random forests tests also the combinations of the features (e.g. X+W) whereas in linear regression you have to build these manually and insert them at the model.

I am quite confused, is this true?

Also if it true then is it about any kind of combination of features (e.g. X*W, X+W+Z etc) or only for some specific ones (e.g. X+W)?

I think it is true. Tree based algorithms especially the ones with multiple trees has the capability of capturing different feature interactions. Please see this article from xgboost official documentation and this discussion. You can say it's a perk of being a non parametric model (trees are non parametric and linear regression is not). I hope this will shed some light on this thought.

I would say it is not true as Random forests which are made up of decision trees does perform feature selection but they do not perform feature engineering (feature selection is different from feature engineering). Decision trees use a metric called Information gain (which is total entropy minus the weighted entropy) as per which useful features are separated from bad features. Simply to say whichever feature exhibit the highest information gain on this iteration is chosen as the node on which the tree on this iteration is split or you can say which feature reduces the entropy(aka randomness) the most in this iteration is chosen as the node upon which the tree is split on this iteration. So if you data is text, trees are split upon words. If your data is real valued numbers, tree is split upon that. Hope it helps

For more details check this

No, it is not true.

In Random Forest (or Decision Tree, or Regression Tree), individual features are compared to each other, not a combination of them, then the most informative individual is peaked to split a node. Therefore, there is no notion of "better combination" in the whole process.

Furthermore, Random Forest is a bagging algorithm which does not favor the randomly-built trees (or their sub-trees) over each other, they all have the same weight in the aggregated output.

It is worth noting that "Rotation forest" first applies PCA to features, which means each new feature is a linear combination of original features. However, this does not count since the same pre-processing can be used for any other method too.

Here is a quote from this wiki page for more details:

A decision tree is a flow-chart-like structure, where each internal
(non-leaf) node denotes a test on an attribute.

EDIT:

@tam provided a nice counter-example for XGBoost, which is not the same as Random Forest. That is, tree boosting algorithms have a notion of "better combination" that, for example, favors (in regression) $f(X, Y, W)=X*Y+textexp(X+Y) + 0 times W$ over $g(X, Y, W)=0 times X + Y+W$ by placing more weight on $f$. Note that a sub-tree, or a complete tree represents a function $f(boldsymbolX=(X,Y,Z,...))$ over a specific region $R subset (Bbb X, Bbb Y, Bbb Z,...)$ in the feature space.

In XGBoost, without going into details, more weight is put on a function that leads to more decrease in the overall error, loosely similar to AdaBoost algorithm (XGBoost in more detail, Adaboost vs XGBoost). This is equivalent to favoring a complicated-combination-of-features over another one.

Note that, a tree can approximate any continuous function $f$ over training points, since it is a universal approximator just like neural networks.