Why does everyone hate or is negative when you want to take 10k penalty free out of your 401k for a first house down payment!!

I’ve read so many articles being negative towards taking out 10k (WHICH is penalty free) for first time homebuyers. They always say “oh well that 10k could be 100k in 40 years”. OKAY and life is short!? The housing market has went up in value more than the stock market so that 10k maybe 100k in 40 years statement is old!! A home is an investment maybe that could grow in 40 years!!