My original idea of making sure all smaller elements are demanded when bitcasting from a vector with small elements to a vector/scalar with larger elements seem to result in lots of lit test regressions. It is often fine to simplify based on some elements not being demanded, as long as not making the vector more poisonous. So maybe the problem here really is how we deal with INSERT_VECTOR_ELT in TargetLowering::SimplifyDemandedVectorElts. It is only ok to skip the INSERT_VECTOR_ELT if either the inserted value is known to be POISON, or if the element in the source vector is known not to be POISON.

No idea if similar problems then exists for other node types, or if it only is INSERT_VECTOR_ELT that has such problems.

I'm struggling a bit with this, trying to find a suitable solution.

Base problem as I see it is that we can have BITCAST of vectors, that merge smaller elements into larger vector elements (or just a scalar). For a vector individual elements can be poison without making the full vector poison (e.g. if only demanding certain elts). When doing SimplifyDemandedBits/SimplifyDemandedVectorElts in DAGCombiner we do not (always) demand all smaller elements when recursing through such a BITCAST.
Simply fixing this by making sure we demand all "not allowed to turn into poison" elements would result in several regressions in lit tests afaict.

One point of view would be that SimplifyDemanded* shouldn't be allowed to make things more poisonous when doing simplifications. More specifically not even turning undemanded elements from non-poison to poison, which typically can happen when dealing with INSERT_VECTOR_ELT. If the inserted element isn't demanded we simplify by just taking the source vector (which could have poison elements). This kind of "making the vector more poisonous" can even happen in SelectionDAG::getNode when trying to create a INSERT_VECTOR_ELT that inserts an UNDEF element (which otherwise could be a way of making sure the element isn't POISON).
Simply fixing this by making sure we avoid to turn undemanded elements into poison by eliminating INSERT_VECTOR_ELT operations that overwrite a maybe poison element also result in several lit test regressions. A typical example would be

        t992: v4i32 = insert_vector_elt undef:v4i32, t943, Constant:i32<1>
      t1000: v4i32 = insert_vector_elt t992, undef:i32, Constant:i32<3>
    t318: i32 = extract_vector_elt t1000, Constant:i32<1>
  t293: i32,i32 = ARMISD::LSRL t957, t318, Constant:i32<28>

when the irrelevant t1000 node would block collapsing of t318+t992 into t943, unless we make more advanced DAG combines to handle such situations.

Maybe there is a bit of philosophy around this. Should we allow simplifications that make a vector more poisonous during SimplifyDemanded* or not? Is there perhaps a difference between SimplifyDemandedBits and SimplifyDemandedVectorElts that I haven't understood? Maybe the whole idea with SimplifyDemandedVectorElts is that undemanded elements really should be irrelevant for the result (such as not impacting the poisoness of the result).

One idea I haven't tried yet is to propagate some kind of bool (MayPoisonUndemandedElts) or mask (UndemandedEltsThatMustNotBeMadeMorePosionous) in the SimplifyDemanded* function. This to indicate if we have recursed through a BITCAST that actually depend on the vector elements that aren't demanded (via the DemandedElts mask) but that isn't allowed to be turned into POISON. That would introduce lots of code changes just to propagate the flag when recursing, but could give us a way to use SimplifyDemanded* both in situations when we want to simplify through a BITCAST (when poison can be an issue even for elements that aren't used in the end) as well as for example simplifying an EXTRACT_VECTOR_ELT (when posion for undemanded elts isn't an issue).

Do you have any advice/recommendations regarding this (@RKSimon , @arsenm , @nikic, @dtcxzyw, or maybe someone else)?

I haven't looked into your example in detail, but I believe the way it should work is that you can turn non-demanded elements poison, but you can't turn non-demanded bits poison. This also means that you cannot transfer non-demanded bits into non-demanded elements for bitcasts. At least this is the way it would work in the middle end.