It will give single tuple having value $0$.
select count(*) from Mystery
where A not in( select B from Mystery)
Inner query evaluates to $S: (null, 4, 5, 6)$
Outer query check for each tuple $t$ in table whether there exists any value $b \in S$ such that $t.A = b$. If that’s the case comparison returns false.
Now point here is that every $t.A$ is compared to $null$ exactly once and that comparison returns $null$. So that $t$ will not counted into result.
Likewise every tuple fails to counted into result and we get count $0$.