About 20% of offices sit empty while affordable homes are in dire need. Conversions alone won't bridge the gap.