Francis Franks (c1827-c1870)

I have this brick wall about an ancestor who lived in Hackney in London England and only appears once and that is in the 1861 census. He was born ca 1827 in Spitalfields and is a hemp/flax dresser. His name is Francis Franks, his wife Susan Barrett, children Sarah, Jane, Susan, Elenor, Francis and Mary Jane are also included on 1861 census but missing the following years. I wonder where he was in 1841, 1851 and 1871 census? Cant find him on Ancestry.com. There is no record of his marriage, death, immigration either. So odd. Susan his wife is on 1881 census as widow so he died during the 20 year period. Anyone have any ideas? This is a 15 year search so far.