I moved from Hornchurch to Billericay when I was 6 , My Father liked football but had no allegiance to any particular team. When my Brother and I wanted to go to a football match he chose a Southend game because it was a lot easier to get to, a Friday night so he could take us with out it impacting on work (he used to do overtime on Saturday mornings and he could pick Granddad up from Wickford on the way.
Had my Mums father not died when I was 8 months old, it may have been a different story, He was a Spurs season ticket holder (he died of a heart attack on the way home from a 4-4 draw in February 59) So I reckon he would have taken me to my First game at White Hart Lane before I went to Roots Hall in 67.
You could have done worse. My dad was a Spurs regular and took me along to see the Spurs team of the early 60s who were a joy to watch. Being born in Rayleigh and going to school there meant that I had to make the choice of going to Southend with my school mates or going to Spurs with the old man. Southend won easily, on local loyalty and games, and after that the Blues were my team. I still have a soft spot for the Spurs though.