This thread is bound to be very subjective to personal taste. We've been to STX 4 times and we enjoyed it, how could you not. BUT for us STJ is our happy place. We're heading back to a Villa called Wings Over Water in April, check it out, the location is outstanding.(although we prefer Coral Bay, and this villa is not on that side of the island, it is just so close to the water we love it). We've lost count but we've been to STJ 15 to 20 times. Personally we prefer the Coral Bay side as it still has that very laid back felling to us. We really like the places to eat on the Coral Bay side and It's a reasonable drive to the North Shore beaches with Jumbie being our favorite.
If being close to a town is important to you, it's not to us, we're old and have no social life anymore

we just want to chill, but STX probably has better choices around things to do in a town. On STX we prefer the East End.
If I was going to live on an Island STX would be appealing due to the variety of things you have to chose from. The beaches are nice but in general smaller than the ones on STJ, and the water colors are stunning. But for pure relaxation STJ gets our vote.
Good luck with whatever you chose, you can't go wrong. But if you go to STJ try to spend some extended time on the Coral Bay side. You'll be hooked.
For my money it would be STJ on the Coral Bay side or near Ditliff Point. On St. Croix, the East End but is not too close to a town as I recall, but it's been a few years.