One way to resolve this is to have some kind of multiverse theory where you don’t travel back in time to your universe, but to a narrow slection of parallel universes that are also shifted slightly so that it spits you out in an analogous location to your initial departure.