What do you mean by “when I come back and use it” - do you mean, powering the whole thing down and starting from scratch much later? Or leaving it on and just returning to a specific point?
2-3mm is quite a bit off; it shouldn’t be off that much, even for stepper motors. The home routine is what establishes the machine origin. It’s possible something is loose or not accurate with homing, and I’d look into that. Also, parking at repeating the homing process from a known location each time may help with repeatability.. And just to be clear: you have to home the machine after turning it off and on. For better repeatability, you could move to servo motors, which will save position more accurately when powered on/off.