The timing is very consistent, at least when using SDK 1.4.0. I use the nodeMCU lua firmware rather than the raw SDK but the issue is probably the same.
In your case I would measure the time across, say, 48 readings (about two days) and set a factor that makes this period show as 48 hours. I expect that the timing will then show periods when it is fast and others where it is slow, I clearly see it between the 10dC temperature at night and 30dC later in the day. The adjustments I made a few months ago are now not good (as summer rushes in) and I will need to readjust. one could, of course, do temperature compensation but I do not need such accuracy.
cheers