A packet's timebase is that of the stream it belongs to. Storing the timebase and manipulating it ourselves leads to headaches.
For instance when encoding video, the "flush" packets carry a valid dts and pts but their timebase doesn't get set, causing _rebase_time to destroy these timestamps. This leads to an exception upon flushing the output stream.