Can't use stream added from template for anything other than remuxing

Overview

When I create a new Container, and add a new stream to it using a template, I can only remux packets. I cannot decode and then re-encode packets.

Expected behavior

I would expect to be able to roundtrip a packet like decode packet -> frame -> encode frame -> packet.

Actual behavior

When trying to do this roundtrip, I am presented with an error.

Traceback:

File "av/stream.pyx", line 152, in av.stream.Stream.encode
File "av/codec/context.pyx", line 476, in av.codec.context.CodecContext.encode
File "av/codec/context.pyx", line 391, in av.codec.context.CodecContext._send_frame_and_recv
File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument

Investigation

I've dug about as far in to the pyav source as I can, but I'm not very familiar with Cython

Reproduction

input_container = av.open('in.mp4', 'r')
input_stream = input_container.streams.get(video=0)[0]

output_container = av.open('out.mp4', 'w')
output_stream = output_container.add_stream(template=input_stream)

for frame in input_container.decode(input_stream):
    # encode and mux
    for packet in output_stream.encode(frame):
        packet.pts = frame.pts
        packet.time_base = frame.time_base
        # packet.stream = output_stream
        output_container.mux(packet)

output_container.close()

Versions

OS: Ubuntu 18.04
PyAV runtime:

PyAV v8.0.2
git origin: git@github.com:PyAV-Org/PyAV
git commit: v8.0.2
library configuration: --disable-doc --disable-static --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-shared --enable-version3 --enable-zlib --prefix=/tmp/vendor
library license: GPL version 3 or later
libavcodec     58. 54.100
libavdevice    58.  8.100
libavfilter     7. 57.100
libavformat    58. 29.100
libavutil      56. 31.100
libswresample   3.  5.100
libswscale      5.  5.100

FFmpeg:

ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil      55. 78.100 / 55. 78.100
libavcodec     57.107.100 / 57.107.100
libavformat    57. 83.100 / 57. 83.100
libavdevice    57. 10.100 / 57. 10.100
libavfilter     6.107.100 /  6.107.100
libavresample   3.  7.  0 /  3.  7.  0
libswscale      4.  8.100 /  4.  8.100
libswresample   2.  9.100 /  2.  9.100
libpostproc    54.  7.100 / 54.  7.100

Research

I have done the following:

Checked the PyAV documentation
Searched on Google
Searched on Stack Overflow
Looked through old GitHub issues
Asked on PyAV Gitter
... and waited 72 hours for a response.

Additional context

I only installed ffmpeg so I could inspect some files. I did not build PyAV against it.