Determining depth from a single camera in motion is a challenging problem that has numerous applications, including autonomous navigation of an unmanned aerial vehicle (UAV). Using traditional computer vision techniques such as structure-from-motion (SfM), a depth estimate can be generated using two image pairs from the video stream. The choice of image pairs directly impacts the quality of reconstruction, which is based largely on the camera extrinsics and image features. In this article, we discuss frame selection algorithms to select appropriate image pairs to process for depth estimation. Frames are stored in a rolling buffer, and several measures are computed on potential frame pairs based on camera extrinsics. We use a customized SfM algorithm, EpiDepth, which has been designed for handling sequences of aerial imagery with embedded GPS and camera pose metadata. We demonstrate our technique on a simulated dataset created using Unreal Engine and AirSim.