tutorial 5 音視頻同步初印象 · FFmpeg入門實踐與分析

### 同步視頻基礎時間戳 dts解碼時間戳 pts顯示時間戳 ### 動手實踐目標 1. 想不想動手實踐看看**p**ts第一次被賦值。 1. 重點研究那個video_fresh-timer函數 1. 鑒于分析的視頻只有12幀， - 理解actual_delay 期望值 - Audio的時間 - 理解一下那個時間戳 **vs**音頻時間的言論** 1. video_fresh-timer關注delay :（現在的時間戳-上一個時間戳）同步閾值兩個變量 1. 提前告知 tutorial同步音頻到視頻， 1. 認識校正方法，是修正delay,要么加倍延遲，要么沒有延遲。下一節對于音頻來說，要么插值，要么刪減？一句話，同步的手段各有。 ### ?表面變化：代碼變化（tutorial 4 vs tutorial 5）增加的宏如下： ~~~ #define AV_SYNC_THRESHOLD 0.010 //minimum sync time #define AV_NOSYNC_THRESHOLD 10 //the differ of video and audio exceeds the time, do not sync with audio ~~~ 值得注意的數據結構變化如下： ~~~ struct VideoState{ ... double video_clock; double audio_clock; double frame_last_pts; double frame_last_delay; double frame_timer; } VideoState; ~~~ 關于音頻當前時鐘字段，暫時忽略，詳見官方tutorial ### 視頻線程代碼及變遷 ~~~ int video_thread(void *arg) { VideoState *is = (VideoState *) arg; AVPacket pkt1, *packet = &pkt1; int len1, frameFinished; AVFrame *pFrame; double pts = 0; pFrame = avcodec_alloc_frame(); for (;;) { if (packet_queue_get(&is->videoq, packet, 1) < 0) { // means we quit getting packets break; } pts = 0; //save global pts to be stored in pFrame in first call global_video_pkt_pts = packet->pts; // printf("packet->pts %d\n", packet->pts ); // printf("packet->dts %d\n", packet->dts ); // Decode video frame len1 = avcodec_decode_video2(is->video_st->codec, pFrame, &frameFinished, packet); if (packet->dts == AV_NOPTS_VALUE && pFrame->opaque && *(uint64_t*) pFrame->opaque != AV_NOPTS_VALUE) { pts = *(uint64_t*) pFrame->opaque; } else if (packet->dts != AV_NOPTS_VALUE) { pts = packet->dts; } else { pts = 0; } // printf("pts %d\n", pts ); pts *= av_q2d(is->video_st->time_base); // Did we get a video frame? if (frameFinished) { pts = synchronize_video(is, pFrame, pts); if (queue_picture(is, pFrame, pts) < 0) { break; } } av_free_packet(packet); } av_free(pFrame); return 0; } ~~~ 差別有如下： Pts值的初始化,詳細如下圖圖中紅色部分是額外添加的dts處理 ![](https://box.kancloud.cn/2016-02-22_56cae4b827038.jpg) 下圖中表明還有視頻同步函數的加入。 ![](https://box.kancloud.cn/2016-02-22_56cae4b839dea.jpg) 以上是所有的兩處大的修改。 ### 視頻刷新定時器video_refresh_timer 一句話：請看英文解釋，一次性解釋清楚 <table border="1" cellspacing="0" cellpadding="7" width="569"><colgroup><col width="553"/></colgroup><tbody><tr><td valign="top" width="553">void video_refresh_timer(void *userdata) { VideoState *is = (VideoState *) userdata;VideoPicture *vp;double actual_delay, delay, sync_threshold, ref_clock, diff; if (is->video_st) {if (is->pictq_size == 0) {schedule_refresh(is, 1);} else {vp = &is->pictq[is->pictq_rindex]; /* Timing code goes here */delay = vp->pts - is->frame_last_pts; //the pts from lastif (delay <= 0 || delay >= 1.0) {//if incorrect delay, use previous one過大或者多小，超過1s,不過理喻delay = is->frame_last_delay;}/* save for next time */is->frame_last_delay = delay;is->frame_last_pts = vp->pts; /* update delay to sync to audio */ref_clock = get_audio_clock(is);diff = vp->pts - ref_clock; /* Skip or repeat the frame. Take delay into accountFFPlay still doesn't "know if this is the best guess." */sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay: AV_SYNC_THRESHOLD;if (fabs(diff) < AV_NOSYNC_THRESHOLD) {if (diff <= -sync_threshold) {delay = 0;//video is slow, do not delay，視頻已太慢（視頻預測時間-當前時間都是負值，請趕緊播）} else if (diff >= sync_threshold)delay = 2 * delay;//video is fast, delay double，視頻太快，刷新時間變慢點}}is->frame_timer += delay;/* computer the REAL delay */actual_delay = is->frame_timer - (av_gettime() / 1000000.0);if (actual_delay < 0.010) {/* Really it should skip the picture instead */actual_delay = 0.010;}schedule_refresh(is, (int) (actual_delay * 1000 + 0.5));// schedule_refresh(is, 80); /* show the picture! */video_display(is); /* update queue for next picture! */后續略} </td></tr></tbody></table> ### 實驗結果分析， #### Video_thread代碼： #### 所有亮點都在顏色里。 Video_thread{ ~~~ Video_thread{ ... if (packet->dts == AV_NOPTS_VALUE && pFrame->opaque && *(uint64_t*) pFrame->opaque != AV_NOPTS_VALUE) { pts = *(uint64_t*) pFrame->opaque; } else if (packet->dts != AV_NOPTS_VALUE) { pts = packet->dts;//12幀測試案列執行此代碼覆蓋 } else { pts = 0; } pts *= av_q2d(is->video_st->time_base); // Did we get a video frame? if (frameFinished) { pts = synchronize_video(is, pFrame, pts); //除第一幀沒有pts賦值，其他幀都有賦值。其他備注本案例沒有重復幀， if (queue_picture(is, pFrame, pts) < 0) { break; } } } ~~~ ### video_refresh_timer代碼運行結果這里的數據結果較多，尤其替換了上一小節定時刷新，值得反復體會。 ~~~ void video_refresh_timer(void *userdata) { ... if (is->video_st) { if (is->pictq_size == 0) { } else { vp = &is->pictq[is->pictq_rindex]; /* Timing code goes here */ delay = vp->pts - is->frame_last_pts; //the pts from last //實驗結果非常理想，所有12幀處理的結果都是1.0 if (delay <= 0 || delay >= 1.0) { //if incorrect delay, use previous one delay= is->frame_last_delay; } /* save for next time */ is->frame_last_delay = delay; is->frame_last_pts = vp->pts; /* update delay to sync to audio */ ref_clock = get_audio_clock(is); diff = vp->pts - ref_clock;//具體的值見下面的表格 /* Skip or repeat the frame. Take delay into account FFPlay still doesn't "know if this is the best guess." */ sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD; if (fabs(diff) < AV_NOSYNC_THRESHOLD) { if (diff <= -sync_threshold) { delay = 0; //video is slow, do not delay //實驗結果非常理想，都是說明視頻太慢， } else if (diff >= sync_threshold) { delay = 2 * delay; //video is fast, delay double } } is->frame_timer += delay;//鑒于12幀delay=0，這個值就沒有增長過 /* computer the REAL delay */ actual_delay = is->frame_timer - (av_gettime() / 1000000.0); if (actual_delay < 0.010) {//詳細的真實delay數據如下，可見都是大大小于閾值 /* Really it should skip the picture instead */ actual_delay = 0.010; } logtime=actual_delay * 1000 + 0.5; schedule_refresh(is, (int) (actual_delay * 1000 + 0.5));//所有填入的時間都是10.5 // schedule_refresh(is, 80); /* show the picture! */ video_display(is); //ingore the the other code } } else { schedule_refresh(is, 100); } } ~~~ ### 12幀的時延分析表格，對于所有12幀的delay初步結果分析，可以看到所有的幀都太慢，不執行delay.**但是視覺上看來，視頻幀還是太慢**，這說明音頻還得同視頻同步，期待下一小節。 <table border="1" cellspacing="0" cellpadding="7" width="569"><colgroup><col width="553"/></colgroup><tbody><tr><td valign="top" width="553">// 10 //the differ of video and audio exceeds the time, do not sync with audio// if(diff <= -sync_threshold)所有出現的sync_threshold都是0.04diff=-2.0393626991565137, //video is slow, do not delay，diff=-5.8978444236176193, //video is slow, do not delay，diff=-9.8762886597938149, //video is slow, do not delay，diff=-8.9662605435801321, //video is slow, do not delay，diff=-7.9662605435801321, //video is slow, do not delay，diff=-6.9662605435801321, //video is slow, do not delay，diff=-5.9662605435801321, //video is slow, do not delay，diff=-4.9662605435801321, //video is slow, do not delay，diff=-3.9662605435801321, //video is slow, do not delay，diff=-2.9662605435801321, //video is slow, do not delay，diff=-1.9662605435801321, //video is slow, do not delay，diff=-0.96626054358013214, //video is slow, do not delay</td></tr></tbody></table> 在計算真實的delay時，涉及到的表格如下， <table border="1" cellspacing="0" cellpadding="7" width="569"><colgroup><col width="553"/></colgroup><tbody><tr><td valign="top" width="553">actual_delay= -3.2161841392517090 ,frame_timer= 1401626707.2479169actual_delay= -8.3304760456085205 ,frame_timer= 1401626707.2479169actual_delay= -13.639780044555664 ,frame_timer= 1401626707.2479169actual_delay= -14.218813180923462 ,frame_timer= 1401626707.2479169actual_delay= -14.572834014892578 ,frame_timer= 1401626707.2479169actual_delay= -14.932854175567627 ,frame_timer= 1401626707.2479169actual_delay= -15.305875062942505 ,frame_timer= 1401626707.2479169actual_delay= -15.662896156311035 ,frame_timer= 1401626707.2479169actual_delay= -16.020915985107422 ,frame_timer= 1401626707.2479169actual_delay= -16.381937026977539 ,frame_timer= 1401626707.2479169actual_delay= -16.737957000732422 ,frame_timer= 1401626707.2479169actual_delay= -16.947968959808350 ,frame_timer= 1401626707.2479169 </td></tr></tbody></table> ## 其他：當播放結束，沒有額外的多調用刷屏函數，不像上一個版本。結果中，也有多線程的隨機性出現，情況在預料之中。未完成分析日志log的一輪起始，下次補充就是。現在認識依然很淺，比如i b b p的認識 ## 小結：邏輯很復雜，測試用例很理想，入門在路上。不要忘了英文注釋不要忘了。。視頻即使同步了，也比音頻慢！！回顧，主要是看vedio_thread ,和播放的timer ### 附件完整log分析簡化版（沒有帶修正的log） Function: video_thread(void *), [pts 2]=0 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=0.00000000000000000 Function: video_refresh_timer(void *), pts之delay=0.00000000000000000 Function: video_refresh_timer(void *), diff=-2.0393626991565137, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=1 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=1.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-5.8978444236176193, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=2 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=2.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-9.8762886597938149, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=3 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=3.0000000000000000 Function: video_thread(void *), [pts 2]=4 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=4.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-8.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=5 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=5.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-7.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=6 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=6.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-6.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=7 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=7.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-5.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=8 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=8.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-4.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=9 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=9.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-3.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=10 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=10.000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-2.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=11 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=11.000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-1.9662605435801321, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-0.96626054358013214, //video is slow, do not delay Function: video_refresh_timer(void *),【core time】=10.500000000000000 ### 最終用到的log Function: our_get_buffer(AVCodecContext *, AVFrame *),pts初始化in自定義alloc，pts=0 Function: video_thread(void *), [pts 2]=0 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we aren't given a pts, set it to the clock */, 0.00000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=0.00000000000000000 Function: video_refresh_timer(void *), pts之delay=0.00000000000000000 Function: video_refresh_timer(void *), diff=-2.0393626991565137, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -3.2161841392517090 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=1 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 1.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=1.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-5.8978444236176193, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -8.3304760456085205 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=2 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 2.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=2.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-9.8762886597938149, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -13.639780044555664 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=3 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 3.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=3.0000000000000000 Function: video_thread(void *), [pts 2]=4 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 4.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=4.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-8.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -14.218813180923462 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=5 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 5.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=5.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-7.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -14.572834014892578 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=6 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 6.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=6.0000000000000000 Function: video_refresh_timer(void *), diff=-6.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -14.932854175567627 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=7 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 7.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=7.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-5.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -15.305875062942505 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=8 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 8.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=8.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-4.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -15.662896156311035 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=9 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 9.0000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=9.0000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-3.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -16.020915985107422 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=10 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 10.000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=10.000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-2.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -16.381937026977539 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_thread(void *), [pts 2]=11 Function: synchronize_video(VideoState *, AVFrame *, double), /* if we have pts, set video clock to it */ 11.000000000000000 Function: video_thread(void *),【pts同步后？,經過計算重復幀等】=11.000000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-1.9662605435801321, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -16.737957000732422 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000 Function: video_refresh_timer(void *), pts之delay=1.0000000000000000 Function: video_refresh_timer(void *), diff=-0.96626054358013214, //video is slow, do not delay，0.040000000000000001 must>=10ms Function: video_refresh_timer(void *), /* computer the REAL delay */ actual_delay= -16.947968959808350 ,frame_timer= 1401626707.2479169 Function: video_refresh_timer(void *),【core time】=10.500000000000000