PJSIP使用视频:本地预览、视频解码,对端预览

实现需求,全志IPC,PJSIP本地预览视频,解码并显示对端视频。先梳理PJSIP本地预览和解码显示流程。

 

本地预览:默认配置 vid_preview_enable_native 是开启的。

PJ_DEF(void) pjsua_media_config_default(pjsua_media_config *cfg)
{
    //....
    cfg->vid_preview_enable_native = PJ_TRUE;
}

static pj_status_t create_vid_win(pjsua_vid_win_type type,
				  const pjmedia_format *fmt,
				  pjmedia_vid_dev_index rend_id,
				  pjmedia_vid_dev_index cap_id,
				  pj_bool_t show,
                                  unsigned wnd_flags,
                                  const pjmedia_vid_dev_hwnd *wnd,
				  pjsua_vid_win_id *id)
{
    pj_bool_t enable_native_preview;
    pjsua_vid_win_id wid = PJSUA_INVALID_ID;
    pjsua_vid_win *w = NULL;
    pjmedia_vid_port_param vp_param;
    pjmedia_format fmt_;
    pj_status_t status;
    unsigned i;

    enable_native_preview = pjsua_var.media_cfg.vid_preview_enable_native;
    
    //...
    
    //...
       /*
         * Determine if the device supports native preview.
         */
        status = pjmedia_vid_dev_get_info(cap_id, &vdi);
        if (status != PJ_SUCCESS)
            goto on_error;

        if (enable_native_preview &&
             (vdi.caps & PJMEDIA_VID_DEV_CAP_INPUT_PREVIEW))
        {
            /* Device supports native preview! */
            w->is_native = PJ_TRUE;
        }
    //...

}

本地预览视频流程:

呱牛笔记


视频本地采集,编码后传输:

摄像头采集回调:

呱牛笔记

on_clock_tick驱动编码,rtp发送。

呱牛笔记


显示对端视频,收包,解码,然后显示。

channel->stream->transport

/**
 * Media channel.
 */
typedef struct pjmedia_vid_channel
{
    pjmedia_vid_stream     *stream;         /**< Parent stream.             */
    pjmedia_dir             dir;            /**< Channel direction.         */
    pjmedia_port            port;           /**< Port interface.            */
    unsigned                pt;             /**< Payload type.              */
    pj_bool_t               paused;         /**< Paused?.                   */
    void                   *buf;            /**< Output buffer.             */
    unsigned                buf_size;       /**< Size of output buffer.     */
    pjmedia_rtp_session     rtp;            /**< RTP session.               */
} pjmedia_vid_channel;


/**
 * This structure describes media stream.
 * A media stream is bidirectional media transmission between two endpoints.
 * It consists of two channels, i.e. encoding and decoding channels.
 * A media stream corresponds to a single "m=" line in a SDP session
 * description.
 */
struct pjmedia_vid_stream
{
    pj_pool_t               *own_pool;      /**< Internal pool.             */
    pjmedia_endpt           *endpt;         /**< Media endpoint.            */
    pjmedia_vid_codec_mgr   *codec_mgr;     /**< Codec manager.             */
    pjmedia_vid_stream_info  info;          /**< Stream info.               */
    pj_grp_lock_t           *grp_lock;      /**< Stream lock.               */

    pjmedia_vid_channel     *enc;           /**< Encoding channel.          */
    pjmedia_vid_channel     *dec;           /**< Decoding channel.          */

    pjmedia_dir              dir;           /**< Stream direction.          */
    void                    *user_data;     /**< User data.                 */
    pj_str_t                 name;          /**< Stream name                */
    pj_str_t                 cname;         /**< SDES CNAME                 */

    pjmedia_transport       *transport;     /**< Stream transport.          */
 }

建立数据编码通道:    create_channel( pool, stream, PJMEDIA_DIR_ENCODING,  info->tx_pt, info, &stream->enc);


建立数据解码通道:  create_channel( pool, stream, PJMEDIA_DIR_DECODING, info->rx_pt, info, &stream->dec);


音频
Stream.c (pjmedia\src\pjmedia):static pj_status_t create_channel( pj_pool_t *pool,
Stream.c (pjmedia\src\pjmedia):    status = create_channel( pool, stream, PJMEDIA_DIR_DECODING,
Stream.c (pjmedia\src\pjmedia):    status = create_channel( pool, stream, PJMEDIA_DIR_ENCODING,
视频

Vid_stream.c (pjmedia\src\pjmedia):static pj_status_t create_channel( pj_pool_t *pool,
Vid_stream.c (pjmedia\src\pjmedia):    status = create_channel( pool, stream, PJMEDIA_DIR_DECODING,
Vid_stream.c (pjmedia\src\pjmedia):    status = create_channel( pool, stream, PJMEDIA_DIR_ENCODING,


/*
 * Create stream.
 */
PJ_DEF(pj_status_t) pjmedia_vid_stream_create(
                                        pjmedia_endpt *endpt,
                                        pj_pool_t *pool,
                                        pjmedia_vid_stream_info *info,
                                        pjmedia_transport *tp,
                                        void *user_data,
                                        pjmedia_vid_stream **p_stream)

/*
 * Create media channel.
 */
static pj_status_t create_channel( pj_pool_t *pool,
                                   pjmedia_vid_stream *stream,
                                   pjmedia_dir dir,
                                   unsigned pt,
                                   const pjmedia_vid_stream_info *info,
                                   pjmedia_vid_channel **p_channel)     
{                                                                      
    /* Create RTP and RTCP sessions: */
    {
        pjmedia_rtp_session_setting settings;

        settings.flags = (pj_uint8_t)((info->rtp_seq_ts_set << 2) |
                                      (info->has_rem_ssrc << 4) | 3);
        settings.default_pt = pt;
        settings.sender_ssrc = info->ssrc;
        settings.peer_ssrc = info->rem_ssrc;
        settings.seq = info->rtp_seq;
        settings.ts = info->rtp_ts;
        status = pjmedia_rtp_session_init2(&channel->rtp, settings);
    }  

}


解码流程:

呱牛笔记


问题1:

编码发送驱动是由on_clock_tick驱动,那解码显示驱动呢?参考音频的,音频的播放是音频设备的play_cb驱动port的get_frame驱动。

但是视频的解码包显示是由on_clock_tick驱动的。

呱牛笔记

        /* Call sink->put_frame()
         * Note that if transmitter_cnt==0, we should still call put_frame()
         * with zero frame size, as sink may need to send keep-alive packets
         * and get timestamp update.
         */
        pj_bzero(&frame, sizeof(frame));
        frame.type = PJMEDIA_FRAME_TYPE_VIDEO;
        frame.timestamp = *now;
        if (frame_rendered) {
            frame.buf = sink->put_buf;
            frame.size = sink->put_frm_size;
        }
        status = pjmedia_port_put_frame(sink->port, &frame);
        if (frame_rendered && status != PJ_SUCCESS) {
            sink->last_err_cnt++;
            if (sink->last_err != status ||
                sink->last_err_cnt % MAX_ERR_COUNT == 0)
            {
                if (sink->last_err != status)
                    sink->last_err_cnt = 1;
                sink->last_err = status;
                PJ_PERROR(5, (THIS_FILE, status,
                              "Failed (%d time(s)) to put frame to port %d"
                              " [%s]!", sink->last_err_cnt,
                              sink->idx, sink->port->info.name.ptr));
            }
        } else {
            sink->last_err = status;
            sink->last_err_cnt = 0;
        }


/**
 * Get a frame from the port (and subsequent downstream ports).
 */
PJ_DEF(pj_status_t) pjmedia_port_get_frame( pjmedia_port *port,
                                            pjmedia_frame *frame )
{
    PJ_ASSERT_RETURN(port && frame, PJ_EINVAL);

    if (port->get_frame)
        return port->get_frame(port, frame);
    else {
        frame->type = PJMEDIA_FRAME_TYPE_NONE;
        return PJ_EINVALIDOP;
    }
}


断点调试发现,sdl显示设备的驱动是on_clock_tick定时器。

呱牛笔记

sink->port = 0x044f70fc {info={name={ptr=0x044f70ec "SDL renderer" slen=12 } signature=1448038479 dir=PJMEDIA_DIR_DECODING (2) ...} ...}


这个src是:

port = 0x040fb7cc {info={name={ptr=0x040fb860 "vstdec040FAE74" slen=14 } signature=1347834708 dir=PJMEDIA_DIR_DECODING (2) ...} ...}


原来解码完的数据是放在stream->dec_frame.buf中的,get_frame方法发现dec_frame.size大于0,则拷贝到frame中,用来显示。

static pj_status_t get_frame(pjmedia_port *port,
                             pjmedia_frame *frame)
{
    pjmedia_vid_stream *stream = (pjmedia_vid_stream*) port->port_data.pdata;
    pjmedia_vid_channel *channel = stream->dec;

    /* Return no frame is channel is paused */
    if (channel->paused) {
        frame->type = PJMEDIA_FRAME_TYPE_NONE;
        frame->size = 0;
        return PJ_SUCCESS;
    }

    /* Report pending events. Do not publish the event while holding the
     * stream lock as that would lead to deadlock. It should be safe to
     * operate on fmt_event without the mutex because format change normally
     * would only occur once during the start of the media.
     */
    if (stream->fmt_event.type != PJMEDIA_EVENT_NONE) {
        pjmedia_event_fmt_changed_data *fmt_chg_data;

        fmt_chg_data = &stream->fmt_event.data.fmt_changed;
        
        	#if 1	//realloc size	
        	pj_int32_t new_size = fmt_chg_data->new_fmt.det.vid.size.h*fmt_chg_data->new_fmt.det.vid.size.w*1.5;
        	if (stream->dec_max_size < new_size)
                {
                    PJ_LOG(5, (THIS_FILE, "Reallocating vid_stream dec_buffer %u --> %u",
                        (unsigned)stream->dec_max_size,
                        (unsigned)new_size));
                    pj_mutex_lock(stream->jb_mutex);
        
                    stream->dec_max_size = new_size;
                    stream->dec_frame.buf = pj_pool_alloc(stream->own_pool, stream->dec_max_size);
                    pj_mutex_unlock(stream->jb_mutex);
                }
        	#endif//realloc size
        	 
        /* Update stream info and decoding channel port info */
        if (fmt_chg_data->dir == PJMEDIA_DIR_DECODING) {
            pjmedia_format_copy(&stream->info.codec_param->dec_fmt,
                                &fmt_chg_data->new_fmt);
            pjmedia_format_copy(&stream->dec->port.info.fmt,
                                &fmt_chg_data->new_fmt);

            /* Override the framerate to be 1.5x higher in the event
             * for the renderer.
             */
            fmt_chg_data->new_fmt.det.vid.fps.num *= 3;
            fmt_chg_data->new_fmt.det.vid.fps.num /= 2;
        } else {
            pjmedia_format_copy(&stream->info.codec_param->enc_fmt,
                                &fmt_chg_data->new_fmt);
            pjmedia_format_copy(&stream->enc->port.info.fmt,
                                &fmt_chg_data->new_fmt);
        }

        dump_port_info(fmt_chg_data->dir==PJMEDIA_DIR_DECODING ?
                        stream->dec : stream->enc,
                       "changed");

        pjmedia_event_publish(NULL, port, &stream->fmt_event,
                              PJMEDIA_EVENT_PUBLISH_POST_EVENT);

        stream->fmt_event.type = PJMEDIA_EVENT_NONE;
    }

    if (stream->miss_keyframe_event.type != PJMEDIA_EVENT_NONE) {
        pjmedia_event_publish(NULL, port, &stream->miss_keyframe_event,
                              PJMEDIA_EVENT_PUBLISH_POST_EVENT);
        stream->miss_keyframe_event.type = PJMEDIA_EVENT_NONE;
    }

    pj_grp_lock_acquire( stream->grp_lock );

    if (stream->dec_frame.size == 0) {
        /* Don't have frame in buffer, try to decode one */
        if (decode_frame(stream, frame) != PJ_SUCCESS) {
            frame->type = PJMEDIA_FRAME_TYPE_NONE;
            frame->size = 0;
        }
    } else {
        if (frame->size < stream->dec_frame.size) {
            PJ_LOG(4,(stream->dec->port.info.name.ptr,
                      "Error: not enough buffer for decoded frame "
                      "(supplied=%d, required=%d)",
                      (int)frame->size, (int)stream->dec_frame.size));
            frame->type = PJMEDIA_FRAME_TYPE_NONE;
            frame->size = 0;
        } else {
            frame->type = stream->dec_frame.type;
            frame->timestamp = stream->dec_frame.timestamp;
            frame->size = stream->dec_frame.size;
            pj_memcpy(frame->buf, stream->dec_frame.buf, frame->size);
        }

        stream->dec_frame.size = 0;
    }

    pj_grp_lock_release( stream->grp_lock );

    return PJ_SUCCESS;
}


显示stream的port如何与解码输出的port关联的呢?

呱牛笔记


pjmedia_vid_port_create方法中,有如下代码:

    vp_param.active = PJ_FALSE;
    
    pjmedia_vid_port_create中:
    vp->role = prm->active ? ROLE_ACTIVE : ROLE_PASSIVE;

PJ_DEF(pj_status_t) pjmedia_vid_port_create( pj_pool_t *pool,
                                             const pjmedia_vid_port_param *prm,
                                             pjmedia_vid_port **p_vid_port)
{
        ///......

    } else if (vp->role==ROLE_PASSIVE) {
        vid_pasv_port *pp;

        /* Always need to create media port for passive role */
        vp->pasv_port = pp = PJ_POOL_ZALLOC_T(pool, vid_pasv_port);
        pp->vp = vp;
        if (prm->vidparam.dir & PJMEDIA_DIR_CAPTURE)
            pp->base.get_frame = &vid_pasv_port_get_frame;
        if (prm->vidparam.dir & PJMEDIA_DIR_RENDER)
            pp->base.put_frame = &vid_pasv_port_put_frame;
        pp->base.on_destroy = &vid_pasv_port_on_destroy;
        pjmedia_port_info_init2(&pp->base.info, &vp->dev_name,
                                PJMEDIA_SIG_VID_PORT,
                                prm->vidparam.dir, &prm->vidparam.fmt);

        need_frame_buf = PJ_TRUE;
    }

    
    
    //pjsua_vid.c
static pj_status_t create_vid_win(pjsua_vid_win_type type,
                                  const pjmedia_format *fmt,
                                  pjmedia_vid_dev_index rend_id,
                                  pjmedia_vid_dev_index cap_id,
                                  pj_bool_t show,
                                  unsigned wnd_flags,
                                  const pjmedia_vid_dev_hwnd *wnd,
                                  pjsua_vid_win_id *id)


    /* Create renderer video port, only if it's not a native preview */
    if (!w->is_native) {
        status = pjmedia_vid_dev_default_param(w->pool, rend_id,
                                               &vp_param.vidparam);
        if (status != PJ_SUCCESS)
            goto on_error;

        vp_param.active = PJ_FALSE;
        vp_param.vidparam.dir = PJMEDIA_DIR_RENDER;
        vp_param.vidparam.fmt = *fmt;
        vp_param.vidparam.disp_size = fmt->det.vid.size;
        vp_param.vidparam.flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_HIDE;
        vp_param.vidparam.window_hide = !show;
        vp_param.vidparam.flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW_FLAGS;
        vp_param.vidparam.window_flags = wnd_flags;
        if (wnd) {
            vp_param.vidparam.flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW;
            vp_param.vidparam.window = *wnd;
        }

        status = pjmedia_vid_port_create(w->pool, &vp_param, &w->vp_rend);
        if (status != PJ_SUCCESS)
            goto on_error;

        /* Register renderer to the video conf */
        status = pjsua_vid_conf_add_port(
                                w->pool,
                                pjmedia_vid_port_get_passive_port(w->vp_rend),
                                NULL, &w->rend_slot);
        if (status != PJ_SUCCESS)
            goto on_error;

        /* For preview window, connect capturer & renderer (via conf) */
        if (w->type == PJSUA_WND_TYPE_PREVIEW && show) {
            status = pjsua_vid_conf_connect(w->cap_slot, w->rend_slot, NULL);
            if (status != PJ_SUCCESS)
                goto on_error;
        }

        PJ_LOG(4,(THIS_FILE,
                  "%s window id %d created for cap_dev=%d rend_dev=%d",
                  pjsua_vid_win_type_name(type), wid, cap_id, rend_id));
    } else {
        PJ_LOG(4,(THIS_FILE,
                  "Preview window id %d created for cap_dev %d, "
                  "using built-in preview!",
                  wid, cap_id));
    }
}    
    
static pj_status_t vid_pasv_port_put_frame(struct pjmedia_port *this_port,
                                           pjmedia_frame *frame)
{
    struct vid_pasv_port *vpp = (struct vid_pasv_port*)this_port;
    pjmedia_vid_port *vp = vpp->vp;

    if (vp->pasv_port->is_destroying)
        return PJ_EGONE;

    handle_format_change(vp);

    if (vp->stream_role==ROLE_PASSIVE) {
        /* We are passive and the stream is passive.
         * The encoding counterpart is in vid_pasv_port_get_frame().
         */
        pj_status_t status;
        pjmedia_frame frame_;

        if (frame->size != vp->src_size) {
            if (frame->size > 0) {
                PJ_LOG(4,(THIS_FILE, "Unexpected frame size %lu, expected %lu",
                                     (unsigned long)frame->size,
                                     (unsigned long)vp->src_size));
            }

            pj_memcpy(&frame_, frame, sizeof(pjmedia_frame));
            frame_.buf = NULL;
            frame_.size = 0;

            /* Send heart beat for updating timestamp or keep-alive. */
            return pjmedia_vid_dev_stream_put_frame(vp->strm, &frame_);
        }
        
        pj_bzero(&frame_, sizeof(frame_));
        status = convert_frame(vp, frame, &frame_);
        if (status != PJ_SUCCESS)
            return status;

        //这里就输出到显示设备的put_frame了
        return pjmedia_vid_dev_stream_put_frame(vp->strm, (vp->conv.conv?
                                                           &frame_: frame));
    } else {
        /* We are passive while the stream is active so we just store the
         * frame in the buffer.
         * The encoding counterpart is located in vidstream_cap_cb()
         */
        if (frame->size == vp->src_size)
            copy_frame_to_buffer(vp, frame);
    }

    return PJ_SUCCESS;
}


基本上对端视频的解码然后显示的流程就梳理清楚了,要实现一个显示对端摄像头视频的功能就有了基本的思路了。

1、参考sdl_dev.c 实现一个显示的dev,然后注册到factory。

2、解码显示适配。


---------补充分析---

显示窗口:

    pjmedia_vid_port*vp_cap;/**< Capture vidport.*/

    pjmedia_vid_port *vp_rend; /**< Renderer vidport */ 

typedef struct pjsua_vid_win
{
    pjsua_vid_win_type		 type;		/**< Type.		*/
    pj_pool_t			*pool;		/**< Own pool.		*/
    unsigned	 		 ref_cnt;	/**< Reference counter.	*/
    pjmedia_vid_port		*vp_cap;	/**< Capture vidport.	*/
    pjmedia_vid_port		*vp_rend;	/**< Renderer vidport	*/
    pjsua_conf_port_id		 cap_slot;	/**< Capturer conf slot */
    pjsua_conf_port_id		 rend_slot;	/**< Renderer conf slot */
    pjmedia_vid_dev_index	 preview_cap_id;/**< Capture dev id	*/
    pj_bool_t			 preview_running;/**< Preview is started*/
    pj_bool_t			 is_native; 	/**< Preview is by dev  */
} pjsua_vid_win;
static pj_status_t create_vid_win(pjsua_vid_win_type type,
				  const pjmedia_format *fmt,
				  pjmedia_vid_dev_index rend_id,
				  pjmedia_vid_dev_index cap_id,
				  pj_bool_t show,
                                  unsigned wnd_flags,
                                  const pjmedia_vid_dev_hwnd *wnd,
				  pjsua_vid_win_id *id)
{
	    if (w->is_native) {
		strm = pjmedia_vid_port_get_stream(w->vp_cap);
	    } else {
		strm = pjmedia_vid_port_get_stream(w->vp_rend);
	    }
	    pj_assert(strm);
	    
}


注册:

选择哪个dev根据什么来决定呢,注册先后顺序。




    pj_strdup2_with_null(pool, &vp->dev_name, di.name);

    vp->stream_role = di.has_callback ? ROLE_ACTIVE : ROLE_PASSIVE;


/* API: Open video stream object using the specified parameters. */
PJ_DEF(pj_status_t) pjmedia_vid_dev_stream_create(
                                        pjmedia_vid_dev_param *prm,
                                        const pjmedia_vid_dev_cb *cb,
                                        void *user_data,
                                        pjmedia_vid_dev_stream **p_vid_strm)
{
    /* Normalize rend_id */
    if (prm->dir & PJMEDIA_DIR_RENDER) {
        unsigned index;

        if (prm->rend_id < 0)
            prm->rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;

        status = lookup_dev(prm->rend_id, &rend_f, &index);
        if (status != PJ_SUCCESS)
            return status;

        prm->rend_id = index;
        f = rend_f;
    }
/* Internal: lookup device id */
static pj_status_t lookup_dev(pjmedia_vid_dev_index id,
                              pjmedia_vid_dev_factory **p_f,
                              unsigned *p_local_index)
{
    int f_id, index;

    if (id < 0) {
        unsigned i;

        if (id <= PJMEDIA_VID_INVALID_DEV)
            return PJMEDIA_EVID_INVDEV;

        for (i=0; i<vid_subsys.drv_cnt; ++i) {
            pjmedia_vid_driver *drv = &vid_subsys.drv[i];
            if (id==PJMEDIA_VID_DEFAULT_CAPTURE_DEV && 
                drv->cap_dev_idx >= 0) 
            {
                id = drv->cap_dev_idx;
                make_global_index(i, &id);
                break;
            } else if (id==PJMEDIA_VID_DEFAULT_RENDER_DEV && 
                drv->rend_dev_idx >= 0) 
            {
                id = drv->rend_dev_idx;
                make_global_index(i, &id);
                break;
            }
        }

        if (id < 0) {
            return PJMEDIA_EVID_NODEFDEV;
        }
    }

    f_id = GET_FID(vid_subsys.dev_list[id]);
    index = GET_INDEX(vid_subsys.dev_list[id]);

    if (f_id < 0 || f_id >= (int)vid_subsys.drv_cnt)
        return PJMEDIA_EVID_INVDEV;

    if (index < 0 || index >= (int)vid_subsys.drv[f_id].dev_cnt)
        return PJMEDIA_EVID_INVDEV;

    *p_f = vid_subsys.drv[f_id].f;
    *p_local_index = (unsigned)index;

    return PJ_SUCCESS;

}



lvgl_dev.c 参考

/*
 * Copyright (C) 2008-2011 Teluu Inc. (http://www.teluu.com)
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 */
#include <pjmedia-videodev/videodev_imp.h>
#include <pjmedia/event.h>
#include <pj/assert.h>
#include <pj/log.h>
#include <pj/os.h>

#if defined(PJMEDIA_HAS_VIDEO) && PJMEDIA_HAS_VIDEO != 0 && \
    defined(PJMEDIA_VIDEO_DEV_HAS_LVGL) && PJMEDIA_VIDEO_DEV_HAS_LVGL != 0
  

#if defined(PJ_DARWINOS) && PJ_DARWINOS!=0
#   include "TargetConditionals.h"
#   include <Foundation/Foundation.h>
#endif

#define THIS_FILE               "lvlg_dev.c"
#define DEFAULT_CLOCK_RATE      90000
#define DEFAULT_WIDTH           640
#define DEFAULT_HEIGHT          480
#define DEFAULT_FPS             25

typedef struct lvgl_fmt_info
{
    pjmedia_format_id   fmt_id;
    pj_uint32_t              sdl_format;
    pj_uint32_t              Rmask;
    pj_uint32_t              Gmask;
    pj_uint32_t              Bmask;
    pj_uint32_t              Amask;
} lvgl_fmt_info;

typedef struct LVGL_Rect
{
    int x, y;
    int w, h;
} LVGL_Rect;

static lvgl_fmt_info lvgl_fmts[] =
{
#if PJ_IS_BIG_ENDIAN
    {PJMEDIA_FORMAT_RGBA,  (pj_uint32_t)SDL_PIXELFORMAT_RGBA8888,
     0xFF000000, 0xFF0000, 0xFF00, 0xFF} ,
    {PJMEDIA_FORMAT_RGB24, (pj_uint32_t)SDL_PIXELFORMAT_RGB24,
     0xFF0000, 0xFF00, 0xFF, 0} ,
    {PJMEDIA_FORMAT_BGRA,  (pj_uint32_t)SDL_PIXELFORMAT_BGRA8888,
     0xFF00, 0xFF0000, 0xFF000000, 0xFF} ,
#else /* PJ_IS_BIG_ENDIAN */
    {PJMEDIA_FORMAT_RGBA,  (pj_uint32_t)0,
     0xFF, 0xFF00, 0xFF0000, 0xFF000000} ,
    {PJMEDIA_FORMAT_RGB24, (pj_uint32_t)1,
     0xFF, 0xFF00, 0xFF0000, 0} ,
    {PJMEDIA_FORMAT_BGRA,  (pj_uint32_t)2,
     0xFF0000, 0xFF00, 0xFF, 0xFF000000} ,
#endif /* PJ_IS_BIG_ENDIAN */

    {PJMEDIA_FORMAT_DIB , (pj_uint32_t)3,
     0xFF0000, 0xFF00, 0xFF, 0} ,

    {PJMEDIA_FORMAT_YUY2, 4, 0, 0, 0, 0} ,
    {PJMEDIA_FORMAT_UYVY, 5, 0, 0, 0, 0} ,
    {PJMEDIA_FORMAT_YVYU, 6, 0, 0, 0, 0} ,
    {PJMEDIA_FORMAT_I420, 7, 0, 0, 0, 0} ,
    {PJMEDIA_FORMAT_YV12, 8, 0, 0, 0, 0} ,
    {PJMEDIA_FORMAT_I420JPEG, 9, 0, 0, 0, 0} ,
    {PJMEDIA_FORMAT_I422JPEG, 10, 0, 0, 0, 0}
};
typedef enum
{
    LVGL_WINDOW_FULLSCREEN = 0x00000001,         /**< fullscreen window */
    LVGL_WINDOW_OPENGL = 0x00000002,             /**< window usable with OpenGL context */
    LVGL_WINDOW_SHOWN = 0x00000004,              /**< window is visible */
    LVGL_WINDOW_HIDDEN = 0x00000008,             /**< window is not visible */
    LVGL_WINDOW_BORDERLESS = 0x00000010,         /**< no window decoration */
    LVGL_WINDOW_RESIZABLE = 0x00000020,          /**< window can be resized */
    LVGL_WINDOW_MINIMIZED = 0x00000040,          /**< window is minimized */
    LVGL_WINDOW_MAXIMIZED = 0x00000080,          /**< window is maximized */
    LVGL_WINDOW_MOUSE_GRABBED = 0x00000100,      /**< window has grabbed mouse input */
    LVGL_WINDOW_INPUT_FOCUS = 0x00000200,        /**< window has input focus */
    LVGL_WINDOW_MOUSE_FOCUS = 0x00000400,        /**< window has mouse focus */
    LVGL_WINDOW_FULLSCREEN_DESKTOP = (LVGL_WINDOW_FULLSCREEN | 0x00001000),
    LVGL_WINDOW_FOREIGN = 0x00000800,            /**< window not created by SDL */
    LVGL_WINDOW_ALLOW_HIGHDPI = 0x00002000,      /**< window should be created in high-DPI mode if supported.
                                                     On macOS NSHighResolutionCapable must be set true in the
                                                     application's Info.plist for this to have any effect. */
                                                     LVGL_WINDOW_MOUSE_CAPTURE = 0x00004000,   /**< window has mouse captured (unrelated to MOUSE_GRABBED) */
                                                     LVGL_WINDOW_ALWAYS_ON_TOP = 0x00008000,   /**< window should always be above others */
                                                     LVGL_WINDOW_SKIP_TASKBAR = 0x00010000,   /**< window should not be added to the taskbar */
                                                     LVGL_WINDOW_UTILITY = 0x00020000,   /**< window should be treated as a utility window */
                                                     LVGL_WINDOW_TOOLTIP = 0x00040000,   /**< window should be treated as a tooltip */
                                                     LVGL_WINDOW_POPUP_MENU = 0x00080000,   /**< window should be treated as a popup menu */
                                                     LVGL_WINDOW_KEYBOARD_GRABBED = 0x00100000,   /**< window has grabbed keyboard input */
                                                     LVGL_WINDOW_VULKAN = 0x10000000,   /**< window usable for Vulkan surface */
                                                     LVGL_WINDOW_METAL = 0x20000000,   /**< window usable for Metal view */

                                                     LVGL_WINDOW_INPUT_GRABBED = LVGL_WINDOW_MOUSE_GRABBED /**< equivalent to LVGL_WINDOW_MOUSE_GRABBED for compatibility */
} SDL_WindowFlags;
/* sdl_ device info */
struct lvgl_dev_info
{
    pjmedia_vid_dev_info         info;
};

/* Linked list of streams */
struct stream_list
{
    PJ_DECL_LIST_MEMBER(struct stream_list);
    struct lvgl_stream   *stream;
};

#define INITIAL_MAX_JOBS 64
#define JOB_QUEUE_INC_FACTOR 2

typedef pj_status_t (*job_func_ptr)(void *data);

typedef struct job {
    job_func_ptr    func;
    void           *data;
    unsigned        flags;
    pj_status_t     retval;
} job;

#if defined(PJ_DARWINOS) && PJ_DARWINOS!=0
@interface JQDelegate: NSObject
{
    @public
    job *pjob;
}

- (void)run_job;
@end

@implementation JQDelegate
- (void)run_job
{
    pjob->retval = (*pjob->func)(pjob->data);
}
@end
#endif /* PJ_DARWINOS */

typedef struct job_queue {
    pj_pool_t      *pool;
    job           **jobs;
    pj_sem_t      **job_sem;
    pj_sem_t      **old_sem;
    pj_mutex_t     *mutex;
    pj_thread_t    *thread;
    pj_sem_t       *sem;

    unsigned        size;
    unsigned        head, tail;
    pj_bool_t       is_full;
    pj_bool_t       is_quitting;
} job_queue;

/* lvgl_factory */
struct lvgl_factory
{
    pjmedia_vid_dev_factory      base;
    pj_pool_t                   *pool;
    pj_pool_factory             *pf;

    unsigned                     dev_count;
    struct lvgl_dev_info         *dev_info;
    job_queue                   *jq;

    pj_thread_t                 *lvgl_thread;        /**< SDL thread.        */
    pj_sem_t                    *sem;
    pj_mutex_t                  *mutex;
    struct stream_list           streams;
    pj_bool_t                    is_quitting;
    pj_thread_desc               thread_desc;
    pj_thread_t                 *ev_thread;
};

/* Video stream. */
struct lvgl_stream
{
    pjmedia_vid_dev_stream       base;              /**< Base stream        */
    pjmedia_vid_dev_param        param;             /**< Settings           */
    pj_pool_t                   *pool;              /**< Memory pool.       */

    pjmedia_vid_dev_cb           vid_cb;            /**< Stream callback.   */
    void                        *user_data;         /**< Application data.  */

    unsigned                     stream_id;

    struct lvgl_factory          *sf;
    const pjmedia_frame         *frame;
    pj_bool_t                    is_running;
    pj_timestamp                 last_ts;
    struct stream_list           list_entry;

    void* window;//实际显示视频的窗口对象
    int                          pitch;             /**< Pitch value.       */
    LVGL_Rect                     rect;              /**< Frame rectangle.   */
    LVGL_Rect                     dstrect;           /**< Display rectangle. */

    pjmedia_video_apply_fmt_param vafp;
};

/* Prototypes */
static pj_status_t lvgl_factory_init(pjmedia_vid_dev_factory *f);
static pj_status_t lvgl_factory_destroy(pjmedia_vid_dev_factory *f);
static pj_status_t lvgl_factory_refresh(pjmedia_vid_dev_factory *f);
static unsigned    lvgl_factory_get_dev_count(pjmedia_vid_dev_factory *f);
static pj_status_t lvgl_factory_get_dev_info(pjmedia_vid_dev_factory *f,
                                            unsigned index,
                                            pjmedia_vid_dev_info *info);
static pj_status_t lvgl_factory_default_param(pj_pool_t *pool,
                                             pjmedia_vid_dev_factory *f,
                                             unsigned index,
                                             pjmedia_vid_dev_param *param);
static pj_status_t lvgl_factory_create_stream(
                                        pjmedia_vid_dev_factory *f,
                                        pjmedia_vid_dev_param *param,
                                        const pjmedia_vid_dev_cb *cb,
                                        void *user_data,
                                        pjmedia_vid_dev_stream **p_vid_strm);

static pj_status_t lvgl_stream_get_param(pjmedia_vid_dev_stream *strm,
                                        pjmedia_vid_dev_param *param);
static pj_status_t lvgl_stream_get_cap(pjmedia_vid_dev_stream *strm,
                                      pjmedia_vid_dev_cap cap,
                                      void *value);
static pj_status_t lvgl_stream_set_cap(pjmedia_vid_dev_stream *strm,
                                      pjmedia_vid_dev_cap cap,
                                      const void *value);
static pj_status_t lvgl_stream_put_frame(pjmedia_vid_dev_stream *strm,
                                        const pjmedia_frame *frame);
static pj_status_t lvgl_stream_start(pjmedia_vid_dev_stream *strm);
static pj_status_t lvgl_stream_stop(pjmedia_vid_dev_stream *strm);
static pj_status_t lvgl_stream_destroy(pjmedia_vid_dev_stream *strm);

static pj_status_t resize_disp(struct lvgl_stream *strm,
                               pjmedia_rect_size *new_disp_size);
static pj_status_t lvgl_destroy_all(void *data);

/* Job queue prototypes */
static pj_status_t job_queue_create(pj_pool_t *pool, job_queue **pjq);
static pj_status_t job_queue_post_job(job_queue *jq, job_func_ptr func,
                                      void *data, unsigned flags,
                                      pj_status_t *retval);
static pj_status_t job_queue_destroy(job_queue *jq);

/* Operations */
static pjmedia_vid_dev_factory_op factory_op =
{
    &lvgl_factory_init,
    &lvgl_factory_destroy,
    &lvgl_factory_get_dev_count,
    &lvgl_factory_get_dev_info,
    &lvgl_factory_default_param,
    &lvgl_factory_create_stream,
    &lvgl_factory_refresh
};

static pjmedia_vid_dev_stream_op stream_op =
{
    &lvgl_stream_get_param,
    &lvgl_stream_get_cap,
    &lvgl_stream_set_cap,
    &lvgl_stream_start,
    NULL,
    &lvgl_stream_put_frame,
    &lvgl_stream_stop,
    &lvgl_stream_destroy
};

/*
 * Util
 */
static void sdl_log_err(const char *op)
{
    PJ_LOG(1,(THIS_FILE, "%s error: %s", op, SDL_GetError()));
}

/****************************************************************************
 * Factory operations
 */
/*
 * Init sdl_ video driver.
 */
pjmedia_vid_dev_factory* pjmedia_lvgl_factory(pj_pool_factory *pf)
{
    struct lvgl_factory *f;
    pj_pool_t *pool;

    pool = pj_pool_create(pf, "lvgl_dev video", 4000, 4000, NULL);
    f = PJ_POOL_ZALLOC_T(pool, struct lvgl_factory);
    f->pf = pf;
    f->pool = pool;
    f->base.op = &factory_op;

    return &f->base;
}

static pj_status_t lvgl_init(void * data)
{
    PJ_UNUSED_ARG(data); 

    return PJ_SUCCESS;
}

static struct lvgl_stream* find_stream(struct lvgl_factory *sf,
                                      pj_uint32_t windowID,
                                      pjmedia_event *pevent)
{
    struct stream_list *it, *itBegin;
    struct lvgl_stream *strm = NULL;

    itBegin = &sf->streams;
    for (it = itBegin->next; it != itBegin; it = it->next) {
        if (it->stream->stream_id == windowID)
        {
            strm = it->stream;
            break;
        }
    }
 
    if (strm)
        pjmedia_event_init(pevent, PJMEDIA_EVENT_NONE, &strm->last_ts,
                           strm);

    return strm;
}

static pj_status_t handle_event(void *data)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)data;

    if (!pj_thread_is_registered())
        pj_thread_register("sdl_ev", sf->thread_desc, &sf->ev_thread);
     

    return PJ_SUCCESS;
}

static int lvgl_ev_thread(void *data)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)data;

    while(1) {
        pj_status_t status;

        pj_mutex_lock(sf->mutex);
        if (pj_list_empty(&sf->streams)) {
            pj_mutex_unlock(sf->mutex);
            /* Wait until there is any stream. */
            pj_sem_wait(sf->sem);
        } else
            pj_mutex_unlock(sf->mutex);

        if (sf->is_quitting)
            break;

        job_queue_post_job(sf->jq, handle_event, sf, 0, &status);

        pj_thread_sleep(50);
    }

    return 0;
}

static pj_status_t lvgl_quit(void *data)
{
    PJ_UNUSED_ARG(data); 

    return PJ_SUCCESS;
}

/* API: init factory */
static pj_status_t lvgl_factory_init(pjmedia_vid_dev_factory *f)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)f;
    struct lvgl_dev_info *ddi;
    unsigned i, j;
    pj_status_t status; 
    pj_list_init(&sf->streams);

    status = job_queue_create(sf->pool, &sf->jq);
    if (status != PJ_SUCCESS)
        return PJMEDIA_EVID_INIT;

    job_queue_post_job(sf->jq, lvgl_init, NULL, 0, &status);
    if (status != PJ_SUCCESS)
        return status;

    status = pj_mutex_create_recursive(sf->pool, "lvgl_factory",
                                       &sf->mutex);
    if (status != PJ_SUCCESS)
        return status;

    status = pj_sem_create(sf->pool, NULL, 0, 1, &sf->sem);
    if (status != PJ_SUCCESS)
        return status;
#if 0
    /* Create event handler thread. */
    status = pj_thread_create(sf->pool, "lvgl_thread", lvgl_ev_thread,
                              sf, 0, 0, &sf->lvgl_thread);
    if (status != PJ_SUCCESS)
        return status;
#endif//

    sf->dev_count = 1;

    sf->dev_info = (struct lvgl_dev_info*)
                   pj_pool_calloc(sf->pool, sf->dev_count,
                                  sizeof(struct lvgl_dev_info));

    ddi = &sf->dev_info[0];
    pj_bzero(ddi, sizeof(*ddi));
    pj_ansi_strxcpy(ddi->info.name, "lvgl_dev renderer", 
                    sizeof(ddi->info.name));
    ddi->info.fmt_cnt = PJ_ARRAY_SIZE(lvgl_fmts);

#if PJMEDIA_VIDEO_DEV_SDL_HAS_OPENGL
    ddi = &sf->dev_info[OPENGL_DEV_IDX];
    pj_bzero(ddi, sizeof(*ddi));
    pj_ansi_strxcpy(ddi->info.name, "LVGL openGL renderer", 
                    sizeof(ddi->info.name));
    ddi->info.fmt_cnt = 1;
#endif /* PJMEDIA_VIDEO_DEV_SDL_HAS_OPENGL */

    for (i = 0; i < sf->dev_count; i++) {
        ddi = &sf->dev_info[i];
        pj_ansi_strxcpy(ddi->info.driver, "LVGL-dev", 
                        sizeof(ddi->info.driver));
        ddi->info.dir = PJMEDIA_DIR_RENDER;
        ddi->info.has_callback = PJ_FALSE;
        ddi->info.caps = PJMEDIA_VID_DEV_CAP_FORMAT |
                         PJMEDIA_VID_DEV_CAP_OUTPUT_RESIZE;
        ddi->info.caps |= PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW;
        ddi->info.caps |= PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW_FLAGS;

        for (j = 0; j < ddi->info.fmt_cnt; j++) {
            pjmedia_format *fmt = &ddi->info.fmt[j];
            pjmedia_format_init_video(fmt, lvgl_fmts[j].fmt_id,
                                      DEFAULT_WIDTH, DEFAULT_HEIGHT,
                                      DEFAULT_FPS, 1);
        }
    } 
    PJ_LOG(4, (THIS_FILE, "LVGL %d.%d initialized",
                          0, 1));

    return PJ_SUCCESS;
}

/* API: destroy factory */
static pj_status_t lvgl_factory_destroy(pjmedia_vid_dev_factory *f)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)f;
    pj_pool_t *pool = sf->pool;
    pj_status_t status;

    pj_assert(pj_list_empty(&sf->streams));

    sf->is_quitting = PJ_TRUE;
    if (sf->lvgl_thread) {
        pj_sem_post(sf->sem); 
        pj_thread_join(sf->lvgl_thread);
        pj_thread_destroy(sf->lvgl_thread);
    }

    if (sf->mutex) {
        pj_mutex_destroy(sf->mutex);
        sf->mutex = NULL;
    }

    if (sf->sem) {
        pj_sem_destroy(sf->sem);
        sf->sem = NULL;
    }

    job_queue_post_job(sf->jq, lvgl_quit, NULL, 0, &status);
    job_queue_destroy(sf->jq);

    sf->pool = NULL;
    pj_pool_release(pool);

    return PJ_SUCCESS;
}

/* API: refresh the list of devices */
static pj_status_t lvgl_factory_refresh(pjmedia_vid_dev_factory *f)
{
    PJ_UNUSED_ARG(f);
    return PJ_SUCCESS;
}

/* API: get number of devices */
static unsigned lvgl_factory_get_dev_count(pjmedia_vid_dev_factory *f)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)f;
    return sf->dev_count;
}

/* API: get device info */
static pj_status_t lvgl_factory_get_dev_info(pjmedia_vid_dev_factory *f,
                                            unsigned index,
                                            pjmedia_vid_dev_info *info)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)f;

    PJ_ASSERT_RETURN(index < sf->dev_count, PJMEDIA_EVID_INVDEV);

    pj_memcpy(info, &sf->dev_info[index].info, sizeof(*info));

    return PJ_SUCCESS;
}

/* API: create default device parameter */
static pj_status_t lvgl_factory_default_param(pj_pool_t *pool,
                                             pjmedia_vid_dev_factory *f,
                                             unsigned index,
                                             pjmedia_vid_dev_param *param)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)f;
    struct lvgl_dev_info *di = &sf->dev_info[index];

    PJ_ASSERT_RETURN(index < sf->dev_count, PJMEDIA_EVID_INVDEV);
    
    PJ_UNUSED_ARG(pool);

    pj_bzero(param, sizeof(*param));
    param->dir = PJMEDIA_DIR_RENDER;
    param->rend_id = index;
    param->cap_id = PJMEDIA_VID_INVALID_DEV;

    /* Set the device capabilities here */
    param->flags = PJMEDIA_VID_DEV_CAP_FORMAT;
    param->fmt.type = PJMEDIA_TYPE_VIDEO;
    param->clock_rate = DEFAULT_CLOCK_RATE;
    pj_memcpy(&param->fmt, &di->info.fmt[0], sizeof(param->fmt));

    return PJ_SUCCESS;
}

static lvgl_fmt_info* get_sdl_format_info(pjmedia_format_id id)
{
    unsigned i;

    for (i = 0; i < PJ_ARRAY_SIZE(lvgl_fmts); i++) {
        if (lvgl_fmts[i].fmt_id == id)
            return &lvgl_fmts[i];
    }

    return NULL;
}

static pj_status_t lvgl_destroy(void *data)
{
    struct lvgl_stream *strm = (struct lvgl_stream *)data;
     
    strm->window = NULL;

    return PJ_SUCCESS;
}

static pj_status_t lvgl_destroy_all(void *data)
{
    struct lvgl_stream *strm = (struct lvgl_stream *)data;  

    lvgl_destroy(data);

    return PJ_SUCCESS;
}

static pj_status_t lvgl_create_window(struct lvgl_stream *strm, 
                                     pj_bool_t use_app_win,
                                     pj_uint32_t sdl_format,
                                     pjmedia_vid_dev_hwnd *hwnd)
{
    //((pjmedia_coord *)pval)->x,        &((pjmedia_coord*)pval)->y
    if (!strm->window) { 

    } 

    return PJ_SUCCESS;
}

static pj_status_t lvgl_create_rend(struct lvgl_stream * strm,
                                   pjmedia_format *fmt)
{
    lvgl_fmt_info *sdl_info;
    const pjmedia_video_format_info *vfi;
    pjmedia_video_format_detail *vfd;

    sdl_info = get_sdl_format_info(fmt->id);
    vfi = pjmedia_get_video_format_info(pjmedia_video_format_mgr_instance(),
                                        fmt->id);
    if (!vfi || !sdl_info)
        return PJMEDIA_EVID_BADFORMAT;

    strm->vafp.size = fmt->det.vid.size;
    strm->vafp.buffer = NULL;
    if (vfi->apply_fmt(vfi, &strm->vafp) != PJ_SUCCESS)
        return PJMEDIA_EVID_BADFORMAT;

    vfd = pjmedia_format_get_video_format_detail(fmt, PJ_TRUE);
    strm->rect.x = strm->rect.y = 0;
    //strm->rect.w = (Uint16)vfd->size.w;
    //strm->rect.h = (Uint16)vfd->size.h;
    if (strm->param.disp_size.w == 0)
        strm->param.disp_size.w = strm->rect.w;
    if (strm->param.disp_size.h == 0)
        strm->param.disp_size.h = strm->rect.h;
    strm->dstrect.x = strm->dstrect.y = 0;
    //strm->dstrect.w = (Uint16)strm->param.disp_size.w;
    //strm->dstrect.h = (Uint16)strm->param.disp_size.h;

    lvgl_destroy(strm);
     
    return lvgl_create_window(strm, 
                         (strm->param.flags & PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW),
                         sdl_info->sdl_format,
                         &strm->param.window);
}

static pj_status_t lvgl_create(void *data)
{
    struct lvgl_stream *strm = (struct lvgl_stream *)data;
    return lvgl_create_rend(strm, &strm->param.fmt);
}

static pj_status_t resize_disp(struct lvgl_stream *strm,
                               pjmedia_rect_size *new_disp_size)
{
    pj_memcpy(&strm->param.disp_size, new_disp_size,
              sizeof(strm->param.disp_size));
     
#if PJMEDIA_VIDEO_DEV_SDL_HAS_OPENGL
    else if (strm->param.rend_id == OPENGL_DEV_IDX) {
        lvgl_create_rend(strm, &strm->param.fmt);
    }
#endif /* PJMEDIA_VIDEO_DEV_SDL_HAS_OPENGL */

    return PJ_SUCCESS;
}

static pj_status_t change_format(struct lvgl_stream *strm,
                                 pjmedia_format *new_fmt)
{
    pj_status_t status;

    /* Recreate SDL renderer */
    status = lvgl_create_rend(strm, (new_fmt? new_fmt :
                                   &strm->param.fmt));
    if (status == PJ_SUCCESS && new_fmt)
        pjmedia_format_copy(&strm->param.fmt, new_fmt);

    return status;
}

static pj_status_t put_frame(void *data)
{
    struct lvgl_stream *stream = (struct lvgl_stream *)data;
    const pjmedia_frame *frame = stream->frame;

    //实际的视频绘制
    //todo

    return PJ_SUCCESS;
}

/* API: Put frame from stream */
static pj_status_t lvgl_stream_put_frame(pjmedia_vid_dev_stream *strm,
                                        const pjmedia_frame *frame)
{
    struct lvgl_stream *stream = (struct lvgl_stream*)strm;
    pj_status_t status;

    stream->last_ts.u64 = frame->timestamp.u64;

    /* Video conference just trying to send heart beat for updating timestamp
     * or keep-alive, this port doesn't need any, just ignore.
     */
    if (frame->size==0 || frame->buf==NULL)
        return PJ_SUCCESS;

    if (frame->size < stream->vafp.framebytes)
        return PJ_ETOOSMALL;

    if (!stream->is_running)
        return PJ_EINVALIDOP;

    stream->frame = frame;
    job_queue_post_job(stream->sf->jq, put_frame, strm, 0, &status);

    return status;
}

/* API: create stream */
static pj_status_t lvgl_factory_create_stream(
                                        pjmedia_vid_dev_factory *f,
                                        pjmedia_vid_dev_param *param,
                                        const pjmedia_vid_dev_cb *cb,
                                        void *user_data,
                                        pjmedia_vid_dev_stream **p_vid_strm)
{
    struct lvgl_factory *sf = (struct lvgl_factory*)f;
    pj_pool_t *pool;
    struct lvgl_stream *strm;
    pj_status_t status;

    PJ_ASSERT_RETURN(param->dir == PJMEDIA_DIR_RENDER, PJ_EINVAL);

    /* Create and Initialize stream descriptor */
    pool = pj_pool_create(sf->pf, "sdl-dev", 1000, 1000, NULL);
    PJ_ASSERT_RETURN(pool != NULL, PJ_ENOMEM);

    strm = PJ_POOL_ZALLOC_T(pool, struct lvgl_stream);
    pj_memcpy(&strm->param, param, sizeof(*param));
    strm->pool = pool;
    strm->sf = sf;
    pj_memcpy(&strm->vid_cb, cb, sizeof(*cb));
    pj_list_init(&strm->list_entry);
    strm->list_entry.stream = strm;
    strm->user_data = user_data;

    /* Create render stream here */
    job_queue_post_job(sf->jq, lvgl_create, strm, 0, &status);
    if (status != PJ_SUCCESS) {
        goto on_error;
    }
    pj_mutex_lock(strm->sf->mutex);
    if (pj_list_empty(&strm->sf->streams))
        pj_sem_post(strm->sf->sem);
    pj_list_insert_after(&strm->sf->streams, &strm->list_entry);
    pj_mutex_unlock(strm->sf->mutex);

    /* Done */
    strm->base.op = &stream_op;
    *p_vid_strm = &strm->base;

    return PJ_SUCCESS;

on_error:
    lvgl_stream_destroy(&strm->base);
    return status;
}

/* API: Get stream info. */
static pj_status_t lvgl_stream_get_param(pjmedia_vid_dev_stream *s,
                                        pjmedia_vid_dev_param *pi)
{
    struct lvgl_stream *strm = (struct lvgl_stream*)s;

    PJ_ASSERT_RETURN(strm && pi, PJ_EINVAL);

    pj_memcpy(pi, &strm->param, sizeof(*pi));

    if (lvgl_stream_get_cap(s, PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW,
                           &pi->window) == PJ_SUCCESS)
    {
        pi->flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW;
    }
    if (lvgl_stream_get_cap(s, PJMEDIA_VID_DEV_CAP_OUTPUT_POSITION,
                           &pi->window_pos) == PJ_SUCCESS)
    {
        pi->flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_POSITION;
    }
    if (lvgl_stream_get_cap(s, PJMEDIA_VID_DEV_CAP_OUTPUT_RESIZE,
                           &pi->disp_size) == PJ_SUCCESS)
    {
        pi->flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_RESIZE;
    }
    if (lvgl_stream_get_cap(s, PJMEDIA_VID_DEV_CAP_OUTPUT_HIDE,
                           &pi->window_hide) == PJ_SUCCESS)
    {
        pi->flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_HIDE;
    }
    if (lvgl_stream_get_cap(s, PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW_FLAGS,
                           &pi->window_flags) == PJ_SUCCESS)
    {
        pi->flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW_FLAGS;
    }
    if (lvgl_stream_get_cap(s, PJMEDIA_VID_DEV_CAP_OUTPUT_FULLSCREEN,
                           &pi->window_fullscreen) == PJ_SUCCESS)
    {
        pi->flags |= PJMEDIA_VID_DEV_CAP_OUTPUT_FULLSCREEN;
    }

    return PJ_SUCCESS;
}

struct strm_cap {
    struct lvgl_stream   *strm;
    pjmedia_vid_dev_cap  cap;
    union {
        void            *pval;
        const void      *cpval;
    } pval;
};

static pj_status_t get_cap(void *data)
{
    struct strm_cap *scap = (struct strm_cap *)data;
    struct lvgl_stream *strm = scap->strm;
    pjmedia_vid_dev_cap cap = scap->cap;
    void *pval = scap->pval.pval;

    if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW)
    { 
        return PJ_SUCCESS; 
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_POSITION) { 
        //((pjmedia_coord *)pval)->x,        &((pjmedia_coord*)pval)->y
        return PJ_SUCCESS;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_RESIZE) {
        //((pjmedia_coord *)pval)->x,        &((pjmedia_coord*)pval)->y
        return PJ_SUCCESS;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_HIDE) {
        //pj_uint32_t flag = SDL_GetWindowFlags(strm->window);
        //*((pj_bool_t *)pval) = (flag & LVGL_WINDOW_HIDDEN)? PJ_TRUE: PJ_FALSE;
        return PJ_SUCCESS;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW_FLAGS) {

        return PJ_SUCCESS;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_FULLSCREEN) {

        return PJ_SUCCESS;
    }

    return PJMEDIA_EVID_INVCAP;
}

/* API: get capability */
static pj_status_t lvgl_stream_get_cap(pjmedia_vid_dev_stream *s,
                                      pjmedia_vid_dev_cap cap,
                                      void *pval)
{
    struct lvgl_stream *strm = (struct lvgl_stream*)s;
    struct strm_cap scap;
    pj_status_t status;

    PJ_ASSERT_RETURN(s && pval, PJ_EINVAL);

    scap.strm = strm;
    scap.cap = cap;
    scap.pval.pval = pval;

    job_queue_post_job(strm->sf->jq, get_cap, &scap, 0, &status);

    return status;
}

static pj_status_t set_cap(void *data)
{
    struct strm_cap *scap = (struct strm_cap *)data;
    struct lvgl_stream *strm = scap->strm;
    pjmedia_vid_dev_cap cap = scap->cap;
    const void *pval = scap->pval.cpval;

    PJ_ASSERT_RETURN(data && strm, PJ_EINVAL);

    if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_POSITION) { 
        return PJ_SUCCESS;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_HIDE) { 
        return PJ_SUCCESS;
    } else if (cap == PJMEDIA_VID_DEV_CAP_FORMAT) {
        pj_status_t status;

        status = change_format(strm, (pjmedia_format *)pval);
        if (status != PJ_SUCCESS) {
            pj_status_t status_;
            
            /**
             * Failed to change the output format. Try to revert
             * to its original format.
             */
            status_ = change_format(strm, &strm->param.fmt);
            if (status_ != PJ_SUCCESS) {
                /**
                 * This means that we failed to revert to our
                 * original state!
                 */
                status = PJMEDIA_EVID_ERR;
            }
        }
        
        return status;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_RESIZE) {
        pjmedia_rect_size *new_size = (pjmedia_rect_size *)pval; 
        pj_status_t status = 0;
         
        return status;
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW) {
        pjmedia_vid_dev_hwnd *hwnd = (pjmedia_vid_dev_hwnd*)pval;
        pj_status_t status = PJ_SUCCESS;
        lvgl_fmt_info *sdl_info = get_sdl_format_info(strm->param.fmt.id);
        /* Re-init SDL */
        status = lvgl_destroy_all(strm);
        if (status != PJ_SUCCESS)
            return status;      

        status = lvgl_create_window(strm, PJ_TRUE, sdl_info->sdl_format, hwnd);
        PJ_PERROR(4, (THIS_FILE, status,
                      "Re-initializing SDL with native window %p",
                      hwnd->info.window));
        return status;  
    } else if (cap == PJMEDIA_VID_DEV_CAP_OUTPUT_FULLSCREEN) {
        pj_uint32_t flag;
        pjmedia_vid_dev_fullscreen_flag val =
                                    *(pjmedia_vid_dev_fullscreen_flag*)pval;
         
         

        return PJ_SUCCESS;
    }

    return PJMEDIA_EVID_INVCAP;
}

/* API: set capability */
static pj_status_t lvgl_stream_set_cap(pjmedia_vid_dev_stream *s,
                                      pjmedia_vid_dev_cap cap,
                                      const void *pval)
{
    struct lvgl_stream *strm = (struct lvgl_stream*)s;
    struct strm_cap scap;
    pj_status_t status;

    PJ_ASSERT_RETURN(s && pval, PJ_EINVAL);

    scap.strm = strm;
    scap.cap = cap;
    scap.pval.cpval = pval;

    job_queue_post_job(strm->sf->jq, set_cap, &scap, 0, &status);

    return status;
}

/* API: Start stream. */
static pj_status_t lvgl_stream_start(pjmedia_vid_dev_stream *strm)
{
    struct lvgl_stream *stream = (struct lvgl_stream*)strm;

    PJ_LOG(4, (THIS_FILE, "Starting sdl video stream"));

    stream->is_running = PJ_TRUE;
    stream->window = NULL;

    return PJ_SUCCESS;
}


/* API: Stop stream. */
static pj_status_t lvgl_stream_stop(pjmedia_vid_dev_stream *strm)
{
    struct lvgl_stream *stream = (struct lvgl_stream*)strm;

    PJ_LOG(4, (THIS_FILE, "Stopping sdl video stream"));

    stream->is_running = PJ_FALSE;

    return PJ_SUCCESS;
}


/* API: Destroy stream. */
static pj_status_t lvgl_stream_destroy(pjmedia_vid_dev_stream *strm)
{
    struct lvgl_stream *stream = (struct lvgl_stream*)strm;
    pj_status_t status;

    PJ_ASSERT_RETURN(stream != NULL, PJ_EINVAL);

    lvgl_stream_stop(strm);

    job_queue_post_job(stream->sf->jq, lvgl_destroy_all, strm, 0, &status);
    if (status != PJ_SUCCESS)
        return status;

    pj_mutex_lock(stream->sf->mutex);
    if (!pj_list_empty(&stream->list_entry))
        pj_list_erase(&stream->list_entry);
    pj_mutex_unlock(stream->sf->mutex);

    pj_pool_release(stream->pool);

    return PJ_SUCCESS;
}

/****************************************************************************
 * Job queue implementation
 */
#if PJ_DARWINOS==0
static int lvgl_job_thread(void * data)
{
    job_queue *jq = (job_queue *)data;

    while (1) {
        job *jb;

        /* Wait until there is a job. */
        pj_sem_wait(jq->sem);

        /* Make sure there is no pending jobs before we quit. */
        if (jq->is_quitting && jq->head == jq->tail && !jq->is_full)
            break;

        jb = jq->jobs[jq->head];
        jb->retval = (*jb->func)(jb->data);
        /* If job queue is full and we already finish all the pending
         * jobs, increase the size.
         */
        if (jq->is_full && ((jq->head + 1) % jq->size == jq->tail)) {
            unsigned i, head;
            pj_status_t status;

            if (jq->old_sem) {
                for (i = 0; i < jq->size / JOB_QUEUE_INC_FACTOR; i++) {
                    pj_sem_destroy(jq->old_sem[i]);
                }
            }
            jq->old_sem = jq->job_sem;

            /* Double the job queue size. */
            jq->size *= JOB_QUEUE_INC_FACTOR;
            pj_sem_destroy(jq->sem);
            status = pj_sem_create(jq->pool, "thread_sem", 0, jq->size + 1,
                                   &jq->sem);
            if (status != PJ_SUCCESS) {
                PJ_PERROR(3, (THIS_FILE, status,
                              "Failed growing SDL job queue size."));
                return 0;
            }
            jq->jobs = (job **)pj_pool_calloc(jq->pool, jq->size,
                                              sizeof(job *));
            jq->job_sem = (pj_sem_t **) pj_pool_calloc(jq->pool, jq->size,
                                                       sizeof(pj_sem_t *));
            for (i = 0; i < jq->size; i++) {
                status = pj_sem_create(jq->pool, "job_sem", 0, 1,
                                       &jq->job_sem[i]);
                if (status != PJ_SUCCESS) {
                    PJ_PERROR(3, (THIS_FILE, status,
                                  "Failed growing SDL job queue size."));
                    return 0;
                }
            }
            jq->is_full = PJ_FALSE;
            head = jq->head;
            jq->head = jq->tail = 0;
            pj_sem_post(jq->old_sem[head]);
        } else {
            pj_sem_post(jq->job_sem[jq->head]);
            jq->head = (jq->head + 1) % jq->size;
        }
    }

    return 0;
}
#endif

static pj_status_t job_queue_create(pj_pool_t *pool, job_queue **pjq)
{
    unsigned i;
    pj_status_t status;

    job_queue *jq = PJ_POOL_ZALLOC_T(pool, job_queue);
    jq->pool = pool;
    jq->size = INITIAL_MAX_JOBS;
    status = pj_sem_create(pool, "lvgl_thread_sem", 0, jq->size + 1, &jq->sem);
    if (status != PJ_SUCCESS)
        goto on_error;
    jq->jobs = (job **)pj_pool_calloc(pool, jq->size, sizeof(job *));
    jq->job_sem = (pj_sem_t **) pj_pool_calloc(pool, jq->size,
                                               sizeof(pj_sem_t *));
    for (i = 0; i < jq->size; i++) {
        status = pj_sem_create(pool, "lvgl_job_sem", 0, 1, &jq->job_sem[i]);
        if (status != PJ_SUCCESS)
            goto on_error;
    }

    status = pj_mutex_create_recursive(pool, "lvgl_job_mutex", &jq->mutex);
    if (status != PJ_SUCCESS)
        goto on_error;

#if defined(PJ_DARWINOS) && PJ_DARWINOS!=0
    PJ_UNUSED_ARG(status);
#else
    status = pj_thread_create(pool, "lvgl_job_th", lvgl_job_thread, jq, 0, 0,
                              &jq->thread);
    if (status != PJ_SUCCESS)
        goto on_error;
#endif /* PJ_DARWINOS */

    *pjq = jq;
    return PJ_SUCCESS;

on_error:
    job_queue_destroy(jq);
    return status;
}

static pj_status_t job_queue_post_job(job_queue *jq, job_func_ptr func,
                                      void *data, unsigned flags,
                                      pj_status_t *retval)
{
    job jb;
    int tail;

    if (jq->is_quitting) {
        jb.retval = PJ_EBUSY;
        goto on_return;
    }

    jb.func = func;
    jb.data = data;
    jb.flags = flags;
    jb.retval = PJ_SUCCESS;

#if defined(PJ_DARWINOS) && PJ_DARWINOS!=0
    PJ_UNUSED_ARG(tail);
    NSAutoreleasePool *apool = [[NSAutoreleasePool alloc]init];
    JQDelegate *jqd = [[JQDelegate alloc]init];
    jqd->pjob = &jb;
    [jqd performSelectorOnMainThread:@selector(run_job)
         withObject:nil waitUntilDone:YES];
    [jqd release];
    [apool release];
#else /* PJ_DARWINOS */
    pj_mutex_lock(jq->mutex);
    jq->jobs[jq->tail] = &jb;
    tail = jq->tail;
    jq->tail = (jq->tail + 1) % jq->size;
    if (jq->tail == jq->head) {
        jq->is_full = PJ_TRUE;
        PJ_LOG(4, (THIS_FILE, "lvgl-dev job queue is full, increasing "
                              "the queue size."));
        pj_mutex_unlock(jq->mutex);
        pj_sem_post(jq->sem);
        /* Wait until our posted job is completed. */
        pj_sem_wait(jq->job_sem[tail]);
    } else {
        pj_mutex_unlock(jq->mutex);
        pj_sem_post(jq->sem);
        /* Wait until our posted job is completed. */
        pj_sem_wait(jq->job_sem[tail]);
    }
#endif /* PJ_DARWINOS */

on_return:
    if (retval)
        *retval = jb.retval;

    return jb.retval;
}

static pj_status_t job_queue_destroy(job_queue *jq)
{
    unsigned i;

    jq->is_quitting = PJ_TRUE;

    if (jq->thread) {
        pj_sem_post(jq->sem);
        pj_thread_join(jq->thread);
        pj_thread_destroy(jq->thread);
    }

    if (jq->sem) {
        pj_sem_destroy(jq->sem);
        jq->sem = NULL;
    }
    for (i = 0; i < jq->size; i++) {
        if (jq->job_sem[i]) {
            pj_sem_destroy(jq->job_sem[i]);
            jq->job_sem[i] = NULL;
        }
    }
    if (jq->old_sem) {
        for (i = 0; i < jq->size / JOB_QUEUE_INC_FACTOR; i++) {
            if (jq->old_sem[i]) {
                pj_sem_destroy(jq->old_sem[i]);
                jq->old_sem[i] = NULL;
            }
        }
    }
    if (jq->mutex) {
        pj_mutex_destroy(jq->mutex);
        jq->mutex = NULL;
    }

    return PJ_SUCCESS;
} 

#endif  /* PJMEDIA_VIDEO_DEV_HAS_LVGL */


本文为呱牛笔记原创文章,转载无需和我联系,但请注明来自呱牛笔记 ,it3q.com

请先登录后发表评论
  • 最新评论
  • 总共0条评论