Development direction: SRS4 is on the way...

[winlinvip]

About the Development Direction of SRS

SRS3 is currently being enhanced for stability and is being released intensively. This means that SRS3 will not support major changes but will only focus on improving stability, indicating that SRS4 is coming.

SRS4's codename is Leo. Thanks to all the brothers and sisters who have been fighting together for over a decade. You can find the story about the codename Leo here.

This issue is to discuss what SRS4 should support. I understand that there is a strong demand for new protocols such as WebRTC, SRT, QUIC, and KCP. However, I have already decided that supporting new protocols will not be the focus. SRS4 will probably support some new protocols, but the main focus will be on usability and ease of use, in simple terms, it will be more stable and user-friendly.

Is SRS unstable? For an open-source project, it may meet the requirements. There are very few open-source projects with a core protocol coverage rate of over 95% and an overall coverage rate of around 50% (of course, stability has many more aspects that are not discussed here). However, the requirements for SRS are to directly provide services on industrial servers, requiring even higher stability. There are many areas that need careful handling, especially in cases of inconsistency, such as player timeouts, source cleaning, race conditions, and cooperation among multiple coroutines. Stability is a fundamental capability that cannot be compromised. The more investment, the more stability.

Is SRS difficult to use? Based on feedback and continuous observations, it should not be considered difficult to use. However, the requirements for SRS are to directly provide services on commercial servers, so usability needs to be further improved. This mainly involves Dockerization, integration with cloud services, HTTP protocol enhancements such as monitoring the online audience of HLS, and providing operational data solutions through JSON-structured logs. Currently, the HTTP part of SRS mainly refers to the implementation of HTTP 1.1 in Go. In the future, consideration will be given to porting more underlying capabilities from NGINX. For example, it may be possible to directly run NGINX modules on SRS, which gives us something to look forward to. See reference #1657.

In addition, security and licensing have always been the focal points of SRS. In terms of security, it mainly involves anti-leeching measures, enhanced restrictions such as rate limiting and client limitations, encryption and decryption, and code vulnerability detection. As for licensing, the only issue currently is replacing ST with at least one optional version that does not have licensing limitations.

So, dear reader, what key capabilities do you expect SRS4 to support? Please reply to this issue 😄 😄

Make sure to maintain the markdown structure.

TRANS_BY_GPT3

[tczf1128]

Can srs3 be used in production now?

TRANS_BY_GPT3

[winlinvip]

@smellyCat

Can SRS3 be used in a production environment now?

SRS3 is currently in alpha-4, and it will be available for production use once it reaches beta stage.

TRANS_BY_GPT3

[springjk]

The milestone plan has almost covered everything, and there are no new expected abilities to propose. If used for production, stability should be the most important indicator. I hope stability can be further improved. Additionally, I would like to give you a thumbs up for your efforts. This project is a great work. 👍

TRANS_BY_GPT3

[machh]

Don't you plan to support RTSP? For example: input RTSP and output RTMP/RTSP.

TRANS_BY_GPT3

[objnf-dev]

The milestone plan is already comprehensive, and everything I wanted to say is included in it.
However, if there is energy available, I suggest introducing support for protocols like WebRTC. After all, supporting a new protocol means meeting the demands of more platforms and commercial servers, and the usage range of SRS will also be broader.
But this should indeed be discussed after stability, CI/CD, and data monitoring.
Finally, I am extremely grateful to the author for developing and consistently maintaining such a fantastic project! Thumbs up! Keep up the good work, author!

TRANS_BY_GPT3

[kongling]

Don't you plan to support RTSP? For example, input RTSP and output RTMP/RTSP.

This is supported. Capture through RTSP, then use FFmpeg transcoding command to output RTMP. You can refer to the following configuration file:
https://github.com/ossrs/srs/blob/3.0release/trunk/conf/ingest.rtsp.conf

TRANS_BY_GPT3

[ciaoamigoschat]

Webrtc is essential for streaming via browser. Rtmp without flash doesn't work well via browser (ex. I can't publish streaming).
I believe webrtc is essential to create a server that can work in all scenarios.

[winlinvip]

Webrtc is essential for streaming via browser. Rtmp without flash doesn't work well via browser (ex. I can't publish streaming).
I believe webrtc is essential to create a server that can work in all scenarios.

@ciaoamigoschat Agreed. I think RTC is one of the next extreme important things for live streaming. And I am sure SRS will support WebRTC in future, mabye in SRS4 or SRS5, it depends on how many time I have for working on this project, it also depends on whether it's stable enough to enable us to add more big/huge features like this.

[ciaoamigoschat]

This is the right time to implement webrtc. Many sites are abandoning flash and looking for a viable alternative.
Giving this functionality is a great move to rapidly spread the product and to find valid supporters.

[dolpstar]

I wish we could support the conversion of WebRTC streams, so that we can play them directly on mobile devices. It would be more real-time compared to HLS.

TRANS_BY_GPT3

[winlinvip]

It would be great if we could support streaming via WebRTC, so we can play directly on mobile devices. It is more real-time compared to HLS.

WebRTC is mainly used natively on mobile devices, and it may not be readily available for use on mobile web pages. It is primarily stable on PCs, but it can also be integrated into native mobile applications.

• red5/ant claims to support WebRTC to RTMP conversion. After reviewing the code and the wiki, it seems that this feature is only available in the enterprise edition and not in the open-source version.

TRANS_BY_GPT3

[hy05190134]

QUIC should be supported

TRANS_BY_GPT3

[winlinvip]

QUIC should be able to support it
Dear, thank you for your attention. Could you please provide more details on why it needs support, what the use case is, and why it needs to be supported now instead of at a slower pace?

TRANS_BY_GPT3

[ciaoamigoschat]

You are losing sight of the problem.
With mobile (Native application) streaming (Publish, Play) can be done in rtmp.
With the browser for streaming (Publish, Play) the only solution is Webrtc.

[canglongxiang]

As a complete novice in the field of live streaming, video processing, and streaming media, my expectations for SRS are "plug-and-play" and "enterprise-level application stability".

I used SRS2 before and encountered many problems during compilation. It was only when I used an "unofficial" Docker image that I was able to partially solve the "plug-and-play" issue. Of course, now that SRS has provided an official Docker image, I immediately switched to the official one.

As for stability, as it was my first attempt to use a streaming media solution for real-time access, I encountered many issues during the process. However, as long as the infrastructure (such as the network) is sufficiently reliable, I have not observed any "unstable" situations, which exceeded my expectations. I am very grateful to SRS for this.

Lastly, regarding the application scenario, the cameras are distributed in independent network environments in 20 cities. They are connected to the public cloud network through a VPN and ffmpeg is used to pull the camera's RTSP stream according to policies, and then push it to the SRS server in the public cloud for distribution.

TRANS_BY_GPT3

[winlinvip]

As a complete layman in the fields of live streaming, video processing, and streaming media, my expectations for SRS are "plug and play" and "enterprise-level stability".

Previously, I used SRS2 and encountered many problems during compilation. It wasn't until I used an "unofficial" Docker image that I was able to partially solve the "plug and play" issue. Of course, now SRS has provided an official Docker image, and I immediately switched to the official image.

As for stability, this is my first attempt to use a streaming media solution for "real-time access" scenarios. I encountered many problems during this process, but as long as the infrastructure (such as the network) is guaranteed, I haven't observed any "instability" situations, which has exceeded my expectations. I am very grateful to SRS.

Finally, let me talk about the application scenario. Cameras distributed in 20 different cities are connected to the public cloud through VPN, and ffmpeg is used to pull the RTSP streams from the cameras and push them to the public cloud's SRS server for distribution, based on certain policies."

Thank you for your recognition of SRS and sharing the application scenario, which is important for everyone. Docker is actually a mature open-source solution. It was through continuous learning that I realized the difference between Docker and installation packages, and I will continue to improve myself to ultimately improve the quality of the SRS open-source project.

Open-source is a closed loop of sharing and improvement. Everyone's feedback, whether it's a problem or a thumbs up, a solution or a pull request, an idea or a suggestion, can influence the entire community. The whole open-source community is like a big melting pot, and each of our participation will make this flame burn even brighter. Welcome everyone to join in, learn from each other, and progress together~

TRANS_BY_GPT3

[charlestamz]

ST may be replaced with https://github.com/tencent/libco, which is with Apache License.
Easy of Use should be the first goal, which means an installer, a web UI and streaming pulling features.

[winlinvip]

@charlestamz I heard wechat libco before and I will do some research. We do need a new coroutine library to resolve the license issue, it doesn't mater whether it's wrote by ourself or others like libco.

[winlinvip]

I roughly looked at libco, and I cannot express complex ideas in English yet, so I will continue in Chinese:

1. It seems to be forked from the bsnes/libco ISC project, with an Apache license. It appears that these two licenses can be converted into each other. I raised an issue to confirm this: LICENSE issue. Tencent/libco#134
2. Its logic provides a coroutine library, not a network IO library. It still needs additional wrapping to implement a network call. For example, in the example_poll, co_create is called to create a certain number of coroutines, and then co_eventloop starts the epoll event loop for event scheduling.
3. Some people say that accept is not mocked, but it seems to have been done (I will spend some time to confirm this later). co_hook_sys_call.cpp uses dlsym or some other method as a hook. I didn't look closely, but it seems like a hook.
4. There is no ARM implementation. Instead of using setjmp and longjmp, it directly swaps the context of two coroutines with co_swap(a,b), which is a similar implementation.

I will look into the details when I have time. I will try using this library, examine the source code, and debug it. I am relatively familiar with the previous implementation by ST and have done complete debugging on it.

TRANS_BY_GPT3

[charlestamz]

I compared bsnes/libco and tencent/libco and found that the code differences are quite significant. If it is Apache License, it should be usable.

TRANS_BY_GPT3

[winlinvip]

https://github.com/AirenSoft/OvenMediaEngine

OvenMediaEngine (OME) is an Open Source, Ultra-Low Latency Streaming Server. OME receives video via RTMP from live encoders such as OBS, XSplit and transmits it on WebRTC. So, Ultra-Low Latency Streaming from OME can work seamlessly in your browser without plug-ins. Also, OME provides OvenPlayer, the HTML5 standard web player.

Our goal is to make it easier for you to build a stable broadcasting/streaming service with Ultra-Low Latency.

• LICENSE: GPLv2
• RTMP to WebRTC.

[winlinvip]

https://github.com/runner365/nginx-rtp-module
The transmission part is basically written, the bandwidth estimation using the remb protocol is still being worked on.

• LICENSE: Unknown, maybe BSD?

TRANS_BY_GPT3

[tczf1128]

QUIC should be able to support it.

Dear, thank you for your concern. Can you please provide a detailed explanation of why it should be supported, what the use case is, and why it should be supported now instead of later?"

"QUIC can be supported, it is very suitable for the current live streaming scenario.

TRANS_BY_GPT3

[winlinvip]

QUIC should be able to support it.

Dear, thank you for your attention. Can you please provide more details on why it should be supported, what the scenario is, and why it should be supported now instead of later.

Quic can be supported, it is very suitable for the current scenario of live streaming.

For live streaming, you can learn about WebRTC, which supports H5 live streaming. Additionally, the protocol standard will soon be supported (or already supported) by various CDN providers.

TRANS_BY_GPT3

[zhubblego]

For the solution you mentioned about resolving the issue with ST, why not completely switch to using the Go language for development, as it seems SRS also has a Go version?

TRANS_BY_GPT3

[winlinvip]

Regarding the issue of solving ST, why not switch completely to using Go language for development, since it seems that SRS also has a Go version?

Language is not a sufficient condition for rewriting a product, just like open-source media servers are not sufficient for providing good media services. This is why many people ask which open-source media server Alibaba Cloud uses, which media server Tencent Cloud uses, and which media server ChinaNet uses - media services are not equivalent to media servers, let alone open-source media servers.

Explaining why we don't use Go language may be difficult, but explaining the difference between "media services" and "media servers" may be somewhat easier. For example, everyone knows that cloud computing uses Linux, even the famous AWS uses Linux. So why not use a new OS? I want to say that this question itself is a non-existent question. Why do people think that the Linux used by cloud computing vendors is the same as the Linux we know? Even if everyone is based on Linux to provide services, it does not mean that cloud computing only has one Linux. Moreover, each vendor has a Linux kernel team and will have customized Linux. So, can we consider the Linux used by cloud vendors as the Linux we know? Definitely not the same.

Aside from Linux, what else do cloud vendors have? Thousands of people in the team, and there are definitely not many who work on the kernel. So, by analogy, if someone provides media services, is it still the same SRS that we know? Is it still the same Nginx that we know even if they use it? How many people are actually working on media servers? Most of them are not directly involved in server development.

Why not use Go language? Media servers are not just a language issue, just like cloud computing is not just about the Linux kernel.

TRANS_BY_GPT3

[winlinvip]

For the issue you mentioned about solving the problem of ST, why not completely switch to using Go language for development, as there seems to be a Go version of srs as well.

Of course, I can also answer this question directly, and I have some mature thoughts on it, hahaha.

In short, there is no pain point that C/C++ cannot solve, and Go does not have an absolute advantage over SRS:

1. Both ST and Go are lightweight threads and can simplify business logic, avoiding the callback hell (Async callback hell). This is a domain-specific problem for streaming media servers, which I think is the inherent complexity of the system, and it is also the fundamental advantage of SRS.
2. Go can avoid crashes caused by memory wild pointers, and SRS has crash risks, but it can be avoided through various technical means such as controlling team size, simplifying solutions, improving stability, increasing unit testing and coverage, prolonging development cycles, and controlling the desire to introduce new product features. It is not an unsolvable problem.
3. Go has the advantage of utilizing multiple cores with multi-threading, and SRS has multi-core REUSEPORT and Cluster solutions. Although SRS had a single-thread bottleneck in the past, it has been completely solved in SRS3. REUSEPORT allows for multi-core usage in Edge, and Edge/Origin Cluster can expand to multiple origin servers. These solutions are simpler from the business perspective, and simplicity means stability.
4. Go has a GC and will not have memory leaks, but the GC will have stop-the-world (STW) pauses and performance overhead (approximately 30%) in multi-core scenarios. SRS does not have a GC and requires manual memory release, but most of the memory in SRS is managed by SrsAutoFree, which is stack-based and not a pain point. Additionally, SRS does not have STW issues.
5. Go supports multiple operating systems, while SRS only supports Linux servers. This is an advantage of Go, but it is not an advantage that SRS should have. Supporting multiple OSes requires considering more general issues. Why should a server support Windows? Now, Docker can run Linux on Windows.
6. ST can be deeply customized, such as easily supporting SRT in SRS. ST's thread and coroutine models can be clearly understood and used to solve problems. However, Go cannot use libsrt because it opens its own threads and calls io functions, which cannot be handled by cgo. Moreover, if hardware is needed in the future, such as DPDK, ST can also support it, but Go will definitely struggle.

Of course, I'm not saying that Go cannot be used for streaming media servers. It can definitely be used, and in fact, I also recommend and prefer using Go in practice. I'm just saying that it is not necessary for SRS to use Go. I recommend Go because:

1. Go's goroutines are at the language level, so there is no need to implement and understand ST. It is more suitable for teamwork. Open-source projects can be maintained for a long time, but it doesn't necessarily mean that the product can be developed by oneself in the long term. If someone else has to work on ST, they might curse why they were given such a challenge. Go is different; it is a standard language with better universality.
2. Go is more suitable for scenarios with rapidly changing business requirements where the product manager's desires cannot be effectively controlled. When you are being forced to rapidly release a C++ server, using Go is necessary; otherwise, it is easy for the C++ server to crash. If you are lucky enough to have the business grow hundreds of times, one crash can cause a major problem. So it is usually worth exchanging stability for CPU usage.
3. Go has a good ecosystem and can be used to build one server after another. Personally, I prefer Go and have also written a guide on Go technology. Sometimes, in business development, we cannot refuse demands from someone like SRS and we cannot carefully consider and think about the optimal solution.

What is the optimal solution? It is not doing anything, for example, there was a lot of talk about H.265, but now it seems that more people are talking about AV1. I will wait for them to talk about it for a few more years before doing anything. Can you achieve this in the real world of product development? If not, then use Go, and use CPU to exchange for stability.

TRANS_BY_GPT3

[winlinvip]

If it supports WebRTC streaming, it would be great, then we can directly play it on mobile devices, which is more real-time than HLS.

WebRTC live streaming has already been put on the agenda and the discussion of solutions is currently ongoing. It will be implemented soon, so please wait patiently. 😄

TRANS_BY_GPT3

[KISSMonX]

我们目前实时监控就是用到这个 RTSP，摄像头厂家的方案，延迟有点大，但也基本满足要求，我们是在办公室监控实验室（本地和远程都有）的化学反应。因为要配合控制操作，想要更加实时的方式，移动端也十分需要，看了你们上面的讨论，WebRTC 似乎是目前最佳的应用。

[winlinvip]

srs3现在可以生产环境使用吗

SRS3 b0已经发布了，可以在生产环境中灰度了。

[xiaozhihong]

大致看了下libco，还不能用英文表达比较复杂的想法，就还是用中文吧：

1. 应该是从bsnes/libco ISC项目fork出来的，LICENSE是Apache，看起来这两个协议可以互相转换，我发了一个Issue问这个问题待确认，Tencent/libco#134
2. 它的逻辑是提供coroutine库，而不是网络IO的库，还需要在这个基础上再封装一层网络调用。比如example_poll，调用co_create创建一定数量的coroutine，然后co_eventloop启动epoll事件循环，在这个循环中进行事件调度。
3. 有人说没有mock掉accept，我看是有做的（后续我会花时间确认这个），co_hook_sys_call.cpp用的dlsym还是啥办法，没仔细看，看起来像是个hook。
4. 没有ARM实现，它并不是使用setjmp和longjmp，而是直接co_swap(a,b)交换两个协程上下文，也是差不多的实现。

其他的等我有空了详细看，我会用下这个库，看下源码和调试下。之前ST的实现相对比较了解一些，有完整调试过。

关于SRS是否需要以及怎么替换成libco, 我的提交在这里
xiaozhihong@7c8a35a

通过对libco和state-thread的对比和替换state-thread的过程, 我觉得SRS短期内没必要替换成libco, 有以下几个原因

1. SRS的"主线程"大量运用了st_usleep, 用来做定时操作. libco原生不支持在"主线程"调用任何co_API, 这样导致用st_usleep做定时器的操作需要改用libco的方法来适配
2. SRS中存在不少recv_thread, send_thread 操作同一个fd的场景, ST能支持的很好, libco的底层实现限定了只能在同一个"线程"里操作同一个描述符
   (核心原因是ST会自动判断用EPOLL_CTL_ADD还是EPOLL_CTL_MOD, 而libco一律用EPOLL_CTL_ADD, 导致epoll_ctl报EEXIST错误, 而且没有暴露给用户的API)
3. ST内置了IDLE_THREAD(其实就是epoll_wait,事件循环), libco没有这种内置实现, 需要在"主线程"中调用co_eventloop, 接管了用户循环, 侵入性较大, 这也是为什么ST能够在任何地方调用st_usleep的原因
4. 如果启用了libco HOOK系统函数的功能, 系统调用多了间接性, 可能会带来一些性能损失, 如果不启用, 需要对常见的系统调用做一次封装, 参考https://github.com/xiaozhihong/srs/blob/7c8a35aea9b6108bffdc39390cdb00db525be2ac/trunk/src/service/srs_service_st.cpp
   不建议开启hook, 可能会改变程序行为, 导致一些难以发现的错误

[dolpstar]

不打算支持一下RTSP吗，譬如：输入rtsp，输出rtmp/rtsp

最近了解了下，对于监控行业，比RTSP更常用的是GB28181，当然SRS完全也是可以搞定的，特别现在监控行业也在互联网化。GB28181目前已经在讨论了，但具体时间还没有定，请看 #1500

是的，监控行业基本都离不开GB28181，我们现在用SRS也都是接入的28181摄像头，再处理流推送到SRS

[dolpstar]

要是支持转webrtc流就好了，这样就可以直接在移动端播放了，比hls又要实时

WebRTC直播已经提上议程了，正在讨论方案，会很快实现出来的，待卿长发及腰时

期待，更期待28181的支持 哈哈

[kn007]

首先感谢作者和各位贡献者辛勤工作。。。
想请教下，mms流媒体，能通过srs直接转换成http么？
谢谢。。
在github搜了下，没相关issue。

[winlinvip]

首先感谢作者和各位贡献者辛勤工作。。。
想请教下，mms流媒体，能通过srs直接转换成http么？
谢谢。。
在github搜了下，没相关issue。

SRS聚焦在互联网流媒体，也就是浏览器可以支持的标准格式，可以考虑用FFMPEG做协议和编码的转换。

[maxid]

QUIC 应该可以支持下

亲呐，感谢你的关注，麻烦你要详细说明下为什么要支持，场景是什么，为什么要现在支持而不是再缓缓。

主要是在网络环境较差的情况下（如3/4G等弱网络环境下）可以提高推流质量

[winlinvip]

QUIC 应该可以支持下

亲呐，感谢你的关注，麻烦你要详细说明下为什么要支持，场景是什么，为什么要现在支持而不是再缓缓。

主要是在网络环境较差的情况下（如3/4G等弱网络环境下）可以提高推流质量

可以考虑用SRT或WebRTC推流，SRT已经支持，WebRTC在路上了。

[Isaac-1010]

I would live to contribute and help building the WevRTC support.

[winlinvip]

@itzikgili Thanks, SRS4 has supported publishing by RTMP and playing by WebRTC, please read #307 , welcome to join us to make it better. 😄

[wujiang0630]

希望可以考虑下H265转码后在网页的播放

[Isaac-1010]

@itzikgili Thanks, SRS4 has supported publishing by RTMP and playing by WebRTC, please read #307 , welcome to join us to make it better.

Sure thing, I would love to!

[IIPoliII]

Damn i will give a look at WebRTC too !

[freeman1974]

QUIC 应该可以支持下

亲呐，感谢你的关注，麻烦你要详细说明下为什么要支持，场景是什么，为什么要现在支持而不是再缓缓。

因为对直播来说，延迟至关重要。RTMP是基于TCP的，哪怕用商业CDN，延迟也在2~3s。这个延迟还是很大，对于一些连麦直播或在线游戏直播来说，还是大。所以，希望能够在SRS5支持QUIC。

[winlinvip]

@freeman1974 延迟问题，SRS已经支持了RTMP推流WebRTC播放，以及WebRTC推流和播放。

[leegkon]

JT1078要是也能支持，那么srs在交通行业就完美了

[luocj]

要是支持转webrtc流就好了，这样就可以直接在移动端播放了，比hls又要实时

WebRTC在移动端以native为主，移动端网页可能基本没法用的。主要以PC比较稳定，移动端native当然也可以接入的。

* [red5/ant](https://github.com/ant-media/Ant-Media-Server) 说是支持了WebRTC转RTMP，看了下代码和wiki，貌似只有企业版才有，开源版本没有。

SRS 4.0会考虑支持webrtc推流转RTMP或者hls之类的播放不

[luocj]

@freeman1974 延迟问题，SRS已经支持了RTMP推流WebRTC播放，以及WebRTC推流和播放。

@winlinvip SRS 4.0会考虑支持webrtc推流转RTMP或者hls之类的播放不

[lizige]

SRS 4.0 会支持webrtc视频会议么？

[wanghuan578]

kcp应该支持下呢？

[cgoder]

你能做得到吗？做不到就用Go吧，用CPU换稳定性。

是不是go的版本已经处于放弃状态了？

在webrtc风口之上，go对网络的亲和性相对比ST更好。从团队维护产品及自动化运维的角度来说也是更优的（单从效率来说）。
最在在看pion/ion这俩项目，用go 实现的rtc比c++的更易让团队成员理解和掌握。

SRS现在的release版本已经是非常稳定且生产可用的了。

是不是有团队可以继续维护go的版本？

[likunde]

监控上云是现在监控行业想摆脱设备厂商的痛点，28181 解决痛点的关键技术

[likunde]

监控上云是现在监控行业想摆脱设备厂商的痛点，28181 解决痛点的关键技术。