This thread is obsolete, blueven patch is now trunk!!!
All oscam Cache-ex users should patch latest SVN with "blueven HT cache patch"
Latest Patch:
blueven_cache_V1.0_svn9072.rar
blueven_cache_V1.0_svn9071.rar
Here is the modified patch for Modern Webif
Latest Patch:
blueven_cache_V1.0_svn9072.modern.rar
blueven_cache_V1.0_svn9071.modern.rar
This implements long awaited Hash Tables which drastically reduce cpu load & much more..
I have tested & in my situation with 6 cacheex peers My average load has gone from 80% to 15%
There are a few tweaks and adjustments to make to take full advantage of this patch which we will discuss in detail later...
Please report all bugs/issues directly to blueven @ streamboard
& Dont forget to leave some positive feedback for all the hard work put into this massive overhaul..
>>>>Streamboard Thread - Finally: blueven HT cache patch<<<<
To build with patch, use simplebuild:
Tutorial: Updating Oscam to latest SVN - Ubuntu & debian with Simplebuild
FTF patch to /opt/simplebuild/patches
Select patch in features menu & Build
Taken from blueven streamboard:
QuoteDisplay MoreHi people!
Finally, after 3 months we got a stable HT implementation!
This patch make changes mainly over cache handler, but also add other usefull stuff in oscam.
These are overall changes:
- Cache is now stored in Hash Table! Cpu load dropped!
The "TommyDS" library (http://tommyds.sourceforge.net/) is used for hash tables implementation.
It implments a chained linear hash table, called "tommy_hashlin".
Anyway, it should be tested on different systems.
- Cache is stored in HT using only csp_hash. To match ecm with cws, we use only csp_hash without check on caid/prid/srvid!
- Now, cheking cw in cache is done each 10ms. So, answer from cache could happen at any time, if ecm not answered yet.
By default, each different cw (max 10) for same hash is stored in cache, and only cw with highest counter (that is, most received cw) is served at checking cache.
This help avoid fake cws. Obviously, we should have more than one cacheex peer to let counters increase.
It will works only with cacheex cws, cw from normal reader have priority and are considered good!
To force a specific counter, we can handle it with these new parameters in cache section:
- "check_cw_count" : set minimum cw counter to allow cw is used.
Default=1, use default behaviour: use cw with highest counter when cache is checked.
e.g. check_cw_count=2 -> use cw only if its counter is equal 2 (That is, 2 same cw received)
- "check_cw_count_mode" : specify behaviour for "check_cw_count".
Possible values:
0: when wait_time expires, serve highest counter's cw got anyway, even if no check_cw_count reached.
1: never serve cw (coming from cacheex) stored in cache if its counter not reaches check_cw_count. When wait_time expires, requests will go to normal readers! Only when a cw reaches check_cw_count, it can be served to clients.
Default 0.
- Each different cw is now pushed to cacheex peers only ONE time! No more risk same cw is pushed twice for same client!
Anyway this depends on max_time parameter, because if a cw arrives after its ecm hash is freed from cache, we lost pushing out informations for clients. So, you should set max_time as cw life at least to be sure this works properly.
- CAMD35 and CS378X:
- add keepalive stuff (handled only by reader).
Parameter: "keepalive", 1 to enable it, 0 disabled (default).
When keepalive is used, reconnetiontimeout is set to 60s, because it cannot be checked before keepalive exchange (each 30s).
- fix a bug (existing in svn) reading packets.
- Revisited all packets exchange in cacheex mode 2 and 3.
Compatibility with svn oscam is ok.
- Added in cache and in cacheex protocols(camd35 and cccam) odd/even byte info.
As bowman told me, this will let CSP able to fix problem in 0963, because now oscam send odd/even byte to it.
In fact, in CSP, it's required all cws sent trough UDP port oscam->CSP have this byte set. If not, CSP cannot detect correct cw part to swapp.
So, if you send cache (from cacheex) to CSP, make sure you and all your peers (and peers of yours peers...) run this version, to correctly send 0963 cache to CSP.
Anyway this require a new CSP plugin, that use this new byte for correctly swap cws. Ask to CSP developers fot it.
Atm, commit at #9036 is keeping here, until this patch will be spreaded. After, it could be removed without problems, because now CW swapp for cache is done in cache handler itself, using odd/even byte.
- Fix cacheex mode 2 for camd35.
REAL problem of mode 2 was that without CONNECTED status, clients would not push cws, because starting connection is done by reader and (if cacheex_allow_request=0) it does not send nothing (ecm requests) for initializing connection.
This is reason mode 2 camd35 readers are NOT WORKING CORRECT in svn atm. They receive nothing if NOT connected.
With cccam this is not a problem, because it connects at startup and have keepalive stuff.
For this reason i add "keepalive" parameter in reader section to work with camd35(tcp and udp) readers. It will connect reader at startup and will take connection alive. It works for all camd35(tcp and udp) readers, cacheex and "normal".
Keepalive stuff is mandatory for cacheex mode 2 readers. ("keepalive=1" for camd35, and "ccckeepalive=1" for cccam)
For cacheex (mode 2/3) camd35 (tcp and udp) readers, it is now enabled by default.
Camd35 keepalive send msg and check connection each 30s, and if network failure, it will re-conenct reader.
When used on a cacheex mode 2/3 readers, it will send its nodeid as keepalive msg. So nodeid is updated each 30s. No more request/answer keepalive msgs each 256 cws pushed as svn.
This decrease network traffic and make communication more cleaned.
On client side, the clientmaxidle parameter > 30-40s should be used (deafult is 120s, so can you leave default value). If not, client will close connection and reader will reopen it each 30s.
- To filter incoming and outgoing cache, now we use caid, ident and services parameters.
"cacheex_ecm_filter" is leaved for compatibility with svn oscam. But we can use caid/ident/services now.
Take attention, filters on prid will be ignored for cache coming from csp, because csp cache not send prid information (prid=0 always).
- Added check on block_same_name and block_same_ip to avoid cacheex loopback in cacheex mode 1.
- Added new parameter "no_wait_time" in user section. E.g. If we would a client that ask real readers without wait for cache, we could use this parameter.
This could be also usefull e.g. for cacheex mode 2 client when cacheex_allow_request=1. If client receive ecm request (when cacheex_allow_request=1), it should not wait because no cw was pushed to it and it would search for real readers. But obviously, thid depends from configuration.
Possible values:
0: use wait_time set in cache config
1: no use wait_time
Default 0.
- Added config parameter "reconnectdelay" for max tcp connection block delay, in milliseconds (reader section). Default 60000 ms (1min, as always been in svn)
- check in cache for "oscam.cacheex" file for mode1 readers removed. Miss documentation about it and maybe no one use that file.
- Fix always accept prid=000000 for services. It has to be explicity set.
- added column to show number of requests per minute in webif user page.
- move reader table pending to max 4096. So now you can set -p4096 when start oscam.
- In webif, if one of stats overflow (become negative), reset all stats automatically.
- Try fix some random crashes found in svn too.
- This patch revert CWC Check changes in svn #9037, #9059, #9061, #9062: code changes are not compatible with this new code. New re-code is nedeed.
- ATM, this patch revert changes in svn #9066, #9067, #9070, #9071: See my posts here: http://www.streamboard.tv/wbb2/thread.php?threadid=39787.
Most of the new parameters are not in the webif (yet), it will be added later by me or by some voluntary Zunge raus
Considerations:
Initially i thinked to remove cacheex mode 3, because now mode 2 works great. But, for compatibility with svn oscam, i leave it in code.
My advise: use cs378x protocol and cacheex mode 2! It is more clean and light. Now cacheex works very well with it!
With this patch, you will see your cache size grown than svn, be quite that all is normal because now oscam process MORE cws per second and cws are taken more time in cache, because free happens at (last cw got time+max_time), while before was (first cw got time+max_time).
Also, you can see now cw diff increases more, it is normal because now each new cw is compared with the heighest "counted" cw, that can be change at each cw arrived.
Special thanks goes to Capncook, support me in development ideas and tests. A lot of nights without sleep! smile
Also thanks to Gorgone, Mo0211 and mumbua, helped me a lot with tests.
This is an example how reader and client could be configured for cacheex mode 2 communication:
CodeDisplay More***** CACHEEX MODE 2 ****** Reader receive cws (and if cacheex_allow_ecm_request=1 send ecm requests too): [reader] label = protocol = device = user = password = group = keepalive = 1 //if camd35/cs378x it is useless, because is enabled by deafult. For cccam,use ccckeepalive=1. cacheex = 2 cacheex_allow_request = //allow send ecm requests (default=1, as svn) #####cacheex incoming filter##### cacheex_maxhop = //max incoming cw hop (default=10, as svn) cacheex_drop_csp = //drop incooming cache from csp (deafult=0, as svn) caid = //incoming caid filter (default all) ident = //incoming ident filter (default all) services = //incoming services filter (default all). Take attention: if we have too much sids to check, incoming push could slowly! User send cws (and if cacheex_allow_ecm_request=1, answer to ecm requests too): [account] user = pwd = group = no_wait_time = //ignore wait_time at incoming ecm request (it works for not cacheex users too) cacheex = 2 cacheex_allow_request = //allow incoming ecm requests (default=1, as svn) #####cacheex outgoing filter##### cacheex_maxhop = //max outgoing cw hop (default=10, as svn) cacheex_drop_csp = //not send cache recived from csp (deafult=0, as svn) caid = //outgoing caid filter (default all) ident = //outgoing ident filter (default all) services = //outgoing services filter (default all). Take attention: if we have too much sids to check, outgoing push could slowly!
Original Post: http://www.streamboard.tv/wbb2…?postid=489241#post489241
