EPG Import mod from Dorik1972 Support Thread

There are 446 replies in this Thread which was already clicked 54,880 times. The last Post () by Panda555.

    Just tested the updated version of 1.91 and external resetting epg.db to 23KB does not trow anymore the can not connect to database.


    Sadly, the switch to clear the database before import has been neutralized because it is started before dumping the current in memory EPG data. So the just emptied file is overwritten by EPG data from the memory.


    Code
    Aug 30 11:14:18 dm7080 enigma2[287]: [EPGImport] Oudeis patch not detected, using eEPGCache.load
Aug 30 11:14:18 dm7080 enigma2[287]: [EPGDB] create empty /media/SanDisk240GB/XMLTV-EPG/epg.db
Aug 30 11:14:18 dm7080 enigma2[287]: [EPGImport] EPG database located at /media/SanDisk240GB/XMLTV-EPG/epg.db
Aug 30 11:14:18 dm7080 enigma2[287]: [EPGImport] Save the EPGchache to database file /media/SanDisk240GB/XMLTV-EPG/epg.db ...
Aug 30 11:14:18 dm7080 enigma2[287]: [EPGC] db thread stopped
Aug 30 11:14:18 dm7080 enigma2[287]: [EPGC] Saving database from memory
Aug 30 11:14:19 dm7080 enigma2[287]: [EPGC] db thread started
Aug 30 11:14:19 dm7080 enigma2[287]: [EPGC] start caching events(1661850859)
Aug 30 11:14:19 dm7080 enigma2[287]: [EPGImport] EPGCache save finished
    Quote from msatter

    Sadly, the switch to clear the database before import has been neutralized because it is started before dumping the current in memory EPG data. So the just emptied file is overwritten by EPG data from the memory.

    At the moment I view this menu item as "Clear DB file before importing data" ...


    Do you want all "traces" of an existing EPG to be deleted? It's not difficult, I'll just don't dump the cache data if this option is selected.... But what if EIT is used to import DVB events, and IPTV events to import - EPGImport? Clearing the EPG will result in the loss of current EIT data and the resulting epg.db will only contain events for IPTV

    In the past I could only EPGimport when being tuned to a station without any EPG DVB data.


    This option to clear is mostly used by users that load EPG during the night and so have each time, a clean database to start with. After EPGimport is ready it will then only contain the just imported data.


    The data can become outdated and when you go to a station that has current EIT data then the database will not be updated. However Now/Next on the screen is updated to reflect current data. the data from the database is ignored for Now/Next.


    I don't know if the EIT data is overwritten by the data that is imported by EPGimport?


    Then EIT data is dynamic and updated as soon you tune into that station and thrown away as soon you tune into an other station except for the non present data in the EPG database.


    I prefer to make a clean cut each time a import is made and have a empty epg database. But then I am not the only user of EPGimport.

    Quote from msatter

    The data can become outdated a

    You do not understand how it works =) .. A special "semaphore" monitors the "freshness" of data, which cleans outdated data from eEPDCache at a specified frequency (see the system log!) ... The EPG data that you see on the screen is ALWAYS data from eEPGCache ... Similarly, there is a "semaphore" for dumping eEPGCache data into a file and overwriting epg.db ... Everything can be seen in the log and sources... if desired


    part of code of epgcache.cpp

    Code
    /* Interval between "garbage collect" cycles */
#define CLEAN_INTERVAL 60000    //  1 min
/* Restart EPG data capture */
#define UPDATE_INTERVAL 3600000  // 60 min
/* Time to wait after tuning in before EPG data capturing starts */
#define ZAP_DELAY 2000          // 2 sec

    pepsik Sorry to bother you again about Twisted. I ran with and without and looked if there is any difference in the journal and I noticed this extra line missing when running Twisted:


    Code
    Aug 30 12:59:48 dm7080 enigma2[2716]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py(241): afterChannelDownload


    Those extra lines are only shown when I use Rytec type of lists. The IPTV lists runs with Twisted.


    Then, on the clearing the egd.db file, I see these kinds of messages when I tune to a loaded EPG station:


    At first I see the data from the epg.deb and a few seconds later the event text changes to the data from the DVB. I am not a programmer and only know what I observe.

    Edited once, last by msatter ().

    Quote from msatter

    Those extra lines are only shown when I use Rytec type of lists. The IPTV lists runs with Twisted.

    You do not understand what twisted.thread is ... And the log that you showed has nothing to do with this. These are not errors - these are warnings that some of the links to channels.xml - give HTML Error 404 ... If you look at the structure of the sources.xml file of Rytec sources, you will understand that there is a list of links to the same channels.xml ... If it doesn’t download from one hosting, we try the next one ... and so on until we download channels.xml


    Quote from msatter

    Then, on the clearing the egd.db file, I see these kinds of messages when I tune to a loaded EPG station:

    This is the EPG "update" and the EPGRefresh plugin log... it just confirms my assertions about the EPGCache update "semaphores"... Just look at the source text of EPGRefresh.py :winking face:

    pepsik I wrote over here what I thought Twisted was and I still don't know if that is correct or not. :winking face:


    The 404 error is generated because Twisted does not work on the DreamOS here. That can be ignored because there is a backup to download the EPG and channels file.


    So back to what I wanted to convey to you.


    When using Twisted (parse - update - createIterator - doThreadRead) missing here afterChannelDownload

    The doRead is when using Twisted doThreadRead.


    Code
    Aug 30 12:23:31 dm7080 enigma2[2238]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGConfig.py(162): parse
Aug 30 12:23:31 dm7080 enigma2[2238]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGConfig.py(180): update
Aug 30 12:23:31 dm7080 enigma2[2238]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py(194): createIterator
Aug 30 12:23:31 dm7080 enigma2[2238]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py(278): doThreadRead


    These lines are show when using Twisted or disable Twisted (parse - update - createIterator - afterChannelDownload - writeFile)

    Code
    Aug 30 12:59:48 dm7080 enigma2[2716]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGConfig.py(162): parse
Aug 30 12:59:48 dm7080 enigma2[2716]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGConfig.py(180): update
Aug 30 12:59:48 dm7080 enigma2[2716]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py(193): createIterator
Aug 30 12:59:48 dm7080 enigma2[2716]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py(241): afterChannelDownload
Aug 30 12:59:48 dm7080 enigma2[2716]:     /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py(409): writeFile

    The second part, reading the EPG data is later. Then does the Twisted version knows the there are two files downloaded? EPG and channels. When I download an IPTV file then the channels is combined with the EPG data in one file when I remember it correct.

    Edited once, last by msatter ().

    pepsik On the EPG being updated when tuning to a station.


    I don't see epgRefresh being installed.

    Code
    enigma2-plugin-extensions-epgrefresh/unknown 4.3.3+git2+a7a8ee2266-r0.0 mipsel
  Plugin to refresh EPG Data when Receiver is inactive

    If installed then it displayed

    Code
    enigma2-plugin-extensions-epgimport-mod-dorik1972/now 1.9.1 all [installed,local]
  EPGImport mod Dorik1972

    Searching for epgrefresh:

    # find / -name epgre*

    /etc/enigma2/epgrefresh.xml

    #


    Content of the epgrefresh.xml file


    XML
    <?xml version="1.0" ?>
<epgrefresh version="1">




</epgrefresh>

    Edited once, last by msatter ().

    pepsik


    Then indulge me and explain what Twisted does. Did you read the rest of my posting or did you stop at that line?


    I could now find EPGRefresh.py but it is not in the location you show. It is sitting in WebInterface/WebChilds/Externals

    Code
    # find / -iname EPGRe*
/usr/lib/enigma2/python/Plugins/Extensions/WebInterface/WebChilds/External/EPGRefresh.pyc
/usr/lib/enigma2/python/Plugins/Extensions/WebInterface/WebChilds/External/EPGRefresh.pyo
/usr/lib/enigma2/python/Plugins/Extensions/WebInterface/WebChilds/External/EPGRefresh.py


    Moving forwards. We know IPTV files work and the separate files crashes DreamOS here and with other users of EPGimport.


    If IPTV works under Twisted as being one file containing channels.xml and epg.xml I suggest, if not already being the case, to also merge those to files to one file after the download. Then EPGImport can use Twisted on it, if they are not separate files anymore, just like the IPTV files I download.

    Edited once, last by msatter ().

    Does it come to mind that the ones that not have the same problem are not complaining about it. There is however there a group that have the same problem since version 1.6


    I am vocal about it and try to find out what could solve it. But I am close to loose interest again like it happened when version 1.6 was released.


    If you want to try, then take the Rytec sources that comes included in EPGimport and select one (yellow button) of the lists and see if Enigma is restarted or locks-up like it is happening with us.

    Quote from msatter

    I am vocal about it and try to find out what could solve it.

    I wrote 100500 times in this forum thread that until I repeat this mistake with twisted.thread, no logs will help. Since it is impossible to understand the reason for its occurrence from the log. Moreover, this error is not observed by all users.


    Quote from msatter

    If you want to try, then take the Rytec sources that comes included in EPGimport and select one (yellow button) of the lists and see if Enigma is restarted or locks-up like it is happening with us.

    As you wish .....

    Снимок экрана 2022-08-31 в 21.21.38.png

    log in attachment ... I repeated any combinations and variations hundreds of times on my DM900HD ... and I never "caught" an error with twisted.thread ...

    • Official Post

    msatter Can I suggest that you change your settings for location of your epg.db

    It's strongly not recommended to store the epg.db on an external storage device and special not the sd card on older Dreamboxes like yours.

    You have plenty of space in the flash and /data folder to store the epg.db.

    Move it to /etc/enigma2 and check if your problem improves.

    And there is no reason to allways reset the epg.db before you run your epg update. This is only needed if you have corruped epg data.

    pepsik


    Thanks for log and it does indeed work with you and others and I did not doubt that:


    So I marked the same sources and this is an unfiltered log showing the import and the restart:


    Maybe you can see here some more that could explain the difference between my DreamOS and your DreamOS.


    pepsik

    seagen


    I try next the suggestion to move epg.db /etc/enigma and I only resetted the epg.db for testing a few days ago.


    Update: Tried it, no improvement:


    Update: install epgrefresh ( /usr/lib/enigma2/python/Plugins/Extensions/EPGRefresh/EPGRefresh.py )


    Still not importing.

    Edited 3 times, last by msatter: with /etc/enigma2 for epg.db Installed EPGRefresh ().

    Quote from msatter

    FATAL!: addTimer must be called from thread 4487 but is called from thread 4624


    Googling the error, I found the ONLY mention that it was solved. All other links lead to reports that this is a DreamOS image bug and will be resolved.


    https://github.com/s3n0/e2plug…ns/src/changelog.txt#L102


    If someone is familiar with the author of ChocholousekPicons or he reads this message - contact me. Maybe you can help me with this error. I don't see any bugs in the plugin code

    pepsik How is your German language knowledge?


    I found some info and also the definition of Twisted and I was in the right direction on that. They say you can't call Enigma2 objects from Twisted because it is Enigma2 not thread safe.


    The problem: https://dreambox.de/board/inde…&postID=207390#post207390

    How he solved it: https://dreambox.de/board/inde…&postID=207406#post207406


    So it most likely an other plug-in that catch the call and I will try with only the EPGimport plug-in present and if it then works add plug-ins one by one to see if the problem returns.


    Do you have Gemini 4 installed on your DM900? I use that and many other but you might not have that installed.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login