Posts by msatter

This a small one and when selecting recording to be placed local so I can watch those in a time I want.

The change is in the flow and it makes it easier to select more than one recording.

            if exists is False:
                if self.categoryname == "vod":
                    downloads_all.append([_("Movie"), title, stream_url, "Not Started", 0, 0])
                elif self.categoryname == "series":
                    downloads_all.append([_("Series"), title, stream_url, "Not Started", 0, 0])
                elif self.categoryname == "catchup":
                    downloads_all.append([_("Catch-up"), title, playurl, "Not Started", 0, 0, description]) #### adding description to download2.json

                with open(downloads_json, "w") as f:
                    json.dump(downloads_all, f)

                ####self.session.openWithCallback(self.opendownloader, MessageBox, _(title) + "\n\n" + _("Added to download manager") + "\n\n" + _("Note recording acts as an open connection.") + "\n" + _("Do not record and play streams at the same time.") + "\n\n" + _("Open download manager?"))
                self.session.openWithCallback(self.opendownloader, MessageBox, _(title) + "\n\n" + _("Added to download manager") + "\n\n" + _("Note recording acts as an open connection.") + "\n" + _("Do not record and play streams at the same time.") + "\n\n" + _("Select more recordings?"))
            else:
                self.session.open(MessageBox, _(title) + "\n\n" + _("Already added to download manager"), MessageBox.TYPE_ERROR, timeout=5)

    def opendownloader(self, answer=None):
        ####if not answer:
        if answer:
            return
        else:
            from . import downloadmanager
            self.session.openWithCallback(self.createSetup, downloadmanager.XStreamity_DownloadManager)

I have marked the lines that are marked out now with #### and added the changed line underneath.

If you by accident still want go to the download manager screen then you can select a other recording and the then select no, showing then the screen or you go the main menu where you can enter the download manager screen also.

Quote from biko

a friend ask me abut problem he faced with sending series to bouquet

the folder is empty and that not happen with vod or live it done perfect

can explain what cause that kiddac

Did you thought about that you are now displaying the domain of the provider?

I have also the same thing of not being displayed but then for the categories. When I go back an try again the category list is being populated. I assume the provider is not delivering the the data fast enough or incomplete data.

When using the download manager and having more than one recording the marking the the recording is a bit tricky. When marking, and at the same time a screen update is being made, the selected recording is deselected and the just updated is being selected.

This can lead to unexpected outcomes when wanting to apply changes by the user.

So I changed the order of the lines in the download manager screen.

 def buildList(self):
        self.drawList = []
        self.drawList = [self.buildListEntry(x[0], x[1], x[2], str(x[3]), x[4], x[5]) for x in self.downloads_all]
        self["downloadlist"].setList(self.drawList)

    def updatescreen(self):
        self.diskspace()
        self.getprogress()
        
    #### here the change of position of the list can be archieved
    def sortlist(self):
        ####order = {"In progress": 0, "Waiting": 1, "Not Started": 2, "Downloaded": 3}
        order = {"In progress": 2, "Waiting": 1, "Not Started": 0, "Downloaded": 3}
        self.downloads_all.sort(key=lambda x: order[x[3]])
        # self.buildList()

The changed order is: note started / waiting / progress / last downloaded

So the not stared are always at the top of the list and can so be started more easy without that it gets tricky. The downloaded lines are default automatically removed after successful download.

disclaimer: the shown line number is an estimate because my code has more modifications than I already have written here. So more to come.

Quote from Hendra02

always refreshing every 5 minutes, where's the harm?

I don't see any context and can you explain what you exactly mean because your line can be interpreted in two ways who are opposite to of each other?

Next step for the EIT file is to read it in downloadmanager.py and store it with the help of an external script named "EvenInformationTable.py" written by cmikula.

This file is attached and has be copied to the X-streamity directory and after a restart of the box it should be active.

This is a new code part and you can place it above "def createMetaFile(self, filename, filmtitle):" at the bottom of the downloadmanager.py code.

The shown line numbers are indicative because in my files there also other changes that I will post at a later moment.

from Components.ActionMap import ActionMap
from Components.Sources.List import List
from Screens.MessageBox import MessageBox
from Screens.Screen import Screen
from enigma import eTimer, eServiceReference
from requests.adapters import HTTPAdapter, Retry
#### for writing the EIT file after downloading a Catch-up item
from EvenInformationTable import writeEIT 

    def createEitFile(self, filename, filmtitle): ####
        if os.path.isfile(downloads_json):
            with open(downloads_json, "r") as f:
                try:
                    self.downloads_all = json.load(f)
                except Exception as e:
                    print(e)

        for video in self.downloads_all:
            if str(video[1]) == str(filmtitle):
                urlParts = re.split("/", video[2])
                startDateTime = urlParts[-2]
                duration = urlParts[-3] # duration here in minutes
                description = str(video[6])
                eitMJD = str(datetime.datetime.strptime(startDateTime, "%Y-%m-%d:%H-%M").strftime("%Y-%m-%d"))
                filename = filename.replace(".m3u8", ".ts")
                eit_file = filename.replace(".ts", ".eit") # includes the whole path
                overview = filmtitle[14:16]+" - "+duration+" "+"minutes"
                break
         
        try:
            #self.session.open(MessageBox, _("WriteEIT: "+"\r\n"+str(filename)+"\r\n"+str(filmtitle)+"\r\n"+str(eit_file)+"\r\n"+str(description)), type=MessageBox.TYPE_INFO)           
            writeEIT(filename, eit_file, (filmtitle[17:]), filmtitle[17:], "Catch-up", description, eitMJD, duration, (filmtitle[14:16]+" "))
        except Exception as e:
            print(e)
        return

    def download_finished(self, filename, filmtitle):
        self.createEitFile(filename, self.filmtitle)  ####
        global ui
        if os.path.isfile(downloads_json):
            with open(downloads_json, "r") as f:
                try:
                    self.downloads_all = json.load(f)
                except Exception as e:
                    print(e)

        x = 0
        for video in self.downloads_all:
            if str(video[1]) == str(filmtitle):
                break
            x += 1
        del self.downloads_all[x]
        if ui:
            self.sortlist()
            self.buildList()
        self.createMetaFile(filename, self.filmtitle)
        #self.createEitFile(filename, self.filmtitle)
        self.saveJson()

The needed file: EvenInformationTable.py

This is the first step to enable EIT (Extended Information Table) file support also for Catch-up recordings.

The first step is to put the short description taken form the EPG into the download2.json file so that it can be read by the download manager code.

I have marked the changes as always with ####

def downloadVideo(self):
        # print("*** downloadVideo ***")
        # load x-downloadlist.json file

        if self.categoryname == "series" and self.level != 4:
            return

        if self.categoryname == "catchup" and self.selectedlist != self["epg_short_list"]:
            return

        if self["main_list"].getCurrent():

            if self.categoryname == "catchup":
                next_url = self["main_list"].getCurrent()[3]
                stream = next_url.rpartition("/")[-1]
                date = str(self["epg_short_list"].getCurrent()[4])
                duration = str(self["epg_short_list"].getCurrent()[5])
                playurl = "%s/timeshift/%s/%s/%s/%s/%s" % (self.host, self.username, self.password, duration, date, stream)

                date_all = str(self["epg_short_list"].getCurrent()[1]).strip()
                time_all = str(self["epg_short_list"].getCurrent()[2]).strip()
                time_start = time_all.partition(" - ")[0].strip()
                current_year = int(datetime.now().year)
                date = str(datetime.strptime(str(current_year) + str(date_all) + str(time_start), "%Y%a %d/%m%H:%M")).replace("-", "").replace(":", "")[:-2]

                otitle = str(self["epg_short_list"].getCurrent()[0])
                channel = str(self["main_list"].getCurrent()[0])
                title = str(date) + " - " + str(channel) + " - " + str(otitle)
                # define and set description with the text for the EIT file
                description = str(self["epg_short_list"].getCurrent()[3]) ####

            else:
                title = self["main_list"].getCurrent()[0]
                stream_url = self["main_list"].getCurrent()[3]

            downloads_all = []
            if os.path.isfile(downloads_json):
                with open(downloads_json, "r") as f:
                    try:
                        downloads_all = json.load(f)
                    except:
                        pass

            exists = False
            for video in downloads_all:
                url = video[2]

                if self.categoryname == "vod" or self.categoryname == "series":
                    if stream_url == url:
                        exists = True

                if self.categoryname == "catchup":
                    if playurl == url:
                        exists = True

            if exists is False:
                if self.categoryname == "vod":
                    downloads_all.append([_("Movie"), title, stream_url, "Not Started", 0, 0])
                elif self.categoryname == "series":
                    downloads_all.append([_("Series"), title, stream_url, "Not Started", 0, 0])
                elif self.categoryname == "catchup":
                    downloads_all.append([_("Catch-up"), title, playurl, "Not Started", 0, 0, description]) #### adding description to download2.json

                with open(downloads_json, "w") as f:
                    json.dump(downloads_all, f)

                self.session.openWithCallback(self.opendownloader, MessageBox, _(title) + "\n\n" + _("Added to download manager") + "\n\n" + _("Note recording acts as an open connection.") + "\n" + _("Do not record and play streams at the same time.") + "\n\n" + _("Open download manager?"))

            else:
                self.session.open(MessageBox, _(title) + "\n\n" + _("Already added to download manager"), MessageBox.TYPE_ERROR, timeout=5)

The next one is a adaptation to the download manager screen. When a Catch-up can be marked as to be recorded but the recording is in fact not present.

This can happen because EPG and the actual recording have separate sources.

Default this will be displayed as the recording being "0B" and changed that "File N/A". If the recording is restored with your provider then then this should be detected and then you can still get the recording.

def convert_size(size_bytes):
    if size_bytes == 0:
        return "File N/A"
        #return "0B" #### replaced by "File N/A" in the downloadmanager screen
    size_name = ("B", "KB", "MB", "GB", "TB")
    i = int(math.floor(math.log(size_bytes, 1024)))
    p = math.pow(1024, i)
    s = round(size_bytes / p, 2)
    return "%s %s" % (s, size_name[i])

I am using since a while X-streamity and modified thing to be more to my liking and extend the capabilities of it.

Thanks to Kiddac for creating X-streamity and it very structured to finding stuff is not that difficult.

My latest modification is able to keep watching live while having started a instant recording of same live channel. Even if you have chosen for .m3u8 instead of .ts as output extension. What it does, when a instant recording is started while you watch the same channel, the current stream is terminated on start of the recording and new stream is started matching the way a recording is made.

This way you keep watching live while the recording is made in background. You can change/stop the recording through the menu button and go to timers.

Also what is added is the actual end time of the transmission so the recording will stop when the transmission is over. If no end time is found the the default recording duration will be one hour.

instantRecordActive = False #### allows to instant record a channel

    def IPTVstartInstantRecording(self, limitEvent=True):
        from . import record
        begin = int(time.time())
        end = begin + 3600
        
        # if now-next is available, take the starttime of next as endtime for current
        if (glob.currentepglist[glob.currentchannellistindex][5]): ####
            endstring = (glob.currentepglist[glob.currentchannellistindex][5])
            shortend_dt = datetime.strptime(endstring, "%H:%M")
            end = int(time.mktime(shortend_dt.timetuple())) ####
            
            #self.session.open(MessageBox, _("Switched live view, to streaming type 1 to match the recording being made!"), MessageBox.TYPE_INFO, timeout=5)

        if glob.currentepglist[glob.currentchannellistindex][3]:
            name = glob.currentepglist[glob.currentchannellistindex][3]
        else:
            name = glob.currentchannellist[glob.currentchannellistindex][0]

        self.name = NoSave(ConfigText(default=name, fixed_size=False))
        self.date = time.time()
        self.starttime = NoSave(ConfigClock(default=begin))
        self.endtime = NoSave(ConfigClock(default=end))
        self.session.openWithCallback(self.RecordDateInputClosed, record.RecordDateInput, self.name, self.date, self.starttime, self.endtime, True)
              
    def RecordDateInputClosed(self, ret=None):
        if ret:
            begin = ret[1]
            end = ret[2]
            name = ret[3]
            
            #### for instant recording .m3u8 extension should to be .ts
            streamurl = self.streamurl.replace(".m3u8", ".ts") #### create a local non class variable

            description = glob.currentepglist[glob.currentchannellistindex][4]
            eventid = int(self.streamurl.rpartition("/")[-1].partition(".")[0])
            #serviceref = eServiceReference(1, 0, self.streamurl)
            serviceref = eServiceReference(1, 0, streamurl)
            
            #### keep playing live when the livestream is not streamtype 1
            if self.servicetype != "1":
                self.session.nav.stopService()
                self.playStream("1", streamurl, self.direct_source)

            self.session.open(MessageBox, _("Live view is probably stopped, allowing it to be recorded. \r\n Please do not change channel or use reconnect! \r\n Pressing menu allows to change or stop the started recording."), MessageBox.TYPE_INFO, timeout=9)
                   
            if isinstance(serviceref, eServiceReference):
                serviceref = ServiceReference(serviceref)
                                
            recording = RecordTimerEntry(serviceref, begin, end, name, description, eventid, dirname=str(cfg.downloadlocation.getValue()))
            recording.dontSave = True
            instantRecordActive = True #### allows to instant record without interupption

            simulTimerList = self.session.nav.RecordTimer.record(recording)

            if simulTimerList is None:  # no conflict
                recording.autoincrease = False
                self.session.open(MessageBox, _("Recording Timer Set."), MessageBox.TYPE_INFO, timeout=5)
            else:
                self.session.open(MessageBox, _("Recording Failed."), MessageBox.TYPE_WARNING)
                return
        else:
            return

The toggle and I shorted it now so that it won't change the servicetype

    def streamTypeFailed(self, data=None):
        if instantRecordActive == False: #### allow instant recording run uninterupted
            if str(self.servicetype) == "1":
                self.servicetype = "4097"
            elif str(self.servicetype) == "4097":
                self.servicetype = "1"

            self.streamcheck = 2
            self.playStream(self.servicetype, self.streamurl, self.direct_source)
        instantRecordActive = False

IPTVstartInstantRecording proposed changes have been implemented in X-streamity 4.16 adding the actual end time of a recording instead of default one hour if EPG data is available.

Quote from KiddaC

msatter any developer that is deleting other peoples plugins shouldn't really be applauded.
We all know he is a very talented and dedicated developer but his recent actions are pissing a lot of people off.

I have had some of discussions in the past ten years and also specific about Jedi. Caught between a rock and a hard place is best covering the situation here.

Only chipping small parts away is the way to go, but if that is going to be completed in our live times is questionable......

Quote from tud2007

Quote from KiddaC

Heval I just noticed in your crash log

enigma2-plugin-systemplugins-gutemine - 4.0-r0 - gutemine Feed3

as far as I have been made aware, and for reasons unknown. gutemine deletes other peoples plugins including jedi files.

If you want to use my plugins, it seems the only solution is to totally remove gutemine from your box.

Rather mature of him. Maybe all us plugin developers should delete his .so file. But that would be lowering to his level. My stuff is better, so you decide

Display More

he doesnt allow plugins with no support with his plugin together. after you marked jedi as dead it‘s then removed automatically. Therefore I had to reflash my box again since his plugin is not easy to remove fully.

gutemine out jedi in

Display More

GM can be removed. Gutemine did state how to remove his plug-in via terminal:

Quote

python

import gm

gm.purge()

exit python with CTRL-D or an other way

then reboot

Display More

BTW Gutemine is doing a LOT of good work now Dream is gone. We got a final goodby gift by the DreamOS developers which was not perfect but Gutemine jumped in that spot with other coders/developers and we have now multiboot on/with GPT partitions and some other fixes.

More to come but it will take time. The One and Two are really nice boxes and will get me an other one soon.

I was browsing through plugin.py and noticed that streamtype 8193 was defined.

In the past I thought that that streamtype was specific for DreamOS but I was pointed out that that 8193 is a dummy.

Sep 11 20:28:44 dreambox enigma2[493]: [eServiceUri] 8193:0:0:0:0:0:0:0:0:0:http%3a....:NL| NPO 1 HD resolved to 4097:0:0:0:0:0:0:0:0:0:http%3a.....:NL| NPO 1 HD - with suburi

You see that 8193 is being converted/resolved to 4097 and so it is easier in the end to just use streamtype 4097.

I got an specific question about downloading EPG the EPG. When I select in the main "Manual EPG Update" is then the /etc/epgimport/xstreamity.sources.xml used? Or is playlists.txt uses?

Why, I have two accounts with the same provider and that leads to twice downloading the XMLTV file. So I removed the second account line in the xstreamity.sources.xml file and made symbolic link in /etc/enigma2/xstreamity/epg/ to the directory name of the first account line.

Update: could now test it and both directories are created. So an double import in this special situation.

Quote from KiddaC

Xstreamity downloads the full xmltv epg every time you visit live categories page after a certain length of time.
From this epg data it parses the xmltv xml and creates a json file with only the information I need.

I am not sure that I ever noticed this and can you give an indication what the time is before a XMLTV automatic download is initiated? Is this also done when I am watching a live channel on that moment?

Quote from KiddaC

Quote from msatter

Python wants TAB's and not spaces in front of the line.

“Spaces are the preferred indentation method. Tabs should be used solely to remain consistent with code that is already indented with tabs. Python disallows mixing tabs and spaces for indentation.” Furthermore, the section on Indentation states that 4 spaces per indentation level should be used

I made a strike through of that text in my earlier posting.

I went back to downloader.py instead of channels. The channels change corrects wget not being able handle m3u8. In meantime I don't use wget anymore for only m3u8 files.

It has an reason but not a one you want to hear.

ps. just now, I noticed that the empty space in your posting contains a explanation why and how. Thanks for that, I assume I mixed tabs and spaces at that moment.

Request for change: when using download in chatch-up and when in Playlist Settings m3u8 as output is set then the item can't be downloaded. This because wget does not know how to handle a m3u8 item and that is two separate downloads.

Request is then to replace the extension m3u8 by ts, so that wget can download the item.

In other places in Xsteamity it not a problem because for Live it is then sent to steam device. In Vod and TV series other formats are being used like mpg, mkv etc.

Update: test now and the change has to be done in channels.py then def downloadVideo(self):

            if exists is False:
                if self.categoryname == "vod":
                    downloads_all.append([_("Movie"), title, stream_url, "Not Started", 0, 0])
                elif self.categoryname == "series":
                    downloads_all.append([_("Series"), title, stream_url, "Not Started", 0, 0])
                elif self.categoryname == "catchup":    
                    playurl = playurl.replace(".m3u8", ".ts")
                    downloads_all.append([_("Catch-up"), title, playurl, "Not Started", 0, 0])

The new line in the code displayed above is:

playurl = playurl.replace(".m3u8", ".ts")

Python wants TAB's and not spaces in front of the line.

It took me rather long to find the correct location. I expected it to tackle it in downloader.py but that is called much later. So it was a search for the source and that was in channels.py

Quote from KiddaC

Why are people suddenly having issues with the infobar??

I don't know if it is suddenly and I saw from the moment I started to use Xstreamity.

Then if there is no EPG available then it is still being shown. After the timeout it goes away. Pressing the OK button shows both again.

I activate during watching the infobar (lower) to but then in Xstreamity then the top bar is also show.

Maybe the when activating with OK the bar then show only the lowerbar and when you press OK again show also the topbar. And when you press OK again the both are hidden.

Lower bar shown, on OK -> show then show topbar also

Lower bar shown and also the topbar, on OK -> hide both

After zap both bars are shown, on OK -> hide both or wait for timeout

Why should I revise vision on the log Enigma2 is producing. Do you want to tell what stated it is not what is happening?

Also, as mentioned earlier, I included your newly introduced vacuum the database after EPGImport in the general information I wrote.

pepsik

Quote

And the "pieces" of the log that you cite in the post above should not be stupidly read and stated in the form of "fantasies on a free theme" that have NO relation to reality....

Can you more specific on what is fantasy and what is a correct interpretation of the journal/log?

Thank for making the difference between clean database and clean EPG clear.

Yes , I stick like gum.

On 1 to 5, no I did not read the information you referred to on the sqlite site because it gives here a : Error code: SSL_ERROR_RX_RECORD_TOO_LONG" error. Looking at the short version on duckduckgo:

Quote

 

The VACUUM command rebuilds the database file, repacking it into a minimal amount of disk space. There are several reasons an application might do this: Unless SQLite is running in "auto_vacuum=FULL" mode, when a large amount of data is deleted from the database file it leaves behind empty space, or "free" database pages

On 6, a journal of the data being retrieved on zapping:

Sep 06 08:58:53 dm920 enigma2[350]: [EPGC] channel 0x36f6940 running

Sep 06 08:58:53 dm920 enigma2[350]: [EPGC] next update in 2 sec

Sep 06 08:58:55 dm920 enigma2[350]: [EPGC] start caching events(1662447535)

Sep 06 08:58:55 dm920 enigma2[350]: [EPGC] 5274 0c99 0003 00eb0000 has external data! won't update...

Sep 06 08:58:55 dm920 enigma2[350]: [EPGC] 17d1 0c82 0003 00eb0000 has external data! won't update...

.

.

Sep 06 08:58:58 dm920 enigma2[350]: [EPGC] 5287 0c96 0003 00eb0000 has external data! won't update...

Sep 06 08:58:58 dm920 enigma2[350]: [EPGC] 52ec 0c96 0003 00eb0000 has external data! won't update...

Sep 06 08:59:05 dm920 enigma2[350]: [EPGC] nownext finished(1662447545)

Sep 06 09:00:32 dm920 enigma2[350]: [EPGC] schedule finished(1662447632)

Sep 06 09:00:32 dm920 enigma2[350]: [EPGC] schedule other finished(1662447632)

Sep 06 09:00:32 dm920 enigma2[350]: [EPGC] stop caching events(1662447632)

Sep 06 09:00:32 dm920 enigma2[350]: [EPGC] next update in 60 min

Sep 06 09:00:35 dm920 enigma2[350]: [EPGC] cleanup invalid data

Sep 06 09:00:35 dm920 enigma2[350]: [EPGC] cleanupOutdated

Starts caching the events and does not update the events that are already in database in RAM. Nownext is always updated and overwrites the short description of the EPG but, so only on the screen not in the database. If you zap forward and back you will see again the short description from the EPG database and a few seconds later the short nownext description is overwritten again on on the screen.

There is no need to vacuum the database as long it is in RAM because it is fast enough to handle all. When writing back to disc it writes a clean database, which is ready to be imported again.

On EPGimport the database on disc is being used to insert/update events. Now most likely, you do a import after vacuum to clean out the obsolete events that are replaced by updated events and maybe you also remove outdated events keeping the events defined by: config.misc.epgcache_outdated_timespan=hours

Vacuum reduces then size of file on disk and that smaller epg.dat file is then loaded into RAM to be used. When the box is restarted or shutdown then a new optimized epg.dat file is overwriting the previous version.

The epg.dat file on disc is renamed, not erased when the EPG from RAM is saved. When the box crashes before restart/shutdown the epg.dat file is still on disc and on start it will be used to fill the EPG in RAM. This way you don't loose EPG data except for the EPG data that is in RAM. This because it is not written back to disk.

Above is general description how the EPG works and I updated that with the vacuum you are now doing. The EPG working is simple on the outside and works very well since a long time.

If you want, please correct any errors I made and add what I forgot. This could be used as a general bit of information for users who want to know how EPG is handled.

Looking forward to the EPGImport version so I test it also.

I am looking at the vacuum. When there is restart or shutdown the database is written to disc, removing so the outdated events.

However you can give events a grace period that they can stay in the database despite being outdated. The value is named "Keep outdated EPG" and is stated in hours.

On import (EPGimport) normally only events that or not outdated or falling in the grace period are stored. It might be that your version stores everything matching a station, and then a mark delete and vacuum is needed leaving the actual/future and the keep outdated events.

I am refering to how DreamOS EPG is working and was implemented in 2013/2014.

Progress is great BTW!

pepsik My DM920 does the same. I tested that this morning thinking that maybe the might be the Mipsel processor that might causing it.

Open for me is, having a clean install and only EPGimport installed to see if any other plug-ins interfere.

I know, and I am patient. For the time being I will have lean on 'crutches' untill I can walk again without them.