EPG translation script - an open project - help required.

My code slows down considerably adding in title. And that not even translating sub-title. Is sub-title actual used on enigma2?

Maybe best for Chris230291 to post a version of your full code. It seems faster than mine. Although mine is quickish if you just want to translate the description.

No idea what's used on e2.

I think most people would only really want the title translating.

I did post the full code, try it.

Just set proxies to None

The time I posted (0h 6m 58s) is to translate the complete "basic" guide.

I think there is a rate limit.

Nothing is getting translated now without a proxy.

With these settings:

#####################################
# CONFIG##############################
desired_language = "en"
translate_titles = True
translate_sub_titles = False
translate_descriptions = False
proxies = {"http": "192.168.1.200:8118", "https": "192.168.1.200:8118"}
workers = 40
days_to_translate = 1
#####################################

It translates the basic sky de epg in 0h 1m 49s

I also spotted an error in my if statement that decides if we want to translate it or not.

Here's the revised script:

import xml.etree.ElementTree as ET
import concurrent.futures
from deep_translator import GoogleTranslator
from threading import Lock
from datetime import datetime

#####################################
# CONFIG##############################
desired_language = "en"
translate_titles = True
translate_sub_titles = False
translate_descriptions = False
proxies = {"http": "192.168.1.200:8118", "https": "192.168.1.200:8118"}
workers = 40
days_to_translate = 1
#####################################


def translate_programe(programme):
    def translate_attribute(attribute):
        element = programme.find(attribute)
        if element != None:
            text = element.text
            lang = element.get("lang", "auto")
            translation = translation_cache.get(text)
            if translation is None and text is not None and lang != desired_language:
                translation = GoogleTranslator(
                    source=lang, target=desired_language, proxies=proxies
                ).translate(text)
            if translation:
                ET.SubElement(programme, attribute, lang=desired_language).text = str(
                    translation
                )
                with lock:
                    translation_cache[text] = translation

    programme_stop = datetime.strptime(programme.get("stop"), "%Y%m%d%H%M%S %z")
    if (
        days_to_translate == 0
        or (programme_stop.date() >= today and (programme_stop.date() - today).days <= days_to_translate)
    ):

        if translate_titles:
            translate_attribute("title")

        if translate_sub_titles:
            translate_attribute("sub-title")

        if translate_descriptions:
            translate_attribute("desc")


start_time = datetime.now()
today = start_time.date()
translation_cache = {}
lock = Lock()

tree = ET.parse("tv.xmltv")
root = tree.getroot()
programmes = root.findall("programme")

with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
    executor.map(translate_programe, programmes)

ET.indent(tree, space="\t", level=0)
tree.write("translated_tv.xmltv", encoding="utf-8", xml_declaration=True)

elapsed_time = datetime.now() - start_time
hours, remainder = elapsed_time.seconds // 3600, elapsed_time.seconds % 3600
minutes, seconds = remainder // 60, remainder % 60
print(f"Completed in: {hours}h {minutes}m {seconds}s")

EDIT: I also suggest something like this to combine all guides into 1, before translating.

This way it doesn't really save anything to disk and I think it's nicer.

If there's a reason you can't do that on e2, then that's fine, just an idea.

import requests
import lzma
import xml.etree.ElementTree as ET

urls = [
    "http://epgspot.com/rytec_epg/rytecDE_Basic.xz",
    "http://epgspot.com/rytec_epg/rytecDE_OtherMovies.xz",
    "http://epgspot.com/rytec_epg/rytecCH_Basic.xz",
    "http://epgspot.com/rytec_epg/rytecDE_Common.xz",
    "http://epgspot.com/rytec_epg/rytecDE_SportMovies.xz"
]

combined_root = ET.Element("tv")
combined_tree = ET.ElementTree(combined_root)
channels = []
programmes = []

for url in urls:
    try:
        response = requests.get(url)
        data = lzma.decompress(response.content)
        tree = ET.ElementTree(ET.fromstring(data))
        root = tree.getroot()
        channels.extend(root.findall("channel"))
        programmes.extend(root.findall("programme"))
    except:
        pass

for channel in channels:
    combined_root.append(channel)

for programme in programmes:
    combined_root.append(programme)

ET.indent(combined_tree, space="\t", level=0)
combined_tree.write("combined_tv.xmltv", encoding="utf-8", xml_declaration=True)

Parsing the combined guide took me 0h 3m 59s to translate the titles for 1 day.

i dont know how fast your computer is - mine isn't slow. But your original code (chris) is killing my cpu. maxing out 100%

1 url.

pasted-from-clipboard.png

Does someone want to double-check this, perhaps actually on a device?

The output from my scripts (translating titles only for 1 day 3 mins 49 secs)

combined_tv.xmltv

ZZzz.. time for me

Quote from jenseneverest

Code

.According to google, you are allowed to make 5 requests per secondand up to 200k requests per day. You can wait and try again later oryou can try the translate_batch function

As always its taking me a while to digest what you guys have posted,

im currently trying to sort a decent proxie list out (back to the IPTV hacking days lol )

I use TOR.

Not sure if there's a precompiled version for your box or if you would have to compile it.

I actually use this docker image: https://github.com/dperson/torproxy

Quote from jenseneverest

Code

.According to google, you are allowed to make 5 requests per secondand up to 200k requests per day. You can wait and try again later oryou can try the translate_batch function

As always its taking me a while to digest what you guys have posted,

im currently trying to sort a decent proxie list out (back to the IPTV hacking days lol )

well i was running my examples non stop for about 3 hours yesterday. Personally I don't think I got blocked anywhere.

I am following this thread from start, the project looks very interesting and from the input of you all, it looks more promising.

In my personal openion, offline translation with cache is the best option for several reasons already mentioned in line and online should come next in priority. Maybe I am wrong.

Anyway, I think basic users like me, with zero coding background, will be happy if all valueble controputions come together in one scrip or plugin....

Do you think this is possible at this stage?

Regards.

Just ask if you need help.

It's really very simple.

The problem with offline translators is... last time I checked, they sucked.

Slow and simply not as good as google.

Google tends to do a very good job at not translating literally and instead giving the answer you are actually looking for.

I.e. giving you the regional name of a show instead of a simple translation.

And regarding ChatGPT.

It's really good, but it also tends to overcomplicate things.

You are better off breaking down your questions into individual tasks rather than “make the thing”.