Recent changes to 325: Datetime fields with microsecond shows as None

#325 Datetime fields with microsecond shows as None

Dmitry T. — Thu, 06 Feb 2014 17:10:04 -0000

Hi,

I'm sorry, this my fail. I was inattentive when read your comment
Please try replace functions code in times.py. Working code below.

def DateTime_or_None(s):
    if ' ' in s:
        sep = ' '
    elif 'T' in s:
        sep = 'T'
    else:
        return Date_or_None(s)

    try:
        d, t = s.split(sep, 1)
        if '.' in t:
            t, ms = t.split('.',1)
            ms = ms.ljust(6, '0')
        else:
            ms = 0
        return datetime(*[ int(x) for x in d.split('-')+t.split(':')+[ms] ])
    except (SystemExit, KeyboardInterrupt):
        raise
    except:
        return Date_or_None(s)

def TimeDelta_or_None(s):
    try:
        h, m, s = s.split(':')
        if '.' in s:
            s, ms = s.split('.')
            ms = ms.ljust(6, '0')
        else:
            ms = 0
        h, m, s, ms = int(h), int(m), int(s), int(ms)
        td = timedelta(hours=abs(h), minutes=m, seconds=s,
                       microseconds=ms)
        if h < 0:
            return -td
        else:
            return td
    except ValueError:
        # unpacking or int/float conversion failed
        return None

#325 Datetime fields with microsecond shows as None

Dmitry T. — Wed, 05 Feb 2014 12:04:09 -0000

Please can you provide dataset sample for testing.
Try get sources of MySQLdb from https://github.com/farcepest/MySQLdb1
I have pushed changes to github repo.

#325 Datetime fields with microsecond shows as None

Mark Wilby — Wed, 05 Feb 2014 00:50:43 -0000

I just ran into this problem while using twisted.enterprise.adbapi. A simple query of datetime objects returned a None object an all records. The patch here did not fix the problem as it was in the DateTime_or_None method which is not modified.

I guess its already fixed in the next version, but I have posted this just in case. Again its a problem of handling the milliseconds. I made a slight change to this method, adding an if statement after the split and it seems to work.

def DateTime_or_None(s):
    if ' ' in s:
        sep = ' '
    elif 'T' in s:
        sep = 'T'
    else:
        return Date_or_None(s)

    try:
        d, t = s.split(sep, 1)
        if '.' in t:
            t = t.replace('.', ':')
        else:
            t = t + ':0'
        return datetime(*[ int(x) for x in d.split('-')+t.split(':') ])
    except (SystemExit, KeyboardInterrupt):
        raise
    except:
        return Date_or_None(s)

I think the string replace is not too costly and everything else is the same.

I apologise, I have limited expertise with Python and its tools, so due to my ignorance I am just posting the code change, not the patch.

#325 Datetime fields with microsecond shows as None

Dmitry T. — Mon, 15 Jul 2013 14:57:59 -0000

New patch for solve this problem. Without float numbers.

#325 Datetime fields with microsecond shows as None

Alexey Malashkevich — Fri, 14 Jun 2013 08:40:48 -0000

Agree, there is no need in float here. Below is one of the ways to fix the problem:

def DateTime_or_None(s):
    if not ' ' in s and 'T' in s:
        return Date_or_None(s)

    try:
        if len(s) == 23: s = s + '000'
        s = s.replace('-', ' ').replace(':', ' ').replace('.', ' ').replace('T', ' ')
        return datetime(*[ int(x) for x in s.split()])
    except (SystemExit, KeyboardInterrupt):
        raise
    except:
        return Date_or_None(s)

#325 Datetime fields with microsecond shows as None

Zmicier Zaleznicenka — Wed, 29 May 2013 15:08:06 -0000

Math module is used now and causes rounding errors. In my post I suggest using string method ljust instead of math.modf. Ljust is simpler and faster (i believe) than rounding the way you propose.

#325 Datetime fields with microsecond shows as None

Dmitry T. — Wed, 29 May 2013 15:03:56 -0000

Yep, and this problem exist.

May be I can provide more simply way and without math module.

>>> s = '50.991'
>>> s = float(s)
>>> int(s)
50
>>> int(round(s - int(s),6) * 1000000)
991000
>>>

#325 Datetime fields with microsecond shows as None

Zmicier Zaleznicenka — Thu, 28 Mar 2013 16:22:04 -0000

Also, if talking about these ms issues, while Time_Or_None() parses microseconds, rounding errors may appear. Instead, I suggest to use string methods.

This happens now:

>>> s='50.991'
>>> s = float(s)
>>> int(s)
50
>>>int(math.modf(s)[0]*1000000)
990999

This is what I think it should be:

>>> s='50.991'
>>> s,ms = s.split('.')
>>> int(s)
50
>>> int(ms.ljust(6,'0'))
991000

#325 Datetime fields with microsecond shows as None

Dmitry T. — Thu, 28 Mar 2013 15:52:56 -0000

I have tried in 1.2.4b4, result tuple have None if DateTime field contained microsecond.

Bug in DateTime_or_None() on times.py when func try to return value it can't split microsecond from time

return datetime(*[ int(x) for x in d.split('-')+t.split(':') ])

import datetime.datetime
d = '2013-03-28'
t = '11:11:11.000011'
datetime(*[ int(x) for x in d.split('-')+t.split(':') ])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '11.000011'

#325 Datetime fields with microsecond shows as None

Zmicier Zaleznicenka — Thu, 28 Mar 2013 14:49:52 -0000

The example works for the new versions of MySQLdb, but Date_Or_None() still parses datetimes with microseconds incorrectly, which leads to problems for clients using it directly (see i.e. https://code.djangoproject.com/ticket/19716). If it is possible, please make changes to Date_Or_None().