Datetime failures with MinGW (Trac #2108) #568

numpy-gitbot · 2012-10-19T15:08:32Z

Original ticket http://projects.scipy.org/numpy/ticket/2108 on 2012-04-16 by @rgommers, assigned to unknown.

These failures are hard to fix for MinGW 3.4.5. Until we can do everything with MinGW 4.x (including distributing binaries) we do have a problem.

numpy-gitbot · 2012-10-19T15:08:32Z

@rgommers wrote on 2012-04-16

It's not yet clear to me how much of datetime would be unusable on Windows if we would do a release right now. Can anyone comment on that? If it's a major part, then I think this is a blocker. If it's a minor issue, then we may decide to ignore it (there's an experimental label for a reason).

numpy-gitbot · 2012-10-19T15:08:32Z

@rgommers wrote on 2012-07-15

Latest output from our XP buildbot:

======================================================================
ERROR: test_datetime_arange (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_datetime.py", line 1351, in test_datetime_arange
    assert_raises(ValueError, np.arange, np.datetime64('today'),
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_datetime.py", line 1706, in test_datetime_y2038
    a = np.datetime64('2038-01-20T13:21:14')
OSError: Failed to use '_gmtime64_s' to convert to a UTC time

======================================================================
ERROR: test_pydatetime_creation (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_datetime.py", line 467, in test_pydatetime_creation
    a = np.array(['today', datetime.date.today()], dtype='M8[D]')
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_string_parser_variants (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_datetime.py", line 1054, in test_string_parser_variants
    assert_equal(np.array(['1980-02-29T01:02:03'], np.dtype('M8[s]')),
OSError: Failed to use '_gmtime64_s' to convert to a UTC time

======================================================================
ERROR: test_timedelta_scalar_construction_units (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_datetime.py", line 287, in test_timedelta_scalar_construction_units
    assert_equal(np.datetime64('2010-03-12T17').dtype,
OSError: Failed to use '_gmtime64_s' to convert to a UTC time

======================================================================
ERROR: Failure: OSError (Failed to use '_gmtime64_s' to convert to a UTC time)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\nose\loader.py", line 382, in loadTestsFromName
    addr.filename, addr.module)
  File "C:\Python26\lib\site-packages\nose\importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "C:\Python26\lib\site-packages\nose\importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_multiarray.py", line 916, in <module>
    class TestArgmax(TestCase):
  File "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_multiarray.py", line 938, in TestArgmax
    np.datetime64('1994-06-21T14:43:15'),
OSError: Failed to use '_gmtime64_s' to convert to a UTC time

numpy-gitbot · 2012-10-19T15:08:33Z

@rgommers wrote on 2012-07-15

Related tickets / PRs:

numpy-gitbot · 2012-10-19T15:08:33Z

@rgommers wrote on 2012-07-15

Last discussion on this: http://thread.gmane.org/gmane.comp.python.numeric.general/50907/focus=50918

certik · 2012-12-26T22:19:27Z

As of today (69c06da), here is how to reproduce it:

$ cd ~/repos/numpy  # numpy directory
$ wine "C:\Python27\python" setup.py build -c mingw32 install
$ cd ..
$ wine "C:\Python27\python"
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.test('full') 
Running unit tests for numpy
NumPy version 1.7.0rc1.dev-Unknown
NumPy is installed in C:\Python27\lib\site-packages\numpy
Python version 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]
nose version 1.1.2
...............S.........S..................E.........................................................................................................................................................................................................E.E..........................................................................................................................................................................................S.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K..................................................................................................................................................................................................................................................................1
.....................................................................................................................................................................................................................................................K..................................................................K..K..............................K...SK.S.......S..............................................................................SSSSSSSSSSSSSSSSSSS....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............K...............................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\numpy\core\tests\test_datetime.py", line 1748, in test_datetime_y2038
    assert_equal(str(a)[:-5], '2038-01-20T13:21:14')
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmax)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1289, in test_combinations
    assert_equal(np.argmax(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1357, in test_combinations
    assert_equal(np.argmin(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time

----------------------------------------------------------------------
Ran 4299 tests in 31.268s

FAILED (KNOWNFAIL=9, SKIP=26, errors=3)
<nose.result.TextTestResult run=4299 errors=3 failures=0>

certik · 2012-12-26T23:52:43Z

The problem seems to be in numpy/core/src/multiarray/datetime_strings.c in get_localtime() in this part of the code:

 #elif defined(NPY_MINGW_USE_CUSTOM_MSVCR)
    if (_localtime64_s(tms, ts) != 0) {
        func_name = "_localtime64_s";
        goto fail;
    }
 #else

where the _localtime64_s function returns nonzero exit status and then the get_localtime() function fails.

certik · 2012-12-27T01:12:39Z

Applying the following patch:

diff --git a/numpy/core/src/multiarray/datetime_strings.c b/numpy/core/src/multi
index 32009bb..e1c6130 100644
--- a/numpy/core/src/multiarray/datetime_strings.c
+++ b/numpy/core/src/multiarray/datetime_strings.c
@@ -47,14 +47,12 @@ get_localtime(NPY_TIME_T *ts, struct tm *tms)
         func_name = "localtime_s";
         goto fail;
     }
- #elif defined(NPY_MINGW_USE_CUSTOM_MSVCR)
-    if (_localtime64_s(tms, ts) != 0) {
-        func_name = "_localtime64_s";
-        goto fail;
-    }
  #else
     struct tm *tms_tmp;
+    printf("localtime1: %d\n", ts);
+    printf("localtime2: %d\n", *ts);
     tms_tmp = localtime(ts);
+    printf("localtime3: %d\n", tms_tmp == NULL);
     if (tms_tmp == NULL) {
         func_name = "localtime";
         goto fail;

I have found, that sometimes, the *ts is negative. In that case (only) the localtime function returns NULL.

I also noticed, that it doesn't matter if _localtime64_s or localtime is used. Moreover, I have noticed that in majority of times, *ts is positive and in that case, both _localtime64_s and localtime seem to be working fine. In the cases when *ts is negative, the _localtime64_s function returns error code 22 (EINVAL), which means "Invalid argument".

This all seems to suggest, that *ts must be positive, otherwise the localtime function fails. Now the big question is why does it sometimes get negative. It might be, that the actual numpy test is wrong and not supported on 32 bit platforms, or something like that.

certik · 2012-12-27T01:33:47Z

Here is a minimal example to reproduce the problem:

>>> np.datetime64('2005-01-04T03:14:12')
localtime1: 4389388
localtime2: 1104808440
localtime3: 0
numpy.datetime64('2005-01-04T03:14:12+0000')
>>> np.datetime64('2041-12-03T14:05:03')
localtime1: 4389388
localtime2: -2025274996
localtime3: 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: Failed to use 'localtime' to convert to a local time

With a little experimenting, it turns out that dates up to 2038 work, but not over 2038. Citing the reference:

_localtime64_s, which uses the __time64_t structure, allows dates to be expressed up through 23:59:59, December 31, 3000, coordinated universal time (UTC), whereas _localtime32_s represents dates through 03:14:07 January 19, 2038, UTC.

this suggests that for whatever reason, the wine that we use in Ubuntu is 32bit inside and thus the localtime does not allow dates over 2038. This is obvious. What is not obvious is why do we test for dates over 2038 in numpy? Or is this a bug in how datetime64 is implemented?

certik · 2012-12-27T01:54:34Z

The real problem is exposed by this C code:

#include <stdio.h>
#include <time.h>

int main ()
{
   time_t rawtime;
   struct tm *info;
   char buffer[80];

   //time( &rawtime );
   rawtime = -2025246196;
   printf("localtime2: %d\n", rawtime);

   info = localtime(&rawtime);
   if (info == NULL) {
      printf("sorry, it's null...\n");
   } else {
      printf("Current local time and date: %s", asctime(info));
   }

   return 0;
}

Which in the 32 bit Ubuntu (in Vagrant) prints

localtime2: -2025246196
Current local time and date: Sat Oct 28 15:36:44 1905

while in Wine under mingw in the same Ubuntu prints:

localtime2: -2025246196
sorry, it's null...

But if you use something positive (just uncomment the time( &rawtime ); line above and comment the next line), Ubuntu gives:

localtime2: 1356573204
Current local time and date: Thu Dec 27 01:53:24 2012

and Wine gives

localtime2: 1356573206
Current local time and date: Thu Dec 27 01:53:26 2012

So for positive values, localtime works identically in both Ubuntu and Wine. For negative times, it only works in Ubuntu.

certik · 2012-12-27T02:14:38Z

I think this problem was introduced by b1e5fa0 in the convert_datetimestruct_utc_to_local function. That patch assumes, that after some "pre-processing", the localtime function will work (on all platforms), but it doesn't in Wine.

certik · 2012-12-27T03:05:02Z

From this, it follows, that if time_t is 32 bit (sizeof(time_t) == 4), then it suffers from the Y2038 problem. If it is 64 bit, then it can represent billion of years, so there is no problem.

charris · 2012-12-27T03:16:35Z

On Wed, Dec 26, 2012 at 8:05 PM, Ondřej Čertík [email protected]:

From this http://www.devx.com/tips/Tip/13129, it follows, that if time_tis 32 bit (sizeof(time_t)
== 4), then it suffers from the Y2038http://en.wikipedia.org/wiki/Year_2038_problemproblem. If it is 64 bit, then it can represent billion of years, so there
is no problem.

IIRC, this was discussed back in the datetime threads in the summer of
2011. Datetime on windows was the main blocker last January...

Chuck

certik · 2012-12-27T03:45:01Z

I am still digging into this, I really need to get at the bottom of this datetime + time_t issue first, to really understand what is going on here. I've already found at least part of the real problem. In the patch 1ef79b7 from Feb 19, 2012 Mark (correctly) introduced a fix not to call localtime() for dates < 1970, because on Windows this does not work, as I independently found out above.

However, as I found out above, the localtime is still being called with "negative" values on 32 bits, that is, with values over 2038. So something does not work here.

I strongly suspect the hack introduced previously in b1e5fa0 from Sep 30, 2011 (just search for the word "HACK" in that patch), either it's doing something wrong, or it works but doesn't come into effect, so the years roll over on 32 bits, and then localtime() fails. I am still digging into this.

charris · 2012-12-27T04:04:22Z

On Wed, Dec 26, 2012 at 8:45 PM, Ondřej Čertík [email protected]:

I am still digging into this, I really need to get at the bottom of this
datetime + time_t issue first, to really understand what is going on here.
I've already found at least part of the real problem. In the patch 1ef79b71ef79b71e18e6ab00002447128f617003b05af29from Feb 19, 2012 Mark (correctly) introduced a fix
not to call localtime() for dates < 1970, because on Windows this does
not work, as I independently found out above.

However, as I found out above, the localtime is still being called with
"negative" values on 32 bits, that is, with values over 2038. So something
does not work here.

I strongly suspect the hack introduced previously in b1e5fa0b1e5fa05ad4803856ec3272a900707eaf4a7f92efrom Sep 30, 2011 (just search for the word "HACK" in that patch), either
it's doing something wrong, or it works but doesn't come into effect, so
the years roll over on 32 bits, and then localtime() fails. I am still
digging into this.

Yes, it's a mess. Linus has made some, well, Linus like remarks on the
time_t structure and, IIRC, essentially said that it was a case where POSIX
conformity needed to be discarded, that 64 bits was the only sane choice
for the time. Also, I think part of the difficulty was that the mingw
library didn't expose the needed windows alternative that would solve the
problem. It's all sort of hazy now, perhaps Ralph or David would remember
more of that discussion.

Chuck

certik · 2012-12-27T04:17:02Z

Yep, I read pretty much everything that was written on this issue, in numpy, Linus and mingw.

I will keep posting random stuff here as I learn things. Then I'll look at all this tomorrow with fresh eyes, I feel I am close to nail this down.

I found out, that in Wine, sizeof(NPY_TIME_T) == 8, which would be consistent with the line at the top of datetime_strings.c:

/* Platform-specific time_t typedef */
#if defined(NPY_MINGW_USE_CUSTOM_MSVCR)
 typedef __time64_t NPY_TIME_T;
#else
 typedef time_t NPY_TIME_T;
#endif

At the same time sizeof(time_t) == 4 in my Wine.
It is now clear that the "HACK" is not executed, because sizeof(NPY_TIME_T) == 8. So we can ignore it for now, that's not the problem.

I printed the NPY_TIME_T *ts variable in the get_localtime using:

    printf("ts = %d\n", *ts);
    printf("ts = %ld\n", *ts);
    printf("ts = %lld\n", *ts);

and I got:

ts = -2025274996
ts = -2025274996
ts = -2025274996

when I executed

$ wine "C:\Python27\python" -c "import numpy; print numpy.datetime64('2041-12-03T14:05:03')"

I am still confused whether the ts variable actually contains a negative number (in which case neither _localtime64_s or localtime works), or whether it contains a positive number larger > 2038 (i.e. 2041 per my command above) and somehow _localtime64_s can't handle dates over 2038, due to a bug in mingw. This still needs to be clarified, I'll try to use a simple C program. But from the above, it seems that ts is actually a negative number, due to some 32/64 bit conversion bug earlier in numpy.

Also, I need to test how numpy behaves in 32 bit Ubuntu directly (i.e. not in Wine).

certik · 2012-12-27T04:26:56Z

I found it!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Here is a fix:

diff --git a/numpy/core/src/multiarray/datetime_strings.c b/numpy/core/src/multi
index 32009bb..794f13b 100644
--- a/numpy/core/src/multiarray/datetime_strings.c
+++ b/numpy/core/src/multiarray/datetime_strings.c
@@ -48,10 +48,16 @@ get_localtime(NPY_TIME_T *ts, struct tm *tms)
         goto fail;
     }
  #elif defined(NPY_MINGW_USE_CUSTOM_MSVCR)
+    printf("sizeof(NPY_TIME_T) = %d\n", sizeof(NPY_TIME_T));
+    printf("ts = %d\n", *ts);
+    printf("ts = %ld\n", *ts);
+    printf("ts = %lld\n", *ts);
     if (_localtime64_s(tms, ts) != 0) {
+       printf("fail\n");
         func_name = "_localtime64_s";
         goto fail;
     }
+    printf("ok\n");
  #else
     struct tm *tms_tmp;
     tms_tmp = localtime(ts);
@@ -189,7 +195,7 @@ convert_datetimestruct_utc_to_local(npy_datetimestruct *out_
      * we drop the seconds value from the npy_datetimestruct, everything
      * is ok for this operation.
      */
-    rawtime = (time_t)get_datetimestruct_days(out_dts_local) * 24 * 60 * 60;
+    rawtime = (NPY_TIME_T)get_datetimestruct_days(out_dts_local) * 24 * 60 * 60
     rawtime += dts_utc->hour * 60 * 60;
     rawtime += dts_utc->min * 60;

You can ignore the print statements, they are there for clarification. Without the time_t -> NPY_TIME_T part:

$ wine "C:\Python27\python" -c "import numpy; print numpy.datetime64('2041-12-03T14:05:03')"
sizeof(NPY_TIME_T) = 8
ts = -2025274996
ts = -2025274996
ts = -2025274996
fail
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: Failed to use '_localtime64_s' to convert to a local time

and with it:

$ wine "C:\Python27\python" -c "import numpy; print numpy.datetime64('2041-12-03T14:05:03')"
sizeof(NPY_TIME_T) = 8
ts = -2025274996
ts = -2025274996
ts = 2269692300
ok
2041-12-03T14:05:03+0000

That's it. This was typing a 64 bit integer into a 32 bit one, thus rolling over, becoming a negative number, and then when you pass a negative number into a 64 bit localtime(), it fails on Windows... I would bet that on 32 bit linux, it returns something like the year 1905, so we have a bug there as well, I am surprised that nobody noticed this before...

All clear now.

The rest should follow quite straightforwardly now. But why do we need the to use _localtime64_s in Mingw in the first place? Why not to use localtime()? It would simplify things, and if we remove the NPY_TIME_T, then actually things will start working. So the real bug was actually introduced in this patch: 5d0e0aa

certik · 2012-12-27T04:31:55Z

Ok, actually 1/2 of the bug was introduced in the patch b1e5fa0, because it introduced a macro NPY_TIME_T and made it equal to time_t, but did not use it everywhere in the code. Then in the patch 5d0e0aa just the macro NPY_TIME_T was changed to __time64_t in Wine, thinking (rightfully so!) that it would change it everywhere in the code (but it didn't due to the bug in the first patch). Nevertheless, this was the other 1/2 of the bug.

certik · 2012-12-27T04:41:51Z

I just posted a PR #2856 that fixes this issue. Now I get:

$ wine "C:\Python27\python" -c "import numpy; numpy.test('full')"
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
err:systray:initialize_systray Could not create tray window
Running unit tests for numpy
NumPy version 1.7.0rc1.dev-Unknown
NumPy is installed in C:\Python27\lib\site-packages\numpy
Python version 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]
nose version 1.1.2
...............S.........S.........................................................................................................................................................................................................................................................................................................................................................................................................................S.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K..................................................................................................................................................................................................................................................................1
.....................................................................................................................................................................................................................................................K..................................................................K..K..............................K...SK.S.......S..............................................................................SSSSSSSSSSSSSSSSSSS....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............K...............................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
----------------------------------------------------------------------
Ran 4299 tests in 30.274s

OK (KNOWNFAIL=9, SKIP=26)

Yes!!!

I started working on this at 6:30am and finished 8:40pm, so good 14 hours... This was definitely a bug. If we are lucky, it actually fixes everything. But this datetime/time_t issue is a horrible mess... Anyway, I am going to bed now. Feel free to merge the PR if tests pass.

Previously, two (critical) parts of the code used `time_t` instead of `NPY_TIME_T`. Due to the fact, that most of the time `NPY_TIME_T` was equal to `time_t`, this bug didn't show up. But in mingw, `NPY_TIME_T` is actually equal to `__time64_t` and then this causes 64 bit integers to be cast into 32 bit integers (thus becoming negative), which causes localtime() to fail in mingw. Fixes numpygh-568.

certik · 2012-12-27T16:27:46Z

As Chuck suggested, I am reopening this issue, until we carefully verify that things are fixed. Also I want to better document what is going on.

After understanding the issues with datetime (see numpygh-568), I have documented my understanding as comments in the code, so that the next time there are problems with datetime on some platform, one can quickly figure out what is going on from the code + comments.

certik · 2012-12-27T17:37:45Z

I've documented my current understanding in #2858. After that is in, I think we can close this issue.

rgommers · 2012-12-27T20:38:41Z

Tested it on my MinGW install, works like a charm with both Python 2.6 and 3.2. Thanks Ondrej! Will now have a look if I understand your changes and documentation.

certik · 2012-12-27T23:21:09Z

Thanks Ralf!

After understanding the issues with datetime (see numpygh-568), I have documented my understanding as comments in the code, so that the next time there are problems with datetime on some platform, one can quickly figure out what is going on from the code + comments.

certik mentioned this issue Dec 27, 2012

FIX: Use the NPY_TIME_T macro everywhere #2856

Merged

certik closed this as completed in 15022cb Dec 27, 2012

certik mentioned this issue Dec 27, 2012

FIX: Use the NPY_TIME_T macro everywhere #2857

Merged

certik reopened this Dec 27, 2012

certik mentioned this issue Dec 27, 2012

DOC: Thoroughly document datetime issues #2858

Merged

rgommers closed this as completed Dec 27, 2012

Uh oh!

Datetime failures with MinGW (Trac #2108) #568

Datetime failures with MinGW (Trac #2108) #568

Comments

numpy-gitbot commented Oct 19, 2012

numpy-gitbot commented Oct 19, 2012

Uh oh!

numpy-gitbot commented Oct 19, 2012

Uh oh!

numpy-gitbot commented Oct 19, 2012

Uh oh!

numpy-gitbot commented Oct 19, 2012

Uh oh!

certik commented Dec 26, 2012

Uh oh!

certik commented Dec 26, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

charris commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

charris commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!

rgommers commented Dec 27, 2012

Uh oh!

certik commented Dec 27, 2012

Uh oh!