Enabled imdb_list to cache credentials per login name #1209

liiight · 2016-05-26T12:03:01Z

Motivation for changes:

IMDB blocks login attempts after a certain number, thus preventing using imdb_list.
By caching login cookies, user ID & list ID, we can directly retrieve IMDB list without having to re-login, circumventing the issue

Detailed changes:

~~Using SimplePersistence to create dict with the following data: KEY is config['login'], JSON value is {"watchlist": "ls032578336", "cookies": "cookie_dict, "user_id": "ur65953086"}~~ Using dedicated tables
Multiple logins are cached
Multiple list IDs per login are cached

Addressed issues:

Fixes imdb login failure #1109

Log and/or tests output (preferably both):

First run:

C:\Python27\python.exe C:/Flexget/flexget_vanilla.py -L debug execute --tasks imdb_list_test
2016-05-26 14:58 DEBUG    manager                       Figuring out config load paths
2016-05-26 14:58 DEBUG    manager                       Found config: C:\Flexget\config.yml
2016-05-26 14:58 DEBUG    manager                       Config file C:\Flexget\config.yml selected
2016-05-26 14:58 DEBUG    manager                       sys.defaultencoding: ascii
2016-05-26 14:58 DEBUG    manager                       sys.getfilesystemencoding: mbcs
2016-05-26 14:58 DEBUG    manager                       os.path.supports_unicode_filenames: True
2016-05-26 14:58 DEBUG    plugin                        Trying to load plugins from: [u'C:\\Flexget\\plugins', 'C:\\Flexget\\flexget\\plugins']
2016-05-26 14:58 DEBUG    safe                          Reading from cache file c:\users\orc\appdata\local\temp\safe-0.4.words.cache
2016-05-26 14:58 DEBUG    plugin                        Plugin `memusage` requires `ext lib `guppy`` to load.
2016-05-26 14:58 DEBUG    deluge                        Looking for deluge install in C:\Program Files (x86)\Deluge
2016-05-26 14:58 DEBUG    plugin                        Plugins took 4.73 seconds to load
2016-05-26 14:58 DEBUG    manager                       Connecting to: sqlite:///C:\\Flexget\\db-config.sqlite
2016-05-26 14:58 DEBUG    manager                       config_name: config
2016-05-26 14:58 DEBUG    manager                       config_base: C:\Flexget
2016-05-26 14:58 DEBUG    manager                       New config data loaded.
2016-05-26 14:58 DEBUG    schema                        current flexget version already exist in db 2.0.34.dev
2016-05-26 14:58 DEBUG    parsing                       setting default movie parser to internal. (options: {u'internal': <flexget.plugins.parsers.parser_internal.ParserInternal object at 0x0559EDB0>, u'guessit': <flexget.plugins.parsers.parser_guessit.ParserGuessit object at 0x05B6B1F0>})
2016-05-26 14:58 DEBUG    parsing                       setting default series parser to internal. (options: {u'internal': <flexget.plugins.parsers.parser_internal.ParserInternal object at 0x0559EDB0>, u'guessit': <flexget.plugins.parsers.parser_guessit.ParserGuessit object at 0x05B6B1F0>})
2016-05-26 14:58 DEBUG    cron_env                      Encoding mbcs stored
2016-05-26 14:58 DEBUG    util.simple_persistence                 setting key terminal_encoding value 'mbcs'
2016-05-26 14:58 DEBUG    task_queue                    task queue shutdown requested
2016-05-26 14:58 INFO     ipc.rpyc                      server started on [127.0.0.1]:64581
2016-05-26 14:58 VERBOSE  task_queue                    There are 1 tasks to execute. Shutdown will commence when they have completed.
2016-05-26 14:58 DEBUG    task          imdb_list_test  executing imdb_list_test
2016-05-26 14:58 DEBUG    disable       imdb_list_test  Disabled built-in plugin(s): seen
2016-05-26 14:58 DEBUG    imdb_list     imdb_list_test  no credentials saved for login name [email protected]
2016-05-26 14:58 DEBUG    util.simple_persistence imdb_list_test  setting key [email protected] value {}
2016-05-26 14:58 DEBUG    utils.requests imdb_list_test  Fetching https://www.imdb.com/ap/signin?openid.return_to=https%3A%2F%2Fwww.imdb.com%2Fap-signin-handler&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=imdb_mobile_us&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0
2016-05-26 14:58 DEBUG    imdb_list     imdb_list_test  [email protected], password=flexget16
2016-05-26 14:58 VERBOSE  utils.requests imdb_list_test  Waiting 3.61 seconds until next request to imdb.com
2016-05-26 14:58 DEBUG    utils.requests imdb_list_test  Fetching https://www.imdb.com/ap/signin
2016-05-26 14:58 VERBOSE  imdb_list     imdb_list_test  successfully logged in, saving credentials
2016-05-26 14:58 DEBUG    util.simple_persistence imdb_list_test  setting key [email protected] value {u'cookies': {'session-id': '739-2237622-8151272', 'uu': 'BCYqm4R0gn4V0W1rWN1HguSTD1N34Lu6R_H2F8u47bUismJzBYukOGCGdoOjDWsv32gxol3OFqIj%0D%0ACInHPHRD0equE4pE2yDAOswAy1Tng2NpvV71ELKBz1FR078ghSNqledaRPtW2UDTfNpm3dGaXjSc%0D%0Asrq3qVv1fxzc9EO-68-zfog9vlIaaBTuiIOIuNIobw0HVi7VvVJ57qfZDZmQSmYj2BdnZPumFZZV%0D%0Aw01qXnygS8wDeGpjXwNkr2hp-rXez8bGX8hvJI-LnTNuM8w4gjuwgmBnBjI6_3F7W31oA2CRY2Y%0D%0A', 'sid': 'BCYj-CHhsAGmDTiQiHGKbb_cut6v-xvwoCMMwVxtqWxgomqGlwcTGOUQ3w9UbS4YSRJdmLKb5zMC%0D%0A-vvDU-3c-7Q57A8QLupvaK1FcrMwonU1H_M%0D%0A', 'id': 'BCYk_Vxy7mHOrJ234l4-jkqdhqMRWcR909aidju1Vl4eJIZ8M4v-A20gb8XYJBsRu3RbDviyF7MY%0D%0AtT3_mlNy3gUUW-jNSB20DPobp5T4CWoRGfE9YsPZLK416zQxURHTiDJGX0Gbqw6Mofif4Pkvwbd4%0D%0Aj10mwj9M1Uva_GTGBkYTKzeR6PFt3-GTQTWEYQTTlLI1%0D%0A', 'session-id-time': '1621943935'}}
2016-05-26 14:58 VERBOSE  utils.requests imdb_list_test  Waiting 4.39 seconds until next request to imdb.com
2016-05-26 14:58 DEBUG    utils.requests imdb_list_test  Fetching http://www.imdb.com/profile
2016-05-26 14:59 DEBUG    imdb_list     imdb_list_test  found user_id ur65953086
2016-05-26 14:59 VERBOSE  utils.requests imdb_list_test  Waiting 4.61 seconds until next request to imdb.com
2016-05-26 14:59 DEBUG    utils.requests imdb_list_test  Fetching http://www.imdb.com/list/_ajax/watchlist_has
2016-05-26 14:59 DEBUG    imdb_list     imdb_list_test  retrieved list_id ls032578336
2016-05-26 14:59 VERBOSE  utils.requests imdb_list_test  Waiting 4.66 seconds until next request to imdb.com
2016-05-26 14:59 DEBUG    utils.requests imdb_list_test  Fetching http://www.imdb.com/list/export?list_id=ls032578336&author_id=ur65953086
2016-05-26 14:59 DEBUG    imdb_list     imdb_list_test  parsing line from csv: 1, tt0133093, Thu May 26 00:22:37 2016, Thu May 26 00:22:37 2016, , The Matrix, Feature Film, The Wachowski Brothers, The Wachowski Brothers, , 8.7, 136, 1999, action, sci_fi, 1194354, 1999-03-24, http://www.imdb.com/title/tt0133093/
2016-05-26 14:59 DEBUG    backlog       imdb_list_test  0 entries purged from backlog
2016-05-26 14:59 VERBOSE  details       imdb_list_test  Produced 1 entries.
2016-05-26 14:59 VERBOSE  task          imdb_list_test  ACCEPTED: `The Matrix (1999)` by accept_all plugin
2016-05-26 14:59 DEBUG    urlrewriter   imdb_list_test  Checking 1 entries
2016-05-26 14:59 VERBOSE  details       imdb_list_test  Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2016-05-26 14:59 VERBOSE  list_add      imdb_list_test  adding accepted entries into movie_list - movie_test
2016-05-26 14:59 DEBUG    movie_list    imdb_list_test  trying to match movie based off id imdb_id: tt0133093
2016-05-26 14:59 DEBUG    movie_list    imdb_list_test  found movie <MovieListMovie title=The Matrix,year=1999,list_id=2>
2016-05-26 14:59 DEBUG    movie_list    imdb_list_test  adding entry <Entry(title=The Matrix (1999),state=accepted)>
2016-05-26 14:59 DEBUG    disable       imdb_list_test  Re-enabled builtin plugin(s): seen
2016-05-26 14:59 DEBUG    util.simple_persistence imdb_list_test  Flushing simple persistence for task imdb_list_test to db.
2016-05-26 14:59 DEBUG    task_queue                    task queue shut down
2016-05-26 14:59 INFO     ipc.rpyc                      listener closed
2016-05-26 14:59 DEBUG    util.simple_persistence                 Flushing simple persistence for task None to db.
2016-05-26 14:59 DEBUG    manager                       Shutting down
2016-05-26 14:59 DEBUG    manager                       Removed C:\Flexget\.config-lock

Process finished with exit code 0

2nd run:

C:\Python27\python.exe C:/Flexget/flexget_vanilla.py -L debug execute --tasks imdb_list_test
2016-05-26 15:01 DEBUG    manager                       Figuring out config load paths
2016-05-26 15:01 DEBUG    manager                       Found config: C:\Flexget\config.yml
2016-05-26 15:01 DEBUG    manager                       Config file C:\Flexget\config.yml selected
2016-05-26 15:01 DEBUG    manager                       sys.defaultencoding: ascii
2016-05-26 15:01 DEBUG    manager                       sys.getfilesystemencoding: mbcs
2016-05-26 15:01 DEBUG    manager                       os.path.supports_unicode_filenames: True
2016-05-26 15:01 DEBUG    plugin                        Trying to load plugins from: [u'C:\\Flexget\\plugins', 'C:\\Flexget\\flexget\\plugins']
2016-05-26 15:01 DEBUG    safe                          Reading from cache file c:\users\orc\appdata\local\temp\safe-0.4.words.cache
2016-05-26 15:01 DEBUG    plugin                        Plugin `memusage` requires `ext lib `guppy`` to load.
2016-05-26 15:02 DEBUG    deluge                        Looking for deluge install in C:\Program Files (x86)\Deluge
2016-05-26 15:02 DEBUG    plugin                        Plugins took 4.40 seconds to load
2016-05-26 15:02 DEBUG    manager                       Connecting to: sqlite:///C:\\Flexget\\db-config.sqlite
2016-05-26 15:02 DEBUG    manager                       config_name: config
2016-05-26 15:02 DEBUG    manager                       config_base: C:\Flexget
2016-05-26 15:02 DEBUG    manager                       New config data loaded.
2016-05-26 15:02 DEBUG    schema                        current flexget version already exist in db 2.0.34.dev
2016-05-26 15:02 DEBUG    parsing                       setting default movie parser to internal. (options: {u'internal': <flexget.plugins.parsers.parser_internal.ParserInternal object at 0x0565EDB0>, u'guessit': <flexget.plugins.parsers.parser_guessit.ParserGuessit object at 0x05C2B1F0>})
2016-05-26 15:02 DEBUG    parsing                       setting default series parser to internal. (options: {u'internal': <flexget.plugins.parsers.parser_internal.ParserInternal object at 0x0565EDB0>, u'guessit': <flexget.plugins.parsers.parser_guessit.ParserGuessit object at 0x05C2B1F0>})
2016-05-26 15:02 DEBUG    cron_env                      Encoding mbcs stored
2016-05-26 15:02 DEBUG    util.simple_persistence                 setting key terminal_encoding value 'mbcs'
2016-05-26 15:02 DEBUG    task_queue                    task queue shutdown requested
2016-05-26 15:02 INFO     ipc.rpyc                      server started on [127.0.0.1]:64600
2016-05-26 15:02 VERBOSE  task_queue                    There are 1 tasks to execute. Shutdown will commence when they have completed.
2016-05-26 15:02 DEBUG    task          imdb_list_test  executing imdb_list_test
2016-05-26 15:02 DEBUG    disable       imdb_list_test  Disabled built-in plugin(s): seen
2016-05-26 15:02 VERBOSE  imdb_list     imdb_list_test  found stored credentials for login [email protected]
2016-05-26 15:02 DEBUG    utils.requests imdb_list_test  Fetching http://www.imdb.com/list/export?list_id=ls032578336&author_id=ur65953086
2016-05-26 15:02 DEBUG    imdb_list     imdb_list_test  parsing line from csv: 1, tt0133093, Thu May 26 00:22:37 2016, Thu May 26 00:22:37 2016, , The Matrix, Feature Film, The Wachowski Brothers, The Wachowski Brothers, , 8.7, 136, 1999, action, sci_fi, 1194354, 1999-03-24, http://www.imdb.com/title/tt0133093/
2016-05-26 15:02 DEBUG    backlog       imdb_list_test  0 entries purged from backlog
2016-05-26 15:02 VERBOSE  details       imdb_list_test  Produced 1 entries.
2016-05-26 15:02 VERBOSE  task          imdb_list_test  ACCEPTED: `The Matrix (1999)` by accept_all plugin
2016-05-26 15:02 DEBUG    urlrewriter   imdb_list_test  Checking 1 entries
2016-05-26 15:02 VERBOSE  details       imdb_list_test  Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2016-05-26 15:02 VERBOSE  list_add      imdb_list_test  adding accepted entries into movie_list - movie_test
2016-05-26 15:02 DEBUG    movie_list    imdb_list_test  trying to match movie based off id imdb_id: tt0133093
2016-05-26 15:02 DEBUG    movie_list    imdb_list_test  found movie <MovieListMovie title=The Matrix,year=1999,list_id=2>
2016-05-26 15:02 DEBUG    movie_list    imdb_list_test  adding entry <Entry(title=The Matrix (1999),state=accepted)>
2016-05-26 15:02 DEBUG    disable       imdb_list_test  Re-enabled builtin plugin(s): seen
2016-05-26 15:02 DEBUG    util.simple_persistence imdb_list_test  Flushing simple persistence for task imdb_list_test to db.
2016-05-26 15:02 DEBUG    task_queue                    task queue shut down
2016-05-26 15:02 INFO     ipc.rpyc                      listener closed
2016-05-26 15:02 DEBUG    util.simple_persistence                 Flushing simple persistence for task None to db.
2016-05-26 15:02 DEBUG    manager                       Shutting down
2016-05-26 15:02 DEBUG    manager                       Removed C:\Flexget\.config-lock

Process finished with exit code 0

To Do:

Absolutely verify that this in fact solves the locking out issue (could not get locked out with this, perhaps more testing is required).
I am unclear about cookie expiration, and what will happen. The current implementation does not handle that, and this will have to be addressed.
Perhaps pushing the meaning of "SimplePersistence" to its edge. Maybe a proper table would be more suitable here. My reasoning was that while this ALLOWS using multiple login names and lists, the typical use case of this plugin will not require that. No longer relevant, using dedicated tables.

…b_list_cache

- Redid cassettes

liiight added 2 commits May 26, 2016 14:48

Enabled imdb_list to cache credentials per login name

0704a43

Forgot to use class value for cookies

7574890

liiight mentioned this pull request May 26, 2016

imdb login failure #1109

Closed

liiight added 14 commits May 29, 2016 14:34

Refactored plugin to use table and not persistence

55106e5

Fixed list_id caching

00655c5

Merge branch 'develop' of https://github.com/Flexget/Flexget into imd…

243b7b1

…b_list_cache

Correctly accessed session

03e08e5

Enabled imdb_list to cache credentials per login name

efd2720

Forgot to use class value for cookies

aa97aa5

Refactored plugin to use table and not persistence

9d6aadb

Fixed list_id caching

afa8831

Correctly accessed session

6b648e6

Added cached credentials test

ee00d65

Correctly pass cookies to all methods

e73adf8

- Changed user_id column type to string

31c0cc7

- Redid cassettes

Redid cassettes, again

36633e5

Reredid cassettes

7330073

liiight merged commit ef91017 into Flexget:develop May 30, 2016

liiight deleted the imdb_list_cache branch May 30, 2016 11:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enabled imdb_list to cache credentials per login name #1209

Enabled imdb_list to cache credentials per login name #1209

Uh oh!

liiight commented May 26, 2016 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Enabled imdb_list to cache credentials per login name #1209

Enabled imdb_list to cache credentials per login name #1209

Uh oh!

Conversation

liiight commented May 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for changes:

Detailed changes:

Addressed issues:

Log and/or tests output (preferably both):

To Do:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

liiight commented May 26, 2016 •

edited

Loading