[Cache] Add ability to configure predis client. #28175

jralph · 2018-08-09T12:53:18Z

Q	A
Branch?	master
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets
License	MIT
Doc PR

When using predis/predis, there is currently no way to set any options as the 2nd parameter in the Predis\Client constructor. This merge allows you to define these options with a predis_options query parameter in the dsn.

Example:

default_redis_provider: 'redis://localhost:6379?predis_options[cluster]=redis&predis_options[profile]=2.8'

nicolas-grekas · 2018-08-09T20:40:09Z

Hello, thanks for the PR. Here is the options reference:
https://github.com/nrk/predis/wiki/Client-Options

To understand better, which options are useful in your case?
E.g. "profile" is not needed in the context of a cache pool.
For "cluster", is this the canonical way to connect to a cluster? From the doc "redis" is not a legal value, unless I missed something. I don't know predis well so I'd be happy to have more details. Thanks.

jralph · 2018-08-09T21:12:58Z

Hi @nicolas-grekas,

No problem, I'd be happy to provide some more info! The predis docs are a little off in the wiki. The readme.md specifies there are 3 options, the one mentioned in the wiki, plus either redis or predis. The redis option specifies that predis should let the redis servers manage the clustering themselves (for clustered redis). If this is set as predis or a callable, this is for client based redis clustering where you just give it a bunch of non-connected redis servers the client handles the sharding/clustering/replication and so on.

See: https://github.com/nrk/predis#client-configuration
See: https://redis.io/topics/cluster-tutorial

The profile option was only included as an example of passing multiple options to the client.

nicolas-grekas · 2018-08-14T16:16:19Z

Thanks for the links.

Right now, Predis is skipped when ext-redis is installed.
When one sets predis_options, I suppose one expects Predis to be used.
I'd suggest to throw an exception when that's not the case.

jralph · 2018-08-14T16:24:36Z

Do you reckon it would be better to throw an exception in that case, or to enforce the use of predis instead? This could be could be done expanding uppon the if statement that checks for the existence of ext-redis. It could check for ext-redis as well as predis options and default to predis if they are present.

It's possible someone could have ext-redis available, but would prefer to use predis for the clustering capability. But still use ext-redis for other redis instances that are not clustered. Allowing the predis_options to force predis could work out better in this use case.

nicolas-grekas · 2018-08-14T21:16:01Z

Setting the class option to RedisCluster is another way to connect to a cluster directly with the extension.
About your question, I'm not sure. I think I would prefer automatic configuration: when a predis option is used, we could force class accordingly, and fail with a clear error message when the package is not installed. WDYT ?

jralph · 2018-08-15T11:07:46Z

I haven't got much experience with the redis extension, does the RedisCluster class extend the Redis class?

I agree, I think having the code work out what you're trying to use and fail on an invalid configuration setup is ideal.

I've updated the code to reflect the more automated configuration approach as well as added tests to check this config.

If a predis option is specified and predis is not available it will throw a CacheException. If a predis option is specified and a custom class is specified, it will check that the custom class is an instance of \Predis\Client and throw an InvalidArgumentException if not. What are your thoughts on this?

nicolas-grekas

some more comment to help move forward.
LGTM afterwards, thanks.

nicolas-grekas · 2018-08-15T19:27:34Z

src/Symfony/Component/Cache/Traits/RedisTrait.php

@@ -118,8 +118,17 @@ public static function createConnection($dsn, array $options = array())
        $params += $options + self::$defaultConnectionOptions;
        if (null === $params['class'] && !\extension_loaded('redis') && !class_exists(\Predis\Client::class)) {
            throw new CacheException(sprintf('Cannot find the "redis" extension, and "predis/predis" is not installed: %s', $dsn));
+        } elseif (isset($params['predis_options']) && !class_exists(\Predis\Client::class)) {


we prefer replacing such "elseif" by an "if" (can be done when the previous "if" break the code flow, with throw or return typically. (same below)

nicolas-grekas · 2018-08-15T19:30:06Z

src/Symfony/Component/Cache/Traits/RedisTrait.php

@@ -118,8 +118,17 @@ public static function createConnection($dsn, array $options = array())
        $params += $options + self::$defaultConnectionOptions;
        if (null === $params['class'] && !\extension_loaded('redis') && !class_exists(\Predis\Client::class)) {
            throw new CacheException(sprintf('Cannot find the "redis" extension, and "predis/predis" is not installed: %s', $dsn));
+        } elseif (isset($params['predis_options']) && !class_exists(\Predis\Client::class)) {
+            throw new CacheException(sprintf('DSN contains "predis_options" and "predis/predis" is not installed: %s', $dsn));


The option can come from the $options argument, so the message should not tell about the DSN I'd say.
What about `Invalid connection option: "predis_options" provided but "predis/predis" is not installed.', $dsn));

nicolas-grekas · 2018-08-15T19:30:36Z

src/Symfony/Component/Cache/Traits/RedisTrait.php

+            throw new CacheException(sprintf('DSN contains "predis_options" and "predis/predis" is not installed: %s', $dsn));
+        } elseif (isset($params['predis_options']) || !\extension_loaded('redis')) {
+            if (null !== $params['class'] && !is_a($params['class'], \Predis\Client::class, true)) {
+                throw new InvalidArgumentException(sprintf('Cannot find the "redis" extension and expected an instance of %s, instead got: %s', \Predis\Client::class, $params['class']));


the message is confusing: when the extension is installed, we should tell otherwise.

nicolas-grekas · 2018-08-15T19:30:56Z

src/Symfony/Component/Cache/Traits/RedisTrait.php

+                throw new InvalidArgumentException(sprintf('Cannot find the "redis" extension and expected an instance of %s, instead got: %s', \Predis\Client::class, $params['class']));
+            }
+
+            $class = null === $params['class'] ? \Predis\Client::class : $params['class'];


$class = $params['class'] ?? \Predis\Client::class; (same below)

nicolas-grekas · 2018-08-15T19:31:54Z

src/Symfony/Component/Cache/Traits/RedisTrait.php

@@ -167,7 +176,10 @@ public static function createConnection($dsn, array $options = array())
            $params['scheme'] = $scheme;
            $params['database'] = $params['dbindex'] ?: null;
            $params['password'] = $auth;
-            $redis = new $class((new Factory())->create($params));
+
+            $predisOptions = false !== isset($params['predis_options']) && true === \is_array($params['predis_options']) ? $params['predis_options'] : array();


no need for the boolean comparisons:
$predisOptions = isset($params['predis_options']) && \is_array($params['predis_options']) ? $params['predis_options'] : array();
but shouldn't we throw when predis_options is not an array?

jralph · 2018-08-16T11:59:08Z

Great, I've made those suggested changes now and cleaned up a few of the exception messages a bit!

fabpot · 2018-08-19T14:27:07Z

src/Symfony/Component/Cache/CHANGELOG.md

@@ -9,6 +9,7 @@ CHANGELOG
 * added sub-second expiry accuracy for backends that support it
 * added support for phpredis 4 `compression` and `tcp_keepalive` options
 * added automatic table creation when using Doctrine DBAL with PDO-based backends
+ * added support for a `predis_options` query parameter array in the redis dsn.


fabpot · 2018-08-19T14:27:46Z

src/Symfony/Component/Cache/Traits/RedisTrait.php

+
+            $predisOptions = $params['predis_options'] ?? array();
+
+            if (!\is_array($predisOptions)) {


We are not doing this kind of checks anywhere in the code base, not sure if we want it here.

What would you recommend instead? If we don't check and throw an exception predis will, I would imagine this would suffice?

Letting predis handle the situation seems better.

jralph · 2018-08-20T13:48:00Z

I've made the suggested changes in the code and the CHANGELOG.md file now.

Was the line doing $predisOptions = $params['predis_options'] ?? array(); something you wanted changed as well, or is that part ok?

When the predis driver is created, there is no current way to configure the predis client options (such as setting the cluster mode to redis). This merge allows you to add a predis_options parameter to the dsn to allow configuration of options.

jralph · 2018-08-21T11:38:38Z

I'm not sure what's caused the tests to fail for travic-ci there. Any ideas? Seems to be something to do with the Process component?

jralph · 2018-09-10T09:19:11Z

@fabpot Were the changes made suitable for what you mentioned?

nicolas-grekas · 2018-09-10T19:32:51Z

See proposal in #28300 (comment)

jralph · 2018-09-13T10:51:11Z

Having done some more debugging on this, I do not believe my merge is yet complete so please do not accept it yet.

There is an issue where passing in a single host to predis causes it to ignore the clustering setup completely, using the default stream connection instead of creating a cluster. (For example, aws elasticache redis clusters that have a single entry endpoint.)

I'm not yet quite sure on how to approach this, possibly if clustering is expected, to pass in an array of hosts (an array of 1 host works), instead of a single string to the factory.

It also appears the predis factory does not support creating a redis cluster connection.

The current tests pass only because when calling the code to get the cluster option actually causes the creation of a cluster connection, giving a false positive. When passing in a dsn as a single string and not an array of dsn's, redis clustering is never used.

I think this changes the scope of changes required to the RedisTrait to get clustering to work, it may be that #28300 provides a better solution to go with overall instead of my attempt here.

nicolas-grekas · 2018-10-03T18:02:28Z

Closing in favor of #28713
Please review/test this PR and see if this works for you.
It'd be awesome if you could contribute some doc for it :)

… via DSN (nicolas-grekas) This PR was merged into the 4.2-dev branch. Discussion ---------- [Cache] added support for connecting to Redis clusters via DSN | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - Replaces #28300 and #28175 This PR allows configuring a cluster of Redis servers using all available options of either the phpredis extension or the Predis package: - the `redis_cluster=0/1` boolean option configures whether the client should use the Redis cluster protocol; - several hosts can be provided using a syntax very similar to #28598, enabling consistent hashing distribution of keys; - `failover=error/distribute/slaves` can be set to direct reads at slave servers; - extra options are passed as is to the driver (e.g. `profile=2.8`) - Predis per-server settings are also possible, using e.g. `host[localhost][alias]=foo` in the query string, or `host[localhost]=alias%3Dfoo` (ie PHP query arrays or urlencoded key/value pairs) Commits ------- a42e877 [Cache] added support for connecting to Redis clusters via DSN

carsonbot added Status: Needs Review Feature labels Aug 9, 2018

jralph force-pushed the predis-client-config branch 2 times, most recently from effe292 to 05c319f Compare August 9, 2018 13:14

jralph changed the title ~~Predis client config~~ [Cache] Add ability to configure predis client. Aug 9, 2018

jralph force-pushed the predis-client-config branch 3 times, most recently from 0f08f1b to 2fe2f53 Compare August 15, 2018 11:01

jralph force-pushed the predis-client-config branch from 2fe2f53 to 5ee41a7 Compare August 15, 2018 11:20

nicolas-grekas reviewed Aug 15, 2018

View reviewed changes

jralph force-pushed the predis-client-config branch 4 times, most recently from 5794698 to 1ac9b3e Compare August 16, 2018 11:58

nicolas-grekas added this to the next milestone Aug 19, 2018

carsonbot added Status: Reviewed and removed Status: Needs Review labels Aug 19, 2018

nicolas-grekas approved these changes Aug 19, 2018

View reviewed changes

fabpot requested changes Aug 19, 2018

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Reviewed labels Aug 19, 2018

jralph force-pushed the predis-client-config branch from 1ac9b3e to 1671186 Compare August 20, 2018 13:45

jralph force-pushed the predis-client-config branch 2 times, most recently from dcff981 to a260177 Compare August 20, 2018 14:00

jralph force-pushed the predis-client-config branch from a260177 to 2458735 Compare August 20, 2018 14:00

nicolas-grekas mentioned this pull request Sep 7, 2018

[Cache] Define multiple Memcached servers in framework.yaml #27855

Closed

nicolas-grekas mentioned this pull request Oct 3, 2018

[Cache] added support for connecting to Redis clusters via DSN #28713

Merged

nicolas-grekas closed this Oct 3, 2018

nicolas-grekas modified the milestones: next, 4.2 Nov 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cache] Add ability to configure predis client. #28175

[Cache] Add ability to configure predis client. #28175

jralph commented Aug 9, 2018 •

edited

Loading

nicolas-grekas commented Aug 9, 2018

jralph commented Aug 9, 2018 •

edited

Loading

nicolas-grekas commented Aug 14, 2018

jralph commented Aug 14, 2018 •

edited

Loading

nicolas-grekas commented Aug 14, 2018

jralph commented Aug 15, 2018 •

edited

Loading

nicolas-grekas left a comment

nicolas-grekas Aug 15, 2018

nicolas-grekas Aug 15, 2018

nicolas-grekas Aug 15, 2018

nicolas-grekas Aug 15, 2018

nicolas-grekas Aug 15, 2018

jralph commented Aug 16, 2018 •

edited

Loading

fabpot Aug 19, 2018

fabpot Aug 19, 2018

jralph Aug 20, 2018

fabpot Aug 20, 2018

jralph commented Aug 20, 2018 •

edited

Loading

jralph commented Aug 21, 2018

jralph commented Sep 10, 2018 •

edited

Loading

nicolas-grekas commented Sep 10, 2018

jralph commented Sep 13, 2018 •

edited

Loading

nicolas-grekas commented Oct 3, 2018


		$predisOptions = $params['predis_options'] ?? array();

		if (!\is_array($predisOptions)) {

[Cache] Add ability to configure predis client. #28175

[Cache] Add ability to configure predis client. #28175

Conversation

jralph commented Aug 9, 2018 • edited Loading

nicolas-grekas commented Aug 9, 2018

jralph commented Aug 9, 2018 • edited Loading

nicolas-grekas commented Aug 14, 2018

jralph commented Aug 14, 2018 • edited Loading

nicolas-grekas commented Aug 14, 2018

jralph commented Aug 15, 2018 • edited Loading

nicolas-grekas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jralph commented Aug 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jralph commented Aug 20, 2018 • edited Loading

jralph commented Aug 21, 2018

jralph commented Sep 10, 2018 • edited Loading

nicolas-grekas commented Sep 10, 2018

jralph commented Sep 13, 2018 • edited Loading

nicolas-grekas commented Oct 3, 2018

jralph commented Aug 9, 2018 •

edited

Loading

jralph commented Aug 9, 2018 •

edited

Loading

jralph commented Aug 14, 2018 •

edited

Loading

jralph commented Aug 15, 2018 •

edited

Loading

jralph commented Aug 16, 2018 •

edited

Loading

jralph commented Aug 20, 2018 •

edited

Loading

jralph commented Sep 10, 2018 •

edited

Loading

jralph commented Sep 13, 2018 •

edited

Loading