Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[DX] Flaws in the way environment variable are used ? #27962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tristanbes opened this issue Jul 16, 2018 · 23 comments
Closed

[DX] Flaws in the way environment variable are used ? #27962

tristanbes opened this issue Jul 16, 2018 · 23 comments

Comments

@tristanbes
Copy link
Contributor

tristanbes commented Jul 16, 2018

Hello,

After a discussion on Symfony Slack about environment variables and CLI around my question:

I’m interested into how the community solves the environnement variable and the CLI.

It seems that nothing solid came out out of this discussion.

The documentation states that environment variables should be added on the webserver level on production.

Nothing is said reguarding the CLI, so a lot of people (like me) will end up spending time looking for why a crontab entries is not executed.

To me, prefixing the executed commands by the environment variables is not a good solution. I mean, it would involve:

  • variable duplication (so bad maintenance, "oops forgot to add this variable in this place"),
  • and very bad lisibility (prefix with 20+ environment variables 🤔?)

Regarding of this, why push in favor of environement variable if the only way of achieving this is to use a not recommended way (.env file).

One alternate solution would be to create a file on the system in /etc/environment that contains those environment variable, BUT again, what happens if your server hosts multiple projects and you want one app X to be APP_ENV=prod and the other Y APP_ENV=dev, then you have a variable collusion with the name unless you change the APP_ENV variable name in all project to include a vendor prefix ? (I don't even know if it's feasible).

Not all of us use containers to run in production (or doesn't even want to).

PS: I open this here instead of the documentation repository because it reveals more questions than just "how to deal with that in doc is missing".

@dkarlovi
Copy link
Contributor

If you have your env vars in ~/.bashrc, ~/.profile or similar, you'll notice cron will not read them. That's because cron doesn't start your Bash as an interactive login shell.

To fix it, you either have a wrapper which does, or, more simply, source ~/.profile. This makes your env vars available to cron too and it works exactly as you'd expect.

@tristanbes
Copy link
Contributor Author

I don’t have my environement in a .bashrc or .profile or /etc/environement yet. I said that processing them this way on a server that host multiple APP is not doable because of variable name collision I mentionned in my original message.

@Saphyel
Copy link
Contributor

Saphyel commented Jul 16, 2018

I think you are trying to find a good solution for a bad practice, your actual project should be isolated from others projects. Also I think you should avoid to use env. vars in that scenario because you have the risk of write dev data into prod or in the other way around.

The solutions that I think you can take is...

  1. Hardcode values like the ol' times in a parameters.yml and use that file as source of data rather than use %ENV()%.
  2. Use DotEnv.
  3. Even if the console.php is provided by symfony is in your git source code so you can update it to enforce to use the .env that you want or hardcode values or whatever you want.

@Kocal
Copy link
Member

Kocal commented Jul 16, 2018

I totally agree with @tristanbes.

For me, it's really hard to handle env vars at so many places (6):

  • .env.dist (and .env for local development)
  • Inside a web context (nginx/Apache) in production
  • Inside a CLI context in production, because I don't want to create global env vars for a specific project. I already have 2/3 Symfony app on the same server, and I don't wan't to learn and masterize Docker for that 🙁 ...
  • PhpUnit, with <env> from phpunit.xml.dist
  • Behat, with a special .env.test (but I thing I can use a simple .env and update config/packages/test/* configurations, I probably did bad here)

When running PhpUnit and Behat inside Travis, I should either manually define env vars from Travis UI, or use a copy of .env.test.

Docker might be the ideal solution to isolate Symfony apps (and any other apps BTW), but as said before, I don't want to learn nor masterize Docker for something that should be so simple to manage.

At the moment, I followed those steps to make my life easier:

  • During deployment, automatically copy a production-ready .env (resolve CLI context)
  • In production, NEVER create APP_ENV, so Symfony will load .env (resolve Web context)

I'm feeling dirty, but it works, and that's easier to maintain/use for me.


Personally, I found paremeters so much useful and easier...
Yes, OFC they are less flexible, but they are so much easier to maintain/use:

  • Copy parameters.yml.dist => parameters.yml for local development and Travis builds
  • Use a parameters.yml for web and CLI production contexts

and that works!

@stof
Copy link
Member

stof commented Jul 16, 2018

@Kocal and you can still use a parameters.yml file. My own approach is actually a mix of both. As I'm deploying on Heroku, I'm using %env() to read credentials for various services (allowing updates to be taken into account without requiring a cache clearing), but most of my config is still handled in a parameters.yml file (part of it is not supported as runtime config anyway and requires to be known at compile time)

@Kocal
Copy link
Member

Kocal commented Jul 16, 2018

Oh, that's interesting, I wasn't aware about that.

I will try to find some time for migrating to parameters.yml, that would be very interesting.

@curry684
Copy link
Contributor

I don't want to learn nor masterize Docker for something that should be so simple to manage.

It also solves 2000 other problems...

Not to derail the discussion or anything but treating Docker as a solution for this issue you're having is like saying the best thing about a car is that you're dry inside.

Docker solves nearly every deployment and testing issue. Elegantly and reliably.

@tristanbes
Copy link
Contributor Author

tristanbes commented Jul 16, 2018

It also solves 2000 other problems...

and creates 3000 others problems.
Please let's focus on this issue, I doubt that Symfony will adopt Docker as the official way now to deploy applications. Until then, let's have solution that works for all !

(PS: we provision & deploy our applications using Ansible playbooks, so I don't think our infrastructure is like FTP ages.)

@Saphyel
Copy link
Contributor

Saphyel commented Jul 16, 2018

@curry684 I think learn Docker now is a bit like learn Symfony 3.1 today...

Anyway he is not asking for improve his archicture side, so I'll shut up.

@dkarlovi
Copy link
Contributor

dkarlovi commented Jul 17, 2018

I don't think introducing Docker into this conversation solves anything.

I said that processing them this way on a server that host multiple APP is not doable because of variable name collision

"server that hosts multiple apps" doesn't imply "server that hosts multiple apps using the same account", that makes a significant difference. Doing so is hardly recommended.

Having said that, I understand there are various situations in which you'd want to to that, when you might not have the choice to do it better.

The question is all about understanding how env vars inheritance works and scoping:

  • placing them in /etc/environment means they're globally scoped (just like global keyword in PHP), you can do this only if your whole server is dedicated to a single app (but you can still chose the following approaches)
  • placing them in $HOME/.profile means they're scoped to the user to which that $HOME belongs to, you can do this only if your whole user is dedicated to a single app (but you can still choose the following approach) - this is what the approach I would recommend
  • placing them in <approot>/.env means they're scoped to the folder, but, since "a folder" is not a first-class citizen for scoping processes, you need to do it yourself.

You can handle it yourself

  1. by sourcing the env file as linked: source .env; bin/console foo:bar (again, recommended if you must use the non-recommended deployment procedure)
  2. by changing the RC file when invoking Bash bash --rcfile .env -ic "bin/console foo:bar"
  3. by just using DotEnv for CLI, even in production

If you're using Ansible as you've noted before, setting this up and distributing the env vars everywhere should be very simple to do.

@apfelbox
Copy link
Contributor

Somehow related / my take on this: symfony/flex#287

Please note that the biggest issue with using a parameters.yaml in Symfony 4 was that due to an issue in flex something like the the incenteev parameters handler wasn't able to work properly. This issue has since been resolved, so you might be able to replicate the same workflow as in Symfony <= 3.

My workflow is using APP_ENV as env var and everything else in a parameters.yaml.

@Kocal
Copy link
Member

Kocal commented Jul 17, 2018

Issue symfony/flex#251 is closely related to this one.

@xabbuh
Copy link
Member

xabbuh commented Jul 17, 2018

IMO we need to improve the documentation to make it more clear when there is actual value to use environment variables and when you should rather stick with well-known container parameters. Right now we lack to make clear that there is nothing wrong in still using them, but we seems to confuse (too) many developers right now. From what I have seen in different projects and on Slack is that people overcomplicate their setup and deployment process by using environment variables when plain old container parameters would have solved their problems way easier.

@Kocal
Copy link
Member

Kocal commented Jul 18, 2018

@xabbuh If I understand well, we should not use env vars (in .env) when their values will be always identical between local dev/tests/production?

Well, it seems logical just by writing it... and it's even written in the doc:

The .env file is special, because it defines the values that usually change on each server

Also, if we really need to defines env vars but without a .env, we must use this, right?

parameters:
  env(FOO): value

@rufinus
Copy link

rufinus commented Jul 25, 2018

what happens if your server hosts multiple projects and you want one app X to be APP_ENV=prod and the other Y APP_ENV=dev

Don't you use FPM with different pools? (Which also seperates the user processes) You can (and should) define the ENV vars directly in the pool config.

Like:

[mypoolname]
user = nologinwebuser
env[APP_ENV] = prod

@chalasr
Copy link
Member

chalasr commented Oct 13, 2018

I personally load projects env vars via ~/.bashrc for the CLI (~/.zshenv to be exact) and configure cron so that it uses the right shell (via SHELL="/bin/zsh")

@Kocal
Copy link
Member

Kocal commented Oct 13, 2018

@chalasr so you have one *nix user per project?

@chalasr
Copy link
Member

chalasr commented Oct 13, 2018

@Kocal yes, one project per host and one app user per host able to run the project.
If an host (EC2) serves more than 1 project, all projects share the same env in CLI (project1 has project2's env when it runs via cron, env vars names are prefixed to avoid collisions).
In dev, I use Dotenv.

@Kocal
Copy link
Member

Kocal commented Oct 14, 2018

And how do you handle env vars through web server? You still have to duplicate them right?
AFAIK you can't tell nginx/apache to source ~/.bashrc for each app right?

@chalasr
Copy link
Member

chalasr commented Oct 14, 2018

Indeed, an additional env file per FPM pool. Env vars are managed in a vault (single place) and dumped for both CLI and FPM on deploy (via Ansible)

@rufinus
Copy link

rufinus commented Nov 20, 2018

@chalasr why dont you include a own file in your fpm pool?

[www]
.... static pool config
include = /srv/www/website/fpm.conf

fpm.conf looks llike:

env[APP_ENV] = prod
env[DATABASE_URI] = ""

This file can be deployed via ansible, you need to restart the fpm - but you should anyway after deployment to reset the op cache.

But unfortunatly this does'nt solve the problem to get the env vars into CLI (eg. for bin/console)

@nicolas-grekas
Copy link
Member

Closing as this improved a lot recently, with symfony/recipes#501 as the last step.

@CedrickOka
Copy link
Contributor

One solution for docker in entry point
printenv | sed 's/^([^_]|[^=])=(.)$/\1="\2"/g' | sed 's/^(_)=(.*)$//g' > /app/.env.prod.local

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests