Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Failed to launch executor due to invalid group #524

New issue
New issue
Closed
Closed
Failed to launch executor due to invalid group#524
Assignees
philwinder
Labels
bug
Milestone
1.0.1
@solomkinmv

Description

@solomkinmv
solomkinmv
opened on Mar 9, 2016

I use following JSON to start scheduler on Marathon:

{
  id: elasticsearch,
  cpus: 0.2,
  mem: 512,
  instances: 1,
  cmd: wget -q https://github.com/mesos/elasticsearch/releases/download/1.0.0/elasticsearch-mesos-scheduler-1.0.0-withDependencies.jar -O es-scheduler.jar && java -jar es-scheduler.jar --frameworkName elastic-search --frameworkUseDocker false --zookeeperMesosUrl zk://z   ookeeper.service.consul:2181/mesos --elasticsearchCpu 2 --elasticsearchRam 8192 --elasticsearchDisk 30720 --elasticsearchSettingsLocation https://gist.githubusercontent.com/TheStigger/c769b1857c9f0fa7b0af/raw/bcca7c69856cb1062da50bc54e47004abe4a290f/elasticsearch_settings.yaml,
  env: {
    JAVA_OPTS: -Xms256m -Xmx512m
  },
  ports: [31100],
  requirePorts: true,
  healthChecks: [
    {
      gracePeriodSeconds: 120,
      intervalSeconds: 10,
      maxConsecutiveFailures: 6,
      path: /,
      portIndex: 0,
      protocol: HTTP,
      timeoutSeconds: 5
    }
  ]
}

Also, I have CentOS 7 on all instances. Scheduler fails to launch executors. I'm assuming that it is because of incorrect group name on CentOS. There is no such group, and you should use nobody group instead of nogroup (link) (but I'm not sure). Here is stderr output from executor:

I0309 09:31:55.999567  9610 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/mi-worker-003.cisco.com:45454\/get\/public\/elasticsearch.tar.gz"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"https:\/\/gist.githubusercontent.com\/TheStigger\/c769b1857c9f0fa7b0af\/raw\/bcca7c69856cb1062da50bc54e47004abe4a290f\/elasticsearch_settings.yaml"}}],"sandbox_directory":"\/var\/mesos\/slaves\/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5\/frameworks\/7f4c8258-846c-4286-af0b-05a7a62f764e-0001\/executors\/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z\/runs\/8ae5b612-fc09-41eb-b199-0027e4c69160","user":"root"}
I0309 09:31:56.001153  9610 fetcher.cpp:379] Fetching URI 'http://mi-worker-003.cisco.com:45454/get/public/elasticsearch.tar.gz'
I0309 09:31:56.001166  9610 fetcher.cpp:250] Fetching directly into the sandbox directory
I0309 09:31:56.001180  9610 fetcher.cpp:187] Fetching URI 'http://mi-worker-003.cisco.com:45454/get/public/elasticsearch.tar.gz'
I0309 09:31:56.001204  9610 fetcher.cpp:134] Downloading resource from 'http://mi-worker-003.cisco.com:45454/get/public/elasticsearch.tar.gz' to '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/elasticsearch.tar.gz'
I0309 09:31:56.222159  9610 fetcher.cpp:84] Extracting with command: tar -C '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160' -xf '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/elasticsearch.tar.gz'
I0309 09:31:56.491094  9610 fetcher.cpp:92] Extracted '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/elasticsearch.tar.gz' into '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160'
I0309 09:31:56.491161  9610 fetcher.cpp:456] Fetched 'http://mi-worker-003.cisco.com:45454/get/public/elasticsearch.tar.gz' to '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/elasticsearch.tar.gz'
I0309 09:31:56.491181  9610 fetcher.cpp:379] Fetching URI 'https://gist.githubusercontent.com/TheStigger/c769b1857c9f0fa7b0af/raw/bcca7c69856cb1062da50bc54e47004abe4a290f/elasticsearch_settings.yaml'
I0309 09:31:56.491186  9610 fetcher.cpp:250] Fetching directly into the sandbox directory
I0309 09:31:56.491204  9610 fetcher.cpp:187] Fetching URI 'https://gist.githubusercontent.com/TheStigger/c769b1857c9f0fa7b0af/raw/bcca7c69856cb1062da50bc54e47004abe4a290f/elasticsearch_settings.yaml'
I0309 09:31:56.491214  9610 fetcher.cpp:134] Downloading resource from 'https://gist.githubusercontent.com/TheStigger/c769b1857c9f0fa7b0af/raw/bcca7c69856cb1062da50bc54e47004abe4a290f/elasticsearch_settings.yaml' to '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/elasticsearch_settings.yaml'
W0309 09:31:56.803175  9610 fetcher.cpp:272] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: https://gist.githubusercontent.com/TheStigger/c769b1857c9f0fa7b0af/raw/bcca7c69856cb1062da50bc54e47004abe4a290f/elasticsearch_settings.yaml
I0309 09:31:56.803211  9610 fetcher.cpp:456] Fetched 'https://gist.githubusercontent.com/TheStigger/c769b1857c9f0fa7b0af/raw/bcca7c69856cb1062da50bc54e47004abe4a290f/elasticsearch_settings.yaml' to '/var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/elasticsearch_settings.yaml'
I0309 09:31:56.904253  9607 exec.cpp:134] Version: 0.27.1
I0309 09:31:56.906848  9619 exec.cpp:208] Executor registered on slave 268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5
chown: invalid group: ‘nobody:nogroup’
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/es_home/logs/mesos-ha.log (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:204)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:258)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
log4j:ERROR Either File or DatePattern options are not set for appender [file].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/es_home/logs/mesos-ha_deprecation.log (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:204)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:258)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
log4j:ERROR Either File or DatePattern options are not set for appender [deprecation_log_file].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/es_home/logs/mesos-ha_index_indexing_slowlog.log (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:204)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:258)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
log4j:ERROR Either File or DatePattern options are not set for appender [index_indexing_slow_log_file].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/es_home/logs/mesos-ha_index_search_slowlog.log (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:204)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:258)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
log4j:ERROR Either File or DatePattern options are not set for appender [index_search_slow_log_file].
Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.home' (es_home/bin)
Likely root cause: java.nio.file.AccessDeniedException: /var/mesos/slaves/268beb4c-7ff4-4484-aa16-8bc94ff6ff12-S5/frameworks/7f4c8258-846c-4286-af0b-05a7a62f764e-0001/executors/elasticsearch_mi-worker-004.novalocal_20160309T093155.893Z/runs/8ae5b612-fc09-41eb-b199-0027e4c69160/es_home
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
    at java.nio.file.Files.createDirectory(Files.java:674)
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
    at java.nio.file.Files.createDirectories(Files.java:767)
    at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
    at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
    at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:243)
    at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
    at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
ls.

Metadata

Metadata

Assignees

Labels

bug

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions