Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

How things will work on Windows/Mac? Process-Compose is cross platform, so some ideas on at least one more platform could be cool?

Why extra lines to be added by Process-Compose yaml config instead of using just command line like apptainer exec --bind /data:/mnt my_container.sif python3 api.py to execute? Should Process Compose TUI or process lifecycle to be changed? So not clear what is value added by integrating apptainer if fallback option to use script exists?

Imho integration with app sandboxes of Windows/MacOs Sandbox may be better things?

• https://apple.stackexchange.com/questions/469368/app-and-terminal-isolation-sandboxing
• https://learn.microsoft.com/en-us/windows/security/application-security/application-isolation/windows-sandbox/windows-sandbox-overview

And linux:

• Nix buildFHSEnv
• chroot/proot https://news.ycombinator.com/item?id=42262308 , cgroups
• whatever linux has else

These technologies are faster REPL/startup/build time than containers/vms and may appear better fit long term Process-Compose what is alternative to containers?

Also some previous idea about containers #87

Thanks for the response!

How things will work on Windows/Mac? Process-Compose is cross platform, so some ideas on at least one more platform could be cool?

My hope was that this would lay the foundation for additional "container" tools on Windows/Mac. Apptainer works on Mac with some work. I'm not sure if it works on WSL on Windows, but this work would be very similar for podman, which works on both windows and mac and docker.

Why extra lines to be added by Process-Compose yaml config instead of using just command line like apptainer exec --bind /data:/mnt my_container.sif python3 api.py to execute?

I was originally doing what you're talking about here, but these apptainer commands can quickly become very long and complex, especially as bind paths are added. I liked the idea of it matching the docker compose style for things like this, but still being very powerful when not using docker compose. I guess I was trying to get the best of both worlds if that makes sense.

Should Process Compose TUI or process lifecycle to be changed?

I do not think process compose TUI or the process lifecycle need to be changed. One of the benefits of apptainer/singularity is that the containers can just be ran as executables like any other app.

So not clear what is value added by integrating apptainer if fallback option to use script exists?
You are right that the script option does absolutely exist. I felt it was just more convenient to have everything in the compose.yml file.

Imho integration with app sandboxes of Windows/MacOs Sandbox may be better things?
https://apple.stackexchange.com/questions/469368/app-and-terminal-isolation-sandboxing
https://learn.microsoft.com/en-us/windows/security/application-security/application-isolation/windows-sandbox/windows-sandbox-overview

I was not aware of these as I work almost entirely in linux. These are very cool though and I could see the benefit of having something like that as well.

And linux:
Nix buildFHSEnv
chroot/cgroups
whatever linux has else

I havne't personally used Nix so I don't know the benefit there.

I use cgroups as more of a means to control how many resources a process/set of processes has access to.

I've only used chroot to build images. I'm not sure the benefit here.

These technologies are faster REPL/startup/build time than containers/vms and may appear better fit long term Process-Compose what is alternative to containers?

Sure, I could see the benefit for adding support for these too.

Also some previous idea about containers #87

I like your use case here with the separate environments and cgroups. Apptainer has some of that built in so you can control the containers with cgroups which means you wouldn't need process compose to do that.

TLDR; Imho there should be more deep thinking on container integration.

My hope was that this would lay the foundation for additional "container" tools on Windows/Mac. Apptainer works on Mac with some work. I'm not sure if it works on WSL on Windows, but this work would be very similar for podman, which works on both windows and mac and docker.

Unlike containers, I guess people may expect PC(process compose) to work on all platforms with less right or configs. And it to support HOST OS on HOST OS runs. I would expect Mac to work on Mac and Windows on Windows. Because PC does that.
So kind of possible expected, at least for me(and hence 2 companies I worked at and integrated PC), for containers integratios works on all OSes with HOST on HOST mode without need for SUDO or installation same way. Likely it will be subset of containers. I just afraid just planning(doing Linux) may may prevent consistency porting that to Mac/Windows.

I was originally doing what you're talking about here, but these apptainer commands can quickly become very long and complex, especially as bind paths are added. I liked the idea of it matching the docker compose style for things like this, but still being very powerful when not using docker compose. I guess I was trying to get the best of both worlds if that makes sense.

As long as apptainer is linux or mac only, it may put commands into sh scripts and call them, as fallback.
Assuming we have put apptainer commands into nice sh scripts. What else is not here to have apptainer well integrated?

Also containers bring bad things too - need to build image. Most builder to copies of files instead doing links (doing links is not portable). So containers bring worth of both words too.

I do not think process compose TUI or the process lifecycle need to be changed. One of the benefits of apptainer/singularity is that the containers can just be ran as executables like any other app.

If process lifecycle or TUI would need changes, it would mean that there is something which needs active integration of containers with PC. So it would be pro containers argument. Currently argument is only long scripts.

I was not aware of these as I work almost entirely in linux. These are very cool though and I could see the benefit of having something like that as well.
I havne't personally used Nix so I don't know the benefit there.
I use cgroups as more of a means to control how many resources a process/set of processes has access to.
I've only used chroot to build images. I'm not sure the benefit here.
I like your use case here with the separate environments and cgroups. Apptainer has some of that built in so you can control the containers with cgroups which means you wouldn't need process compose to do that.

I am not expert in any of this, so worked with all.

PC allows to build host binaries on host and use them without building containers. Containers may require build in container(HOST<-> GUEST OS), and copy a lot of files.

That is slow. If Apptainer on Mac can run Mac osses and allows to link binaries from host instead of copying (slow), than it is very good and I am for it. But is it?

As for nix, nix is just *nix. In the end any nix thing is just set of traditional bash and configs files.
For example buildFHSEnv allowed me, non power user of linux, to setup light isolation to test if my scripts will run well on CI. CI vs Docker vs Local usually different. So need to test these.

So for linux idea that I do not need to install anything to do buildFHSEnv/cgroups/chroot/etc on Linux to get container like behaviour (for testing at least, not security) and fast REPL(can reuse host files).

Similar for Mac and Windows.

If on Linux Apptainer is shortcut for light isolation without a lot of file copying(without long time to take to build container, but more like just linking host files) than it is super good (I used nix as example as it provides such feature out of box).

Windows/Mac/Linux has similar technologies to:

1. reuse host files
2. run host os on host os, with host os built for and on binaries
3. add some simple isolation (ACL to read/write files, use network, etc)

than these technologies are more likely to be better fit for Process Compose than Apptainer(or Containers).

Specific techs of aternatives/references:

1. how to not copy files from host to container https://github.com/pdtpartners/nix-snapshotter?tab=readme-ov-file#key-features - just hook into container load process and give it links.
2. other tech idea to share xplat binaries Native packaging and updating  #219 (comment) , so that not xplat containers, but xplat apps.

Here is other set of tools for Linux which seems do not require container build step for isolation https://news.ycombinator.com/item?id=42944858 . Process compose awesome as no need to build anything - just use existing binary.

Here is other set of tools for Linux which seems do not require container build step for isolation https://news.ycombinator.com/item?id=42944858 . Process compose awesome as no need to build anything - just use existing binary.

This looks great!

this one has reasonable requirements list for tech which fits for PC:

https://github.com/Zouuup/landrun

• no need for root(admin) to sandbox. unprivileged process can sandbox
• lightweight (given native binary, can run it as is; no need to build oci tar archieve or VM image)
• exec/read/write/list per directory/file control
• bind/connect for sockets control

and can add:

• there should be existing Go modules to use (so can just use modules, instead writing own API)
• for Darwin and Windows

as for darwin, there is no Go module to use libsandbox, nor its is at all well documented(I see some C++ code uses it). more, it requries installation of app via apple store (so cannot just run simple Go binary).

as for windows, seems can use CreateRestrictedToken and ACL can be used, but then need specific users to be ACLed and all that requires elevated permissions to PC. and not Go module.
so in general doable without going via Microsoft Store or driver signature, just elevate permissions of current user and create runner users. but seems dirty.

TLDR; On Linux people can use directly Landrun for example or any other isolation. No viable to do anything for Mac/Windows. But if not support for Mac/Windows, does not seems need it to be part of PC?.

How things will work on Windows/Mac? Process-Compose is cross platform, so some ideas on at least one more platform could be cool?

My hope was that this would lay the foundation for additional "container" tools on Windows/Mac. Apptainer works on Mac with some work. I'm not sure if it works on WSL on Windows, but this work would be very similar for podman, which works on both windows and mac and docker.

You can execute SIF files "on" macOS, by wrapping them in lima. I think the same would work with wsl:

https://learn.microsoft.com/en-us/windows/wsl/filesystems#run-linux-tools-from-a-windows-command-line

For the Lima project there is an apptainer.lima wrapper, that sets up the parameters and environment for you:

https://github.com/lima-vm/lima/blob/master/cmd/apptainer.lima

It is basically the same idea as with Docker or Podman, you set up a VM and then call out to it with SSH or API.

With some added complexity of transporting files back and forth, either over a networked filesystems or tarball.

So when we execute the SIF file with run-singularity, it calls upon apptainer.lima which calls lima apptainer.
The FUSE filesystem on the VM should cache the image file, so it only needs to be transferred to the VM once...

If the initial startup is a concern, then the command could call a local path but then the illusion would break.
For the Docker and Podman containers, they use a image "id" which will map to a local unpacked directory.

@afbjorklund

1. what is quantitative measure on how fast to copy files or not? I came to PC from OCI composers (podman/docker) - copying files is slow (even on NVMes). PC does not copy.
2. how to make what you say xplat?
3. why not use OCI containers for xplat via existing OCI solutions? and if not xplat, why not use Linux only non containers solutions?

Isolation for windows as per Gemini:

https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221XfPZG453Y7-kKvlUqneHRIOqilb-rCYO%22%5D,%22action%22:%22open%22,%22userId%22:%22110329122672250733418%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing

https://learn.microsoft.com/en-us/windows/win32/secauthz/appcontainer-for-legacy-applications-

https://github.com/sandboxie-plus/Sandboxie

I believe that Mac and Windows sandboxes can be commercial proposition of PC on Mac and Windows stores.

• what is quantitative measure on how fast to copy files or not? I came to PC from OCI composers (podman/docker) - copying files is slow (even on NVMes). PC does not copy.

It got a little better with virtiofs, before with virtfs (9p) it was rather slow - like slower than NFS, even if the "server" is really close (localhost)... The only way to get decent performance is to keep the files on the native disk image (in Linux)

• how to make what you say xplat?

The client portion is cross-platform, but the server is Linux-only (we did experiment with FreeBSD, but same difference). The most promising way to make cross-platform binaries is to run wasm containers, which is a separate story...

• why not use OCI containers for xplat via existing OCI solutions? and if not xplat, why not use Linux only non containers solutions?

There are some pros and cons between SIF and OCI, and even new hybrids like OCI-SIF. And yes, if everyone ran Linux it would just be processes. Of course, that also assumes that everyone runs the same distro and there are no dependencies.

But yeah, you could accomplish more or less the same thing with podman run oci-archive:image.tar
It would import the layers every time, but since they might already exist in storage that could go fast...

When docker/podman imports to their storage, it's not very different to apptainer importing into a sif.
The difference in performance between the filesystems is comparable, again there are pros and cons.

But if you want to run OCI containers with compose, I guess you would just use docker-compose?

Or perhaps nerdctl compose, if you want to only use containerd/buildkitd and not all of dockerd.

Here was my comparison, between the technologies. As in SIF versus OCI.

$ apptainer pull docker://python
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Fetching OCI image...
5.9MiB / 5.9MiB [=====================================================================================] 100 % 0.0 b/s 0s
46.2MiB / 46.2MiB [===================================================================================] 100 % 0.0 b/s 0s
22.9MiB / 22.9MiB [===================================================================================] 100 % 0.0 b/s 0s
201.6MiB / 201.6MiB [=================================================================================] 100 % 0.0 b/s 0s
61.4MiB / 61.4MiB [===================================================================================] 100 % 0.0 b/s 0s
26.1MiB / 26.1MiB [===================================================================================] 100 % 0.0 b/s 0s
INFO:    Extracting OCI image...
INFO:    Inserting Apptainer configuration...
INFO:    Creating SIF file...
[=============================================================================================================] 100 % 0s
$ ./python_latest.sif python hello.py
Hello, World!

$ podman pull python
$ podman save --format=oci-archive python:latest >python_latest.tar
$ podman run -v $PWD:/pwd oci-archive:python_latest.tar python /pwd/hello.py
Getting image source signatures
Copying blob 9d9e1c31ad9a skipped: already exists  
Copying blob 579c7fb4fb83 skipped: already exists  
Copying blob 3ae679bbed48 skipped: already exists  
Copying blob 4c5aa3578368 skipped: already exists  
Copying blob a2db8f7484d4 skipped: already exists  
Copying blob 14a930caa7fd skipped: already exists  
Copying blob 81cb21bc73a0 skipped: already exists  
Copying config 07e1786f1f done   | 
Writing manifest to image destination
Hello, World!

Podman normally suggests using systemd (with quadlets), instead of compose.

And of course, normally you would just run "python", rather than from an archive:

$ podman run -v $PWD:/pwd docker.io/library/python:latest python /pwd/hello.py
Hello, World!

But then you need the registry (docker.io) to be available, if image is not present.

Note: the above runs the podman command as root (in the container) for simplicity,
so there are some issues when trying to share files with the host (unlike apptainer).

It also assumes that the current directory is available for mounting in the container.
The default for the tools is to export the home directory to the VM, and work there.

For python only, using a local "virtual environment" (venv) would make more sense.

for mac it is here too https://github.com/AkihiroSuda/alcless (used by coding agents).
(other macs are to heavy via VM or deprecated which is sandbox-exec)