Just a little comment, it says Docker is an abstraction layer on top of LXC. Since version 0.9, Docker uses their own libcontainer go package to communicate with the kernel directly instead of using LXC. It is still possible to use LXC though, but it's not used by default.
A small enhancement to your HOST_IP grepping, the host's public and private IP is available in /etc/environment. docker0 may or may not be present before a docker run as well due to socket activation, so your results may vary with a brand new CoreOS VM where docker has not started yet. For example, we are resolving this in Deis with https://github.com/deis/deis/pull/851
To expand on this a bit, you can source a file to expose variables to your unit. On most providers, CoreOS populates /etc/environment with the public and private IP. You can then use those as env vars in your unit:
[Service]
EnvironmentFile=/etc/environment
You can then use $COREOS_PUBLIC_IPV4 anywhere in that unit file. Makes service registration a breeze.
Hi Rob, thanks for your expansion on it :-). I see Deis does that in a similar fashion. I'll update the article with this approach, because it's clearly the best way to solve the "host IP issue".
Ha, thanks! I actually got the grepping magic from Deis as well, but this is a much more elegant solution. By the way, Deis is looking really good now that it's CoreOS based :-).
I am wondering why you have a 1:1 mapping between Docker containers and processes? I have not used Docker but my impression of it was a good way to group a few inter-related processes together while separating them from unrelated processes. Our thought was basically 1 container per role/service in our platform (app servers, workers, DB, logging, etc). If you have 1 process per container, isn't it just a bit stronger form of sandboxing a process?
You can do it in several different ways, and it really depends on what you're trying to accomplish.
If you were trying to build wordpress-as-a-service, for example, you might create a Docker instance which includes nginx, php-fpm, memcached, and maybe even mysql (which makes it easy to make snapshots of the entire state of the system).
That said, fleet has a really interesting way of making interconnected systems, so you can create a memcached instance, an nginx instance, and a php-fpm-running-wordpress instance, and you can manage them separately. This makes it easy to upgrade memcached without touching nginx or php-fpm, or to change the nginx config without having to affect the running application server (and risk flushing the APC cache), etc.
It also allows you to make everything pluggable in your design. You might want to add varnish in front of your nginx instance, for example, but sometimes you might not. That sort of flexibility is really useful in some circumstances.
In our infrastructure at work, the design I would use would probably be similar to this; we have front-end services (nginx), internal services that they proxy to, internal services that those services access, asynchronous processing services, dispatch services, activemq, redis, memcached, mysql, etc. Having the flexibility to add more of any one of those things without having to scale any of the rest is really important.
For other circumstances, it makes a lot more sense to bundle everything into a faux-VM. Run supervisord to launch ssh, nginx, php-fpm, memcached, and whatever else needs running. Share volumes to local filesystems so that configs, local changes, etc. can be taken care of, and you're good to go.
So in the end, the real question is what do you want out of your containers? Flexibility or simplicity?
The process is the unit in which things fail in an OS, so a collection of processes needs to do its own failure handling. Part of the point of these types of system is to do things like failure handling, rather than everything having to do its own.
That's a fair point. In this case I was especially interested in running multiple instances of the same application on a CoreOS mini-cluster, demonstrating fleet's capabilities of scaling/distributing the containers over the cluster.
However, in a real life application I think I would still keep the containers as "single-process" as possible; in our case that would mean: a single Rails application or a group of Resque workers. It would make the containers quick to boot and would increase security by sandboxing all the separate parts of a system in their own containers. I do think this is up to personal preferences, though :-).
Either way can work. Some time ago, it was difficult to share data and networking between containers.
Right now, it is so easy that I even have (without CoreOS) a Hubot in a container, with its Redis brain in another container. Hubot can talk to Redis just fine, and that's the only container in that machine which can do so.
Maintaining the data storage backend gets easier this way. The Redis data is stored in a volume.
You're welcome! I'm now looking into log aggregation and centralisation with Logstash and logstash-forwarder (previously Lumberjack). So expect a new article on that in the near future.
Have you done any stress testing yet? I've run into some contention problems with more than a couple hundred containers per node. I imagine most other deployments haven't required that kind of density so far...
What types of issues are you seeing? From time to time it's easy to get hundreds (or thousands) of containers on a machine. I know docker is a bit slow to start up with a lot of containers, but because of the socket activation that shouldn't be an issue at all if you're starting containers via systemd or fleet.
I gave Blake the whole rundown (i.e. laundry list of whines) at GopherCon, and it sounds like a lot of the issues I ran into are fixed in more recent images. I was attempting to inject 1000 containers running a small node.js app into 3 168GB AWS instances via fleet 0.1.3 (I think).
Part of my problem may have been that I was sharing some config data between app instances via Redis, when I could've just used etcd for that.
Yeah, one of the motivations for me to experiment with this is to see if we can expand our vanilla EC2 stuff with a CoreOS layer on top, to make deployment and scaling easier and more flexible.
Namely that it forces you to store your data in a shared service and design your app so that it can be destroyed at any time without severe consequences.
It also has primitives built in to get you thinking about fleets of machines instead of individual machines such as a default shared configuration and low level knowledge of the other machines in the fleet.
Lastly, since it is very Docker friendly, it gets you thinking in containers, which is a very neat way of making sure that your software is consistent across machines.
I love the 12 Factor App! I recommend that link all the time to people. It's kind of like functional programming, in that once you grok the fundamentals, it changes your attitude completely.
While all this sounds cool it is a tiny bit frustrating that do not have an ISO to boot. I found at least one other person looking at it, but I could not figure out why that was not an option.
It almost sounded like this is an OS but not one you install? Or somehow the OS is only in memory? Like tinycore maybe?
I don't think it's quite at the point of being stable enough to provide an ISO yet.
Also, while it is possible to run on bare mettle, it would not be most productive to do so in most cases. One of the strengths of CoreOS is the ability to quickly add new machines to a cluster and manage applications not on the individual server level, but as collections of services running via Docker. This means, unless you have quite a few computers ready to scale, it would work best on something like AWS or Google Cloud Platform.
For small to medium clusters cloud providers work great. As deployments of CoreOS get larger, PXE booting will probably become the most popular way to add capacity. Plug in 10/100/1000 servers, turn them on and instant capacity added. The upper end of the target scale for CoreOS consists of companies that own their own datacenters or have many points of presence.
The ISO is undocumented as it is still being tweaked, but you can try i t out if you'd like: http://storage.core-os.net/coreos/amd64-usr/alpha/coreos_pro.... You'll have to use config-drive to provide cloud-config. Report back to the coreos-dev mailing list with any issues.
