It seems like it is only affecting one of our clusters and it is possible that cluster may have been running another fork of fleet that suffers from this issue. Rebooting the machines seems to fixed it.

I'm closing this for now until I see this happening again.

I'm reopening this bug because we updated our fleet to use master branch HEAD and we started seeing this problem again.

@jonboulle
We've recently rebased our fleetd that we run to be on the HEAD of master and patched it with the monitoring patches (#1415). The branch is here (https://github.com/improbable-io/fleet/tree/imp_release_20160316_INF-metrics).

We think something changed between fleetd version 0.11.5 and HEAD that caused the instancename=%i interpolation not to work in engine scheduling. We've seen similar problems before when we were experimenting with head before (hence the bug).

We're rolled back to a version branched off of:

3178aed

Merge pull request #1263 from

miekg/followonly-hack

Implement follow_only flag

This version does not have this problem, so this issue has been introduced since then.

Confirmed.

$ fleetctl list-machines
MACHINE         IP      METADATA
1e1a8655...     coreos3 hostname=coreos3
74dce006...     coreos2 hostname=coreos2
cd29d103...     coreos1 hostname=coreos1

template:

[Service]
ExecStart=/bin/bash -c "while true; do echo Hello, World %i!; sleep 1; done"

[X-Fleet]
MachineMetadata="hostname=%i"

$ fleetctl start hello@random.service

v0.11.5 just hangs when it can not find corresponding machine.
master branch schedules unit randomly independently on metadata.

Can you try with this instead?

[Service]
ExecStart=/bin/bash -c "while true; do echo Hello, World %i!; sleep 1; done"

[X-Fleet]
MachineMetadata=hostname=%i

IIRC, the way the code parses the MachineMetadata is special when there is only one condition and when there are more than one.

Also, doesn't fleetctl start hello@random.service try to schedule this on machines with hostname=random? If so, then it wouldn't be scheduled on any of the machines in the cluster.

@daniellowtw what do you mean? I use only one condition.

I @daniellowtw was considering the case of AND vs OR in: https://github.com/coreos/fleet/blob/master/Documentation/unit-files-and-scheduling.md#schedule-unit-to-machine-with-specific-metadata

@kayrus https://github.com/coreos/fleet/blob/master/job/job_test.go#L323
The test uses

MachineMetadata=hostname=%i

when there is one condition and

MachineMetadata="hostname=%i" "foo=bar"

when there's more than one.

I think it doesn't parse it if there are quotes when there is only one required metadata
i.e. MachineMetadata="hostname=%i"

I've tried to cherry-pick this to the v0.11.5 tag and reproduced the issue. Will try to investigate what is wrong.

Looks like this loop doesn't recognize whether unit is a template or not and returns hello@.service instead of hello@nodename.service. Then https://github.com/coreos/fleet/blob/master/unit/unit.go#L233 doesn't recognize specifiers of the "naked" template.

Weird. When you submit templated unit, etcdctl shows that two instances were created:

etcdctl get /_coreos.com/fleet/unit/79699e32624ad46c366ef37de89006ae9da617cd
{"Raw":"[Service]\nExecStart=/bin/bash -c \"while true; do echo Hello, World %i!; sleep 1; done\"\n"}
etcdctl get /_coreos.com/fleet/unit/bd40daa16d1d9d8962b12c6f512f912be98a5a4f
{"Raw":"[Service]\nExecStart=/bin/bash -c \"while true; do echo Hello, World %i!; sleep 1; done\"\n\n[X-Fleet]\nMachineMetadata=\"hostname=%i\"\n"}

Closed by #1520